LLM Evaluator – Model Response Analyst (Remote)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. You will analyze AI‑generated text for accuracy, coherence, safety, bias, and alignment with our guidelines, working remotely with a global team.

Key responsibilities

• Evaluate and rank model outputs using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple responses to the same prompt and justify the preferred output.
• Provide concise feedback to modeling and training teams on recurring failure patterns.
• Craft adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines and handle ambiguous edge cases.
• Participate in cross‑checking sessions to ensure inter‑rater reliability.
• Analyze underperforming outputs, hypothesize root causes, and flag novel model behaviors.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP QA, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Strong ability to explain why a model output is good or bad based on logic, tone, factuality, and instruction adherence.
• Experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation calibration.
• Proven track record of improving consistency among evaluation teams.

Required skills

• Prompt engineering
• RLHF data collection
• Data annotation and cleaning
• A/B testing concepts applied to AI models
• Evaluation rubric design and calibration