- Hyderabad, Telangana, India
- Senior
- 5+ years
Lead the design, development, and deployment of AI agent evaluation pipelines and scoring logic to validate agent performance before customer release.
Responsibilities
- Design evaluation methodologies and benchmarks for agent reasoning, planning, tool use, reliability, and safety across LLM-as-a-Judge, trajectory-based, and human evaluation
- Take problems from research question to prototype to shipped feature, owning them end to end
- Build and harden the pipelines and scoring logic behind customer-facing evaluation
- Curate synthetic and real-world datasets; measure the evaluator itself for consistency and agreement with human labels
Requirements
- Nice to have: AI-assisted development tools
- Nice to have: Claude code
- Nice to have: Windsurf