- Los Angeles, California, United States
- Hybrid
- Contract
- 14+ years
AI Engineer focused on evaluating LLM-powered agents, designing and running evaluation frameworks, and developing golden datasets.
Responsibilities
- Design, implement, and run LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
- Collaborate with Business Analysts and stakeholders to create, validate, and maintain golden datasets for agent evaluation benchmarks.
- Instrument AI agents using Langfuse, including tracing execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
- Evaluate agent behavior, tool usage, reasoning workflows, and response quality to ensure high standards.
- Apply expertise in Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, and AI Observability & Evaluation.