- Remote, United States
- Remote
- Full-time · Principal
- $200K–$300K per year
Lead the design and build of an evaluation framework for an enterprise planner-based agent, benchmarking agents across planner models, executor LLMs, and inference optimization levers.
Compensation
Our salary range is $200,00 - $300,000
Responsibilities
- Design the end-to-end evaluation workflow for a planner-executor agent: scenarios, user simulation, configuration, execution, grading, comparison and insights
- Define the scenario schema, grading spec and validators with the client's team
- Build an LLM-based user simulator with a hidden goal and a stop rule that holds realistic multi-turn conversations
- Build isolated, parallel experiment execution
- Package optimization levers as plug-ins and implement a shortlist of 3 to 4 for v1
Requirements
- Nice to have: Publications or open-source contributions in LLM evaluation, agents or efficient inference
- Nice to have: Building internal dashboards or web UIs that a client team can adopt as their own
- Nice to have: Hands-on use of an LLM gateway or tracing stack (LangFuse or LangSmith)