- Remote, Canada +1 more
- Remote
- Full-time · Mid
- 3+ years
As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run, you build the systems that measure it across every conversation, and you are the person the company trusts when the question is “how good is it, really?”
Locations (2)
- Canada
- United States
Compensation
In addition to base pay, total compensation includes equity and a comprehensive benefits package for you and your family.
Responsibilities
- Define and maintain the evaluation rubric for the agent across three axes: adherence, outcome, and conversation experience
- Write the annotation guidelines and failure taxonomy
- Classify and prioritize failure modes by severity and frequency
- Review production conversations by hand every week
- Build and operate the evaluation pipeline that scores every production conversation
Requirements
- Bachelor’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field