- San Francisco
- Hybrid
- Full-time
Design, build, and maintain internal evaluation suites and infrastructure for post-training AI models, ensuring reliability, scalability, and reproducibility.
Compensation
Compensation & Equity: Competitive base salary plus equity, so you share in what we build. Final offers reflect skills, experience, and location.
Responsibilities
- Work on the internal eval suite and infrastructure: keep it reliable, optimized, and reproducible at scale.
- Design held-out evals for priority capabilities such as agentic workflows, computer use, and cybersecurity, with documented construction and contamination checks.
- Build evals targeted at small edge models, where most public evaluations built for frontier models don't apply.
- Read generated outputs to debug unexpected scores before anyone acts on them.
- Turn customer and open-source feedback into environments usable for data, training, and evaluation, and partner with other teams to design evals for their specific needs.
Requirements
- Nice to have: Model training experience
- Nice to have: Publications or research background
- Nice to have: Deep systems expertise
- Nice to have: Familiarity with small edge models