- San Jose, California, United States
- Full-time
- $180K–$450K per year
Build and maintain evaluation harnesses for multimodal foundation models, design and analyze benchmarks, and collaborate with researchers to improve training recipes and capability assessments.
Compensation
The US base salary range for this full-time position is between $180,000 - $450,000 annually.
Responsibilities
- Build and maintain an evaluation harness for held-out loss, performance across data domains, and capabilities in text, code, math, and reasoning.
- Design evaluations for early checkpoints and smaller models using log-likelihood scoring, few-shot prompts, and generated completions.
- Assess benchmark coverage and saturation, quantify uncertainty in results, and work with data researchers to detect training-data contamination.
- Version datasets, prompts, graders, and inference settings so results can be reproduced and compared across checkpoints and training runs.
- Analyze experiments and work with researchers to identify useful recipe changes, select checkpoints, and prioritize capability improvements.
Requirements
- Nice to have: human ratings
- Nice to have: model-based graders
- Nice to have: code execution sandboxes
- Nice to have: math verifiers
- Nice to have: programmatic graders