- San Francisco, California, United States
- On-site
- Full-time · Staff
- $225K–$550K per year
- Visa sponsorship
The Evals team measures the performance and capabilities of internal and external models, helping teams across Magic make informed decisions. As a Member of Technical Staff on Evals, you will develop new benchmarks and evaluation methodologies, run data ablations, assess dataset quality, and investigate whether our measurements accurately reflect model performance.
Compensation
Annual salary range between $225K - $550K depending on experience Equity is a significant part of total compensation, in addition to salary
Responsibilities
- Design and validate new evaluations and benchmarks across model training and deployment
- Investigate model behavior and identify gaps in existing evaluations
- Measure dataset quality and identify opportunities to improve training data
- Improve evaluation correctness, reproducibility, and reliability
- Audit and improve upon public benchmarks, research papers, and open-source implementations
