- New York City, New York, United States +2 more
- Hybrid
- $200K–$340K per year
Research Scientist / Research Engineer to identify benchmark weaknesses, develop and validate fixes, and create reusable methods to improve benchmark quality across tasks and datasets.
Locations (3)
- New York City, New York, United States
- San Francisco, California, United States
- United States
Compensation
$200,000—$340,000 USD
Responsibilities
- Identify and reproduce issues such as reward hacking, grader failures, task ambiguity, shortcuts, and environment vulnerabilities
- Propose next steps for validated issues, implement selected fixes with benchmark authors, and verify fixes without breaking valid solutions
- Reuse existing tools where possible and develop new checks to detect recurring benchmark weaknesses