- San Francisco, California, United States
- On-site
- Full-time
- $150K–$300K per year
- 1–7 years
Design and build test environments, evaluation harnesses and labeled datasets that measure how AI systems behave across multi-step tasks.
Compensation
Base salary: USD 150,000 to 300,000
Responsibilities
- Construct simulated software environments that mimic the tools and workflows automated systems use.
- Build grading logic and evaluation systems that hold up when systems look for shortcuts.
- Create benchmarks whose scores track real-world results.
- Review interaction logs to find and measure previously unrecorded failure patterns.
- Turn findings into reusable datasets and test suites for internal improvement and external partners.
Requirements
- Nice to have: found a real flaw in a widely used benchmark
- Nice to have: caught a model gaming a grader and fixed the grader
- Nice to have: built environments or benchmarks a leading AI lab used for training or evaluation
- Nice to have: startup founding experience or participation in a selective fellowship
- Nice to have: high-level competitive achievement in mathematics, programming, debate or games