- Remote
- Remote
- Contract · Associate
- $30–$60 per hour
- 0–5 years
Designs and runs human-led evaluations of AI models, measuring model capabilities by creating rigorous questions, rubrics, and comparative assessments.
Compensation
Compensation is $30 to $60 per hour, depending on project, field, and experience. Payment is for hours worked.
Responsibilities
- Write questions and tasks that test specific model capabilities
- Develop rubrics that define strong, partial, and weak answers
- Create reference answers and edge-case scenarios to challenge models
- Grade model outputs against rubrics and explain each score clearly
- Analyze incorrect answers, group error patterns, and document findings
Requirements
- Nice to have: Inspect
- Nice to have: lm-eval-harness
- Nice to have: HELM
- Nice to have: promptfoo
- Nice to have: model comparisons
