- Santa Clara, California, United States
- On-site
- $183K–$248K per year
- 5+ years
Lead evaluation and benchmarking data development for Amazon Quick features.
Compensation
USA, CA, Santa Clara - 183,000.00 - 247,600.00 USD annually
Responsibilities
- Design and build evaluation and benchmark datasets
- Define quality metrics and evaluation frameworks, including human evaluation, model-based evaluation (LLM-as-a-Judge), and checks on how well the two correlate
- Run human annotation programs from start to finish: write guidelines, measure inter-annotator agreement, and control data quality
- Analyze model and agent performance to find gaps, root causes, and the highest-impact improvements
- Partner with science, engineering, and product teams to turn evaluation results into launch decisions and roadmap priorities
Requirements
- Nice to have: data visualization using AWS QuickSight
- Nice to have: Tableau
- Nice to have: R Shiny
- Nice to have: data pipelines
- Nice to have: evaluating LLMs