- Remote
- Remote
- Contract · Senior
- 3+ years
Own the independent proof that retrieval works and that permissions hold by building and running the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates.
Responsibilities
- Build the golden question suite from top contact drivers and needs workshops
- Build the negative entitlement test set
- Build freshness and retirement tests
- Automate the harness to run on schedule and after every content, configuration, or model change
- Define the QA gate content analysts apply to remediated content
Requirements
- Nice to have: AWS Certified AI Practitioner
- Nice to have: AWS Certified Machine Learning Engineer Associate
- Nice to have: RAG evaluation libraries
- Nice to have: LLM-as-judge scoring
- Nice to have: SOC 2
