- Remote, United States
- Remote
- Full-time · Senior
- 3+ years
Help ensure that AI-powered features across Bask Health are accurate, dependable, and ready for real-world use by developing evaluation infrastructure, testing workflows, and performance measurements for large language models, retrieval systems, AI assistants, and automated workflows.
Responsibilities
- Build and maintain evaluation systems for AI-powered applications across Bask Health
- Develop automated testing frameworks for large language models and AI application workflows
- Create and maintain representative datasets for evaluating model behavior
- Implement benchmarks that measure response accuracy, relevance, completeness, and consistency
- Develop evaluation pipelines for prompts, retrieval systems, and model configurations
Requirements
- 3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field
- Nice to have: Evaluation frameworks for large language models
- Nice to have: OpenAI
- Nice to have: Anthropic
- Nice to have: LangSmith