Senior AI Content Specialist to shape conversational AI product for an adult audience, writing prompts, building datasets, testing models, and automating content workflows.
Responsibilities
Write and refine prompts for AI characters
Create and edit text datasets for training and fine-tuning models
Machine Learning Engineer focused on the quality and evaluation of ML models and AI features in creativity applications. Includes defining evaluation strategy, metrics, building datasets/infrastructure, and driving end-to-end quality.
Responsibilities
Define evaluation strategy and metrics for ML models, LLMs, and AI features, tailored to the model, the product, and its use cases.
Curate and create evaluation datasets, measure and improve dataset quality, and run iterative improvement loops so that datasets stay representative of each product.
Design, build, maintain, and optimize evaluation infrastructure at scale, with reproducible runs, tracked baselines, and comparisons between model versions.
Apply ML, LLMs, and agentic workflows to the evaluation itself, including automated judging, test generation, and analysis, to make evaluation more efficient, more accurate, and scalable.
Drive end-to-end evaluation and quality of AI features, from model checkpoints to integrated, shipping features, including triaging issues and driving them to resolution.
AI Research Engineer focused on building evaluation frameworks, experiments, and tooling to improve AI agents across research, engineering, and business workflows at Gauntlet.
Locations (4)
New York City, New York, United States
San Francisco, California, United States
Los Angeles, California, United States
United States
Compensation
The national pay range for this role is $175,000 - $195,000 base plus additional On Target Earnings potential by level and equity in the company.
Responsibilities
Build evaluation frameworks for AI agents, including datasets, sandbox environments and metrics that measure performance on real-world tasks.
Spring Intern, Warfare, Irregular Threats, and Terrorism Program
Center for Strategic and International Studies, Inc. · Posted Nov 6, 2026
Design experiments to compare models, agent harnesses, tools and execution environments, identifying regressions and understanding trade-offs between performance, reliability and cost.
Work with teams across Gauntlet to turn qualitative feedback about agent behaviour into concrete research questions, experiments and measurable outcomes.
Investigate agent failures, understand whether problems come from the model, harness, tooling or prompting, and develop ways to improve performance.
Build and improve agent harnesses, internal tools and integrations, including contributing to Aria and other agents used across the business.
Requirements
Advanced degree or postgraduate research experience in computer science, mathematics, statistics, machine learning or a related field
Senior AI Content SpecialistSocial Discovery Group · Remote
Machine Learning Engineer, Evaluation and Quality, Creativity AppsApple · Vancouver, Canada
Research EngineerGauntlet · Remote, US+3 more
Spring Intern, Warfare, Irregular Threats, and Terrorism ProgramCenter for Strategic and International Studies, Inc. · Washington, US