AI Test Engineering is a hands-on role within VLabs Quality Engineering, responsible for validating the performance, reliability, and integrity of AI-enabled solutions in production environments.
Responsibilities
Translate AI testing strategy into executable test scenarios across LLM outputs, document classification, extraction accuracy, agent workflows, and edge cases
Design adversarial and boundary test inputs to expose hallucination, misclassification, and failure modes
Validate AI outputs for structure, consistency, accuracy, and production readiness against defined performance thresholds
Build reusable Python-based evaluation frameworks, including output validation, hallucination detection, and scoring mechanisms
Develop parameterized test scripts reusable across features, models, and releases
Requirements
Bachelor’s degree
Advanced degree in Computer Science, Data Science, or related field preferred
Nice to have: Background in regulated or compliance-driven environments
The Principal Engineer operates at the intersection of deep technical execution and broad organizational influence, owning the end-to-end technical strategy for iHerb's AI Platform and GenAI product experiences.
Compensation
The expected salary range for this role is $204,000.00 - $260,000.00 USD.
Responsibilities
Define and own the AI platform architecture: retrieval infrastructure, model lifecycle, evals framework, guardrails, and GenAI product feature design.
Lead the build of the shared AI platform layer consumed by all AI product features and reusable by internal business teams
Hands-on contributor: build production AI systems, write proofs of concept, and validate architecture through working software.
Set and enforce technical standards for the AI Platform team; drive architecture reviews and model quality reviews.
Coordinate with the Personalization team to define clear boundaries between the GenAI product layer and existing ML personalization infrastructure.
Requirements
Bachelor’s Degree in Computer Science or related field
A senior individual-contributor role acting as the technical authority guiding advanced AI across the commercial landscape—from traditional ML to Generative and agentic AI.
Responsibilities
Define and drive the technical vision, roadmap and architecture for traditional ML, Generative AI and agentic AI capabilities
Provide hands-on technical leadership to project teams; set design direction, review architectures and code, remove blockers
Lead the design of multi-agent systems and LLM applications, including orchestration, tool use, memory, guardrails and human-in-the-loop patterns
Establish standards for evaluation, including automated and human-in-the-loop methods, benchmarking, and quality gates
Drive approaches to fine-tuning, distillation and reinforcement learning (including RL from feedback)
Senior AI Engineer with strong LLM and product engineering experience to join our AI Platform team and help ship the AI-driven features across our app
Compensation
£120,000—£180,000 GBP
Responsibilities
Feature development: build and ship LLM-powered product features, including AI-generated mobile push notifications, personalized in-app content recommendations and custom user journeys, and conversational AI-powered insights
Latency and UX: reduce perceived and actual response latency through streaming, pregeneration, and inference-time tradeoffs, working closely with product and design on how AI output is experienced by users
Prompt and model behavior: iterate on prompt design and calibration, use fine-tuned health-domain models (Llama, Gemma, MedGemma) where relevant, and work with the Judge-as-a-Service ecosystem to keep outputs safe and high quality
Experimentation: help enable A/B testing for model and prompt changes, using evaluation gates to ship safely and measure impact
Data and evaluation: contribute to synthetic Q&A generation, golden test sets, and evaluation dataset design that supports the features you’re building
AI DevOps Engineer to design, develop, deploy, and support enterprise AI agents and Generative AI solutions for clients in consumer electronics and retail industries.
Compensation
Compensation: up to $10,043.48/M depends on experience
Responsibilities
Design and develop custom AI agents, GenAI applications, and AI-powered automation based on client business requirements.
Develop, test, deploy, monitor, and continuously enhance AI solutions across development and production environments.
Work with client stakeholders to translate business requirements into practical AI solutions.
Implement and maintain RAG pipelines, prompt engineering, agentic workflows, tool/function calling, and LLM integrations.
Provide ongoing DevOps and production support, including monitoring, troubleshooting, performance optimization, upgrades, and incident resolution.
Requirements
Bachelor's degree in Computer Science, Information Technology, Engineering, or related field
Senior Data Scientist for North America Data Analytics focusing on AI and Machine Learning capabilities, responsible for developing AI solutions, building pipelines, and designing architecture to meet business objectives.
Locations (4)
Jersey City, New Jersey, United States
Whitehouse Station, New Jersey, United States
Madison Avenue, New York, United States
Philadelphia, Pennsylvania, United States
Compensation
The pay range for the role is $125,400 to $182,600. The specific offer will depend on an applicant’s skills and other factors. This role may also be eligible to participate in a discretionary annual incentive program.
Lead measurement and insights work for generative music experiences, designing large-scale analyses, integrating evaluation signals, and building metrics and decision systems for Spotify's Artist-First AI Music lab.
Compensation
The United States base range for this position is $144,714–$206,735 USD, plus equity.
Responsibilities
Lead foundational measurement and insights work for generative music experiences
Design and run large-scale analyses and evaluations of generative music systems
Partner with Engineers and AI Researchers to integrate evaluation signals into model development loops
Identify behavioral and market dynamics that shape sustainable consumer demand and creator participation
Build practical, scalable metrics and decision systems that define what "good" looks like for AI music
Senior AI Engineer responsible for building and evolving AI evaluation systems, core agent harness, and scalable AI platform services at ClickUp.
Responsibilities
Design, build, and own ClickUp's AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features.
Build and evolve the core agent harness, and the AI that performs well on those evals, partnering closely with product and engineering to define what "great" looks like.
Build self‑improvement loops where agents modify other agents or prompts, rerun evals, and iterate automatically.
Benchmark new models as they're released and own model selection, routing, and fallback strategies across providers (e.g., OpenAI, Anthropic, Google, open-weight).
Architect, design, and implement scalable AI platform services that support the deployment, orchestration, and lifecycle management of LLMs and other AI models.
Collaborate with business partners and peers to understand and scope problems
Work as an individual contributor or team to develop, implement and test AI/ML models on Azure
Synthesize data to uncover trends and recommend improvements
Perform data validation, preprocessing, and exploratory analyses
Research and implement novel AI/ML predictive modeling approaches
Requirements
Bachelor’s or Master’s degree in a technical field such as Statistics, Computer Science, Data Science, Bioinformatics, Physics, Mathematics, Operations Research, Economics or Engineering
Nice to have: Property and casualty insurance experience
Gen AINewZensar · India, India
Senior AI/ML engineerNewFlo Health · London, United Kingdom
Senior Machine Learning Engineer, Agentic Evals and Self-Improvement (Nova)Iterable · Remote, US
Sr. Data ScientistChubb · Jersey City, US+3 more
Senior Data ScientistSpotify · New York, US
Senior AI DeveloperClickUp · Remote, US