Test Analyst - AI Assurance
Test Analyst - AI Assurance role focusing on GenAI system architecture, RAG, and Agent-based systems, involving AI evaluation, Responsible AI validation, and observability.
- India
- Posted Nov 11, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareAI & Machine LearningData & Analytics
Responsibilities
- Develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
- Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
- Golden dataset curation & Synthetic dataset generation
- Implement AI evaluation frameworks like Ragas, DeepEval, etc.
- Use AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
- Instrumentation to establish AI Observability / tracing using OTEL libraries
- Integrate AI evaluation into CI/CD pipeline
- Responsible AI validation
Requirements
- Experience in a SaaS
Skills
GenAI System ArchitectureRetrieval-Augmented Generation (RAG)Agent-based systemsPythonRagasDeepEvalArizeLangFuseConfident AIResponsible AI (RAI) ValidationGolden dataset curationSynthetic dataset generationOpenTelemetry (OTEL)LLM EvaluationAI Quality EngineeringAI GovernanceAI MonitoringAI Testing & AssuranceRoot Cause Analysis (AI Systems)RagasDeepEvalArizeLangFuseConfident AILangSmithGalileoEvidentlyOpenTelemetry (OTEL)
Full description
Company details are for members
Sign up for a free job alert to see HEXAWARE's profile, website and LinkedIn, and the same for every employer on Referso Jobs.