Back to search

Test Analyst - AI Assurance

Test Analyst - AI Assurance role focusing on GenAI system architecture, RAG, and Agent-based systems, involving AI evaluation, Responsible AI validation, and observability.

  • India
  • Posted Nov 11, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareAI & Machine LearningData & Analytics

Responsibilities

  • Develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
  • Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
  • Golden dataset curation & Synthetic dataset generation
  • Implement AI evaluation frameworks like Ragas, DeepEval, etc.
  • Use AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
  • Instrumentation to establish AI Observability / tracing using OTEL libraries
  • Integrate AI evaluation into CI/CD pipeline
  • Responsible AI validation

Requirements

  • Experience in a SaaS

Skills

GenAI System ArchitectureRetrieval-Augmented Generation (RAG)Agent-based systemsPythonRagasDeepEvalArizeLangFuseConfident AIResponsible AI (RAI) ValidationGolden dataset curationSynthetic dataset generationOpenTelemetry (OTEL)LLM EvaluationAI Quality EngineeringAI GovernanceAI MonitoringAI Testing & AssuranceRoot Cause Analysis (AI Systems)RagasDeepEvalArizeLangFuseConfident AILangSmithGalileoEvidentlyOpenTelemetry (OTEL)

Full description

Test Analyst - AI Assurance Role Overview Strong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on experience in implementing Python-based AI evaluation frameworks such as Ragas, DeepEval, and similar tools. Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving Responsible AI (RAI) validation and reporting key RAI metrics. Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry (OTEL) instrumentation to enable effective monitoring, diagnostics, and continuous quality improvement of AI systems. Key Responsibilities •  Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior •  Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions •  Experience in Golden dataset curation & Synthetic dataset generation •  Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc. •  Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc. •  Experience in instrumentation to establish AI Observability / tracing using OTEL libraries •  Experience in Integrating AI evaluation into CI/CD pipeline •  Experience in Responsible AI validation Required Skills and Capabilities •  Strong understanding of GenAI System Architecture of RAG & Agents •  Proficiency in Python & AI evaluation SDKs •  AI Evaluation Frameworks (Ragas, DeepEval) •  Model Evaluation •  Agent Evaluation •  Responsible AI (RAI) Validation •  RAI Metrics Reporting •  Golden Dataset Curation •  Synthetic Dataset Generation •  AI Observability •  AI Tracing •  OpenTelemetry (OTEL) •  LLM Evaluation •  AI Quality Engineering •  AI Governance •  AI Monitoring •  AI Testing & Assurance •  Root Cause Analysis (AI Systems) •  Knowledge of regulatory considerations for AI systems Role-Specific Requirements •  Proficiency in Python & AI evaluation SDKs •  Strong understanding of RAG & agentic system architecture •  Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior •  Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions •  Experience in Golden dataset curation & Synthetic dataset generation •  Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc. •  Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc. •  Experience in instrumentation to establish AI Observability / tracing using OTEL libraries •  Experience in Integrating AI evaluation into CI/CD pipeline •  Experience in Responsible AI validation •  Knowledge of regulatory considerations for AI systems Experience Requirements More than 4 years of experience in AI Assurance Testing

Test Analyst - AI Assurance at HEXAWARE | Referso Jobs