Back to search

Automation Engineer - AI

  • Posted Nov 18, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareAI & Machine LearningData & Analytics

Responsibilities

  • Design & Build AI Eval Pipelines
  • LLM-as-a-Judge & Heuristic Scoring
  • Adversarial & Safety Testing (Guardrail Validations)
  • Partner with ML/Platform Engineers
  • Synthetic Test Data Generation
  • CI/CD Integration

Requirements

  • Experience in a Conversational Workflows
  • Experience in a Agentic Workflows

Skills

AILLMPrompt InjectionRAGCI/CD

Full description

Key Responsibilities - Design & Build AI Eval Pipelines: Architect automated evaluation frameworks measuring faithfulness, answer completeness, hallucination rates, and task completion across complex conversational and agentic workflows. - LLM-as-a-Judge & Heuristic Scoring: Develop, calibrate, and maintain rubric-based LLM-as-a-Judge evaluators. - Adversarial & Safety Testing (Guardrail Validations): Design systematic stress tests for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and adversarial edge cases. - Partner with ML/Platform Engineers: Inspect distributed traces, logs, and telemetry to pinpoint whether failures stem from prompts, models, RAG retrieval, or downstream connectors. - Synthetic Test Data Generation: Build programmatic pipelines to generate diverse persona simulations, multi-turn edge cases, and high-quality golden evaluation datasets. - CI/CD Integration: Integrate automated regression suites into build/release pipelines to gate prompt changes, model bumps, and backend API updates.

Automation Engineer - AI at ClifyX, INC | Referso Jobs