Senior Agentic AI Engineer
Senior/Lead Agentic AI Engineer – LLM Evaluation & Observability
- Dallas, Texas, United States
- Hybrid
- Competitive salary
- Posted Nov 5, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software
Responsibilities
- Design, develop, and deploy production‑grade agentic and generative AI applications
- Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore
- Implement multi‑agent workflows, tool calling, state management, memory, and orchestration
- Create scalable LLM‑powered applications and reusable AI components
- Deploy agentic workflows with Amazon Bedrock AgentCore
- Design and implement Retrieval‑Augmented Generation (RAG) solutions
- Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management
- Optimize retrieval accuracy, response quality, latency, and cost
- Apply grounding, hallucination reduction, and context‑relevance techniques
- Build automated evaluation frameworks with RAGAS, DeepEval, and similar tools
- Evaluate RAG pipelines using metrics such as Faithfulness, Answer Relevancy, Context Precision, Context Recall, and Answer Correctness
- Implement LLM‑as‑a‑Judge techniques to assess generated responses, retrieval quality, agent behavior, and workflow execution
- Establish evaluation datasets, benchmarks, and regression‑testing pipelines for continuous quality assessment
- Implement end‑to‑end tracing of multi‑agent workflows, including tool invocation, agent handoffs, execution paths, and failure handling
- Establish observability practices to detect hallucinations, reasoning failures, tool‑calling errors, and execution inconsistencies
- Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse
- Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance
- Develop debugging mechanisms for complex multi‑step agentic workflows
- Create monitoring practices to detect performance degradation and operational issues
- Deploy AI applications using Amazon Bedrock and Bedrock AgentCore
- Integrate foundation models, agent frameworks, APIs, and enterprise data sources
- Apply production‑readiness practices covering scalability, reliability, security, and performance
- Support CI/CD integration and automated evaluation of AI applications
- Optimize model selection, inference costs, latency, and resource utilization
- Provide technical guidance to engineering and QA teams
- Establish reusable frameworks, implementation patterns, and best practices
- Collaborate with architecture, DevOps, QA, and engineering teams
- Participate in design discussions, code reviews, proof‑of‑concept development, and implementation
- Mentor engineers and help teams adopt effective AI development and testing practices
Requirements
- Experience in a Generative AI
- Experience in a Agentic AI
- Experience in a LLM Evaluation
- Experience in a AI Observability
Nice to have: 8+ years of software engineering or IT experience, Demonstrated hands‑on experience building and deploying GenAI and Agentic AI applications, Recent experience in LLM engineering, agent orchestration, and AI evaluation, Experience with enterprise‑scale AI implementations, Familiarity with AWS cloud services and enterprise AI security practices, Experience establishing AI quality standards and evaluation methodologies
Skills
LangGraphLangChainAmazon BedrockAmazon Bedrock AgentCoreAI agentsmulti‑agent orchestrationtool callingworkflow managementLarge Language ModelsRetrieval‑Augmented GenerationEmbeddingsvector searchPrompt engineeringLLM application developmentDeepEvalLLM‑as‑a‑Judge techniquesEvaluation metricsBenchmarkingAutomated evaluationRegression testingRAGASOpenTelemetryLangSmithLangfusePythonREST APIsasynchronous programmingbackend developmentLangGraphLangChainAmazon BedrockAgentCoreRAGASDeepEvalOpenTelemetryLangSmithLangfusePythonREST APIs
More jobs nearby
Full description
Company details are for members
Sign up for a free job alert to see Select Minds LLC's profile, website and LinkedIn, and the same for every employer on Referso Jobs.