AI Engineer
Build, test, and support AI workflows and agents on an enterprise AI workflow management platform supporting the Department of the Air Force in Azure IL-5 cloud environments.
- Remote, United States
- Full-time
- Remote
- $115K–$252K per year
- Posted Nov 6, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software
Responsibilities
- Build AI workflows and agents using the platform's frameworks, tools, and reference architectures
- Develop RAG pipelines, including document ingestion, chunking, embedding, vector search, and retrieval tuning
- Build prompt chains, tool calls, and agent logic for document generation, summarization, and task automation
- Integrate AI workflows with Azure OpenAI services, ServiceNow APIs (Flow Designer, REST, scripted integrations), and internal data sources
- Build reusable components, templates, and connectors that both technical and non-technical users can leverage
- Implement workflow orchestration and business process automation
- Develop and execute test plans for AI workflows, including unit, integration, and user acceptance testing
- Build evaluation datasets and automated evaluation harnesses to measure the accuracy, relevance, groundedness, and reliability of LLM outputs
- Define and track success metrics and evaluation criteria for AI-driven workflows
- Identify, document, and triage defects, hallucinations, and performance issues, and drive root-cause analysis and remediation
- Perform regression testing when models, prompts, or platform components change
- Partner with the AI Platform Architect to implement designs, evaluate tools, and give feedback on platform capabilities and limitations
- Prototype solutions and proofs-of-concept for government use cases, and present results to customers and stakeholders
- Work with ServiceNow developers and Azure cloud engineers to troubleshoot integration issues across Azure, ServiceNow, and government networks
- Contribute to design reviews, engineering standards, and technical documentation
- Support technical screening of AI/ML talent when requested
- Support monitoring, logging, and alerting for AI workflows and agents
- Manage prompt and model versioning, and support CI/CD deployments of AI services
- Monitor output quality, model drift, and data quality in production workflows, and report findings
- Track model and compute usage costs and recommend optimizations
- Build and operate within DISA STIGs, IL-5 requirements, and Department of the Air Force security policies
- Implement CUI handling and data classification controls in AI workflows
- Provide engineering evidence and documentation to cybersecurity teams for ATO and continuous monitoring.
Requirements
- Bachelor’s Degree in relevant field
- Experience in a government cloud
- Experience in a defense
Nice to have: Active Secret clearance, Top Secret eligibility, Azure Government, AWS GovCloud, Kubernetes, Azure Kubernetes Service (AKS), DoD compliance frameworks, IL-4/IL-5, FedRAMP, NIST 800-53, STIGs, MLOps, MLflow, CI/CD for ML, PostgreSQL, data modeling, ATO processes, security accreditation, ServiceNow Flow Designer, Sec+ certification
Skills
LLM applicationsRAGprompt engineeringagentic workflowsPythonLangChainLlamaIndexAzure cloud servicesAzure OpenAIAI FoundryDevOpsAPI ManagementDockerREST APIvector databasesembedding strategiestest automationQAServiceNow integrationworkflow orchestrationPythonLangChainLlamaIndexAzureDockerServiceNowDockerPostgreSQL
Benefits
- flexible time off
- healthcare
- wellness
- financial
- retirement
- family support
- continuing education
Full description
CaciCompany details are for members
Sign up for a free job alert to see Caci's profile, website and LinkedIn, and the same for every employer on Referso Jobs.