Lead the team building out AI Observability and Evaluation features used by developers to measure the accuracy, relevance, performance and safety of artificial intelligence models, agents and RAG applications.
Responsibilities
Perform primary research, interviewing end-users and buyers to deeply understand their AI Observability and Evaluation needs and benefits sought.
Partner with engineers and technical stakeholders to articulate, scope, and deliver well integrated AI Observability and Evaluation features and product line extensions.
Build on Chronosphere’s industry leading observability platform to enable the systemic testing, measuring and benchmarking of AI models, particularly Large Language Models and agentic AI applications.
Set KPIs to articulate and measure intended value and impact for customers.
Develop comprehensive roadmaps and deliver plans to achieve stated outcomes for customers and the business.
Lead a team to build and deliver shared AI capabilities across NetDocuments’ platform, driving engineering execution, technical direction, and talent development.
Locations (2)
Lehi, Utah, United States
United States
Compensation
The compensation range for this position is: $170,000 - $190,000
Responsibilities
Drive engineering delivery and execution
Lead planning and execution of engineering initiatives
Partner with Product and Architecture to translate customer needs into scalable, secure solutions
Balance feature development with system reliability, performance, cost efficiency, and operational excellence
Proactively identify technical risks, dependencies, and tradeoffs
Requirements
Bachelor's degree in Computer Science
Bachelor's degree in Engineering
related field
Nice to have: Experience developing AI-powered solutions within legal technology
Design and deliver secure, grounded AI solutions for enterprise customers using Azure AI services, LLM integration, RAG, and agent orchestration.
Responsibilities
Lead technical discovery sessions and translate business problems into prioritized AI use cases
Design and implement grounded AI assistants and agents with Azure AI Services, Azure OpenAI, RAG, prompt engineering, tool calling, and REST APIs
Build agent actions and integrations with enterprise systems, data sources, and Microsoft services using secure authentication and authorization patterns
Develop retrieval, chunking, grounding, and evaluation approaches to improve relevance, accuracy, and traceability
Establish evaluation datasets, task-success measures, safety checks, responsible AI controls, and prompt-injection defenses
Manager, Software EngineeringNetDocuments · Remote, US+1 more
AI / Agent EngineerNetrix Global · Pune, India