Containerized test environments jobs

Agentic Evaluation & Testing Specialist

Siemens · Posted Nov 5, 2026

Kitchener, Ontario, Canada
Hybrid
Full-time
CA$100K–CA$150K per year
4+ years

Apply with RefersoAI Apply on careers page

Build automated test infrastructure and evaluation workflows for AI-driven industrial software

Compensation

Salary is commensurate with experience, and ranges between $ 100,000 CAD - $ 150,000 CAD, excluding bonus and benefits. In addition to base salary, this role includes eligibility for an annual discretionary bonus of 10%

Responsibilities

Build automated test infrastructure for a product developed across multiple sites
Design evaluation methods for agent behavior
Build the assessment workflow into the product
Integrate security checks into the workflow
Keep test expectations independent of implementation

Requirements

Degree in computer science, software engineering, or a related field
Nice to have: agent evaluation frameworks
Nice to have: AI observability tooling (Langfuse, LangSmith, Promptfoo)
Nice to have: safety-relevant or regulated software experience
Nice to have: background in safety-relevant or regulated software

Skills

Python containerized test environments Kubernetes contract testing

Senior QA Lead

Senior QA Lead

Responsibilities

Requirements

Skills

Agentic Evaluation & Testing Specialist

Compensation

Responsibilities

Requirements

Skills

Benefits