Join a small, technical team building benchmarks to evaluate frontier AI agents on realistic, domain-specific workflows. You will own evaluation design and implementation, helping ensure benchmark results are rigorous, credible, and relevant to real-world performance.
Compensation
Base salary: $100,000 to $170,000 annually. Visa sponsorship is available.
Responsibilities
Design, implement, and maintain benchmarks for evaluating frontier agents on domain-specific tasks.
Work with subject-matter experts to translate real workflows into realistic tasks and evaluation criteria.
Build reliable infrastructure for running models and agents against benchmark tasks at scale.
Develop metrics and analyses to assess benchmark difficulty, reliability, and failure modes.
Validate whether benchmark results correlate with real-world evaluations and user needs.
Senior Software Engineer to help build the core libraries, evaluation systems, and reusable harness capabilities that make AI agents safer, faster, and easier for developers to trust.
Locations (2)
United States, United States
Canada, Canada
Compensation
The expected base salary ranges for this position are: 184,000 USD – 287,500 USD for Level 4; and 224,000 USD – 356,500 USD for Level 5.
Responsibilities
Define and evolve the technical approach for evaluating and improving open agent harnesses across frameworks, models, and benchmark environments.
Build and operate scalable, reproducible evaluation systems spanning CI, scheduled compute, sandboxed execution, artifact provenance, trace capture, and analysis.
Diagnose agent failures from traces, logs, tool results, and benchmark artifacts, separating harness, evaluator, environment, inference, and model causes.
Design controlled experiments and promotion criteria that distinguish genuine harness improvements from noise, benchmark artifacts, or model-specific gains.
Build the learning systems that make autonomous laboratories possible by owning the loop from expert demonstrations and robot experience through policy training, evaluation, and deployment.
Responsibilities
Develop mid- and post-training methods for robot policies using real-world demonstrations, robot experience, and task-specific data.
Build systems that learn from expert chemists, turning demonstrations of difficult scientific procedures into capabilities robots can reproduce and generalize.
Develop and adapt world models, world-action models, vision-language-action models, and other learned representations for precise, contact-rich manipulation.
Build the training and evaluation systems needed to ingest, process, curate, and learn from large-scale multimodal robot datasets.
Deploy policies on real robotic systems and diagnose failures across models, data, perception, controls, hardware, and task design.
Build the end-to-end system that lets customers adapt UFORCE models on their own private data without exposing that data to us, spanning data processing, foundation-model-assisted labeling, human-in-the-loop QA, active learning, training, evaluation, and model promotion.
Responsibilities
Own the ML adaptation pipeline from raw customer data to trained, evaluated, deployable models.
Build foundation-model-assisted labeling workflows using tools such as SAM-2, Grounding DINO, open-vocabulary models, LLM steering, and human review.
Design self-serve operator workflows using yes/no/maybe feedback and natural-language corrections.
Create versioned datasets with lineage, data cards, label provenance, class distributions, and known gaps.
Build QA methods that catch systematic pseudo-label errors, missing annotations, and long-tail data gaps.
Design, build, and maintain internal evaluation suites and infrastructure for post-training AI models, ensuring reliability, scalability, and reproducibility.
Compensation
Compensation & Equity: Competitive base salary plus equity, so you share in what we build. Final offers reflect skills, experience, and location.
Responsibilities
Work on the internal eval suite and infrastructure: keep it reliable, optimized, and reproducible at scale.
Design held-out evals for priority capabilities such as agentic workflows, computer use, and cybersecurity, with documented construction and contamination checks.
Build evals targeted at small edge models, where most public evaluations built for frontier models don't apply.
Read generated outputs to debug unexpected scores before anyone acts on them.
Turn customer and open-source feedback into environments usable for data, training, and evaluation, and partner with other teams to design evals for their specific needs.
Ship focused harness and runtime improvements, including upstream open-source contributions, detectors, regression tests, and maintainable documentation.
Requirements
Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Applied Math, or a related field, or equivalent experience
Nice to have: Meaningful contributions to open-source coding agents, harnesses, agent runtimes, evaluation frameworks, developer tools, or observability projects.
Nice to have: Published research, technical writing, patents, or substantive open-source work in autonomous software engineering, agent evaluation, reliability, tool use, or inference efficiency.
Nice to have: Experience with SWE-bench, Terminal-Bench, AgentBench, or comparable public agent benchmarks.
Nice to have: Shipped improvements to context management, tool interfaces, retries, sandboxed repository execution, long-running agent loops, or evaluation environments.