- Remote, United States
- Remote
- Full-time · Senior
- $150K–$350K per year
- 3+ years
As a Software Engineer, Cybersecurity Benchmarks, you will play a key role in evaluating the cybersecurity capabilities of advanced AI models through rigorous, reliable, and publicly accessible benchmarks. You will design, develop, and maintain evaluation frameworks while ensuring that testing environments remain secure against increasingly capable AI systems.
Compensation
Annual Salary: Competitive compensation ranging from $150,000 to $350,000, depending on experience and qualifications.
Responsibilities
- Design, develop, and maintain cybersecurity benchmarks for evaluating AI capabilities within the organization's evaluation infrastructure, primarily using the Inspect library.
- Develop and prototype new benchmark concepts, experiments, and evaluation methodologies to measure emerging AI cybersecurity capabilities.
- Maintain and improve existing benchmark suites, ensuring evaluations can be executed reliably and efficiently whenever new AI models are released.
- Design, implement, and strengthen secure sandboxing environments to prevent highly capable AI models from escaping their designated evaluation environments.
- Monitor emerging developments in AI agents, coding harnesses, and their interactions with software environments to anticipate potential security vulnerabilities.
Requirements
- Nice to have: Benchmarking AI systems
- Nice to have: Capture the Flag (CTF) infrastructure
- Nice to have: AI coding harnesses
- Nice to have: Inspect AI framework
