The ML Training team is dedicated to increasing Stack's AV development velocity by accelerating machine learning iterations. Our core mission is to deliver a training system that is reliable, scalable, user-friendly and observable. In this Senior Staff role, you will drive the design and development of a high-performance multi-tenant AI training platform.
Locations (2)
Pittsburgh, Pennsylvania, United States
United States
Responsibilities
Design & evolve high-performance training platform components including orchestration, scheduling, training abstractions, control plane, observability and performance tuning
Deliver end-to-end ML model pipelines across logs processing, feature extraction, dataset schema design/storage, model configuration management, model training, and profiling/acceleration workflows
Instrument and analyze training infrastructure performance to resolve performance bottlenecks across all layers of the stack
Optimize training throughput across distributed nodes using topology-aware frameworks and hardware solutions (such as InfiniBand, NVLink, NCCL, or similar technologies)
Evangelize system abstractions and tooling that enable MLEs to rapidly iterate on models
Requirements
Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field
Senior System Software Engineer responsible for developing software tools and infrastructure for next-generation NVIDIA data center diagnostics, validating and testing new chip development and manufacturing diagnostic tests.
Compensation
Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary ranges for this position are: 150,000 USD – 286,000 USD for Level 3; and 183,750 USD – 351,000 USD for Level 4.
Responsibilities
Working closely with architecture, hardware and driver teams through the product development lifecycle of computing and graphics processors
Crafting software tools and infrastructure required for new chip development, validation, and productization
Assessing new hardware features and architecting manufacturing diagnostic tests using pre-beta CUDA and OpenGL extensions
Requirements
BS or MS degree in Electrical Engineering, Computer Engineering, Computer Science or equivalent experience
Nice to have: Background with PCIe, NVLink, InfiniBand, Ethernet
Research Interns put inquiry and theory into practice. Alongside fellow doctoral candidates and some of the world’s best researchers, Research Interns learn, collaborate, and network for life. Research Interns not only advance their own careers, but they also contribute to exciting research and development strides.
Compensation
The base pay range for this internship is USD $6,810 - $13,480 per month. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $8,920 - $14,610 per month.
Responsibilities
Using AI agents to automate hardware experimentation, system management, and performance analysis
Reverse engineering the performance and security characteristics of modern GPU servers and their memory systems
Understanding and overcoming performance bottlenecks in high-bandwidth GPU interconnects, such as NVLink and NVLink-C2C
Improving the efficiency of LLM serving by optimizing data movement and communication across CPUs, GPUs, and memory systems
Design and implement high-performance kernels for AI workloads, optimize compiler and runtime stacks, improve communication efficiency across large GPU clusters, and develop tooling for profiling and performance analysis.
Compensation
Depending on background, skills, and experience, the expected annual salary range for this position is $200,000 - $400,000 USD + equity.
Responsibilities
Design and implement high-performance kernels for AI workloads
Optimize compiler and runtime stacks for ML systems
Improve communication efficiency across large GPU clusters
Reduce latency and increase throughput for distributed workloads
Profile and eliminate system bottlenecks across the stack
Design, build, and operate distributed training, rollout, and orchestration systems for large-scale LLM and multimodal post-training across multi-GPU, multi-node environments.
Compensation
Depending on background, skills, and experience, the expected annual salary range for this position is $200,000 to $400,000, plus equity.
Responsibilities
Design, build, and operate distributed training, rollout, and orchestration systems for large-scale LLM and multimodal post-training across multi-GPU, multi-node environments.
Profile and optimize performance across the full-stack — model implementation, parallelism strategies, communication libraries, and GPU kernels — to improve throughput, latency, memory efficiency, hardware utilization, and cost.
Investigate numerical correctness and low-precision issues in distributed training and inference, including train–inference consistency for reinforcement learning.
Improve the reliability of long-running workloads through checkpointing, fault recovery, observability, and operational tooling.
Build supporting infrastructure for reinforcement learning and agentic post-training, including asynchronous rollout, trajectory collection, sandboxed execution, evaluation harnesses, and data pipelines.
Requirements
Nice to have: RL algorithms such as PPO, GRPO
Nice to have: modern post-training frameworks such as Miles, slime, AReaL, verl, Prime-RL
Nice to have: key open-source contributions to training or inference frameworks such as SGLang, vLLM, Megatron-LM
Define what reliability means for a GPU-accelerated AI platform and make it measurable. Own service-level indicators and objectives for the K0rdent Observability Framework, build APIs to expose SLIs, and establish alerting and error-budget practices.
Responsibilities
Define SLIs and SLOs based on the signals available across the platform
Design and build the API that exposes SLIs and reliability state to Platform Administrators and downstream systems
Establish alerting and error-budget practices that maximize signal and minimize noise
Partner with infrastructure, storage, and networking teams to ensure the right signals are instrumented and collected
Diagnose reliability and performance issues across the observability stack and drive their resolution
Creating and refining large-scale GPU cluster architectures within strict power and infrastructure limits, focusing heavily on power-aware design.
Compensation
$200,000 - $220,000
Responsibilities
Architect large-scale GPU clusters within fixed site power budgets that optimizes for maximum GPU density while reserving necessary headroom for compute services, storage, and networking.
Model and validate power consumption across the full cluster bill of materials (GPUs, CPUs, NICs, switches, fabric components, storage, and facility limits).
Evaluate tradeoffs across multiple fabric networking architectures (InfiniBand, RoCE, SpectrumX) as well as multi-plane, 2-tier/3-tier, and rail-optimized topologies.
Determine network scale limits based on switch radix, link speed, topology, and blocking requirements.
Gather, interpret, and maintain detailed SKU-level power and thermal specifications for GPUs, NICs, switches, DPUs, storage, and server platforms.
Senior Infrastructure Engineer responsible for building and maintaining software tools, automation, and data platforms that support hardware engineering lifecycle at Crusoe.
Compensation
Compensation will be paid in the range of $170,000 - $205,000. Restricted Stock Units are included in all offers.
Responsibilities
Design, build, and maintain software tools and frameworks for hardware development and sustaining lifecycle
Develop automated test, diagnostics, and health‑monitoring frameworks for large‑scale server fleets
Build tooling, monitoring and logging systems for hardware engineers
Create automated validation and characterization workflows for GPU, CPU, and high‑performance compute platforms
Build and operate data pipelines, dashboards, and analytics for hardware failure trends
Requirements
Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent experience
Nice to have: GPU clusters
Nice to have: GPU-to-GPU communication diagnostics
Lead performance optimization for large language model training and inference on NVIDIA GPU clusters, improving GPU utilization, memory footprint, and distributed training efficiency.
Locations (2)
Cambridge, Massachusetts, United States
San Francisco, California, United States
Compensation
Expected Base Salary Range
$148,000—$240,000 USD
Responsibilities
Profile and optimize GPU compute kernels for large language models
Optimize GPU cluster performance for collective/MoE frameworks and RDMA/IB/EFA usage
Manage big data pipelines for mid-training scale data
Optimize reinforcement learning pipelines and async cycles
Ensure optimized configurations are tested, accessible, and adaptable for research tasks
Requirements
Bachelor's or Master's in Computer Science or related quantitative field