Sr. Software Engineer, Training/Inference (C++)
Design and optimize large-scale model serving systems for Grok, including distributed infrastructure, GPU kernel work, and low-level optimizations.
- Palo Alto, California, United States
- Full-time
- $220K–$357K per year
- Posted Nov 6, 2026
Responsibilities
- Architect and implement scalable distributed infrastructure for model serving (load balancing, auto-scaling, batch scheduling, global KV cache)
- Optimize latency and throughput of model inference under real production workloads
- Build and operate high-concurrency serving systems with excellent tail latency and high reliability at the scale required to serve millions of users
- Benchmark, fine-tune, and accelerate inference engines (including low-level GPU kernel work and code generation)
- Develop custom tools to trace, replay, and fix issues across the full stack — from orchestration down to GPU kernels
- Create robust CI/CD infrastructure for seamless endpoint deployment, image publishing, and inference engine updates
- Accelerate research on scaling test-time compute, RL rollout, and model-hardware co-design for next-generation systems
Requirements
- Bachelor’s degree in computer science, computer engineering, or science discipline
Nice to have: Experience with large-scale, high-concurrent production serving, Experience with GPU inference engines (e.g., vLLM, SGLang, Triton, TensorRT-LLM), Strong background in system optimizations: batching, caching, load balancing, parallelism, Low-level inference optimizations: GPU kernels, code generation, Algorithmic inference optimizations: quantization, speculative decoding, distillation, low-precision numerics, Experience with testing, benchmarking, and reliability of inference services, Experience designing and implementing CI/CD infrastructure for inference
Skills
Benefits
- equity
- comprehensive medical coverage
- vision coverage
- dental coverage
- 401(k) retirement plan
- short-term disability insurance
- long-term disability insurance
- life insurance
- various discounts and perks
More jobs nearby
Full description
SpaceXAISign up for a free job alert to see SpaceXAI's profile, website and LinkedIn, and the same for every employer on Referso Jobs.