Principal Software Engineer - AI Runtime & Distributed Inference Systems
Lead the architecture of large-scale AI inference systems and distributed runtimes.
- San Jose, California, United States
- Posted Nov 7, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise Software
Responsibilities
- Define the software architecture for large-scale AI inference systems
- Lead the evolution of the distributed runtime stack for dense and MoE foundation models
- Establish long-term technical strategy for performance, scalability, reliability, and programmability
- Influence hardware-software co-design decisions across future accelerator generations
- Architect runtime systems supporting tensor, pipeline, expert, data parallelism and multi-rack inference execution
- Design systems for efficient prefill and decode execution
- Drive continuous batching and request scheduling strategies
- Optimize throughput, utilization, and QoS
- Lead design of token routing and expert execution systems
- Define expert placement and replication strategies
- Develop mechanisms for expert load balancing and congestion control
- Optimize communication and compute overlap across distributed expert clusters
- Architect high-performance collective communication libraries
- Design software interfaces for custom accelerator fabrics
- Drive latency and bandwidth optimization across accelerator clusters
- Define architecture for distributed KV-cache management
- Drive memory placement, eviction, and tiering policies
- Establish performance methodology, benchmarks, and KPIs
- Analyze bottlenecks across compute, memory bandwidth, communication fabric, expert routing, scheduling layers
- Lead root-cause analysis of production performance issues
- Drive reliability, observability, telemetry, and debugging architecture
- Define health monitoring and fleet management capabilities
- Partner with infrastructure teams to operate inference systems at scale
- Establish production readiness standards and operational excellence practices
Requirements
- BS, MS, or PhD in Computer Science, Computer Engineering, Electrical Engineering
- Experience in a AI infrastructure
- Experience in a distributed systems
- Experience in a hardware-software co-design
Nice to have: large language model serving systems, vLLM, SGLang, TensorRT-LLM, Triton, PyTorch Distributed, NCCL, custom accelerators, GPUs, TPUs, AI ASICs, transformer architectures, sparse models, MoE systems, RDMA, InfiniBand, RoCE, PCIe, Ethernet fabrics, compiler technologies, graph optimization, kernel scheduling, HBM, memory hierarchy design, cache management
Skills
C++Rustsystems programmingdistributed systemsAI inference systemsnetworkingdistributed communicationarchitectureperformance engineeringmemory managementcommunication librariescollective communicationvirtualizationrack controllersPCIeNCCLMPIRoCEv2PFCDCQCNC++RustNCCLMPIRoCEv2PFCDCQCN
More jobs nearby
Full description
International Recruiting LLCCompany details are for members
Sign up for a free job alert to see International Recruiting LLC's profile, website and LinkedIn, and the same for every employer on Referso Jobs.