- San Jose, California, United States
- Full-time
- $180K–$450K per year
Improve the performance of Hark's Megatron-based pretraining systems across thousands of GPUs, focusing on model FLOPs utilization, training throughput, and kernel performance.
Compensation
The US base salary range for this full-time position is between $180,000 - $450,000 annually.
Responsibilities
- Own MFU and step time for pretraining runs across dense and mixture-of-experts models
- Write or integrate fused and custom kernels (CUDA, Triton, CUTLASS) for attention, normalization, and mixture-of-experts routing
- Overlap communication with compute, reduce pipeline bubbles, and tune activation recomputation and memory use to raise throughput per GPU
- Implement model and optimizer sharding, and optimize data, tensor, pipeline, sequence, context, and expert parallelism for dense and mixture-of-experts architectures
- Profile GPU kernels, collective communication, data loading, and mixed-precision training to identify performance and numerical stability bottlenecks
Requirements
- Nice to have: mixture-of-experts training at scale
- Nice to have: load balancing
- Nice to have: all-to-all communication optimization
- Nice to have: fault-tolerant training
- Nice to have: elastic training