Building multimodal foundation models, focusing on pretraining stack, model architecture, scaling, and distributed training.
The US base salary range for this full-time position is between $180,000 - $450,000 annually.
- Conduct research on model architecture, optimization, and scaling to improve the capability and efficiency of our largest models.
- Establish baselines and run controlled experiments to evaluate which architectural and optimization changes improve performance at larger scales.
- Develop training recipes, including learning-rate schedules, context lengths, token budgets, and compute allocation.
- Investigate mixture-of-experts architectures, hybrid attention, and support for long-context training.
- Diagnose loss spikes, divergence, and numerical issues, working with training engineers to distinguish modeling problems from infrastructure failures.
- Nice to have: mixture-of-experts routing
- Nice to have: sparse architectures
- Nice to have: long-context methods
- Nice to have: data mixtures
- Nice to have: curriculum
Member of Technical Staff, Mid-trainingNewHark · San Jose, US