Back to search

Machine Learning Infrastructure Engineer

Join an AI team building large physics foundation models to predict real-world systems and identify effective actions. You will develop the distributed infrastructure that supports model training and inference, with weather data as a key application area.

  • San Francisco, California, United States
  • Full-time
  • On-site
  • $200K–$400K per year
  • Posted Nov 19, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareAI & Machine LearningData & Analytics

Responsibilities

  • Design, deploy, and maintain distributed machine learning training and inference clusters.
  • Build scalable end-to-end pipelines for petabyte-scale datasets and model training across the ML lifecycle.
  • Evaluate training approaches, including parallelization methods and numerical precision trade-offs across model sizes.
  • Profile and debug low-level GPU operations to improve performance.
  • Track relevant research and apply promising ideas to infrastructure and workflows.

Requirements

  • Experience in a Physical AI
  • Experience in a AI for science

Nice to have: Physical AI, AI for science, Robotics, Autonomous systems, Biology

Skills

Benefits

  • Relocation support

More jobs nearby

Full description

ABOUT THE ROLE Join an AI team building large physics foundation models to predict real-world systems and identify effective actions. You will develop the distributed infrastructure that supports model training and inference, with weather data as a key application area. WHAT YOU'LL DO - Design, deploy, and maintain distributed machine learning training and inference clusters. - Build scalable end-to-end pipelines for petabyte-scale datasets and model training across the ML lifecycle. - Evaluate training approaches, including parallelization methods and numerical precision trade-offs across model sizes. - Profile and debug low-level GPU operations to improve performance. - Track relevant research and apply promising ideas to infrastructure and workflows. WHAT WE'RE LOOKING FOR - Two to ten years of experience building large-scale infrastructure for pretrained or foundation models, including compute clusters, data infrastructure, inference, and distributed training. - Hands-on experience optimizing large-scale training, data, or inference workloads, and using distributed training frameworks such as FSDP or DeepSpeed. - Experience with low-level GPU performance analysis and debugging, including CUDA and JAX. - Familiarity with Python, C++, Linux, NVIDIA GPUs, cloud platforms such as GCP, AWS, or Azure, and containerization and orchestration tools such as Docker and Kubernetes. - Background in physical AI or AI for science, such as robotics, autonomous systems, or biology, and motivation to work on scientific or physical-world applications. - Comfort working across the ML lifecycle in a fast-paced environment, including analyzing experiments and ablation studies. COMPENSATION & BENEFITS Base salary is $200,000 to $400,000 annually. Visa sponsorship is not available. LOCATION On-site in San Francisco, California. Relocation support is provided.

Machine Learning Infrastructure Engineer at True Talent Solutions Partners | Referso Jobs