Build and operate production-grade model-serving and inference systems supporting high-throughput, low-latency AI workloads.
Compensation
Competitive base pay for Bellevue market
Certain roles are eligible for additional rewards, including merit increases, annual bonus, and long term incentives.
Responsibilities
Build and operate production-grade model-serving and inference systems supporting high-throughput, low-latency AI workloads.
Optimize inference infrastructure for token throughput, latency, scalability, and cost efficiency across different model architectures and workloads.
Design systems that maximize GPU utilization while maintaining predictable performance and reliability.
Improve the scalability and operational maturity of inference platforms as customer demand grows.
Partner with AI training, GPU performance, orchestration, and infrastructure teams to ensure smooth transitions from model development to production serving.
Lead product strategy and roadmap for AI infrastructure stack, defining compiler, runtime, and framework integration features.
Responsibilities
Translate customer, engineering, and business requirements into detailed Product Requirements Documents (PRDs), technical specifications, and feature definitions.
Develop detailed product schedules, milestone plans, and release timelines while managing dependencies across multiple engineering organizations.
Define requirements for the software and AI model compilation toolchain.
Lead product partnerships and technical integrations with major machine learning frameworks and runtimes.
Define performance optimization roadmap for low-level operator libraries.
Requirements
Master of Science (M.Sc.) in Computer Science, Electrical Engineering, Mathematics, or Physics
Doctor of Philosophy (Ph.D.) in Computer Science, Electrical Engineering, Mathematics, or Physics
Nice to have: MBA
Nice to have: open-source AI infrastructure contributions
Lead product management for AI infrastructure stack, defining roadmap for compiler, runtime, and framework integration.
Responsibilities
Translate customer, engineering, and business requirements into detailed Product Requirements Documents (PRDs), technical specifications, and feature definitions.
Develop detailed product schedules, milestone plans, and release timelines while managing dependencies across multiple engineering organizations.
Define requirements for the software and AI model compilation toolchain.
Lead product partnerships and technical integrations with major machine learning frameworks and runtimes.
Define performance optimization roadmap for low-level operator libraries.
Requirements
Master of Science (M.Sc.) in Computer Science, Electrical Engineering, Mathematics, or Physics
Doctor of Philosophy (Ph.D.) in Computer Science, Electrical Engineering, Mathematics, or Physics
Nice to have: MBA
Nice to have: open-source AI infrastructure contributions
Build next-generation green intelligent computing infrastructure and drive breakthroughs in performance and energy efficiency of AI clusters through innovation.
Responsibilities
Build next-generation green intelligent computing infrastructure, driving continuous breakthroughs in performance and energy efficiency of AI clusters through innovative technology.
Participate in cutting-edge sustainable computing technology research and apply results to large-scale AI infrastructure scenarios.
Conduct cutting-edge research in sustainable computing and green AI, exploring energy efficiency optimization solutions for large-scale AI training and inference scenarios.
Design and build fine-grained energy consumption analysis systems to achieve precise energy monitoring, decomposition, and modeling from cluster, node, and device to task levels.
Develop dynamic frequency and voltage scaling (DVFS), resource management, and intelligent scheduling optimization algorithms for CPU/GPU/NPU resources to improve overall system energy efficiency.
Senior member of the LLM inference framework team responsible for building and optimizing production-grade single-node and distributed inference runtimes for large language models on AMD GPUs.
Responsibilities
Architect and optimize distributed LLM inference runtimes based on in-house LLM engines or open-source stacks such as vLLM, SGLang, and llm-d
Design and improve TP / PP / EP (MoE) hybrid execution, including KV-cache management, attention dispatch, and token scheduling
Implement and optimize multi-node inference pipelines using RCCL, RDMA, and collective-based execution
Drive throughput, latency, and memory efficiency across single-GPU and multi-GPU clusters
Lead the design and development of our production inference platform and define the technical roadmap for inference infrastructure, model serving, and runtime optimization.
Compensation
Salary Range ($190- $230K) plus health insurance and equity. United States - Remote Pay Range $190,000—$230,000 USD
Responsibilities
Lead the design and development of our production inference platform
Define the technical roadmap for inference infrastructure, model serving, and runtime optimization
Build and operate scalable, cost-effective systems for serving large language models in production
Evaluate and integrate modern inference technologies, frameworks, and serving runtimes
Optimize latency, throughput, GPU utilization, memory efficiency, and infrastructure cost
Lead a team of software engineers focused on AI/ML infrastructure and performance engineering, driving technical leadership, project delivery, and cross‑team collaboration across Google’s global ML stack.
Responsibilities
Learn and build an intuitive understanding of existing data collection, analysis, and visualization workflows with deep introspection across Frameworks, Accelerated Linear Algebra (XLA) and runtime stack.
Support new and exciting ML paradigms (such as horizontal scaling for upcoming TPU chips) by making contributions across the end to end stack and analysis tools.
Partner with ML Stack leads to understand model optimization use cases and bring debugging to feel native in 3P environments (VSCode, Cursor, Grafana, etc.).
Work with OSS ML inference frameworks such as vLLM, SGLang to provide insights in Xprof about performance improvement opportunities.
Partner with other teams that own various parts of the ML stack to understand performance optimization use cases.
Requirements
Bachelor's degree or equivalent practical experience
Inference Engineer - Member of Technical StaffNewDesignworks Talent · Bellevue, US
AI可持续计算资深研究员NewLenovo · Beijing, China
AI Framework Eng.(VLLM CI)NewAdvanced Micro Devices, Inc · Shanghai
Senior Software Development EngineerAmazon · Seattle, US+1 more
Software Engineering Manager, AI/ML Infrastructure and Performance EngineeringGoogle · Bengaluru, India