Staff Software Engineer, AI Inference
Spearhead the effort to build and optimize the AI inference stack to deliver best-in-class performance, reliability, cost efficiency, and flexibility across the latest generation of open-source language models.
- Remote, United States
- Full-time
- Remote
- $190K–$230K per year
- Posted Jul 13, 2026
INDUSTRYSoftware & InternetSaaS & Enterprise SoftwareAI & Machine Learning
Responsibilities
- Lead the design and development of our production inference platform.
- Define the technical roadmap for inference infrastructure, model serving, and runtime optimization.
- Build and operate scalable, cost-effective systems for serving large language models in production.
- Evaluate and integrate modern inference technologies, frameworks, and serving runtimes.
- Optimize latency, throughput, GPU utilization, memory efficiency, and infrastructure cost.
- Develop systems for model deployment, traffic routing, autoscaling, scheduling, observability, and operational excellence.
- Partner with ML engineers to productionize new models and inference techniques.
- Establish benchmarking methodologies to evaluate new models, runtimes, and hardware.
- Make key architectural decisions around when to build internally versus leverage open-source or commercial solutions.
- Mentor engineers as the team grows and help establish engineering best practices for AI infrastructure.
Requirements
- Experience in a Litigation AI
Skills
AI inference systemsLLM serving infrastructurevLLMSGLangTensorRT-LLMTriton Inference ServerHugging Face TGINVIDIA DynamoDistributed systemsBackend infrastructureHigh-performance platform engineeringGPU workloads optimizationPythonGoRustC++vLLMSGLangTensorRT-LLMTriton Inference ServerHugging Face TGINVIDIA Dynamo
Benefits
- health insurance
- equity
Full description
Syllo
Company details are for members
Sign up for a free job alert to see Syllo's profile, website and LinkedIn, and the same for every employer on Referso Jobs.