Back to search

Cloud Inference Engineer

Deploy and tune models with optimizations like KV caching, paged attention, sequence packing, etc. Conduct model performance reviews, improve scheduler, batcher, autoscaling; profile latency, cost, utilization. Occasionally write kernels and engage in tasteful communication.

  • San Francisco, California, United States
  • Full-time
  • On-site
  • $150K–$250K per year
  • Posted Nov 7, 2026
INDUSTRYSoftware & InternetData & Analytics

Responsibilities

  • Deploy and tune models with optimizations like KV caching, paged attention, sequence packing, etc.
  • Conduct model performance reviews
  • Improve scheduler, batcher, autoscaling; profile latency, cost, utilization
  • Occasionally write kernels and engage in tasteful communication

Skills

CUDAGPU inference optimizationvLLMSGLangTensorRT-LLMKV cachingpaged attentionbatchingtoken streamingdistributed computeGPUsCUDAvLLMSGLangTensorRT-LLM

More jobs nearby

Full description

QUALIFICATIONS - CUDA + GPU inference optimization - vLLM, SGLang, or TensorRT-LLM experience - KV caching, paged attention, batching, token streaming, etc. - Distributed compute (with GPUs is a super plus) - No degree required COMPANY Luminal (YC S25) builds an AI compiler and serving stack that makes models 10x faster and production ready with one line. ROLE Founding, on site in downtown SF. Ship low latency, high throughput model serving on Luminal Cloud. Day to day responsibilities: - Deploy and tune models with optimizations like KV caching, paged attention, sequence packing, etc. - Conducting model performance reviews - Improve scheduler, batcher, autoscaling; profile latency, cost, utilization - Sometimes write kernels and, yes, occasional tasteful shitposting LOCATION AND COMPENSATION San Francisco, CA, US · Full-time · On-site. Annual base salary: $150,000–$250,000 USD. Equity: 0.15%–0.75%. Visa: US citizen/visa only.

Cloud Inference Engineer at Luminal | Referso Jobs