- Seoul, Seoul, South Korea
- On-site
- Full-time · Associate
Software engineer to join the core team behind vllm-sr, owning the whole stack: the router, the serving engine, the decision models and the open-source community, training and serving on AMD Instinct GPUs, and making ROCm a first-class platform for intelligent routing.
Responsibilities
- Build the routing intelligence of vllm-sr, from decision models that pick the right model for every request to semantic, policy-based and workload-aware routing, semantic caching, and routing for safety, privacy, tools and agents.
- Make vllm-sr simple to deploy and operate in production, with first-class integrations for vLLM, Kubernetes, Envoy, Gateway API and AI gateways.
- Build vllm-sr into a universal serving engine for decision models that exposes the System One API and runs efficiently on everything from AMD Instinct GPUs to CPUs.
- Optimize decision model inference across batching, memory, latency and throughput, down to GPU kernels in HIP or Triton on ROCm where it matters.
- Train foundation decision models across sizes and keep pushing their capabilities to the frontier, owning data, training, post-training and evaluation.
Requirements
- Bachelor's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field
- Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field
- Nice to have: Model routing
- Nice to have: API gateways
- Nice to have: LLM inference optimization