Technology Radar
llm-d is an open-source distributed inference serving stack designed to operate model servers such as vLLM and SGLang across multiple Kubernetes nodes. Traditional web applications often rely on stateless routing and horizontal scaling, but LLM inference is highly stateful due to key value (KV) caching. Rather than treating model-serving pods as interchangeable HTTP backends, llm-d uses inference-aware routing to coordinate requests across multiple accelerators, extending optimization beyond individual model servers.
Built around Kubernetes and the Gateway API Inference Extension, llm-d offers several capabilities tailored to stateful AI traffic. It supports model-aware and KV-cache-aware routing, using factors such as cache affinity, queue depth and predicted latency to guide scheduling decisions. A notable feature is prefill and decode disaggregation, which allows operators to place compute-intensive prefill workloads on one set of GPUs and memory-bandwidth-bound decoding tasks on another. This can improve hardware utilization and reduce inference costs. Teams building large-scale GPU inference platforms should assess llm-d as an emerging foundation for their inference layer, particularly when looking to scale while meeting strict latency objectives.
