KServe

KServe is an open-source Kubernetes operator that manages predictive and generative AI inference through custom resources. Rather than acting as an inference engine itself, it orchestrates serving runtimes such as vLLM, SGLang and NVIDIA Triton, separating the lifecycle of a model service from the engine that executes it.

As organizations build AI factories, operating inference across Kubernetes GPU infrastructure becomes a fleet-management problem rather than a matter of running individual model servers. KServe addresses this with two custom resources: InferenceService for conventional machine learning and simpler LLM workloads, with optional Knative-based scale-to-zero, and LLMInferenceService for advanced LLM serving. Built on llm-d and the Gateway API Inference Extension, LLMInferenceService supports multi-node inference, prefill/decode disaggregation and KV-cache-aware routing that directs requests to replicas already holding the relevant cached prefix.

KServe is a CNCF incubating project, and several of our clients use it in production. Our teams prefer it over other tools in this space when building in-house inference platforms on Kubernetes. LLMInferenceService is newer than InferenceService, and its ecosystem is still evolving.