Technology Radar
KAITO (Kubernetes AI Toolchain Operator) is a Kubernetes operator that automates the infrastructure needed to self-host open-weight models on your own clusters. Instead of manually configuring GPU nodes and model servers, teams declare a Workspace, KAITO's custom resource for a single model deployment, specifying the model and GPU instance type. The KAITO controller then provisions matching GPU nodes through Karpenter NodePools, estimates GPU memory requirements, tunes the runtime and serves the model through vLLM, exposing an OpenAI-compatible endpoint via a Kubernetes Service.
Our teams began exploring KAITO on AKS to host small language models at lower cost and latency. Previously, running Ollama and vLLM directly on GPU VMs had left them managing drivers, scaling and node configuration manually. Managing the model lifecycle through the same declarative manifests as other workloads promises to reduce much of that toil.
KAITO's scope is deliberately narrow, though. It handles node provisioning and model deployment rather than the broader inference stack, where KServe is more mature and offers capabilities such as KV-cache-aware routing. KAITO is still a CNCF Sandbox project. Its GPU node auto-provisioning is most mature on AKS, and it supports only Ampere-generation or newer NVIDIA GPUs. We think it's worth assessing, particularly for teams that want to self-host models using their existing Kubernetes tooling.
