Releases · kaito-project/kaito

Release list

v0.11.0 Latest

Latest

github-actions released this 01 Jul 04:21

v0.11.0

7545aa2

v0.11.0 - 2026-07-01

This release closes out the migration off the legacy preset system, tightens node estimation, and rounds out three long-running workstreams — GPU node provisioning via Karpenter, model weight streaming from blob, and prefill/decode disaggregation with MultiRoleInference. It also promotes InferenceSet to v1beta1 on by default, ships the KAITO ProductionStack as an optional Helm subchart, and refreshes the model catalog, base images, and runtime versions.

Breaking Changes

Legacy preset system removed . The old built-in preset API is gone. Workloads that still reference legacy preset names must migrate to modelName in the Model Catalog before upgrading.
Stricter max-model-len and node estimation . KAITO now uses vLLM's native estimation to compute default max-model-len and the number of GPU nodes required. Deployments that previously relied on looser estimates may need to set max-model-len explicitly or request additional nodes.

Major Features

Karpenter-based GPU node provisioning

@rambohe-ch @2170chm

Set --node-provisioner=karpenter at controller startup and KAITO will provision, reconcile, and drift-detect GPU nodes through Karpenter instead of the legacy gpu-provisioner path. NodeClass RBAC is derived from the configured karpenterProvider so operators no longer hand-roll it. The bundled gpu-provisioner is also updated to v0.4.2 for clusters that stay on the classic path.

Model weight streaming from blob storage

@2170chm

Two new resources work together to eliminate per-pod model downloads at startup:

ModelMirror CRD creates a PVC-backed mirror of a HuggingFace model on blob storage.
Workspace ModelStreaming annotation — add kaito.sh/model-streaming: "true" to a Workspace and the controller creates the ModelMirror for you and starts the download in parallel with GPU node provisioning. Once the mirror is ready, inference pods start with --model=az:// and stream weights directly from blob via RunAI Model Streamer, with no per-pod weight volume. Download bandwidth is exposed as a Prometheus metric on the vLLM pod.

Prefill/decode disaggregation — `MultiRoleInference`

@andyzhangx

A new MultiRoleInference CRD lets you run prefill and decode on separate, independently-scaled pod groups. You declare the roles and node requirements; the controller creates the InferencePool, wires the EPP plugin config, and injects the NixlConnector KV-transfer settings so the two groups can exchange KV-cache state. A GWIE + MultiRoleInference deployment guide is included in the docs.

`InferenceSet` promoted to `v1beta1`

@andyzhangx

InferenceSet is now v1beta1 and enabled by default — the feature gate is gone. Two behaviors ship with the promotion: replicas can scale to zero, and the Kubernetes deployment name can be used directly as the served-model-name.

Base image auto-upgrade for Workspaces

@zhehli688

Workspaces can now opt in to automatic base image updates. When a new vLLM base image is published, opted-in Workspaces roll forward without any manual spec change.

ProductionStack as an optional subchart

@rambohe-ch @techworldhello @tnsimon @zhuangqh

The KAITO ProductionStack is now available as an optional Helm subchart — a turnkey inference gateway designed based on the llm-d reference stack. It stitches together Istio, llm-gateway-auth (API-key or Azure Entra ID authz), the llm-d inference scheduler for KV-cache-aware routing, KEDA-driven autoscaling on vllm:* metrics, and KAITO InferenceSet into a single deployable configuration.

Model Catalog Additions

Family	Models added
Gemma 4	Gemma 4 series
NVIDIA Nemotron	Nemotron series
Mistral / Qwen	Mistral 4, Qwen 3.5, Qwen 3.6
MoonshotAI / Minimax	Kimi-K2.5, Minimax series

AWQ 4-bit quantization is now supported as a deployment option for compatible models.

Dependency & Runtime Upgrades

Component	Version
vLLM	0.22.1
PyTorch	2.11.0
transformers	5.6.0
Go	1.26.4
Kubernetes	1.33.8
gpu-provisioner	v0.4.2
local-csi-driver	0.2.18
ProductionStack	0.2.2

Changelog

Features 🌈

4d73506 feat: Add minimal block-only streaming guardrails (#2157)
2853512 feat: use vLLM native estimation for max-model-len (#2156)
a8def1d feat: add RAGEngine streaming buffer window (#2138)
f4b80b3 feat: derive NodeClass RBAC from karpenterProvider config (#2150)
77ddc2c feat: add RAGEngine streaming semantic utilities (#2136)
2007c56 feat: promote InferenceSet to v1beta1 and enable EnableInferenceSetController by default (#2112)
7245974 feat: support P/D disaggregation (#2093)
45266bd feat: Support streaming passthrough for RAGEngine chat completions (#2128)
d5b9af0 feat: model streaming (#2100)
27ad3b0 feat: upgrade gpu-provisioner version to v0.4.2 for kaito (#2111)
ab299f9 feat: add productionstack to kaito chart (#2105)
561a658 feat: expose model download bandwidth metrics in vLLM pod (#2087)
d34d424 feat: support base image auto-upgrade for Kaito Workspaces (#2062)
af0167f feat: model mirror crd + controller (#2082)
70848a5 feat: add invisible text and token limit guardrail scanners (#2064)
f8876c4 feat: scanner pack 3 json reading time length limit (#2061)
f7de6d4 feat: add secrets and sensitive scanners (#2060)
ef5cda7 feat: MRI controller creates InferencePool + EPP plugins config (basic functionalities) (#2048)
410d557 feat: Add response-path guardrails metrics (#2070)
73d51ba feat: add guardrails observability metrics (#2037)
5842a40 feat: enable karpenter and add e2e tests (#2041)
79b2a1c feat: support hybrid Mamba model in max_concurrency calculation (#2052)
6aaff7e feat: add guardrails scanner capability validation (#2039)
0c4ce3e feat: ragengine guardrails reload observability (#2038)
e72797f feat: hot-reload output guardrails policy via watchfiles (#2025)
d0b2168 feat: onboard kimi and minimax models to model catalog (#2047)
b0002fd feat: add MultiRoleInference CRD types, controller, and status aggregation for P/D disaggregated inference (#2005)
75d60ba feat: support per-scanner guardrails actions (#2042)
f2220cd feat: inject NixlConnector kv-transfer-config for P/D disaggregated inference (#2027)
4a4a40b feat: wire default guardrails policy configmap for ragengine (#1992)
464662e feat: onboard mistral 4 and qwen 3.5/3.6 models to model catalog (#2031)
f507b2c feat: Add guardrails fail-open config and policy ConfigMap mounting for RAGEngine (#2022)
ce81a5f feat: 1952 integrate karpenter provisioner (#2017)
6353a00 feat: onboard Gemma 4 models to model catalog (#2021)
898d91a feat: add fail-open and basic error handling for output guardrails (#1963)
cc371e0 feat: add AWQ quantization support (#2007)
ecb905c feat: onboard Nvidia Nemotron models to model catalog (#2008)
23a745c feat: karpenter drift controller (#2003)
dcbd663 feat: support deployment name as served-mode-name for inferenceset (#2004)
7b4a600 feat: add guardrails YAML policy loader to RAG runtime (#1990)
557194d feat: karpenter provisioner (#1982)
afcdfbe feat: add RAGEngine guardrails API and policy validation (#1986)
d325c36 feat: [breaking] complete legacy preset to model catalog migration (#1977)
4cdfaa1 feat: Migrate default EPP from GWIE to llm-d inference scheduler (#1975)
12836f1 feat: add --node-provisioner startup parameter (#1974)
eeb5b3e feat: add karpenter crds dependencies and ensure aksnodeclass resources (#1968)
a60bcf1 feat: onboard more built-in models to model catalog (#1945)
bb5b4db feat: add non-streaming output guardrails hook for chat completions (#1962)
2783b2e feat: add nodes provision interface and gpu-provisoner implementation (#1954)

Bug Fixes 🐞

b023ad9 fix: skip inference config validation when no ConfigMap is specified (#2171)
cb140a3 fix: pin inference pods to provisioned nodes (#2161)
888ec57 fix: isolate inference c...

Contributors

andyzhangx, zhuangqh, and 5 other contributors

Assets 6

v0.9.4

github-actions released this 19 Jun 02:08

v0.9.4

c59cb49

v0.9.4 - 2026-06-19

Changelog

Maintenance 🔧

c59cb49 chore: update chart version
633b48f chore: Update go version in docker file
de7e181 chore: bump the go version to 1.26.4

Assets 6

v0.10.0

github-actions released this 15 Apr 11:22

v0.10.0

c673cad

v0.10.0 - 2026-04-15

Qdrant Vector Store for RAG
RAG Engine now supports Qdrant as a vector backend, including hybrid search. This expands storage flexibility and retrieval quality options for RAG workloads.

vLLM v0.17.1 with Smart Runtime Tuning
KAITO upgrades to vLLM v0.17.1 and introduces a 3-tier parallelism strategy that auto-configures tensor parallelism, pipeline parallelism, and data parallelism.
Runtime dtype is now selected dynamically based on GPU compute capability.
The kaito.sh/performance-mode annotation (interactivity, balanced, throughput) provides intent-based tuning and maps directly to vLLM performance mode controls.

Built-in MT-Bench Evaluation
MT-Bench integration enables out-of-the-box dialogue quality evaluation with automated benchmarking workflows.

Azure GPU Spot Instance Support
KAITO now supports Azure GPU spot instances for node pools, helping reduce inference infrastructure cost.

Improved Inference Observability
TPM benchmark metrics are now written into Workspace and InferenceSet status.
PVC metrics and preset usage tracking are also added for better operational visibility.

LoRA Adapter Volumes
LoRA adapters can now be mounted directly from PVC volumes, offering a flexible alternative to image-bundled adapters.

VS Code Copilot Plugin
The new kaito-workspace Copilot plugin adds an inference skill for richer Copilot workflows.