Back to AI News
MarkTechPost

Buoyant Releases Kinference, an Open-Source Inference Control Plane for Kubernetes

Buoyant has released Kinference v0.1.0-alpha.7, an Apache 2.0 control plane that routes requests across vLLM workers on Kubernetes using declarative policies, queue state and KV-cache signals. Self-reported tests showed substantial latency and goodput improvements over random Kubernetes Service routing for shared-prefix and multiturn workloads, but little median improvement for prompts without shared prefixes. The project remains alpha software, with advertised features including token rate limiting, semantic classifiers and Anthropic API support not yet implemented.

AI Brief

Kinference offers AI infrastructure teams a concrete new option for policy-driven, multi-tenant vLLM routing, while its alpha status and limited benchmark baseline warrant caution.

Buoyant, the company behind Linkerd, has released Kinference, an open source project for building inference platforms on Kubernetes. Kinference is an inference control plane for Kubernetes and vLLM. It sits in front of a fleet of vLLM workers and decides where each request goes. It applies routing policies, token rate limits, KV-cache-aware load balancing and per-tenant metrics in one layer. For dev teams serving many open-weight models to many internal clients, it targets the layer above the model server. The project is Apache 2.0 licensed on GitHub. The current release is v0.1.0-alpha.7.

TL;DR

  • Size: v0.1.0-alpha.7, written in Rust, Apache 2.0, configured through 5 Kubernetes CRDs.
  • Runs on: any GPU-enabled Kubernetes cluster running vLLM. Buoyant’s benchmark used 3 L40S GPUs per pool.
  • Performance: large gains on prefix-heavy traffic versus plain Kubernetes Service routing (self-reported).
  • Best: median TTFT fell from 6.88 s to 0.31 s (95%) on a shared-prefix workload.
  • Worst: no median TTFT gain on prompts without shared prefixes (6.92 s vs 6.83 s).

What is Kinference?

Kinference is a policy-driven control plane that turns Kubernetes plus vLLM into a multi-tenant inference platform. Buoyant’s FAQ puts it simply: vLLM is the data plane and Kinference is the control plane.

How does Kinference work?

The system has 2 parts: a traffic gateway and an inference handler. The GitHub README states that the handler exposes a gRPC interface. It takes a request and returns the chosen endpoint plus a possibly mutated body. It pairs with a proxy such as the Buoyant Gateway.

Kinference watches vLLM KV-cache events and engine metrics. It currently uses NVIDIA’s dynamo-kv-router crate to process cache events. Configuration lives in 5 CRDs: GatewayBackend, InferencePolicy, InferencePool, InferenceProvider and InferenceWorkerSet.

What are inference policies?

Inference policies are declarative rules for how each request is routed or rejected. Rules can match on tenant, requested model, GPU pool state and semantic tags. Example policies on the project site cover 5 patterns:

  • Simple to cheaper: route low-complexity prompts to a smaller model, unless cache overlap makes the large model faster.
  • Burst to cloud: skip workers with over 2 queued prompts, then fall back to providers like Bedrock.
  • PII stays local: remove external providers when a PII tag fires, and fail closed with onEmpty: Reject.
  • Batch on idle GPUs: send batch requests only to workers under 70% KV-cache usage.
  • Model aliasing: publish one stable name, such as standard-model, backed by several models.

Most rules end with minimize: PredictedTTFT. Kinference predicts time-to-first-token for each candidate endpoint, using cache overlap and queue length. Policies live in git and can be validated offline with the kinfctl CLI.

Kinference, explained

Buoyant's open source inference control plane for Kubernetes and vLLM. v0.1.0-alpha.7, Apache 2.0.

TenantsAgent, prod app, coding harness, chat widgetOpenAI-compatible request
Traffic gatewaySingle endpoint. Asks the controller where to send each request.
Kinference controllerPolicy engine, routing, cache-aware balancing, classificationReads 5 CRDs + vLLM KV-cache events
Backends
vLLM pod A prefix cached
vLLM pod B cold
vLLM pod C queue 3
External provider fallback
Kubernetes Service (kube-proxy)Kinference

Self-reported by Buoyant. 2 pools of 3 vLLM 0.31.0 pods, 1 L40S each, Qwen3.8-27B-FP8, GuideLLM 0.8.1. Baseline is random routing, not another cache-aware router.

Kinferencellm-dNVIDIA DynamovLLM Prod. Stack
FocusPolicy and governance layerDistributed serving performanceDatacenter-scale servingReference vLLM stack
Maturityv0.1.0-alpha.7v0.71.xReference
EnginesvLLMvLLM, SGLangvLLM, SGLang, TRT-LLMvLLM
KV-aware routingYesYesYesPrefix-aware WIP
Policy CRDYesNot in READMENot in READMENot in README
Burst to providersYesNot in READMENot in READMENot in README
Disaggregated P/DNot disclosedYesYesRoadmap

All 4 are Apache 2.0. Sources: each project's GitHub README.

NOT YET SHIPPED
Token rate limiting
Promoted at launch, but the README lists it as missing from the current implementation.
NOT YET SHIPPED
Semantic classifiers
PII and complexity tags drive several example policies. The README says classifiers are not implemented yet.
NOT YET SHIPPED
Anthropic /v1/messages
The site says Anthropic-compatible traffic works. The README lists /v1/messages as pending.
NOT YET SHIPPED
Multiple replicas
The controller runs as a single replica for now, a factor for HA planning.
CAVEAT
No gain without shared prefixes
Control workload median TTFT: 6.92 s vs 6.83 s. The p95 still fell from 21.46 s to 12.33 s.
STRENGTH
Agent-friendly config
kinfctl validates policy YAML offline, and every CRD reports status to kubectl.

Tap a card for details. The README calls Kinference alpha software without sustained production exposure.

Sources: GitHub · Benchmarks · Docs© Marktechpost

How fast is Kinference in benchmarks?

Buoyant published a benchmark against plain Kubernetes load balancing. Each arm used 3 vLLM 0.31.0 pods, each on 1 L40S GPU. The model was Qwen3.8-27B-FP8. Load came from GuideLLM 0.8.1. The baseline was a ClusterIP Service, where kube-proxy picks a random pod per connection.

Unloaded latency results (median TTFT):

WorkloadKubernetes ServiceKinferenceChange
Control (no shared prefix)6.92 s6.83 sNo significant difference
Shared-prefix6.88 s0.31 sabout -95%
Multiturn1.70 s0.59 sabout -65%

Median end-to-end latency fell 45% on shared-prefix and 32% on multiturn. On control traffic, p95 TTFT still dropped from 21.46 s to 12.33 s. Buoyant attributes this to queue-aware balancing.

The capacity test measured goodput: requests per minute meeting a TTFT SLO. Peak goodput was 103.0 vs 14.0 req/min on shared-prefix, about 7.4x. On multiturn it was 82.7 vs 23.8, about 3.5x. The Service baseline never met the SLO at any tested concurrency.

These numbers are self-reported. The baseline is random routing, not another cache-aware router like llm-d or Dynamo. Buoyant also notes its SLOs (8 s and 4 s) suit offline agents, not chat.

How does Kinference compare with llm-d, Dynamo and Production Stack?

FeatureKinferencellm-dNVIDIA DynamovLLM Production Stack
BackerBuoyantCNCF sandbox (Red Hat, Google Cloud, IBM, CoreWeave, NVIDIA)NVIDIAvLLM project
LicenseApache 2.0Apache 2.0Apache 2.0Apache 2.0
Maturityv0.1.0-alpha.7v0.7 (May 2026)1.x (1.5.1 containers)Reference stack
EnginesvLLM (SGLang, TensorRT-LLM planned)vLLM, SGLangvLLM, SGLang, TensorRT-LLMvLLM
KV-cache-aware routingYes, predicted TTFTYes, prefix-cache and predicted latencyYes, KV-aware routerSession-based; prefix-aware listed as WIP
Declarative routing policy CRDYes (InferencePolicy)Not documented in READMENot documented in READMENot documented in README
Burst to external providersYes (InferenceProvider)Not documented in READMENot documented in READMENot documented in README
Disaggregated prefill/decodeNot disclosedYesYesOn roadmap
Multi-tenant controlsPer-tenant metrics; rate limits plannedFlow control for multi-tenant servingGateway-level rate limiting via GAIENot documented in README
Main result-95% median TTFT vs kube-proxy (3x L40S)3x throughput, 2x faster TTFT vs round-robin (4x MI300X)2x faster TTFT with KV-aware routing (Baseten)Not disclosed

Kinference’s distinct angle is governance: policies, aliases, provider fallback and tenant accounting. llm-d and Dynamo lead on raw serving performance, disaggregation and engine coverage.

Deployment and availability

Installation takes 5 steps per the getting started guide. Teams install Gateway API v1.5.1 CRDs, the Buoyant gateway Helm chart, then the Kinference chart. Kinference ships Prometheus metrics and Grafana dashboards. Metrics include TTFT, ITL, end-to-end latency and cached versus uncached prompt tokens, mostly per tenant.

Key Takeaways

  • Kinference is an Apache 2.0 control plane for vLLM on Kubernetes, from Linkerd maker Buoyant.
  • Declarative InferencePolicy rules handle model choice, provider bursting, PII routing and batch admission.
  • Self-reported median TTFT fell 95% on shared-prefix traffic versus kube-proxy.
  • Peak goodput rose about 7.4x on shared-prefix and 3.5x on multiturn.
  • It is alpha: rate limiting, classifiers and Anthropic API support are not yet shipped.


Check out the GitHub repo, docs, launch post and benchmark report. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM appeared first on MarkTechPost.

Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!