Amazon SageMaker AI adds KV cache-aware routing for real-time endpoints
Amazon SageMaker AI introduced KV cache-aware routing, a routing strategy that uses cache state reported by model containers to send requests to instances holding the longest cached prompt match while weighing instance load. It is enabled through the SM_VLLM_KV_EVENTS_CONFIG environment variable and RoutingStrategy set to KV_AWARE, and supports real-time single-model and inference component endpoints on the AWS vLLM Deep Learning Container. AWS reports benchmarks with Llama 3.1 70B on seven ml.p5.48xlarge instances showing lower p50 time to first token and higher throughput versus random routing, with results depending on workload.
Why it matters: It gives teams serving LLMs with multi-turn conversations, RAG, or long shared prompts a concrete way to reduce prompt recomputation and improve latency and throughput by routing to instances that actually hold cached prefixes.