Live intelligence

Latest AI updates

2 updates
Products Editor selected

Amazon SageMaker AI adds KV cache-aware routing for real-time endpoints

Amazon SageMaker AI introduced KV cache-aware routing, a routing strategy that uses cache state reported by model containers to send requests to instances holding the longest cached prompt match while weighing instance load. It is enabled through the SM_VLLM_KV_EVENTS_CONFIG environment variable and RoutingStrategy set to KV_AWARE, and supports real-time single-model and inference component endpoints on the AWS vLLM Deep Learning Container. AWS reports benchmarks with Llama 3.1 70B on seven ml.p5.48xlarge instances showing lower p50 time to first token and higher throughput versus random routing, with results depending on workload.

Why it matters: It gives teams serving LLMs with multi-turn conversations, RAG, or long shared prompts a concrete way to reduce prompt recomputation and improve latency and throughput by routing to instances that actually hold cached prefixes.

Amazon SageMaker AILLM inferenceKV cache
Products Editor selected

Google Releases TPU Sync: Open-Source KV Cache Transfer for TPUs

Google Cloud has released TPU Sync, an Apache-2.0 C++ library that moves KV caches and model weights across TPU HBM, pinned host DRAM, shared storage, and data-center networking. It integrates with PyTorch and JAX and serves as the KV connector or transfer backend for vLLM-TorchTPU, SGLang-TorchTPU, and llm-d on GKE. Google reports over 90% lower KV transfer latency and up to 90% lower average TTFT in internal tests on 235B to 480B MoE models, while the README says it is not yet recommended for general use.

Why it matters: It gives AI infrastructure teams a concrete open-source option for reducing KV-cache transfer and TTFT bottlenecks in disaggregated TPU inference, with clear early-stage caveats.

Google CloudTPUKV cache