Amazon SageMaker AI adds KV cache-aware routing for real-time endpoints
Amazon SageMaker AI introduced KV cache-aware routing, a routing strategy that uses cache state reported by model containers to send requests to instances holding the longest cached prompt match while weighing instance l...