Products
Amazon SageMaker AI adds automatic monitoring and recommendations for inference endpoints
Amazon SageMaker AI now monitors generative AI inference endpoints by default for vLLM serving containers on GPU instances. It detects latency drift, underutilized GPUs, and configuration gaps such as disabled speculative decoding or outdated serving containers. Findings are delivered through AWS Health, with links into a SageMaker Studio agentic experience for benchmarking alternatives on separate compute. Monitoring uses endpoint-level latency and GPU telemetry without reading request or response content, and users can opt out per endpoint.
Amazon SageMaker AIAI inferenceGPU utilization