Amazon SageMaker HyperPod Inference adds managed NVIDIA Dynamo support
AWS now supports NVIDIA Dynamo on SageMaker HyperPod Inference, bringing multi-node deployment, disaggregated prefill/decode and KV-aware routing to managed LLM endpoints. The HyperPod Inference add-on installs Dynamo's operators and scheduling components, with configuration available either through a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with a dynamoSpec block. The post walks through aggregated and disaggregated GLM-5.3 deployments on ml.p5.48xlarge instances and reports benchmarks: disaggregated serving reached 2.05x the highest aggregated throughput for Qwen3-235B-A22B-FP8, while a two-node aggregated deployment retained 99.1 percent of single-node per-GPU throughput. It closes with tuning guidance covering recipes, EFA verification and AIPerf-based benchmarking.
Why it matters: Gives teams running large models a concrete, documented path to distributed and disaggregated inference on managed AWS infrastructure, including deployment configs and measured throughput trade-offs.