Live intelligence

Latest AI updates

1 updates
Products Editor selected

Amazon SageMaker HyperPod adds workload recovery and expanded infrastructure checks

AWS introduced three capabilities for Amazon SageMaker HyperPod: the MultiNodeTrainingValidation deep health check, which runs a small Megatron-LM training workload on each instance and then across instances to catch GPUs that pass hardware tests but produce NaN or infinite loss; a Kubernetes Resource Monitoring Agent for EKS-orchestrated clusters that reports device plugin crash loops, missing DaemonSets, node condition and taint changes, and the pods and nodes behind failed training jobs; and hung job detection for Slurm clusters that flags stalled jobs and can cancel them so Slurm requeues them. AWS frames the features as addressing numerical faults, hung jobs and cluster-level failures that node-level health monitoring misses. On a two-node test cluster, the two MultiNodeTrainingValidation tests added about eight minutes to a full deep health check that took roughly 1 hour 28 minutes.

Why it matters: Teams running large distributed training on HyperPod get concrete new failure-detection and recovery features, with setup steps and reported results, that target wasted GPU time from silent corruption, hung jobs and Kubernetes-level faults.

Amazon SageMaker HyperPodAWSDistributed training reliability