Live intelligence

Latest AI updates

1 updates
Tutorial

Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

AWS describes how to detect and recover from silent data corruption (SDC) on SageMaker HyperPod by adding three checks to a PyTorch DDP training loop: non-finite gradient checks, ring replay audits, and weight fingerprint comparisons. The post explains how to localize suspect ranks to nodes, contain the corruption, and replace hardware using HyperPod features. The solution was tested with software-injected faults on a 4-node cluster with 32 NVIDIA H100 GPUs, and full code is available in a GitHub repository.

Silent data corruptionSageMaker HyperPodDistributed training