Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod
AWS describes how to detect and recover from silent data corruption (SDC) on SageMaker HyperPod by adding three checks to a PyTorch DDP training loop: non-finite gradient checks, ring replay audits, and weight fingerprin...