What changed
Amazon SageMaker HyperPod now includes three checks within a PyTorch DistributedDataParallel (DDP) training loop to detect silent data corruption (SDC). These checks identify faulty GPUs by comparing data-parallel replicas and trigger node replacement through SageMaker HyperPod's features.
What this means in practice
The three checks are designed to catch different types of SDC: a non-finite gradient check identifies NaN or Inf values in gradients, a ring replay audit compares micro-batch recomputations across nodes, and a weight fingerprint comparison detects discrepancies in stored weights. When a check triggers, the suspect GPU is mapped to its node, the node is cordoned, and SageMaker HyperPod replaces it. Training resumes from a checkpoint saved before the suspect window.
Limitations and availability
The solution requires a SageMaker HyperPod cluster orchestrated by Amazon Elastic Kubernetes Service (Amazon EKS) with at least three GPU nodes. The ring audit needs three different nodes for comparisons, and the weight comparison requires three copies of each weight. The full code is available in a GitHub repository. Replacing a node destroys local NVMe volumes, and the SageMaker HyperPod observability add-on is needed for correlation steps. The solution was tested on a cluster with 4 ml.p5.48xlarge instances (32 NVIDIA H100 GPUs), and tests were repeated on three instances, the minimum required for the checks.
