AWS released NVRx for fault-tolerant distributed training on Amazon EKS, addressing interruptions in large-scale machine learning workloads. The integration of NVIDIA Resiliency Extension (NVRx) with PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon Elastic Kubernetes Service (Amazon EKS) enables asynchronous checkpointing and in-process restarts.
AWS reported a reduction in idle time by up to 40% in large-scale training jobs, measured on H100 GPUs at 2-node to 8-node scale. This compares with synchronous checkpointing, which previously consumed up to 40% of total wall time.
NVRx is built on PyTorch and targets high-performance multi-node GPU workloads. Availability begins with the release of the solution, initially for users running training jobs on Amazon EKS.
"NVRx handles the application-level resilience (async checkpointing, in-process restart, and in-job restart), while the EKS cluster provides the infrastructure foundation," said the AWS blog post. The solution combines NVRx fault tolerance primitives with an EKS-based training environment designed for high-performance multi-node GPU workloads.
The announcement follows AWS’s focus on improving large-scale distributed training. AWS did not specify the exact timeline for broader availability, and the source raises the question of how well the solution performs in real-world scenarios.
Source: awsml
