AMD has successfully ported the On-Demand Communication (ODC) tool to its Instinct MI300X GPUs running ROCm 7.2. The tool, originally designed for NVIDIA GPUs, now supports AMD hardware by rebuilding its communication framework using XGMI and rocSHMEM/MORI. This adaptation enables both single-node and dual-node training within the Primus framework, significantly improving efficiency in large-scale training scenarios. The implementation focuses on eliminating synchronization barriers and workload-imbalance issues inherent in Fully Sharded Data Parallel (FSDP) training. Source: amd
ODC replaces FSDP’s collective communication with one-sided, on-demand point-to-point transfers, allowing ranks to operate independently. This approach removes the need for all-ranks to synchronize at each layer, reducing idle time caused by uneven workloads. The tool’s implementation on MI300X demonstrates measurable improvements, with single-node training speeds up to 1.201× faster than a load-balanced RCCL baseline. Additionally, dual-node training saw a 1.154× speedup. These results are based on actual experiments and are supported by PyTorch Profiler traces. Source: amd
The blog outlines the challenges of FSDP, including per-layer synchronization barriers and workload-imbalance bubbles caused by variable-length data. ODC addresses these by shifting synchronization from per-iteration to per-minibatch, allowing for asynchronous gradient accumulation. The tool’s effectiveness is validated through three configurations: NCCL_pad (baseline), ODC_pad, and ODC_nopad, each offering varying degrees of load balancing and performance gains. Source: amd
