AMD has published an analysis of network traffic patterns in AI training clusters, highlighting how synchronized data flows and congestion impact performance. The report focuses on the challenges of managing massive data exchanges across thousands of GPUs, which are essential for training large language models and generative AI systems. These networks, which support high-speed, low-latency communication, face unique congestion issues due to their synchronized and bursty nature. As AI clusters scale to tens of thousands of GPUs, the traffic patterns become more complex, leading to periodic congestion and harmonic-like oscillations in data flow. This analysis provides insights into the behavior of AI training networks, helping network architects and data center operators address these challenges.
The report outlines how AI training workloads generate highly bursty and synchronized data flows, with GPUs sending and receiving data at full line rate during collective communication operations like AllReduce. These flows are characterized by low entropy, with limited IP addresses and fixed ports, making them difficult to manage with traditional load balancing techniques. Additionally, the majority of traffic is intra-cluster, with over 90% of data moving between GPUs rather than external systems. This east-west dominance creates unique congestion patterns that differ from the more variable traffic seen in traditional data centers.
The analysis also explores the causes of congestion in AI scale-out networks, including ingress/egress mismatches, slow NICs, and load balancing inefficiencies. These issues can lead to buffer overflows, packet drops, and latency spikes, which degrade training efficiency. The report further notes how these congestion trends evolve as cluster sizes grow, with congestion shifting from localized to fabric-wide and tail latencies increasing significantly.
Source: amd
