AMD has introduced the Cluster Validation Framework (CVF), a Kubernetes-native tool designed to validate GPU clusters before they enter production. CVF ensures hardware and networking components are healthy by running destructive tests in an isolated environment, reducing the risk of operational disruptions. The framework is part of the AMD GPU Operator ecosystem and aims to address the growing reliability challenges as GPU clusters scale to thousands of accelerators. CVF executes a six-phase validation pipeline within an ephemeral k3s cluster, separating validation workloads from the production control plane. This approach allows for stress testing and other destructive validation without affecting ongoing workloads. The framework supports both single-node bring-up and fleet-scale acceptance, using ConfigMap-driven thresholds to make the workflow practical for a wide range of deployments. Source: amd

CVF's six-phase validation pipeline includes node selection, GPU hardware acceptance, intra-node mesh bandwidth checks, NIC health validation, per-rail RDMA testing, and multi-node RCCL scale-out verification. Each phase is executed as Kubernetes Jobs and MPIJobs within the ephemeral k3s cluster, ensuring that validation does not interfere with production operations. The framework uses node labels to gate later phases and support fleet dashboards, providing insights into hardware health and informing production scheduling policies. By running validation in a self-contained environment, CVF enables destructive testing patterns such as GPU stress tests and HBM error injection without risking production workload disruption. Source: amd

The need for CVF emerged from the challenges faced during large-scale GPU training campaigns, such as Meta's Llama 3 project, where 419 unexpected interruptions occurred over 54 days. These interruptions were largely due to GPU and HBM3 memory faults, highlighting the operational risks of silent hardware degradation in large clusters. CVF aims to mitigate these risks by validating the full stack—from GPU silicon health to multi-node RDMA fabric performance—before nodes join a production workload. The framework's design ensures that only nodes with healthy hardware and networking components proceed to subsequent validation phases, thereby reducing the likelihood of disruptions during distributed training jobs. Source: amd