What changed
AMD has detailed a performance issue related to vector memory (VMEM) load scheduling in lock-stepped kernels on its AMD Instinct™ MI300X GPUs. The issue involves stalls that occur before device memory read instructions are issued, which can significantly reduce the rate at which new memory loads are issued during periods of heavy VMEM activity.
What this means in practice
The article explains that VMEM loads can only issue after the instruction decodes and clears all checks, including operand readiness, an available slot in the per-wave VMEM tracker, and acceptance by the VMEM issue port. Once issued, the request traverses the memory hierarchy, hits cache or device memory, returns, retires in the tracker, and finally writes the data into the destination vector register(s) (VGPR(s)).
The article highlights that a period of heavy VMEM activity can significantly reduce the rate at which new memory loads are issued.
Recognizing these patterns, particularly the long red stall lines beneath memory operations, helps pinpoint where optimization efforts can have the biggest impact.
The article also discusses the different types of sixteen-byte VMEM reads on AMD MI300X GPUs, including global_load_dwordx4 and buffer_load_dwordx4, and their respective behaviors.
Limitations and availability
The article focuses on the dwordx4 form of VMEM operations, which are more commonly encountered in AI kernels. It also focuses solely on global_load_dwordx4 instructions, but all presented observations translate to buffer_load_dword,4 instructions as well. The article begins with an extreme, but instructive schedule: all global loads clustered at the top of each inner loop iteration, followed by dependent VALU work, with s_barrier between iterations so waves in each workgroup are lock-stepped. This exaggerated pattern stresses the vector-memory subsystem and exaggerates stalls analyzed in the article. Subsequent sections revisit the same computational intent while giving the scheduler more room to reduce issue stalls and/or to hide vector-memory latency.
The article manually modifies and links the assembly programs for the various scenarios presented in the study to provide insights into how they behave on the GPU. The baseline scenario involves clustered global loads, which lead to severe instruction issue stalls.
Removing barriers allows for natural skew, reducing the number of cycles required for VMEM loads. The article also explores how many loads are too many, showing that even two consecutive VMEM loads can cause stalls under certain conditions.
![Source facts for AMD Details VMEM Load Stalls in MI300X GPUs: Example ATT trace showing stalls attributed to VMEM instruction classes of interest. Figure 1 highlights some of the common optimization opportunities observed with VMEM loads[1]. Scope and Experimental Setup# A vector memory (VMEM) load can issue only after the instruction decodes and clears all checks.](/images/articles/amd-details-vmem-load-stalls-in-mi300x-gpus-editorial-facts.png)
