AMD has introduced a new optimization for attention decode workloads on its MI450 GPU, leveraging Gluon to achieve 85% of the peak HBM bandwidth. The company's blog highlights how the MI450's hardware features are designed to address memory bottlenecks in AI inference, particularly for long prompts and multi-turn reasoning. The MI450 series provides more on-chip resources and higher HBM bandwidth compared to its predecessor, the MI350 series, enabling more efficient data movement and processing. The blog outlines how the Gluon kernel can be used to optimize performance on MI450 by utilizing its advanced memory and compute capabilities.
The MI450 introduces a new hardware unit called TDM to facilitate structured tensor data transfers between global memory and LDS. This unit allows for asynchronous bulk data transfers, making it easier to write pipelined kernels that overlap compute and memory operations. Additionally, the MI450 extends the workgroup model with workgroup clusters, enabling coordinated processing across multiple workgroups. This feature allows workgroups to share data across WGP boundaries without relying on global memory for synchronization.
The blog focuses on attention decode, a critical phase in large language model inference, where each new token must attend to all previous tokens from the KV cache. The MI450's architecture is designed to handle this memory-intensive task efficiently, with the Gluon kernel demonstrating significant performance gains. The blog also discusses the use of Gluon's explicit tensor layout control to optimize memory access patterns and improve kernel performance.
Source: amd
