What changed
AMD has announced the porting of an FP16 GEMM kernel to FP8 on RDNA 4 GPUs, specifically targeting the AMD Radeon RX 9070 XT (gfx1201). The FP8 WMMA instruction allows for smaller element sizes, which in turn enables larger tiles and deeper K slices to fit within the LDS budget. The kernel uses the FP8 and BF8 builtins from the RDNA4 dense WMMA builtins reference in the AMD ROCm Optimization Guide. The FP8 kernel uses a different instruction, which multiplies two 16×16 tiles of FP8 E4M3 values and accumulates into FP32. This change allows for more efficient memory usage and better performance in transformer workloads.
What this means in practice
The porting of the FP16 kernel to FP8 on RDNA 4 GPUs opens up new tile configurations that were not possible with FP16. The smaller element size allows for larger tiles and deeper K slices, which can improve performance in transformer workloads. The post compares four configurations on four problem shapes drawn from transformer workloads. Config 1 is the FP16 final kernel with element types swapped, to establish how much headroom the FP8 instruction leaves on its own. Configs 2 and 3 explore the larger tiles that the smaller element size makes possible, and Config 4 is the tile selected through a sweep of the search space. A complete implementation is in the accompanying repository.
Limitations and availability
The FP8 WMMA instruction is only available on gfx12, so this post covers one GPU: the AMD Radeon RX 9070 XT (gfx1201). All of the structural choices from Part 1 carry over unchanged: the TN memory layout, buffer resource descriptors for bounded global loads, a deep K slice in LDS, the cache-reusing block schedule, and the interleaved prefetch fold. To work through the examples, you need a ROCm installation with the hipBLASLt development package, an RDNA 4 GPU, and working familiarity with Part 1. All timings in this post come from ROCm 10.0.
