AMD has introduced a new software optimization for its Instinct MI355X accelerator that significantly reduces the latency of online rotation for MXFP4 quantization. The company's fused kernel design, based on Gluon (Triton), eliminates the performance overhead associated with traditional separated implementations, making online rotation more practical for production use. This advancement addresses a key challenge in serving large language models with low-bit quantization, allowing for accuracy-preserving MXFP4 online rotation at near-zero cost. The fused kernel maintains the same level of accuracy as the separated approach while reducing latency to within a tolerable range for most applications.
The optimization is particularly beneficial for Dense models like Qwen3 8B/14B/32B, where online rotation was previously associated with a +5–10% TPOT overhead. With the fused kernel, this overhead drops to +0.3–0.8%, making online rotation viable for default deployment. For MoE models, the improvement is even more pronounced, with the overhead nearly halved at RS=128. The fused kernel achieves this by performing all operations in a single launch, keeping intermediate values in registers and eliminating the need for intermediate bf16 global-memory round-trips.
The optimization builds on AMD's previous work with MXFP4 quantization, which demonstrated that learned rotation significantly improves quantization accuracy for Qwen3 Dense models. This new approach further enhances efficiency by leveraging the block-diagonal structure of R₁, allowing for efficient computation without sacrificing accuracy. By reducing the latency cost of online rotation, AMD is making it possible to deploy MXFP4 quantization more broadly, improving the affordability and performance of large language model serving.
Source: amd
