AMD has introduced a series of optimizations for the MiniMax-M3 large language model, enhancing its performance on AMD Instinct MI355X GPUs. These optimizations focus on improving serving throughput and reducing token latency while maintaining accuracy. The changes include online quantization for attention GEMMs, AITER sparse attention, FP8 KV cache, and EAGLE3 speculative decoding. These improvements are part of AMD's ongoing efforts to enhance the efficiency of large language model inference.
The optimizations address key bottlenecks in the inference process, such as attention GEMM cost, KV cache bandwidth, sparse attention scheduling, and target model forward frequency. Online quantization for attention GEMMs allows for efficient execution without regenerating the full model, using formats like FP8 and FP4. This approach is particularly beneficial for MiniMax-M3 quantized checkpoints, where performance-sensitive paths can leverage lower-bit GEMMs or maintain FP8 while switching to different quantization schemes. The selective application of these optimizations covers both MiniMaxAI/MiniMax-M3-MXFP8 and Quark quantized models, ensuring that specific layers retain their original formats.
The optimizations also integrate AITER sparse attention and FP8 KV cache, which require joint design and tuning between ATOM and AITER. This collaboration enables high-performance attention operators for prefill and decode, while also exposing the necessary metadata and layout configurations. These changes allow MiniMax-M3 to reuse standard attention metadata binding while preserving its model-specific sparse behavior, reducing HBM round trips and improving overall efficiency.
Source: amd
