AMD has introduced Quark, a high-performance quantization library optimized for AMD Instinct™ MI350 GPUs, to accelerate diffusion models like FLUX.1-dev. By leveraging MXFP4 quantization, the library significantly improves inference efficiency without compromising image quality. The approach uses AMD's AITER GEMM kernels to enable native inference with FP4/FP8 matrix-core capabilities, achieving notable performance gains.
Quark supports multiple numeric formats, including MXFP4, and is integrated with ROCm™ and optimized for matrix-core acceleration. It allows users to configure per-layer quantization schemes based on layer sensitivity, enabling both uniform and mixed-precision flows. This flexibility, combined with the native FP4/FP8 capabilities of MI350 GPUs, enables near-lossless image quality while significantly improving inference efficiency. The library also supports online quantization for xDiT, making inference more efficient by converting BF16 transformer layers into FP8 or MXFP4 in-memory and routing them to AITER matrix-core kernels.
The blog details the implementation of Quark within the xDiT inference stack, where it quantizes each replica’s linear layers to FP8/MXFP4. The benchmarks measure the impact of this quantization on inference performance, with results showing a 1.92× speedup over BF16 eager and 1.41× speedup over BF16 torch.compile when combined with in-tree torch.compile fixes.
Source: amd
