AMD has introduced an emulation-based serving pipeline that enables its MI355 accelerators to run NVFP4 quantized checkpoints directly, without requiring conversion to other formats. This solution, integrated into vLLM, allows the MI355 to load and serve standard NVFP4 checkpoints by dequantizing weights to BF16 on-the-fly during GEMM operations. The pipeline preserves the numerical behavior of the original format through a quantize-dequantize round-trip for activations. This advancement addresses a gap in the CDNA4 architecture, which lacks native NVFP4 tensor execution support. According to AMD, the pipeline is compatible with MI300/325 and MI350/355, with MI355 serving as the primary validation target. The solution supports both dense and MoE models, with specific layer quantization strategies to balance accuracy and performance. AMD-Quark, a native tool, generates compatible NVFP4 checkpoints from source models, and the pipeline is evaluated using accuracy benchmarks on MI355 across two model families. The results show that AMD-Quark’s quantized Kimi-K2.6 model matches NVIDIA’s checkpoint within statistical noise, while the RedHatAI variant shows a small gap. The dense Qwen3-8B model also follows a similar pattern. The emulation approach is positioned as an accuracy reference for quantized model validation rather than a production deployment due to inference latency constraints. AMD plans to optimize the pipeline with features like streaming dequantization and improved BF16 GEMM performance.
The emulation pipeline for NVFP4 models on MI355 is designed to handle both dense and MoE models by selectively quantizing layers. For MoE models, routed_experts and shared_experts are quantized while attention linears, routers, and vision modules remain in higher precision. This approach minimizes accuracy loss while improving inference throughput when fused with quantized routed_experts. In dense models, MLP and attention linears are quantized, while the lm_head remains in higher precision. AMD-Quark, the company’s native tool, generates compatible NVFP4 checkpoints from source models, supporting artifacts from NVIDIA, AMD, and open-source communities. The quantization process excludes specific layers, such as self-attention and lm_head, to maintain performance and accuracy. The resulting checkpoints are then served using vLLM, which loads them directly into the NVFP4 emulation path. This enables direct deployment of NVFP4 models without conversion, providing a flexible solution for various deployment scenarios.
The accuracy of the NVFP4 models on MI355 was evaluated using the GSM8K and MMLU-Pro benchmarks. Results show that AMD-Quark’s quantized Kimi-K2.6 model matches NVIDIA’s checkpoint within statistical noise across both benchmarks, while the RedHatAI variant shows a small gap, possibly due to differences in quantization strategies. The same pattern holds for the dense Qwen3-8B model. These findings indicate that the emulation pipeline effectively preserves model accuracy while enabling direct deployment of NVFP4 checkpoints. The evaluation was conducted using lm_eval against a live endpoint, with results logged for further analysis. This benchmarking process helps validate the effectiveness of the emulation approach for quantized model deployment.
Source: amd
