AMD has released a guide detailing how to serve the Kimi-K2.5-MXFP4 model on the MI355X GPU using the ATOM framework. The guide highlights how ATOM enables the runtime to recognize MXFP4 metadata and dispatch optimized gfx950 FP4 kernels, allowing for an OpenAI-compatible endpoint without manual wiring of model and kernel components. This approach simplifies the deployment of large language models on AMD hardware. The post also outlines how ATOM integrates with existing frameworks like vLLM, offering an out-of-tree plugin path for users who prefer to retain their current serving infrastructure. ATOM is described as a lightweight implementation focused on optimizing AMD GPU model execution with AITER kernels. It provides a framework layer that routes model operations to AITER kernels, supports OpenAI-compatible endpoints, and enables low-latency decoding through piecewise torch.compile and CUDA graph capture. Additionally, ATOM supports multi-GPU execution for large MoE models and quantization-aware dispatch, mapping HuggingFace quantization metadata to runtime choices. For the Kimi-K2.5-MXFP4 model, this includes recognizing packed FP4 weights and E8M0 scale tensors, which are mapped to the per_1x32 FP4 runtime path and gfx950 kernels that execute scaled MFMA instructions. The guide also discusses the transition from W4A8 configurations to MXFP4, highlighting how MXFP4 retains fine-grained 32-value block scaling to leverage the new scaled MFMA hardware path on gfx950. This shift allows for a more efficient memory and computation trade-off compared to FP8, with FP4 payloads reducing weight and activation traffic for quantized matrix paths. The hardware-visible block scales on gfx950 enable the simultaneous consumption of FP4 payloads and E8M0 scale operands in scaled MFMA, offering a distinct advantage over FP8. This approach provides a more compact payload and a fine-grained scale layout, which is crucial for optimizing performance on the latest AMD GPUs. The post concludes by emphasizing how ATOM’s standalone server and vLLM plugin paths offer different pathways for model deployment, each with its own benefits for validation and operational efficiency. The guide is part of AMD’s ongoing efforts to optimize AI/ML workloads on its GPU platforms.
Source: amd
