What changed
AMD announced optimizations for running GLM-5.2, a large Mixture-of-Experts (MoE) model, on eight AMD Instinct MI355X GPUs. The optimizations include prefill context parallelism (CP8), AITER sparse-attention (DSA) kernels, PyHIP MoE tuning, custom all-reduce, FP8 KV-cache optimization, attention-kernel tuning, small-kernel fusion, GEMM tuning, and EAGLE multi-token prediction (MTP). These changes aim to improve performance for long-context processing and high prefix-cache hit rates.
What this means in practice
The optimized stack significantly boosts throughput for GLM-5.2 inference. EvalScope measurements show peak throughput increasing from 163K to 629K tok/s at 64K input length and from 137K to 553K tok/s at 128K. At a 1M-token input length and concurrency 1, throughput increases from 22K to 157K tok/s, a 7.16× improvement. The optimizations also preserve model accuracy, as confirmed by GSM8K validation.
Limitations and availability
The optimizations are part of an end-to-end stack targeting long-context requests with high prefix-cache hit rates. The work includes deploying the FP8 model with SGLang on eight MI355X GPUs. The required SGLang support for CP metadata, KV reconstruction, and tensor layouts was added to enable reliable CP8 prefill. The optimized server image is available as sabreshao/sglang:glm_0831. The test environment uses ROCm 7.2.4 and specific versions of the baseline and optimized server images. The benchmarks and scripts used are published alongside the blog for reproducibility.
