AMD has demonstrated how to scale the GLM-5.1-FP8 model across 64 MI300X GPUs using WideEP and MoRI. The system enables disaggregated inference with prefill-decode (PD) separation, allowing for efficient scaling of large, sparse Mixture-of-Experts (MoE) models. This approach addresses challenges in maintaining accuracy and performance as models are distributed across multiple nodes. The results show correct long-context serving and clean throughput scaling across four topologies, from a single prefill and decode pair (EP8) up to EP32 on both the prefill and decode all-to-all paths. According to AMD, this is the first in a series of posts exploring how to bring frontier models from single-node demos to production-level multi-node serving on AMD Instinct GPUs.
The post highlights two production-blocking defects that were resolved to enable reliable scaling. One defect involved a long-context accuracy collapse past ~30k tokens, which was fixed by updating the metadata cache to use a per-request (context_len, query_len) pair. The second defect was an 8k-prefill disaggregation crash caused by a second, easy-to-miss KV cache introduced by GLM-5.1’s sparse attention. This issue was resolved by pairing main and indexer caches by layer index and transferring all 156 caches. These fixes ensure that the model can maintain performance and correctness as it scales across multiple nodes.
The post also describes the architecture of the system, which uses xPyD notation to define topologies with x prefill instances and y decode instances, each instance being an 8-GPU MI300X node. The EP width is instances × 8, allowing for scalability from EP8 to EP32. The system uses MoRI-EP for MoE dispatch and combine all-to-all communication, while MoRI-IO handles RDMA KV-cache transfer between prefill and decode roles. The model, GLM-5.1-FP8, combines Multi-head Latent Attention with DeepSeek Sparse Attention, making it suitable for long-context serving. The results show that the system can handle up to 2k–35k tokens with DeepSeek-V3 non-regressed on the shared stack.
Source: amd
