AWS has introduced a new approach to large language model (LLM) inference by disaggregating the prefill and decode phases on SageMaker HyperPod. This method, called Disaggregated Prefill and Decode (DPD), separates the two phases of LLM inference to avoid GPU contention, allowing for more efficient processing of long prompts and high-concurrency workloads. By running prefill and decode on separate GPU pools connected through Elastic Fabric Adapter (EFA) with Remote Direct Memory Access (RD-MA), AWS aims to reduce latency spikes that often occur in streaming applications. The technique is designed to address challenges in multi-node deployments and optimize routing for mixed traffic scenarios. Source: awsml

Disaggregated Prefill and Decode delivers the strongest gains for long-context, high-concurrency streaming workloads such as chat assistants, agentic pipelines, and Retrieval Augmented Generation (RAG) with large retrieved contexts. In these cases, a single long prompt on a colocated GPU can stall in-flight decode for every other request, causing per-token latency spikes. DPD removes this interference by construction, ensuring that long-context prefills do not block ongoing decode requests. The method is particularly effective when input prompts regularly exceed 4,096 tokens, and when multiple concurrent users or requests are involved. The DPD router automatically handles mixed long and short traffic without requiring manual routing logic. Source: awsml

The HyperPod DPD implementation is built on the vLLM Production Stack router, with LMCache providing the KV cache transfer layer over NIXL and EFA. The deployment includes three components plus a transport stack: an intelligent router, a prefiller pod, and a decoder pod. The router tokenizes each prompt and applies a configurable token threshold to decide whether the request takes the disaggregated path or runs end-to-end on a decoder. The prefiller pod computes KV cache for long prompts and pushes it to the chosen decoder, while the decoder pod reserves GPU memory for incoming KV transfers and runs full CUDA graphs for the decode kernel. Source: awsml