AWS Introduces Disaggregated Prefill and Decode for LLM Inference on SageMaker HyperPod
AWS announced a new method for LLM inference using disaggregated prefill and decode, improving performance for long prompts and high-concurrency workloads. The approach uses EFA and RDMA to reduce latency spikes in streaming applications.