AWS released a push-based NIXL connector in vLLM, enhancing disaggregated inference by reducing TTFT and TPOT. It is the company's first update to the vLLM engine since its initial release.
AWS reported up to 3x improvement in TTFT and 30 percent better TPOT for the Qwen3-32B model, measured on a 1P1D deployment using Amazon EC2 P5en instances connected with EFA. That compares with pull-based transfer results.
The push-based transfer is built on the NVIDIA Inference Xfer Library (NIXL) and targets disaggregated inference for large language models. Availability begins with AWS Trainium instance clusters connected with EFA, initially for developers and enterprises.
"Push-based KV transfer removes the serialization of the pull model by overlapping producer and consumer work," said AWS. This approach allows the decode instance to register target blocks in advance while the prefill instance computes the KV cache.
The announcement follows the release of vLLM, which introduced disaggregated inference for LLMs. AWS said the push-based transfer improves resource utilization and throughput under load.
AWS did not specify the exact performance gains for all models, and the results depend on workload and configuration. The company said the feature will be contributed to vLLM as an accelerator-agnostic feature.
Source: awsml
