Amazon SageMaker released prefix-aware routing on Amazon SageMaker Inference, saying it improves LLM latency and throughput by leveraging prefix caching. It is the company's first major update to its inference platform since the launch of vLLM and TensorRT-LLM.
Amazon SageMaker reported a 77% reduction in P50 TTFT and a 16% increase in throughput, measured on Llama 3.1 70B Instruct. That compares with the default random routing baseline.
Prefix-aware routing is built on Amazon SageMaker Inference and targets LLM workloads where requests share common prefixes. Availability begins with the launch of the feature, initially for production variants configured via endpoint settings.
"When a request arrives at your endpoint, Amazon SageMaker looks at the beginning of the payload and uses it to decide which instance should handle it," said the blog post. The endpoint handles routing based on the content of the request without requiring manual tagging.
The announcement follows the release of vLLM and TensorRT-LLM, which introduced prefix caching as a solution for repeated prefixes. Amazon SageMaker said the feature is designed to work with existing serving frameworks.
Amazon SageMaker did not say how the feature will perform with non-LLM models, and it raised the question of whether routing overhead could impact performance in certain scenarios. The company said it will continue to refine the feature based on user feedback.
Source: awsml
