Amazon announced model caching for Amazon SageMaker HyperPod on June 14, 2024, saying it reduces inference cold starts from minutes to seconds. It is the company's first major update to its SageMaker Inference service since its launch in 2017.
Amazon reported that large models like DeepSeek-R1, which can be over 600 GB, previously took 30 minutes or more to start serving traffic due to cold starts. That compares with smaller models, which might take just a few minutes.
Model caching is built on Amazon SageMaker HyperPod and targets large-scale inference workloads. Availability begins with the latest SageMaker Inference update, initially for users on AWS cloud infrastructure.
"When you start your pod, it can read from local NVMe storage at approximately 7 GB/s instead of downloading over the network," said Anand Chandrasekaran, Machine Learning Specialist Solutions Architect. This significantly reduces the time to serve traffic after a pod starts.
The announcement follows increased demand for faster inference services in large-scale machine learning applications. Amazon said the update improves autoscaling efficiency by eliminating the need for repeated downloads during scale-out events.
Amazon did not say how model caching will perform with models larger than 600 GB, and it raised the question of how well the feature will scale with higher traffic volumes. The company said it will continue to refine the feature based on user feedback.
Source: awsml
