What changed
AWS has introduced the Global Inference Gateway for Amazon SageMaker HyperPod, enabling a single endpoint to route requests across multiple clusters and regions. This addresses the issue of idle GPUs in one region while requests queue in a saturated cluster.
What this means in practice
The Global Inference Gateway allows users to consolidate multiple endpoints into one, reducing the complexity of managing numerous endpoints for different models and clusters. It automatically routes requests to the best-suited cluster based on real-time metrics like KV cache utilization and queue depth. This setup also supports automatic failover, ensuring continuous service even if a cluster fails. Additionally, it eliminates the need for custom networking solutions, as AWS handles the routing and private connectivity through AWS PrivateLink.
Limitations and availability
The Global Inference Gateway is available for users who have already set up SageMaker HyperPod clusters. Setting up the gateway involves publishing clusters and creating the gateway through the SageMaker console. The gateway operates on AWS infrastructure, with users responsible for deciding which clusters to include and how to weight routing signals. The feature is currently in preview and requires users to configure routing weights and manage cluster publications.
