What changed

AWS has introduced the Global Inference Gateway for Amazon SageMaker HyperPod, enabling a single endpoint to route requests across multiple clusters and regions. This addresses the issue of idle GPUs in one region while requests queue in a saturated cluster.

What this means in practice

The Global Inference Gateway allows users to consolidate multiple endpoints into one, reducing the complexity of managing numerous endpoints for different models and clusters. It automatically routes requests to the best-suited cluster based on real-time metrics like KV cache utilization and queue depth. This setup also supports automatic failover, ensuring continuous service even if a cluster fails. Additionally, it eliminates the need for custom networking solutions, as AWS handles the routing and private connectivity through AWS PrivateLink.

Limitations and availability

The Global Inference Gateway is available for users who have already set up SageMaker HyperPod clusters. Setting up the gateway involves publishing clusters and creating the gateway through the SageMaker console. The gateway operates on AWS infrastructure, with users responsible for deciding which clusters to include and how to weight routing signals. The feature is currently in preview and requires users to configure routing weights and manage cluster publications.

Source facts for AWS Introduces Global Inference Gateway for SageMaker HyperPod: You have already paid for that capacity, but your requests can’t reach it. It reads real-time signals such as key-value (KV) cache utilization and queue depth to place every request on the best-suited pod. That solves routing inside one cluster, but not across clusters.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. awsml: Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters