What changed
Salesforce has implemented cross-Region inference routing in Amazon SageMaker AI, allowing the company to manage model serving capacity across 30+ global Regions. This change enables the routing of requests to endpoints that can serve them, while maintaining data residency commitments.
What this means in practice
Salesforce’s Agentforce Foundation team serves models for various products across 30+ global Regions and hundreds of AWS accounts. Previously, hosting models separately in each Region led to capacity fragmentation and reliability issues during traffic spikes. With cross-Region inference routing, Salesforce can now route requests to endpoints that have available capacity, improving reliability without compromising data residency.
The inference router is a Regional SageMaker AI resource that your application invokes instead of an endpoint.
It uses routing targets, which are names you choose for one logical model, and lists the endpoints and inference components that may serve that model by their full Amazon Resource Names (ARNs).
The router prefers its own Region and only shifts traffic to other Regions when the local endpoint shows signs of capacity constraints.
Limitations and availability
The cross-Region inference routing is available as a Regional SageMaker AI resource in your account. It requires the use of routing targets, which are names you choose for one logical model, and lists the endpoints and inference components that may serve that model by their full Amazon Resource Names (ARNs). The router does not deploy models, copy artifacts, or add instances. It uses the capacity you have deployed, and each endpoint’s own autoscaling keeps adding instances as its traffic grows.
The router does not move traffic based on model latency or client errors such as invalid input. It also does not deploy models, copy artifacts, or add instances.
The feature is integrated into Salesforce’s Internal Gateway, where the router call sits exactly where the direct InvokeEndpoint call used to be: in the per-request egress path, after the data residency check and before response handling.
