Salesforce faced a challenge in making its AI foundation, Agentforce, highly available across multiple Availability Zones (AZs). The company used Amazon SageMaker AI Inference Components (ICs) to cut GPU costs, but the default placement of these components did not guarantee the Multi-AZ resilience required by its compliance standards. Salesforce achieved an 8x reduction in infrastructure costs by co-hosting multiple models on shared GPUs, but this introduced a new problem: ensuring high availability across multiple AZs.
To address this, Salesforce leveraged the new IC Placement capability introduced through the SchedulingConfig parameter in the CreateInferenceComponent API. This feature gives customers control over how IC copies are distributed across instances and AZs. Two key sub-parameters, AvailabilityZoneBalance and PlacementStrategy, govern the high availability behavior. AvailabilityZoneBalance ensures copies are evenly distributed across AZs, while PlacementStrategy determines whether copies are spread across as many instances as possible (SPREAD) or packed onto fewer instances (BINPACK) for efficiency.
The source explains how Salesforce configured the SchedulingConfig to achieve Multi-AZ high availability by deploying models with SPREAD placement and AZ balancing. This ensures that model copies are spread across multiple instances in different AZs, reducing the risk of single points of failure. The company also emphasized the importance of maintaining a minimum of two copies per AZ for compliance and reliability. Salesforce used the SchedulingConfig to scale out and scale in while preserving AZ balance, ensuring continuous high availability for its AI models.
Source: awsml
