What changed
AWS has introduced a method to replace guesswork with real-time measurements for setting auto scaling targets for text-to-speech (TTS) endpoints on Amazon SageMaker AI. The approach uses two key metrics: Q_max, which measures the highest concurrency one instance can sustain before breaching the service level objective (SLO), and T_total, which measures the time it takes for a scale-out to occur end to end. These metrics are derived from live endpoint data, eliminating the need for guesswork.
What this means in practice
The solution deploys the Kokoro-82M, an 82-million-parameter TTS model, as a custom container on an Amazon SageMaker AI real-time endpoint. AWS Cloud Development Kit (AWS CDK) stacks handle the deployment. The auto scaling policy is based on the measured Q_max and T_total values, which are used to derive scale-out and scale-in thresholds and cooldown periods. This ensures that the endpoint scales appropriately to handle traffic spikes without overprovisioning or underprovisioning resources.
Limitations and availability
The method requires an AWS account with specific permissions and the deployment code from the referenced GitHub repository. The solution is available for models that can be deployed using the AWS CDK stacks, and the auto scaling resources are only synthesized once a real, measured plan exists for the model. The deployment code includes a note about an earlier PyTorch version that may require upgrading before non-proof-of-value use.
