What changed
AWS announced a new method for streaming low-latency speech from AI agents using the Kokoro TTS model on Amazon SageMaker AI. The solution addresses the issue where text-to-speech integrations wait for full responses before starting synthesis, leading to delays and user frustration. Instead, the system streams each sentence to a hosted Kokoro-82M TTS model as soon as the agent produces it, allowing audio playback to start immediately.
What this means in practice
The solution enables real-time speech synthesis by streaming each sentence to the Kokoro-82M model, which runs on a SageMaker AI real-time endpoint. This model, with 82 million parameters, is designed to run efficiently on a mid-tier GPU instance, keeping both latency and inference costs low. The system supports two transport methods: a unidirectional POST /invocations route for complete input responses and a bidirectional WebSocket /invocations-bidirectional-stream route for handling multiple messages in both directions.
Limitations and availability
The solution requires an AWS account with specific permissions and prerequisites such as an IAM role, an ECR repository, and the AWS CDK v2. The deployment involves setting up a SageMaker AI real-time endpoint with a custom container that includes the Kokoro-82M model. The container uses an older version of PyTorch, which should be upgraded before non-proof-of-value use. The solution is available through the AWS SDK for Python (Boto3) and requires familiarity with WebSockets and ASGI frameworks. The deployment code is provided in a GitHub repository referenced in the post.
