AWS and NVIDIA announced a solution that reduces automatic speech recognition (ASR) inference costs by 75% using NVIDIA Multi-Process Service (MPS) on Amazon EC2. The approach addresses the inefficiency of GPU utilization, where a single ASR request typically uses only 15–20% of a GPU’s compute capacity, leaving the majority idle. This inefficiency forces companies like Heidi Health to run 16 GPU instances to maintain sub-second latency during peak traffic. By deploying MPS, the solution reduces infrastructure requirements to just four instances while maintaining sub-second latency at 92.1 requests per second (RPS) per GPU.
The collaboration leverages NVIDIA MPS, which allows multiple processes to share a GPU concurrently without code changes. This eliminates context-switching overhead and enables concurrent kernel execution across different SMs. The solution also integrates NVIDIA Triton Inference Server for request scheduling and batching, supporting dynamic and sequence batching strategies. For transcription, dynamic batching accumulates requests for a configurable delay before dispatching them as a batch, while sequence batching maintains streaming state for diarization. These optimizations, combined with model-level improvements using ONNX Runtime and TensorRT, enhance performance and reduce costs.
Heidi Health, an AI Care Partner, processes over 2.4 million clinical consultations weekly across 190 countries. The company previously required 16 GPUs to sustain sub-second transcription latency at peak traffic. With the new approach, GPU utilization is optimized, reducing infrastructure needs by 75% while maintaining performance standards. The solution is deployed on Amazon EC2 g6e.4xlarge and g7e.4xlarge instances with NVIDIA L40S GPUs, using Docker Compose to orchestrate three containerized components.
Source: awsml
