What changed
Amazon SageMaker AI now monitors generative AI inference endpoints for latency drift, idle GPUs, and missed optimizations. This feature is enabled by default for endpoints using vLLM serving containers and GPU instances, with no setup required. It analyzes operational signals like latency, GPU utilization, and serving configuration to identify three types of findings: latency drift, unused capacity, and configuration gaps.
What this means in practice
The feature sends findings through AWS Health, with one notification per endpoint. These notifications explain observed changes, their impact on latency, cost, or capacity, and recommend actions through the SageMaker AI agentic experience in Amazon SageMaker Studio. Users can benchmark current configurations against alternatives on separate compute, comparing response time, throughput, and cost per million tokens. Changes to the endpoint are only applied after explicit user action.
Limitations and availability
Monitoring is on by default for eligible endpoints but can be opted out. The feature does not read user requests or responses and does not incur additional GPU costs for monitoring. Benchmarks use standard SageMaker AI charges or ML reservations. Notifications are region-specific and provide estimates labeled as such. The feature is available for endpoints using vLLM serving containers and GPU instances, with no setup required.
