What changed

AWS introduced enhanced observability in Amazon SageMaker HyperPod, a new capability that helps users diagnose GPU training issues faster. This feature brings together diagnostic and profiling capabilities for training workloads running on both Amazon Elastic Kubernetes Service (Amazon EKS) and Slurm orchestrators.

What this means in practice

The enhanced observability collects telemetry at two levels: default metrics and advanced metrics. Default metrics are enabled automatically and cover cluster health, CPU/disk/OS-level usage, GPU utilization and temperature, and per-job resource utilization. These metrics are collected every 30 seconds and sent to the chosen metric store with approximately 1–2% CPU overhead and 250–300 MB of memory. Advanced metrics are optional and provide deeper visibility, including full node exporter suite metrics, NVIDIA GPU DCGM and AWS Neuron metrics, and Elastic Fabric Adapter (EFA) metrics.

The diagnostic summary uses Amazon CloudWatch’s DetectGroupOutliers API to compare GPUs within a training job across utilization, memory usage, temperature, power draw, and SM clock frequency. It flags GPUs that deviate from their peers, helping identify issues such as GPU hardware faults, EFA throughput drops, thermal throttling, and overheating.

Limitations and availability

Enhanced observability is delivered as an add-on for EKS-orchestrated clusters and as an extension for Slurm-orchestrated clusters. Users control who can install or change it through AWS Identity and Access Management (IAM). The feature is available through the AWS Management Console, API, or AWS CLI. Metric stores can be chosen between Amazon CloudWatch (new default) or Amazon Managed Service for Prometheus. Advanced metrics are optional due to their higher telemetry volume and cost implications. Most customers running small jobs on small clusters may not need them, but they are essential for long-running training jobs on large clusters.

Source facts for AWS Introduces Enhanced Observability in SageMaker HyperPod for GPU Training Diagnostics: When training throughput drops, the hard part is rarely noticing that something is wrong. It is figuring out why and where. Why you need it Distributed training is expensive, and small inefficiencies compound into significant wasted GPU hours.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. awsml: Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod