Amazon SageMaker HyperPod is expanding its capabilities to support enterprise inference workloads by enabling data capture at multiple points in the inference pipeline. This includes recording inputs and outputs at the endpoint, load balancer, and model pod levels, providing deep observability and auditability through declarative custom resource definition (CRD) configuration. The feature allows teams to independently control and configure data capture at each level, offering flexibility in visibility for different use cases. According to AWS, this enhancement streamlines deployment and operation of large models in production environments. The data capture functionality supports three tiers, each capturing at a different point in the request flow, with the option to enable any combination. These capabilities help reduce cold-start latency by loading weights from node-local NVMe storage and automatically falling back to cloud storage when needed. Source: awsml

To enable data capture, users must configure a dataCapture section within their InferenceEndpointConfig or JumpStartModel CRD. The configuration includes setting an S3 URI, specifying capture options, and optionally adding an AWS KMS key for encryption. Each tier has specific requirements, such as registering the endpoint for Tier 1, enabling the load balancer for Tier 2, and simply enabling the model pod for Tier 3. The system defaults to capturing both input and output at 100% sampling for Tier 3, which provides the deepest visibility closest to the model. Users can also fine-tune buffer settings and payload size limits to optimize storage and performance. Source: awsml

The data capture feature writes all captured data to an Amazon S3 bucket, with a default location if no S3 URI is specified. Each deployment gets a unique path based on a hash derived from the cluster ARN, namespace, CRD type, and deployment name. IAM permissions are required to enable data capture on existing clusters, including access to S3 and KMS resources. These permissions ensure secure and controlled access to the captured data. Source: awsml