What changed
Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) technology, enabling users to divide a single GPU into multiple isolated partitions. This allows for more efficient GPU utilization by enabling multiple models to run on a single GPU without dedicating an entire accelerator to each model.
What this means in practice
With MIG, users can deploy more models than the number of GPUs available, as each partition provides a dedicated share of the GPU’s compute units and memory. This is particularly beneficial for scenarios where multiple models need to be hosted on the same endpoint, as it reduces the need for additional GPUs and lowers costs. The feature also allows users to select a fraction of a newer GPU instead of a whole one, providing more flexibility in resource allocation. Additionally, MIG ensures predictable latency, fault isolation, and tenant isolation, which are critical for production environments.
Limitations and availability
GPU partitioning is only available for endpoints that use inference components, which are used to deploy multiple models to a single endpoint. Single-model endpoints, serverless endpoints, and heterogeneous instance pools cannot be partitioned. The maximum number of partitions depends on the accelerator and never exceeds seven. Furthermore, MIG support is currently available on SageMaker AI and requires users to define a partition layout when creating an endpoint. The feature is not available for models that require more than one GPU.
