AWS released Ray Serve Deep Learning Containers on June 2024, saying it simplifies model serving by providing pre-built, performance-optimized Docker images for inference workloads. It is the company's first major update to its deep learning container offerings since the launch of the training-focused DLCs in 2022.
AWS reported the Ray Serve DLCs include a full stack of dependencies, including PyTorch, Ray Serve, and CUDA, tested together to avoid version drift. That compares with previous DLCs, which focused primarily on training frameworks and lacked the complete inference stack.
Ray Serve Deep Learning Containers are built on the Amazon Linux 2023 base image for CPU and the NVIDIA Amazon Linux 2023 image for GPU, targeting model serving with HTTP endpoints. Availability begins with public access, initially for developers and enterprises using Amazon EKS and EC2.
"The Ray Serve DLC is purpose-built for serving models behind an HTTP endpoint, maintained and tested by AWS, with the full inference stack already assembled," said David Soto, Senior Machine Learning Specialist at AWS. The container supports common inference tasks like video preprocessing with FFmpeg and NVIDIA hardware acceleration.
The announcement follows the deprecation of TorchServe, which AWS noted was no longer actively maintained. AWS itself frames the significance as a move to reduce the operational burden on developers by eliminating version management and security patching.
AWS did not say how many organizations are currently using TorchServe, and the open question remains about the adoption rate of the new DLCs. The company said the complete code is available in the accompanying repository for developers to test and deploy.
Source: awsml
