AMD has introduced SGLang-ATOM, a tool that connects the SGLang serving framework with ATOM’s ROCm-native execution path for AMD Instinct GPUs. This integration allows developers to leverage optimized model execution, weight loading, and attention mechanisms on AMD hardware without rebuilding the entire serving stack. The solution aims to reduce migration effort for existing SGLang applications while improving throughput, latency, and cost efficiency in production environments.
SGLang-ATOM operates within the ROCm LLM inference stack, acting as a bridge between the familiar SGLang serving surface and ATOM’s optimized execution path. By using SGLang’s model registration mechanism, ATOM integrates its optimized model implementations into the upstream SGLang framework, allowing developers to retain the familiar serving workflows while benefiting from ROCm-native acceleration. This approach minimizes changes to the SGLang source tree, enabling teams to adopt ATOM-backed execution where model support and workload requirements align.
The design of SGLang-ATOM includes a compact wrapper system that conforms to SGLang’s model interface, delegating model creation to ATOM. This system allows for dynamic class generation, reducing per-model boilerplate and enabling rapid integration of new model paths. Additionally, ATOM’s optimized attention backend is integrated into SGLang’s attention backend registration, allowing for AMD-optimized kernels to handle prefill, decode, and KV cache operations beneath the SGLang serving layer.
Source: amd
