What changed
Today, Amazon announced that SGLang-Neuron is publicly available and can be accessed through an AWS Deep Learning Container, released by RadixArk in collaboration with Annapurna Labs. With SGLang-Neuron, users can serve models and generate reinforcement learning (RL) rollouts with SGLang on AWS Trainium using the same SGLang interface they use today.
What this means in practice
Trainium is a portfolio of AWS AI chips that delivers strong economics for AI training and inference at scale.
Now, running SGLang on Trainium follows the SGLang workflow users already know: they launch models through SGLang’s server interface and invoke the endpoint using the same generation APIs. Underneath that interface, SGLang-Neuron runs on native PyTorch.
With its out-of-tree plugin design, Trainium support and SGLang core develop independently and iterate faster.
Limitations and availability
To follow this walkthrough, users use a Trainium3 Amazon Elastic Compute Cloud (Amazon EC2) instance with a pytorch-inference-sglang-neuronx container. The steps include starting the container, launching the SGLang model server inside the container, and sending a request to the model server.
The model used in the example is openai/gpt-oss-120b, with specific parameters such as tp-size, ep-size, and context-length. The supplied announcement does not specify the exact performance benchmarks or independent testing results for SGLang-Neuron.
The plugin implements the Trainium-specific models, attention, mixture of experts (MoE), and the graph runner.
SGLang-Neuron also supports PD disaggregation, where a prefill server computes the prompt KV cache and transfers it to a decode server using NIXL over Elastic Fabric Adapter (EFA). The decode server then generates the remaining tokens.
Each server still uses SGLang’s scheduler and request lifecycle, while the plugin manages the Trainium execution and KV-cache transfer path.
