AWS and the SGLang team released SGLang-Neuron 0.1.0 in public preview on November 12, providing a container and source for serving selected models on Trainium.

Serving keeps a model available to receive an application's requests and return generated outputs. Choosing a serving system therefore involves both the model's answers and the software that handles requests while users wait.

The big change

  • The supported hardware depends on the workload: language models use Trn3, while the listed diffusion models use trn2.48xlarge. A model-family name alone does not establish compatibility.
  • This follows a limited October beta. That earlier announcement already described Trainium serving and reinforcement-learning rollouts, so developers should distinguish wider access from a first demonstration.
  • An evaluation should use the application's prompts, expected output lengths and simultaneous traffic. A successful answer to one question cannot establish acceptable response times under the expected load.

From requests to execution

SGLang receives and schedules requests; the plug-in supplies Trainium execution through TorchNeuron, compiling PyTorch operations and specialized kernels. The server interface and chip execution remain separate.

The original SGLang research explains why serving software matters beyond loading a model. Its RadixAttention mechanism reuses a cache of attention state for the shared beginning of prompts. Reusing that state avoids repeating work for text the model has already processed.

The paper's illustrated chat example makes this concrete: a later message can reuse the cached first turn, while another conversation can share the system prompt. Available memory limits that reuse; less recently used entries can be removed to make room. The benefit depends on the workload. The authors found reuse more helpful for short replies than long ones, where generating the answer dominated execution time. These are findings about the earlier system, not a benchmark of the Trainium preview.

A starting configuration

The repository Qwen3-8B example uses four NeuronCores. Requirements include Docker, Neuron SDK 2.33.0 and driver 2.31.7.0 or later; source builds require SGLang revision 43881a7.

Preview APIs and model support may change. BIG CHANGE has not tested the release.

The October announcement described TorchNeuron as a native PyTorch backend and said performance work was ongoing. Its reinforcement-learning example separated generating training experiences from updating the model: rollouts ran on Trainium3, while policy updates ran on an H200 GPU node. That example explains different jobs within a training workflow; it is not evidence that every training step ran on Trainium.

For a purchase or migration decision, set an answer-quality threshold and a response-time target before testing. Include instance charges and the engineering work needed to reproduce the environment. Compare the actual application's results against those targets, rather than treating a documented launch command as a cost or reliability measurement.