NVIDIA’s TensorRT Edge-LLM v0.11.1 adds a unified trtedge command-line interface for preparing and serving supported models, according to its October 28 release.

The big change

  • Developers can use one interface for preparation, engine builds and HTTP serving.
  • The release adds Windows engine building and inference through TensorRT RTX 1.7.
  • NVIDIA’s Spark example has specific hardware and software requirements.

From a model to an application

The release lists model preparation, quantization, ONNX export, direct engine builds and HTTP serving as CLI capabilities. This gives developers a common entry point for several stages between a model checkpoint and an application that can send it requests.

An engine is a runnable artifact rather than the original model alone. In its separate TensorRT architecture documentation, NVIDIA explains that an engine file contains a model graph and compiled execution state. The builder selects implementations using timing, and builds can require substantial temporary device memory. The engine is then available to a runtime without keeping the builder alive.

A specific Spark setup

NVIDIA’s October 30 tutorial demonstrates the text-only Nemotron-3.5 Lightning model on DGX Spark, including a local OpenAI-compatible API. It specifies CUDA 13, Python 3.10–3.12 and TensorRT 10.16.1.11 or newer within 10.x; its examples use Edge-LLM 0.11.1 and TensorRT 10.16.1.11. The initial model download is about 21GB, with space also needed for engines.

The tutorial says later launches reuse matching cached engines. BIG CHANGE has not independently tested NVIDIA’s example.

BIG CHANGE previously covered TensorRT’s separate global performance tuner.

Sources & further reading