High-Performance Large Language Model Inference Framework for NVIDIA Edge Platforms
Overview | Support Matrix | Quick Start | Performance | Documentation | Roadmap
- [2026/09] TensorRT Edge-LLM completes the MLPerf Edge Agentic benchmark 6.4x faster on Jetson AGX Thor.
- [2026/09] Release 0.11.0 adds published Python wheels, Muse-Glimmer, experimental pi0.5, Hunyuan dense models, guided decoding, opt-in in-flight batching, native provider Jinja chat templates, and GPU-free CuTe DSL artifact generation.
- [2026/09] Release 0.10.1 adds experimental TP=2 inference on Dual NVIDIA DGX Spark and redesigns the experimental OpenAI-compatible server for faster cold launches and lower memory usage.
- [2026/08] TensorRT Edge-LLM 0.10.0 adds Day-0 support for Qwen3.8-27B.
- [2026/08] Release 0.10.0 adds support for NVIDIA Nemotron-3.5 Lightning with MTP and DFlash, Cosmos3-Edge, DiffusionGemma, Nemotron-3.5-ASR, and DSpark speculative decoding, alongside an experimental direct TensorRT engine builder without ONNX export, multi-turn KV-cache reuse, and video input for the experimental OpenAI-compatible server.
- [2026/07] Support for the full Gemma 4 family (E2B / E4B / 12B / 26B-A4B / 31B — multimodal text + image + audio, with MTP), Qwen3-Omni and Nemotron-3 NVFP4, and DFlash speculative decoding (with DDTree for Qwen3 / Qwen3.5) landed across releases 0.9.0 and 0.9.1.
TensorRT Edge-LLM is NVIDIA's C++ inference runtime for text, vision, audio, speech, and action models on NVIDIA Jetson, NVIDIA DRIVE, and NVIDIA DGX Spark. The supported frontend exports Hugging Face checkpoints to ONNX for C++ engine building; an experimental direct frontend builds engines from checkpoints without ONNX. Both paths use the same C++ deployment runtimes.
Check the Official Support Matrix, then follow the Quick Start Guide. Checkpoint IDs are listed in Supported Models.
For a supported target, install a published Python wheel
without compiling Edge-LLM. Use tensorrt-edgellm[server] for the high-level
Python API and HTTP serving; see the extras guide
for export/tools dependencies and the minimal base workflow.
Install the base package from PyPI (requires a supported CUDA/TensorRT stack):
pip install tensorrt-edgellm==0.11.0- Overview - What is TensorRT Edge-LLM and key features
- Official Support Matrix - Platform, JetPack, DriveOS, CUDA, TensorRT, and TensorRT Edge-LLM compatibility
- Supported Models - Complete model compatibility matrix
- Checkpoint Exporter - Recommended ONNX export pipeline
- Experimental Direct Engine Builder - Build all model components directly from a checkpoint
- Installation - Set up quantization,
tensorrt_edgellm, and the C++ runtime - Quick Start Guide - Run your first inference in ~15 minutes
- Examples - End-to-end workflows
- Quantization - Create quantized checkpoints for
tensorrt_edgellm - Experimental High-Level Python API and Server - vLLM-style API and OpenAI-compatible server
- Input Format Guide - Request format and specifications
- Chat Template Format - Chat template configuration
- Quantization Package Design - Quantization package architecture
- Engine Builder - Building TensorRT engines
- C++ Runtime Overview - Runtime system architecture
- Customization Guide - Customizing TensorRT Edge-LLM for your needs
- TensorRT Plugins - Custom plugin development
- Tests - Comprehensive test suite for contributors
See the Performance Benchmarks page for released benchmark results covering LLM and VLM prefill, generation throughput, memory usage, and EAGLE speculative decoding speedups.
🚗 Automotive
- In-vehicle AI assistants
- Voice-controlled interfaces
- Scene understanding
- Driver assistance systems
🤖 Robotics
- Natural language interaction
- Task planning and reasoning
- Visual question answering
- Human-robot collaboration
🏭 Industrial IoT
- Equipment monitoring with NLP
- Automated inspection
- Predictive maintenance
- Voice-controlled machinery
📱 Edge Devices
- On-device chatbots
- Offline language processing
- Privacy-preserving AI
- Low-latency inference
- TensorRT Edge-LLM Jetson AI Lab tutorial
- Maximizing Memory Efficiency to Run Bigger Models on NVIDIA Jetson
- Build Next-Gen Physical AI with Edge-First LLMs for Autonomous Vehicles and Robotics
- Accelerate AI Inference for Edge and Robotics with NVIDIA Jetson T4000 and NVIDIA JetPack 7.1
- Accelerating LLM and VLM Inference for Automotive and Robotics with NVIDIA TensorRT Edge-LLM
Follow our GitHub repository for the latest updates, releases, and announcements.
- Documentation: Full Documentation
- Quick Start: Quick Start Guide
- Roadmap: Developer Roadmap
- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Forums: NVIDIA Developer Forums
We welcome contributions! Please see our Contributing Guidelines for details.