domains / families / data / tensorrt-llm

tensorrt-llm

separate

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

When to use

• Deploying on NVIDIA GPUs (A100, H100, GB200)
• Need maximum throughput (24,000+ tokens/sec on Llama 3)
• Require low latency for real-time applications
• Working with quantized models (FP8, INT4, FP4)
• Scaling across multiple GPUs or nodes
• Need simpler setup and Python-first API
• Want PagedAttention without TensorRT compilation
• Working with AMD GPUs or non-NVIDIA hardware

When not to use

• Simpler setup and Python-first API
• PagedAttention without TensorRT compilation
• Working with AMD GPUs or non-NVIDIA hardware
• Deploying on CPU or Apple Silicon
• Edge deployment without NVIDIA GPUs
• Simpler GGUF quantization format

Skill page

Source