Skill · em Dados, IA e pesquisa

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

Procedência

Antes de instalar

4 arquivos · 26,5 KB · só texto, nenhum script

Instalar na sua CLI

O comando baixa a versão fixada (commit 57f899e) direto da origem, para a pasta que a CLI lê. Precisa de curl (macOS e Linux); no Windows não há comando, porque o Rook Labs é para macOS.

Claude Code

Neste projeto: instala em .claude/skills/tensorrt-llm/.

d=".claude/skills/tensorrt-llm"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/multi-gpu.md" "$u/references/multi-gpu.md" \
  -o "$d/references/optimization.md" "$u/references/optimization.md" \
  -o "$d/references/serving.md" "$u/references/serving.md"

Global: instala em ~/.claude/skills/tensorrt-llm/.

d="$HOME/.claude/skills/tensorrt-llm"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/multi-gpu.md" "$u/references/multi-gpu.md" \
  -o "$d/references/optimization.md" "$u/references/optimization.md" \
  -o "$d/references/serving.md" "$u/references/serving.md"

Codex

Neste projeto: instala em .agents/skills/tensorrt-llm/.

d=".agents/skills/tensorrt-llm"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/multi-gpu.md" "$u/references/multi-gpu.md" \
  -o "$d/references/optimization.md" "$u/references/optimization.md" \
  -o "$d/references/serving.md" "$u/references/serving.md"

Global: instala em ~/.agents/skills/tensorrt-llm/.

d="$HOME/.agents/skills/tensorrt-llm"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/multi-gpu.md" "$u/references/multi-gpu.md" \
  -o "$d/references/optimization.md" "$u/references/optimization.md" \
  -o "$d/references/serving.md" "$u/references/serving.md"

Antigravity

Neste projeto: instala em .agents/skills/tensorrt-llm/.

d=".agents/skills/tensorrt-llm"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/multi-gpu.md" "$u/references/multi-gpu.md" \
  -o "$d/references/optimization.md" "$u/references/optimization.md" \
  -o "$d/references/serving.md" "$u/references/serving.md"

Global: instala em ~/.gemini/antigravity-cli/skills/tensorrt-llm/.

d="$HOME/.gemini/antigravity-cli/skills/tensorrt-llm"
u="https://raw.githubusercontent.com/davila7/claude-code-templates/57f899e5394bb8ca166f38eacae8f0853cbfe033/cli-tool/components/skills/ai-research/inference-serving-tensorrt-llm"
curl -fsSL --create-dirs \
  -o "$d/SKILL.md" "$u/SKILL.md" \
  -o "$d/references/multi-gpu.md" "$u/references/multi-gpu.md" \
  -o "$d/references/optimization.md" "$u/references/optimization.md" \
  -o "$d/references/serving.md" "$u/references/serving.md"

Peça ao Rook

Já usa o Rook Labs? Cole no chat do Rook: instale a skill https://rooklabs.sh/marketplace/cct.tensorrt-llm

Prévia do SKILL.md

---
name: tensorrt-llm
description: Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Inference Serving, TensorRT-LLM, NVIDIA, Inference Optimization, High Throughput, Low Latency, Production, FP8, INT4, In-Flight Batching, Multi-GPU]
dependencies: [tensorrt-llm, torch]
---

# TensorRT-LLM

NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.

## When to use TensorRT-LLM

**Use TensorRT-LLM when:**
- Deploying on NVIDIA GPUs (A100, H100, GB200)
- Need maximum throughput (24,000+ tokens/sec on Llama 3)
- Require low latency for real-time applications
- Working with quantized models (FP8, INT4, FP4)
- Scaling across multiple GPUs or nodes

**Use vLLM instead when:**
- Need simpler setup and Python-first API
- Want PagedAttention without TensorRT compilation
- Working with AMD GPUs or non-NVIDIA hardware

**Use llama.cpp instead when:**
- Deploying on CPU or Apple Silicon
- Need edge deployment without NVIDIA GPUs
- Want simpler GGUF quantization format

## Quick start

### Installation

```bash
# Docker (recommended)
docker pull nvidia/tensorrt_llm:latest
…

Ver todo o marketplace