Concept card

LLM Inference Engines — vLLM vs TensorRT-LLM vs llama.cpp vs SGLang

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is LLM Inference Engines — vLLM vs TensorRT-LLM vs llama.cpp vs SGLang?

The inference engine, not the model, is the single biggest lever on LLM serving cost — the same Llama-3-70B can cost 3-10× more on a naive generate loop than on a tuned vLLM or TensorRT-LLM deployment.

An LLM inference engine is the runtime layer that turns a trained model's weights into served tokens: it schedules incoming requests, manages GPU memory, batches prompts together, and decodes outputs token by token. The engine choice is arguably the single biggest lever on serving cost in the entire generative AI stack — the same seventy-billion-parameter open-weight model can cost three to ten times more to run on a naive, unoptimised serving loop than on a properly tuned production engine, before anyone has touched model size or prompt design.

The Four Engines That Matter

vLLM, born out of UC Berkeley research, is the open-source reference implementation most teams reach for first. Its signature technique, PagedAttention, manages the key-value cache the way an operating system manages virtual memory, which lets it pack far more concurrent requests onto the same GPU than earlier serving stacks. Combined with continuous batching — folding new requests into a running batch instead of waiting for the whole batch to finish — vLLM covers a huge range of open-weight model families and is genuinely easy to operate, which is why it has become the default choice for teams self-hosting Llama, Mistral, Qwen or similar models on NVIDIA or AMD hardware.

Why it matters

vLLM is the open-source default for broad model coverage and Python ergonomics; TensorRT-LLM is 1.5-2× faster on H100/H200 but needs a 30-90 minute compilation step per model variant and locks you to NVIDIA.
llama.cpp is the edge/laptop engine (10-15 tok/s on a MacBook M-series), not a data-centre choice; SGLang's RadixAttention is 2-5× faster than vLLM specifically on agentic workloads with heavy prefix reuse.
The decision maps directly to workload: TensorRT-LLM for latency-sensitive single-user chat, vLLM for high-throughput batch RAG, SGLang for multi-turn Joule agents with shared system prompts.

Key points

Four dominant engines — vLLM (open-source default), TensorRT-LLM (max throughput on NVIDIA), llama.cpp (CPU/edge), SGLang (prefix reuse for agents).
vLLM combines PagedAttention (C246) + continuous batching (C245) — 10-24x throughput vs HuggingFace generate() baseline.
TensorRT-LLM compiles per-model engines (FP8/INT8 fused kernels) — 1.5-2x faster than vLLM on H100 but heavier build step + NVIDIA-only.
SGLang's RadixAttention shares KV-cache across requests with common prefixes — 2-5x speedup on agentic + few-shot workloads.
Engine choice is the single biggest cost lever — wrong engine can 3-10x the GPU bill before any model tuning matters.
LLM Inference Engines — vLLM vs TensorRT-LLM vs llama.cpp vs SGLang is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

Inference engine: The runtime that loads model weights into GPU/CPU memory, schedules incoming requests, manages the KV-cache, and decodes output tokens; distinct from the model itself.
vLLM: Open-source LLM inference engine from UC Berkeley (2023) built around PagedAttention and continuous batching; the de-facto reference for self-hosting open-weight models.
TensorRT-LLM: NVIDIA's compiled inference engine that fuses CUDA kernels and applies FP8/INT8 quantization per model; highest absolute throughput on NVIDIA hardware.
RadixAttention: SGLang's technique of organising KV-cache as a prefix tree so multiple requests sharing system prompt / few-shot exemplars reuse the cached compute.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →