Concept card

Continuous Batching — How vLLM Hits 10× Throughput

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Continuous Batching — How vLLM Hits 10× Throughput?

Static batching wastes 60-90% of GPU compute because one long request in a batch forces the GPU to keep running padded slots for requests that finished early — continuous batching frees each slot the instant its request ends.

Continuous batching — also called iteration-level or in-flight batching — is the scheduling technique that lets an LLM inference engine start processing a new request the moment an older one finishes, rather than waiting for an entire batch to complete together. It is the single largest reason engines such as vLLM and TensorRT-LLM reach roughly ten to twenty-four times higher throughput than a naive text-generation loop running on the same GPU hardware, and it is worth understanding even for a non-engineer evaluating an SAP AI deployment, because it directly drives the cost-per-token and the response latency a consultant will be asked to explain to a client.

Why naive batching fails, and how the alternative works

Why it matters

The scheduler reasons in iterations, not requests: it repacks the batch every forward pass, admitting a new pending request the moment any slot frees.
Combined with PagedAttention, continuous batching keeps GPU utilization at 60-90% across requests of wildly different lengths, versus the massive waste of static batching.
Throughput improves 5-10× at the same hardware budget, with P99 tail latency improving even more since short requests no longer queue behind long ones.

Key points

Iteration-level scheduling — new requests admitted to the batch as soon as older ones finish, not at batch boundaries.
Naive static batching wastes 60-90% of GPU compute on real workloads with variable output length.
Combined with PagedAttention (C246), continuous batching keeps GPU at 60-90% effective utilisation across diverse-length requests.
10-24x throughput improvement over HuggingFace generate() baseline on the same hardware.
P99 tail latency improves because short requests do not queue behind long ones.
Implementation complexity is why most teams adopt vLLM or TensorRT-LLM rather than build their own scheduler.
Continuous Batching — How vLLM Hits 10× Throughput is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.

Terms used on this page

Continuous batching: Scheduling LLM inference at the iteration level so new requests are admitted into the batch as soon as older requests finish, rather than waiting for the whole batch to complete.
Static batching: The naive approach where a batch of requests is fixed at admission time and all step through forward passes together until the longest one finishes.
Iteration-level scheduling: Synonym for continuous batching; emphasises that scheduling decisions happen at every forward-pass iteration, not at request boundaries.
Head-of-line blocking: The latency problem in static batching where short requests must wait for the longest request in their batch to finish; eliminated by continuous batching.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →