Concept card

Speculative Decoding — How Claude / GPT-4 / Llama 3 Hit Sub-100ms TTFT

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Speculative Decoding — How Claude / GPT-4 / Llama 3 Hit Sub-100ms TTFT?

Speculative decoding hits sub-100ms time-to-first-token by having a small draft model propose several tokens and verifying all of them with the large target model in one parallel pass — a 2-4x wall-clock speedup with output mathematically identical to running the target model alone.

What it is

Autoregressive generation is sequential by definition — token n+1 depends on token n. On modern GPU hardware that sequential dependency, not raw flops, is the bottleneck: a 70B model takes ~30 ms per token regardless of batch size because each step waits on memory loads, not on arithmetic. Speculative decoding breaks the sequential constraint by using a small, fast draft model to propose k candidate tokens, then verifying all k with the large target model in a single parallel forward pass. When the draft model agrees with the target on j of k tokens, the system advances j+1 tokens (the j accepted + one correction) per target-model step instead of one — a 2-4× wall-clock speedup with mathematically identical output to greedy target-model generation.

The canonical recipe (Leviathan et al., Google, 2023). Target model M_q (large, expensive). Draft model M_p (small, cheap — typically 10-100× smaller). For each generation step: (1) Run M_p autoregressively for k steps producing candidate tokens t 1..t k with probabilities p 1..p k. (2) Run M_q in a single parallel forward pass over the candidate sequence, producing q 1..q k. (3) For each i from 1 to k: accept t i with probability min(1, q i/p i); if accepted continue; if rejected, sample a corrected token from a modified distribution. (4) The expected number of accepted tokens per cycle depends on draft-target agreement — typically 2-4 in practice, giving 2-4× speedup. Crucially the output distribution is mathematically identical to sampling from M_q alone.

Why it matters

The 70B-model bottleneck is memory loads, not arithmetic — each token takes roughly 30ms regardless of batch size, exactly the constraint speculative decoding attacks.
The acceptance probability min(1, q_i/p_i) guarantees the sampled output distribution is identical to greedy target-model generation — no quality trade-off, only a speed one.
Medusa and EAGLE remove the need for a separate draft model (lightweight prediction heads or feature-level prediction instead), avoiding the two-model serving complexity of the original recipe.

Key points

Sequential generation is memory-bandwidth bound, not compute bound — speculative decoding amortises one large-model forward pass across multiple accepted tokens.
Canonical recipe (Leviathan 2023) — small draft model proposes k tokens, target model verifies all k in one parallel pass, accept-reject preserves target distribution exactly.
Modern variants — Medusa (lightweight prediction heads on target, no separate draft); EAGLE (feature-level prediction, ~3-4× speedup); Lookahead (n-gram self-verification, no draft model).
Production users — Anthropic Claude (sub-100 ms TTFT), Meta Llama 3 inference stacks (vLLM/TensorRT-LLM), Mistral/DeepSeek vLLM deployments.
Output is mathematically identical to greedy/sampled target-model generation — speculative decoding is a pure latency optimisation, not a quality trade-off.
Speculative Decoding — How Claude / GPT-4 / Llama 3 Hit Sub-100ms TTFT is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

Speculative decoding: An inference acceleration technique where a small fast draft model proposes k candidate tokens that a large target model then verifies in a single parallel forward pass, achieving 2-4× wall-clock speedup with identical output distribution to target-only sampling.
Draft model: The small, fast auxiliary model in speculative decoding that proposes candidate tokens; typically 10-100× smaller than the target model and ideally trained on the same data distribution to maximise draft-target agreement.
Target model: The large, expensive model whose output distribution speculative decoding preserves exactly; verifies the draft's k candidate tokens in one parallel forward pass.
Medusa: A speculative-decoding variant (Cai et al., 2024) that adds lightweight prediction heads on top of the target model itself, eliminating the need for a separate draft model and simplifying serving.
Time-to-first-token (TTFT): The user-facing latency metric for LLM inference — the wall-clock time between the user submitting a prompt and the first generated token arriving. Speculative decoding is one of the few techniques that improves both TTFT and steady-state throughput.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →