Speculative Decoding — How Claude / GPT-4 / Llama 3 Hit Sub-100ms TTFT

What it is

Autoregressive generation is sequential by definition — token n+1 depends on token n. On modern GPU hardware that sequential dependency, not raw flops, is the bottleneck: a 70B model takes ~30 ms per token regardless of batch size because each step waits on memory loads, not on arithmetic. Speculative decoding breaks the sequential constraint by using a small, fast draft model to propose k candidate tokens, then verifying all k with the large target model in a single parallel forward pass. When the draft model agrees with the target on j of k tokens, the system advances j+1 tokens (the j accepted + one correction) per target-model step instead of one — a 2-4× wall-clock speedup with mathematically identical output to greedy target-model generation.

The canonical recipe (Leviathan et al., Google, 2023). Target model M_q (large, expensive). Draft model M_p (small, cheap — typically 10-100× smaller). For each generation step: (1) Run M_p autoregressively for k steps producing candidate tokens t 1..t k with probabilities p 1..p k. (2) Run M_q in a single parallel forward pass over the candidate sequence, producing q 1..q k. (3) For each i from 1 to k: accept t i with probability min(1, q i/p i); if accepted continue; if rejected, sample a corrected token from a modified distribution. (4) The expected number of accepted tokens per cycle depends on draft-target agreement — typically 2-4 in practice, giving 2-4× speedup. Crucially the output distribution is mathematically identical to sampling from M_q alone.

Why it matters

• The 70B-model bottleneck is memory loads, not arithmetic — each token takes roughly 30ms regardless of batch size, exactly the constraint speculative decoding attacks.
• The acceptance probability min(1, q_i/p_i) guarantees the sampled output distribution is identical to greedy target-model generation — no quality trade-off, only a speed one.
• Medusa and EAGLE remove the need for a separate draft model (lightweight prediction heads or feature-level prediction instead), avoiding the two-model serving complexity of the original recipe.

Key points

• Sequential generation is memory-bandwidth bound, not compute bound — speculative decoding amortises one large-model forward pass across multiple accepted tokens.
• Canonical recipe (Leviathan 2023) — small draft model proposes k tokens, target model verifies all k in one parallel pass, accept-reject preserves the target distribution exactly.
• Modern variants — Medusa (lightweight prediction heads on the target, no separate draft); EAGLE (feature-level prediction, ~3-4× speedup); Lookahead decoding (n-gram self-verification, no draft model at all).
• Production users — Anthropic Claude (sub-100ms TTFT), Meta Llama 3 inference stacks (vLLM/TensorRT-LLM), Mistral/DeepSeek vLLM deployments.
• Output is mathematically identical to greedy/sampled target-model generation — speculative decoding is a pure latency optimisation, not a quality trade-off.
• On a managed generative AI hub endpoint, speculative decoding — like the KV cache and the attention kernel — is invisible: it is the hosting vendor's implementation choice, priced into the per-token rate, not a lever SAP customers configure directly.
• A self-hosted SAP AI Core BYOM deployment (vLLM, per SAP's own sample repository) can enable speculative decoding explicitly with a smaller sibling model as drafter — but only when a suitably matched, same-distribution small model is actually available and licensed for the tenant.
• The single biggest practical risk is a mismatched or stale draft model: once the draft and target diverge (commonly after the target is fine-tuned but the draft is not updated), acceptance collapses toward zero and the technique adds latency instead of removing it.

Terms used on this page

Speculative decoding

An inference acceleration technique where a small fast draft model proposes k candidate tokens that a large target model then verifies in a single parallel forward pass, achieving 2-4× wall-clock speedup with identical output distribution to target-only sampling.

Draft model

The small, fast auxiliary model in speculative decoding that proposes candidate tokens; typically 10-100× smaller than the target model and ideally trained on the same data distribution to maximise draft-target agreement.

Target model

The large, expensive model whose output distribution speculative decoding preserves exactly; verifies the draft's k candidate tokens in one parallel forward pass.

Medusa

A speculative-decoding variant (Cai et al., 2024) that adds lightweight prediction heads on top of the target model itself, eliminating the need for a separate draft model and simplifying serving.

Time-to-first-token (TTFT)

The user-facing latency metric for LLM inference — the wall-clock time between the user submitting a prompt and the first generated token arriving. Speculative decoding is one of the few techniques that improves both TTFT and steady-state throughput.

Sources

1. Fast Inference from Transformers via Speculative Decoding (Leviathan et al., Google, 2023)
2. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al., 2024)
3. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (Li et al., 2024)
4. Anthropic engineering — Claude inference architecture commentary
5. arXiv — Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (Fu, Bailis, Stoica & Zhang, ICML 2024)
6. GitHub hao-ai-lab — LookaheadDecoding official implementation
7. GitHub vllm-project — vLLM (speculative decoding / draft-model serving configuration)
8. GitHub NVIDIA — TensorRT-LLM (Medusa and draft-model speculative decoding support)
9. SAP News Center — How SAP and NVIDIA Advance Enterprise AI Transformation (NIM inference optimisation, SAP-ABAP-1, 2026-03-17)
10. GitHub SAP-samples — btp-generative-ai-hub-use-cases: bring-your-own OSS LLM on SAP AI Core (vLLM serving backend)
11. SAP Help Portal — Choose a Resource Plan for training/inference in SAP AI Core (GPU sizing for a self-hosted draft+target deployment)
12. Anthropic — Context windows documentation (Claude model family referenced for production sub-100ms TTFT claims)