Concept card

KV Cache Optimization — PagedAttention, Quantized KV, Speculative Cache

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is KV Cache Optimization — PagedAttention, Quantized KV, Speculative Cache?

At 32K context, Llama 3 70B's KV cache alone runs roughly 20GB per request — several times the model weights themselves at INT8 — making KV-cache optimisation the single highest-leverage lever in LLM inference economics.

What it is

During autoregressive LLM inference, every previously generated token's key and value tensors must be kept in GPU memory so future tokens can attend to them — the KV cache. At long context lengths the KV cache dominates total memory usage: for Llama 3 70B at 32K context the KV cache is ~20 GB per request, several times larger than the model weights themselves at INT8. KV cache optimisation is therefore the single highest-leverage area for LLM inference economics — every technique here directly translates into either more concurrent users on the same GPU or longer context at the same cost.

Why it matters

PagedAttention eliminates memory fragmentation and enables up to 24x higher throughput by safely overlapping multiple sequences — the single biggest inference-serving advance of 2023-2024.
KV quantization to INT8 halves cache memory for under 1% quality loss; INT4 quarters it for 2-3% loss.
Prefix caching reuses a shared prompt prefix's KV state across requests, often delivering a 5-10x throughput win for chatbot workloads with a common system prompt.

Key points

KV cache (per-token K + V tensors held in GPU memory for future attention) dominates long-context memory — Llama 3 70B at 32K context = ~20 GB KV cache per request.
PagedAttention (vLLM, 2023) — page-table-style KV allocation eliminates fragmentation, 24× throughput improvement; now standard in vLLM, TensorRT-LLM, SGLang.
KV quantisation — INT8 halves memory at <1% quality loss, INT4 quarters at ~2-3% loss; standard in modern serving stacks.
MQA (single K/V across all heads) and GQA (grouped heads share K/V, used in Llama 2/3, Mistral, Mixtral) shrink KV cache 4-8× at architecture level.
Prefix / speculative caching — cache shared prompt prefix KV once, reuse across requests; 5-10× throughput win for chatbot workloads with shared system prompts.
KV Cache Optimization — PagedAttention, Quantized KV, Speculative Cache is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

KV cache: The cached key and value tensors of all previously generated tokens, held in GPU memory so future tokens can attend to them during autoregressive inference; dominates memory consumption at long context lengths.
PagedAttention: An inference-serving technique introduced by vLLM (Kwon et al., 2023) that allocates the KV cache in fixed-size blocks like OS virtual-memory pages, eliminating fragmentation and enabling 24× higher throughput than naive contiguous allocation.
Grouped-Query Attention (GQA): An attention architecture variant where groups of query heads share a single set of K/V tensors (e.g. 8 K/V groups for 32 query heads); used in Llama 2/3, Mistral, Mixtral and gives 4-8× KV cache reduction at near-zero quality loss.
Multi-Query Attention (MQA): An extreme attention variant where all query heads share a single K and V tensor; cuts KV cache by H (number of heads, typically 32-128) but with measurable quality loss compared to GQA.
Prefix caching: An inference optimisation that stores the KV cache for a shared prompt prefix once and reuses it across all requests that share that prefix; canonical in chatbot workloads with shared system prompts; 5-10× throughput win typical.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →