Concept card

KV Cache

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is KV Cache?

The KV cache, not model weights, is the binding VRAM constraint on LLM inference — an 8,000-token context on a 70B model already consumes ~42GB, meaning doubling context length roughly doubles hardware cost.

The KV cache — key-value cache — is the memory structure that makes fast, affordable large language model inference possible, and it sits directly behind cost and latency decisions for any SAP Joule or AI Foundation deployment that runs at real user volume. Understanding it is what separates an architect who can explain why a longer context window is expensive from one who can only point at an invoice.

What it is and why it matters

When a transformer model generates a response one token at a time, each new token has to "attend" to every token that came before it — the whole conversation so far, including the system prompt and any retrieved grounding content. Recomputing the key and value projections for that entire history at every single step would make generation slower with every token, an effect that compounds badly on long enterprise prompts stuffed with SAP context. The KV cache avoids that: it stores the key and value projections for every token the moment they are first computed, so generating token number one thousand only requires processing that one new token, not recomputing the previous nine hundred and ninety-nine. This is the single mechanism that makes chat-style, multi-turn generation practical at all.

How it works

Why it matters

Without caching, generating token N would require an O(N²) recomputation explosion — the cache is what makes autoregressive generation at enterprise scale feasible at all.
Prefix/prompt caching is directly relevant to Joule: sessions prefixed with the same enterprise context block can reuse cached KV projections instead of recomputing them every call.
The concrete memory formula (2 × L × H × d × bytes) lets an architect size hardware (single A100-80GB vs. multi-GPU pod) before committing to a context-length requirement.

Key points

KV cache memory = 2 × n_layers × n_kv_heads × d_head × seq_len × batch × dtype_bytes.
PagedAttention (vLLM) raises GPU utilisation from ~40% to ~90% by virtualising the KV cache.
Prefix caching (RadixAttention) reuses KV representations of shared prompt prefixes across calls — critical for SAP Joule system prompts.
INT8 KV quantisation halves cache memory with ~1% quality loss when per-token dynamic quantisation is applied.
H2O eviction reduces cache by 20× for long generation tasks with <5% quality loss by keeping only high-attention tokens.
KV Cache is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

KV cache: Memory buffer storing key and value matrices for all past tokens; enables O(seq_len) per-step generation cost instead of O(seq_len²) full recomputation.
PagedAttention: Virtual memory-inspired KV cache allocation (Kwon 2023, vLLM) that stores cache in non-contiguous pages, eliminating fragmentation and raising GPU utilisation.
Prefix caching: Mechanism (RadixAttention, Zheng 2023) that indexes KV representations of shared prompt prefixes so repeated calls reuse the cached computation.
H2O: Heavy Hitter Oracle (Zhang 2023): KV cache eviction policy that retains the tokens with highest cumulative attention mass, reducing cache size 20× with <5% quality loss.
INT8 KV quantisation: Compression of KV cache entries from BF16 (2 bytes) to INT8 (1 byte), halving memory with ~1% quality loss when per-token dynamic scales are maintained.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →