Concept card

PagedAttention — Memory Management for Long-Context LLMs

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is PagedAttention — Memory Management for Long-Context LLMs?

PagedAttention borrows OS-style virtual-memory paging for the KV-cache, raising effective GPU memory utilization from 20-40% to over 96% — turning an A100 that served under 4 concurrent 32k-context requests into one that serves 30-50.

PagedAttention is the memory-management technique that turned large-scale LLM serving from a memory-bound bottleneck into a solvable engineering problem. Introduced by the vLLM team in 2023, it takes the oldest idea in operating-systems design — paged virtual memory — and applies it to the one resource that dominates GPU memory during inference: the KV cache, the running store of key and value vectors that every transformer layer accumulates as it generates tokens.

What it is and why it matters

Every token an LLM produces appends one key vector and one value vector per attention head, per layer, to a cache that must stay resident in GPU memory for the life of the request. A single 32k-token conversation against a 70-billion-parameter model can occupy several gigabytes of cache on its own. Before PagedAttention, serving engines allocated this cache as one contiguous block sized to the maximum sequence length the request declared at admission — regardless of how many tokens the request would actually emit. A request that asked for a 32k context but generated 200 tokens still reserved the full 32k worth of slots, and none of that space was released until the request finished. Multiply that waste across dozens of concurrent requests of varying length, and 60-80% of nominally "available" GPU memory becomes structurally unusable: present on the accelerator, but fragmented into pieces too small or too misallocated to serve a new request. The practical consequence was blunt — engines refused new work while sitting on gigabytes of idle memory.

Why it matters

Pre-vLLM engines reserved a full contiguous block sized to the maximum sequence length at request admission, wasting 60-80% of GPU memory to fragmentation with 30-50 concurrent requests of varying length.
Pages (fixed-size KV-cache blocks) map to physical memory via a per-request block table, allocated on demand and freed individually when a request finishes — exactly like an OS's virtual-to-physical page mapping.
Pages with identical token content can be shared between requests, which is the foundation for prefix-caching and SGLang's RadixAttention.

Key points

Applies OS-style virtual-memory paging to the KV-cache (C160) — fixed-size blocks of e.g. 16 token-slots mapped via a per-request block table; frees pages individually as requests complete.
Replaces contiguous allocation that reserved max-sequence space upfront (wasting 60-80% of GPU memory) with on-demand allocation; on an A100 80GB serving 70B + 32k context, naive allocation supports <4 concurrent requests vs 15-20 under PagedAttention.
Lifts effective GPU memory utilisation from 20-40% to 96%+ on diverse-length concurrent workloads — the headline result from the SOSP 2023 paper.
Enables prefix-caching and RadixAttention (SGLang) by allowing pages with identical content to be shared between requests; a shared 2k-token system prompt across 50 concurrent requests saves 50 × 2k × (KV tensor size per token) of GPU memory.
Per-step kernel overhead 5-15% (non-contiguous K/V gather vs contiguous) in exchange for 3-5x more concurrent requests on the same hardware — the core trade-off.
Foundation that makes continuous batching (C245) viable on long-context and agentic workloads — without paged allocation, continuous batching cannot admit new requests while long-context requests hold contiguous memory.
Implemented in: vLLM (original implementation, open-source) · NVIDIA TensorRT-LLM (production-grade, GPU-optimised) · SGLang (extends with RadixAttention) · LMDeploy (Huawei, mobile + edge).
SAP AI Core uses vLLM-compatible paged-allocation engines for its Generative AI Hub model serving; capacity planning for concurrent Joule or SAP AI agent sessions must account for the per-session KV-cache page budget.
Prefix-caching practical impact: on a Joule deployment where all requests share a 2k-token SAP context prompt, prefix-caching reduces TTFT (time-to-first-token) by 40-60% and KV-cache memory by 30-50% — a material cost reduction at scale.
Failure mode: page eviction — when GPU memory is exhausted even with paged allocation (very long contexts + high concurrency), the engine evicts cold pages to CPU RAM (swap); eviction adds 100-500ms latency spikes visible as p99 latency degradation; the signal is high swap-rate in vLLM metrics.

Terms used on this page

PagedAttention: Memory-management algorithm that divides the LLM KV-cache into fixed-size pages mapped via a per-request block table, mirroring OS virtual-memory paging; introduced in the vLLM paper (Kwon et al., SOSP 2023); the primary technique enabling 3-5x GPU throughput improvement on diverse-length concurrent LLM workloads.
KV-cache: The key and value tensors of every token in an LLM's context window, cached in GPU memory to avoid recomputing multi-head attention over prior tokens; can occupy 1-10 GB per long-context conversation; the primary GPU memory bottleneck in LLM serving.
Block table: Per-request data structure mapping logical token positions to the physical pages holding their K/V vectors; analogous to a page table in an OS virtual-memory system; the mechanism that decouples logical sequence length from physical memory layout.
Prefix caching: Reusing the same KV-cache pages for the shared prefix of multiple requests (e.g. a common system prompt or few-shot exemplars); enabled by PagedAttention's page-sharing mechanism; reduces TTFT by 40-60% and KV-cache memory by 30-50% on Joule deployments where all requests share an SAP context prompt.
RadixAttention: Extension of PagedAttention's page-sharing to arbitrary shared prefixes via a radix tree, implemented in SGLang (Zheng et al., 2023); achieves 20-40% additional memory saving over basic prefix-caching by sharing non-prefix substrings as well.
Page eviction: The fallback when GPU memory is exhausted even with paged allocation: cold pages (those not recently accessed) are written to CPU RAM and reloaded on demand; adds 100-500ms latency spikes visible as p99 degradation; the key failure mode to monitor via vLLM's swap_rate metric.
TTFT (Time to First Token): The latency from request submission to the first token appearing in the response; the user-experience metric most sensitive to KV-cache memory pressure and prefix-caching; target for interactive Joule sessions is <500ms TTFT.
Continuous batching: The LLM serving technique (C245) that admits new requests mid-generation rather than waiting for the current batch to complete; only viable with PagedAttention, because naive contiguous allocation holds memory until a full sequence completes.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →