Concept card

Attention Variants

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Attention Variants?

The KV cache, not model weights, is the binding memory constraint in production LLM serving — vanilla multi-head attention pushes a single 70B-model request to ~336GB at 128K tokens, which is why GQA (8× reduction) is now the Pareto-optimal default.

What it is

Attention variants are the family of engineering modifications made to the original multi-head self-attention mechanism inside a transformer, and the choice among them is arguably the single biggest lever a team pulls when it wants to control the cost of running a large language model at scale. Vanilla multi-head attention, the mechanism introduced in the original transformer paper, gives every attention head its own independent set of query, key and value projections. That design maximises the model's representational richness, but it also means that during autoregressive generation the model must cache a key and value vector for every head, at every layer, for every token already generated — the so-called key-value cache. For a large model with many heads, a long context window and a reasonably sized batch of concurrent requests, this cache — not the model's weights — becomes the binding constraint on how much hardware a deployment needs and how many users it can serve at once.

Why it matters

Attention-variant choice is named as the single largest lever on inference cost — bigger than quantisation, batching, or hardware tier in most deployments.
GQA's 8× KV reduction with quality 'competitive with MHA' is the concrete number that justifies why Llama 3.3 70B uses H_kv=8 instead of full MHA.
PagedAttention eliminates up to 60% memory fragmentation, lifting GPU utilisation well above the ~40% typical of contiguous allocation — a direct cost lever for self-hosted inference.

Key points

KV cache (not model weights) is the binding GPU memory constraint in LLM serving — MHA at 128K context requires ~336 GB for Llama 3.3 70B; GQA reduces this 8× to ~42 GB.
GQA (H_q=64, H_kv=8 on Llama 3.3 70B) is the Pareto optimum: 8× KV reduction with quality competitive to MHA — the industry default since 2023 (Llama 3, Mistral, Gemma 2, Falcon 180B).
MQA (H_kv=1) maximises memory savings but degrades quality on multi-step reasoning — use only for ultra-low-latency edge deployments where the regression is measured and accepted.
FlashAttention-3 achieves 75% H100 FLOP utilisation vs 35% for naive SDPA — an orthogonal kernel optimisation with zero quality trade-off, applicable on top of any structural variant.
PagedAttention (vLLM, Kwon 2023) virtualises the KV cache, eliminating up to 60% memory fragmentation and lifting GPU utilisation from ~40% to ~90% in multi-user serving.
PyTorch SDPA since 2.0 automatically dispatches to FlashAttention when inputs meet the CUDA kernel requirements — no explicit opt-in needed for PyTorch-based inference stacks.
SAP BTP AI Core uses vLLM-compatible serving infrastructure for open-weight model deployments — FlashAttention and PagedAttention benefits apply directly when self-hosting on Kyma.
For a 70B model serving 50 concurrent users at 4K context on 4× A100 80GB: GQA + FlashAttention + vLLM (PagedAttention) is the only configuration that fits in memory with acceptable throughput.
Multi-head Latent Attention (MLA, DeepSeek-V2 2024) further compresses KV via low-rank projection — emerging variant, not yet mainstream but directionally relevant for very long context scenarios.
Hardware utilisation difference (FlashAttention: 75% vs naive: 35%) translates directly to ~2× throughput on the same GPU — for a production inference cluster, this is the single highest-ROI software optimisation available.

Terms used on this page

MHA (Multi-Head Attention): Original attention mechanism (Vaswani 2017): H independent Q, K, V projection sets. Maximum expressivity; maximum KV cache memory footprint at long contexts.
GQA (Grouped-Query Attention): Attention variant (Ainslie 2023): H_q query heads share G groups of K/V projections where H_kv = G < H_q. Current Pareto optimum — 8× KV reduction with quality competitive to MHA. Default in Llama 3, Mistral, Gemma 2.
MQA (Multi-Query Attention): Extreme KV compression (Shazeer 2019): H_kv=1 — all query heads share a single K/V projection. H× memory saving; measurable quality regression on multi-step reasoning. Used in PaLM; superseded by GQA in most modern models.
FlashAttention: IO-aware attention implementation (Dao 2022–2024): tiles Q/K/V computation within SRAM to avoid repeated HBM reads. Achieves O(n) activation memory vs O(n²), 75% H100 FLOP utilisation vs 35% naive. Orthogonal to structural variant choice.
KV cache: The cached key-value projection tensors from all previous attention steps in autoregressive decoding. Memory scales as O(n_layers × H_kv × d_head × seq_len × batch_size) — the binding GPU memory constraint in LLM serving, not the model weights.
PagedAttention: Virtual memory-inspired KV cache management (Kwon 2023, vLLM): allocates KV cache in non-contiguous pages using OS-style page tables. Eliminates up to 60% memory fragmentation; lifts GPU utilisation from ~40% to ~90% in multi-user serving.
HBM (High-Bandwidth Memory): The primary GPU memory (e.g. 80 GB on A100/H100): slower than SRAM but much larger. FlashAttention's key insight is to minimise HBM reads by keeping intermediate attention computations in the faster on-chip SRAM.
SRAM (on-chip cache): The fast on-chip memory of a GPU (e.g. ~20 MB on A100, ~256 KB L1 per SM): orders of magnitude faster than HBM but far smaller. FlashAttention tiles attention computation to fit within SRAM, dramatically reducing memory bandwidth cost.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →