Concept card

Multi-head Attention vs Grouped-Query Attention vs Multi-query Attention

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Multi-head Attention vs Grouped-Query Attention vs Multi-query Attention?

Switching from MHA to GQA-8 at equal model size cuts KV-cache memory roughly 8x for under 1% MMLU drop, doubling or tripling how many concurrent users one H100 can serve — the real reason Mistral and Llama 3 serve cheaper than equivalent MHA models.

What it is

Multi-head Attention (MHA), Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) are three points on the same trade-off curve: how many distinct Key/Value projections does each layer compute, given a fixed number of Query heads. The choice does not change what a model can express; it changes how much KV-cache memory inference burns per token, and therefore the per-request cost and the maximum concurrency a given GPU can serve.

Why it exists. The original Transformer (2017) used MHA — every Query head has its own Key and Value head. That doubles representational diversity but also doubles KV-cache memory at inference, which is the dominant memory cost at long context (companion C160). MQA (Shazeer 2019) collapses all heads to a single shared K/V — minimal KV memory, but a measurable quality drop at scale. GQA (Ainslie et al. 2023) is the practical compromise that every frontier model since Llama 2 has adopted.

Why it matters

MHA (Llama 1 70B: 64 Q, 64 KV) maximises representational diversity but doubles KV-cache memory versus alternatives; MQA collapses to one shared K/V head with a measurable quality drop at scale.
GQA (Llama 3 70B: 64 Q, 8 KV) is the practical compromise every frontier model since Llama 2 has adopted.
If you self-host or fine-tune via SAP AI Core, this single architectural choice controls sustainable concurrency per GPU — invisible to a hosted-API consumer, but essential to know why some models cost less to serve.

Key points

Three points on one curve — MHA (n_kv = n_q), GQA (n_kv = n_q / g), MQA (n_kv = 1) — trading KV memory for quality.
GQA-8 (8 Q heads per KV pair) is the default for Llama 3/4, Mistral 7B/Large, Mixtral 8x7B, Gemma 2 — ~8× KV memory reduction vs MHA, <1% MMLU loss.
MQA (n_kv = 1) maximises memory savings but degrades multi-task quality; rarely used in frontier models post-2023.
KV-cache memory at inference = 2 · n_layers · n_kv · d_head · seq_len · batch · bytes — n_kv is the dominant lever.
Quality preservation requires GQA training from scratch or careful 'uptraining' from an MHA checkpoint (Ainslie 2023).
Multi-head Attention vs Grouped-Query Attention vs Multi-query Attention is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

n_q / n_kv: The number of Query heads (n_q) and Key/Value heads (n_kv) in a Transformer layer. MHA has n_kv = n_q; GQA has n_kv < n_q; MQA has n_kv = 1.
KV cache: The per-request memory store of computed Key and Value tensors for all past tokens, reused on each generation step to avoid recomputation. Dominates GPU memory at long context.
Uptraining: The procedure (Ainslie et al. 2023) for converting an existing MHA checkpoint into a GQA model with a small amount of extra training, preserving most original quality at a fraction of from-scratch cost.
Concurrency: The number of simultaneous user sessions a fixed GPU pool can serve at target latency. Inversely proportional to KV-cache memory per session — the key business KPI GQA improves.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →