Concept card

FlashAttention 2 + 3 — Memory-Aware Attention Kernels

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is FlashAttention 2 + 3 — Memory-Aware Attention Kernels?

FlashAttention never materialises the full n-by-n attention matrix in slow HBM memory — it tiles Q/K/V to fit in fast SRAM instead, closing a 3-10x GPU-hour gap at long context versus a naive attention kernel, with mathematically identical output.

What it is

FlashAttention is not a new way of computing attention mathematically — it produces the exact same numbers as standard scaled dot-product attention — it is a different way of moving data while computing it, and that distinction is the entire point. FlashAttention 1 (2022) established the pattern; FlashAttention 2 (2023) restructured the parallelism for roughly a 2x speedup; FlashAttention 3 (2024) is tuned specifically for the Hopper generation of GPUs (H100, H200), using asynchronous memory copy hardware and FP8 arithmetic to reach a high fraction of the chip's theoretical throughput. For an SAP consultant, the relevance is concrete: this is one of the largest, most invisible levers on the cost per token of any self-hosted or BTP AI Core-backed generative AI deployment.

The problem it solves is a memory-hierarchy mismatch, not a compute shortage. A GPU has a small amount of very fast on-chip memory (SRAM) and a much larger pool of slower memory (HBM). Standard attention implementations compute the full n-by-n matrix of token-to-token attention scores and write that entire matrix out to HBM before reading it back in for the softmax and the weighted sum — at long sequence length this traffic between HBM and the compute cores becomes the bottleneck, and the GPU's arithmetic units sit idle waiting for data to arrive. This is why attention at long context is described as memory-bound rather than compute-bound: the chip has FLOPs to spare, it just cannot feed them fast enough.

Why it matters

The math is exact, not an approximation — FlashAttention changes only how attention is computed on the GPU, never what it computes.
FlashAttention 3 exploits Hopper-specific hardware (TMA, warp specialisation, FP8) to hit up to 75% of the H100's theoretical FP16 throughput.
The gap between a FlashAttention-enabled deployment (the default in vLLM, TensorRT-LLM, PyTorch SDPA) and a naive kernel is 3-10x in GPU-hours at long context — a direct line item in any BTP AI Core cost estimate.

Key points

Mathematically exact reimplementation of scaled dot-product attention — same output as naive softmax(QK^T/√d), dramatically less HBM memory traffic.
Core mechanism: block-tiles Q/K/V into SRAM-resident chunks (64-128 rows) and uses an online (streaming) softmax — never materialises the n×n attention matrix in HBM.
FlashAttention 2 (Dao 2023): ~2× speedup over v1 via better work partitioning across CUDA thread blocks and improved backward-pass parallelism.
FlashAttention 3 (Shah et al. 2024): Hopper-specific kernels using TMA async tile copies + warp-specialised producer/consumer pipelines + optional FP8; achieves up to ~75% of H100 FP16 theoretical peak.
Default in PyTorch SDPA (torch.nn.functional.scaled_dot_product_attention), vLLM, TensorRT-LLM, llama.cpp, and Hugging Face Transformers — opt-in required only in custom training code.
Cost impact on BTP AI Core: FlashAttention-enabled vs naive attention at context length 4 k+ = 3–10× fewer GPU-hours for equivalent query throughput; direct multiplier on cost-per-token.
Memory saving enables longer context at the same GPU RAM: a 7B model serving 4 k context with naive attention uses ~2× more activation memory than with FlashAttention — enabling larger batch sizes or longer contexts on the same GPU.
SRAM block size determines tile throughput: block sizes tuned to fit the 256 KB/SM SRAM on H100; misaligned block sizes (e.g. from mismatched head-dim) degrade performance to near-naive levels.
FP8 (FlashAttention 3 only): halves arithmetic bandwidth vs FP16 with < 1% accuracy loss on most tasks when combined with absmax scaling; not yet default in all frameworks as of 2025.
For SAP analytics consultants: when the client asks 'why is our BTP AI Core invoice so high?', the first diagnostic is confirming that FlashAttention is active and context lengths are not inadvertently exceeding the efficient range for the deployed model.

Terms used on this page

HBM (High-Bandwidth Memory): The off-chip GPU memory holding model weights, activations and KV cache. ~3 TB/s on H100 — fast vs CPU RAM but the bottleneck vs on-chip SRAM. Standard attention makes 5–10 full HBM reads/writes per forward pass; FlashAttention reduces this to ~1.
SRAM (on-chip shared memory): The fast on-chip GPU cache (shared memory + L1), ~20 TB/s effective bandwidth, but only ~256 KB per streaming multiprocessor on H100. FlashAttention's block sizes are tuned to fit SRAM — the entire value of the algorithm depends on this fit.
Online softmax: A numerically-stable streaming algorithm that computes softmax incrementally across blocks of input without ever materialising the full normalisation denominator. Uses a running maximum and a correction factor applied at each tile boundary. The key mathematical trick enabling FlashAttention's HBM avoidance.
TMA (Tensor Memory Accelerator): A Hopper-specific hardware unit that asynchronously copies large tensor blocks between HBM and SRAM, freeing the streaming multiprocessors to compute during the transfer. FlashAttention 3's primary Hopper-specific innovation — enables the overlap of memory and compute that achieves 75% H100 utilisation.
Warp specialisation: A Hopper programming pattern where different warps within a thread block are assigned permanently to either data-loading (producer) or computation (consumer) roles, enabling sustained overlap of TMA transfers and matrix multiplications. Used by FlashAttention 3.
IO-aware algorithm: An algorithm designed to minimise reads and writes to the slow memory tier (HBM), treating memory access as the primary cost to optimise rather than floating-point operation count. FlashAttention is the canonical IO-aware implementation of attention.
SDPA (Scaled Dot-Product Attention): PyTorch's built-in attention function (torch.nn.functional.scaled_dot_product_attention), which automatically dispatches to FlashAttention, memory-efficient attention, or naive math depending on hardware and input shape. The correct entry point for any custom model using PyTorch — calling raw softmax(QK^T) bypasses this dispatch.
KV cache: The cache of Key and Value tensors from previous tokens, used by autoregressive models during inference to avoid recomputing attention over the full context. FlashAttention's memory efficiency directly enables larger KV caches (= longer serving context) at the same GPU RAM budget.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →