Sparse Attention Patterns — Longformer / BigBird / Mixtral Style

What it is

Sparse attention patterns exist to solve one specific arithmetic problem: standard self-attention costs O(n²) in sequence length n, because every token attends to every other token. At 100,000 tokens the attention matrix alone needs roughly 40 GB of FP16 memory, and that cost dominates both compute time and latency well before the rest of the model becomes the bottleneck. Sparse attention is the family of techniques that gets the cost down to roughly O(n·k) or O(n·log n) by restricting each token to a structured subset of k positions rather than the full sequence — and it is the specific architectural reason long-context models like 200K-token Claude, 1M-token Gemini, and 128K-token GPT-4 Turbo are economically viable at all, rather than theoretical curiosities.

Why it matters

• Mistral 7B's w=4096 sliding window across 32 layers yields an effective 131K-token receptive field despite an 8K nominal window.
• BigBird's global-local pattern adds a small set of global tokens that attend to everything, mitigating the core weakness of pure locality-based sparsity.
• Sparse attention assumes locality dominates relevance, which breaks on true global-reasoning tasks (a fact at position 5K needing to inform generation at position 95K) unless global tokens or hybrid dense-sparse stacks compensate.

Key points

• Vanilla self-attention is O(n²) — at 100K tokens dense attention alone needs ~40 GB FP16; sparse drops to O(n·k) or O(n·log n).
• Sliding-window (Longformer, Mistral 7B with w=4096) — receptive field grows linearly with depth; 32 layers × 4096 = 131K effective receptive at 8K nominal window.
• Global-local (BigBird) — sliding window + global tokens (CLS, anchors) + random sparse links preserves theoretical expressiveness.
• Block-sparse / dilated — partition sequence into blocks, dense within, selective across; underpins Mixtral 8x7B attention layers.
• Trade-off — assumes locality dominates; multi-document global reasoning needs global tokens or hybrid dense-sparse stacks (some Claude variants reportedly hybrid).
• For a hosted generative AI hub call, the specific sparse-attention pattern underneath Claude, GPT, Gemini or Mistral is neither configurable nor disclosed — SAP's governed alternative (document grounding, SAP Knowledge Graph) is what a consultant actually controls.
• Naming the specific pattern matters when comparing self-hosted candidates on SAP AI Core: 'uses sparse attention' predicts nothing on its own, while sliding-window vs global-local vs block-sparse each implies a different cost-versus-recall trade-off.

Terms used on this page

Sliding-window attention

Each token attends to w neighbours on each side; cost O(n·w); receptive field grows linearly with depth. Used in Longformer (2020) and Mistral 7B (2023, w=4096).

Global tokens

A small set of special tokens that attend to every position and that every position attends to; used in BigBird to preserve global information flow despite local-window attention.

Receptive field

The set of input positions that can influence a given output position; in sliding-window attention it grows by w per layer, so L layers × w window = L·w effective receptive field.

Block-sparse attention

Attention pattern where the sequence is partitioned into blocks; attention is dense within each block and selective (or none) across blocks; underpins efficient long-context inference in Mixtral and similar models.

Dense (full) attention

The baseline O(n²) pattern where every token attends to every other token; the reference point every sparse pattern trades against, and still the right default under roughly 8K tokens, where the quadratic cost is not yet painful.

Attention-pattern audit

The practice of asking a vendor or checking a model card for the exact sparse-attention pattern and its layer-by-layer composition — which layers are sliding-window, global-local or full — before trusting a context-length claim for a specific SAP use case.

Sources

1. Longformer: The Long-Document Transformer (Beltagy et al., 2020)
2. Big Bird: Transformers for Longer Sequences (Zaheer et al., 2020)
3. Mistral 7B technical report (sliding-window attention)
4. Anthropic Claude 200K context — engineering blog
5. SAP Generative AI — official product page
6. arXiv — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022)
7. arXiv — FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (Dao, 2023)
8. Mistral AI — Announcing Mistral 7B (2023): confirms sliding window attention w=4096
9. Mistral AI — Mixtral of Experts announcement (2023)
10. SAP Help Portal — Generative AI hub in SAP AI Core (hosted model endpoints)
11. SAP Help Portal — Orchestration service (document grounding pipeline module)
12. SAP Community — Why SAP needs a Knowledge Graph: giving enterprise AI a map of the business (2026)
13. Databricks — Model Serving documentation (self-hosting open-weight checkpoints)