Concept card

Context Window Mechanics

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Context Window Mechanics?

Doubling context length quadruples attention compute (O(n²)) and can push a single 70B-model request's KV cache to ~42GB at 128K tokens — before model weights — making long context a hardware-budget decision, not a free setting.

What it is

The context window is the maximum number of tokens an LLM can attend to simultaneously in a single forward pass. It governs what information the model can 'see' when generating each output token and determines whether a workload can be handled in a single call or requires chunking strategies such as RAG or sliding-window overlap.

The problem it addresses is that useful enterprise documents — SAP ABAP class files, procurement contracts, annual reports — routinely exceed 8,000 tokens, so naive truncation loses critical context and naive concatenation exceeds the model's capacity. For SAP analytics consultants, this matters directly: a Joule agent reasoning over a complete plant maintenance history (potentially 200+ pages of service orders) or a BDC pipeline ingesting full SAP HANA tables via Datasphere must handle volumes that strain even 128K-token models.

Why it matters

FlashAttention-2 is what makes 128K inference practical at all — it caps activation memory at O(n) instead of O(n²) via tiled SRAM computation.
'Lost in the middle' retrieval accuracy drops from ~95% at context edges to ~60% mid-context at 100K tokens — long windows don't guarantee uniform attention quality, a real risk for document-heavy Joule use cases.
Directly relevant to SAP: full plant-maintenance histories (200+ pages) or full HANA table ingestion via Datasphere routinely strain even 128K-token models.

Key points

KV cache memory formula: 2 × n_layers × n_kv_heads × d_head × seq_len × batch × dtype_bytes — every input field is a variable an architect must size for the target model and sequence length.
Llama 3.3 70B at 128K tokens, batch=1, BF16: ~42 GB KV cache alone, requiring at least two H100 80 GB GPUs before accounting for the 140 GB model weights.
'Lost in the middle' (Liu et al. 2023): retrieval accuracy falls from ~95% at context edges to ~60% for items at the midpoint of a 100K-token context — position your most critical facts at the beginning or end of the prompt.
FlashAttention-2 reduces activation memory from O(n²) to O(n) by tiling attention computation in SRAM — the enabling technology for practical 128K inference on current hardware.
YaRN extends a trained RoPE model from its training context to a larger inference context (Llama 3.1: 128K → 1M) with <5% perplexity increase by rescaling the RoPE θ values through NTK interpolation.
RoPE θ_base controls effective context length: θ=10,000 (original) → ~4K effective; θ=500,000 (Llama 3.3) → 128K effective — when evaluating a model's stated context length, confirm the θ_base used in training.
For SAP BDC: the hybrid RAG + long-context pattern is the standard architecture — RAG narrows the candidate set to under 32K tokens, long-context model reasons over the reduced set for holistic coherence.
SAP Joule uses a long-context model hosted via SAP AI Core; the effective context available to a Joule agent prompt is reduced by the system prompt, tool call overhead, and conversation history — budget 30–40% of the stated context window for non-user content.
Chunking strategies for documents exceeding the context window: fixed-size overlap (simplest, loses cross-chunk entities), semantic chunking (splits at paragraph/section boundaries), hierarchical chunking (summary at top level, detail on retrieval) — the right strategy depends on whether the document has clear semantic sections.
Cost implication: long-context inference is significantly more expensive than short-context — 128K-token calls typically cost 8–16× more per call than 8K-token calls at the same model tier; design workflows to avoid unnecessary long-context calls.

Terms used on this page

Context window: Maximum tokens an LLM processes in one forward pass; determines the information horizon for generation and sets the architecture choice between single-call and chunking strategies.
KV cache: Stored key and value matrices from all prior sequence positions, enabling O(1) incremental token generation without recomputing past attention — the dominant memory cost in long-context inference.
FlashAttention-2: IO-aware attention algorithm (Dao 2023) that tiles computation to stay in SRAM, reducing HBM reads and activation memory from O(n²) to O(n); the enabling technology for practical 128K inference on A100/H100.
Lost in the middle: Empirical finding (Liu et al. 2023) that LLM retrieval accuracy drops from ~95% at the edges of a long context to ~60% at the midpoint — critical information should be placed at the beginning or end of the prompt.
YaRN: Yet another RoPE extensioN (Peng 2023): NTK-scaled fine-tuning that extends a trained RoPE model's effective context window beyond its training length with <5% perplexity increase.
RoPE θ_base: Base frequency parameter in Rotary Position Embedding controlling the rotation speed of Q/K vectors; higher θ_base (500,000 in Llama 3.3) enables longer effective context before positional information wraps around.
Semantic chunking: Chunking strategy that splits documents at natural semantic boundaries (paragraphs, sections, headings) rather than fixed token counts — preserves cross-sentence coherence at the cost of variable chunk sizes.
Hierarchical chunking: Chunking strategy that stores document summaries at a top level and detailed passages at a child level; retrieval first finds the relevant summary, then fetches the associated detail passage — reduces unnecessary long-context calls.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →