Concept card

Positional Encoding

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Positional Encoding?

RoPE encodes relative position directly into the attention dot product, which is why θ_base alone — 10,000 in the original Transformer vs. 500,000 in Llama 3.3 — is the single parameter separating 4K coherence from 128K coherence.

What it is

Positional encoding is the mechanism that tells a transformer where each token sits in a sequence, and without it the model has no way to distinguish "the client escalated after the invoice" from "the invoice escalated after the client" — the attention computation itself is a set operation, blind to order unless order is injected some other way. This matters directly for anything built on top of a large language model inside an SAP-adjacent workflow: a Joule reasoning chain that needs to track that a payment term applies after a contract amendment, not before it, depends entirely on the positional-encoding scheme working correctly at the context lengths the deployment actually uses.

Why it matters

Without positional encoding, attention is permutation-invariant — 'Q2 revenue decreased 15%' and 'revenue decreased Q2 15%' would be indistinguishable inside a Joule reasoning chain, a concrete enterprise risk.
Sinusoidal and learned-absolute encodings both fail to generalise past training length — ruling them out for any long-context enterprise deployment.
ALiBi trades near-zero inference overhead and robust length generalisation for a ~1-2% MMLU cost versus RoPE at 7B scale — a real, quantified trade-off to weigh.

Key points

RoPE encodes relative position in the Q·Kᵀ dot product — the model sees position-i attending to position-j via their angular difference, not absolute indices. This is why RoPE models generalise to longer sequences: they never learn absolute position lookup tables.
θ_base controls effective context length: θ=10,000 (Llama 1/2, original) → effective ~4K; θ=500,000 (Llama 3.3) → effective 128K. Always confirm θ_base before accepting a vendor's stated context length — it is the ground truth, not the marketing figure.
ALiBi adds a fixed negative linear bias per attention head, scaled by a geometric sequence of slopes — no parameters, no memory overhead, generalises to arbitrary lengths. The ~1–2% MMLU penalty at 7B scale makes it the correct choice when memory is the binding constraint or when document length is truly unbounded.
YaRN achieves context extension without full retraining via NTK-aware interpolation: it rescales the RoPE frequency components so the model experiences the extended sequence as a compressed version of the training distribution. Llama 3.1 extended 128K→1M tokens with <5% perplexity increase.
Sinusoidal and learnable absolute encodings both fail to generalise beyond training length — they are the baseline from 'Attention Is All You Need' (Vaswani 2017) and should not be selected for any SAP enterprise deployment targeting documents longer than the training maximum.
The production standard in 2025–2026 is RoPE with high θ_base — every major open-source foundation model uses it: Llama 3 (θ=500,000), Mistral (θ=1,000,000), Gemma 2 (θ=10,000 with sliding window), Falcon 3 (RoPE). ALiBi is used by MPT and early BLOOM derivatives.
For SAP BDC custom agent fine-tuning: if the base model uses RoPE and the fine-tuning dataset contains sequences longer than the base model's training context, apply YaRN before fine-tuning — fine-tuning on out-of-range sequences without YaRN degrades positional coherence across the full length.
The practical consequence of 'lost in the middle' (C155) and positional encoding interact: even with a properly-configured RoPE model at 128K, items at the midpoint receive less attention. Position-aware prompt structuring (critical content at edges) is a complementary mitigation that does not require changing the positional encoding scheme.
Multi-head attention with RoPE applies a different rotation frequency to each head's dimension pairs — lower-dimensional pairs rotate faster (higher frequency, captures short-range dependencies); higher-dimensional pairs rotate slowly (lower frequency, captures long-range dependencies). This is why RoPE naturally handles both local and global context within a single attention layer.
Context extension at inference time via position interpolation (the simpler approach before YaRN) linearly maps positions beyond training length into the training range — it works for modest extensions (2×) but degrades for larger extensions (8×) because interpolated positions collide with training-range semantics, which YaRN's NTK approach avoids.

Terms used on this page

RoPE (Rotary Position Embedding): Su et al. 2021: applies a position-dependent rotation matrix to Q and K vectors in each attention head, encoding relative position directly in the dot product. The current de facto standard for long-context transformer models.
θ_base: Base frequency parameter in RoPE controlling the rotation speed; higher θ_base (500,000 in Llama 3.3) enables longer effective context before positional information wraps or becomes ambiguous. The ground-truth metric for evaluating a model's effective context length.
ALiBi (Attention with Linear Biases): Press et al. 2022: adds a fixed negative linear penalty to each attention logit proportional to the query-key token distance, scaled per-head by a geometric sequence. Zero parameters, zero memory overhead, generalises to arbitrary lengths — best for unbounded-length streaming workloads.
YaRN (Yet another RoPE extensioN): Peng et al. 2023: NTK-aware fine-tuning technique that rescales RoPE frequency components to extend a trained model's effective context window beyond its training length with less than 5% perplexity increase.
NTK-aware interpolation: The interpolation strategy in YaRN that rescales RoPE frequencies using Neural Tangent Kernel theory to avoid the position-collision artefact of naive linear interpolation, enabling reliable 8–16× context extension.
Length generalisation: The ability of a positional encoding scheme to maintain coherent attention on sequences longer than those seen during training. RoPE with high θ_base and ALiBi both generalise well; sinusoidal and learnable absolute encodings do not.
Sinusoidal encoding: Vaswani et al. 2017: adds fixed sinusoidal waves of different frequencies to token embeddings before the first transformer layer. The original baseline encoding — mathematically elegant but empirically fails to generalise beyond training length. Do not use for enterprise long-context workloads.
Learnable absolute embedding: A position-indexed lookup table trained alongside the model — each position index maps to a learned embedding vector. Fails to generalise to positions beyond the training maximum because those indices were never in the training distribution.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →