Transformer Architecture for SAP Practitioners

What the Transformer actually is, and why every senior SAP analytics consultant needs a working mental model of it, comes down to one fact: it is the neural architecture underneath every large language model in SAP's AI stack—Joule, the models hosted on SAP AI Core, the BTP Generative AI Hub, and every third-party model SAP routes through its foundation model hub. You do not need to derive the mathematics from scratch, but you do need to understand what the architecture makes possible, what it makes expensive, and where it breaks—because those three facts drive real decisions about Joule sizing, prompt design, and hallucination triage.

The problem it solves

Before Transformers, sequence models were recurrent: a network read a sentence token by token, carrying a hidden state forward. Recurrence has two structural weaknesses. It cannot be parallelised during training, because token N's computation depends on token N-1's output—so training time scales with sequence length. And it forgets: long-range dependencies decay after roughly 500 tokens, so a fact stated early in a document is effectively invisible by the time the model reaches the end. The Transformer removes recurrence entirely. Every token attends to every other token in the sequence simultaneously through self-attention, which means training parallelises across the full sequence and dependency capture is, in principle, unbounded—limited only by the context window you can afford to compute.

Why it matters

• Self-attention (Q/K/V, softmax(QK^T/sqrt(d))) lets any token inform any other regardless of distance, unlike RNNs which lose dependencies past roughly 512 tokens.
• Factual knowledge lives in the feed-forward sublayer, not the attention weights — a distinction that matters when deciding fine-tuning vs. RAG.
• Joule's 128K-token context window means every token in it burns KV-cache memory linearly, and facts placed mid-prompt get attended to less reliably past 32K tokens ("lost in the middle").

Key points

• Self-attention: Q·K score determines how much each token attends to every other; output is a weighted sum of V vectors. Multi-head = h parallel attention mechanisms.
• Feed-forward sublayer stores factual knowledge; attention heads specialise in syntactic and semantic relationship types.
• RoPE positional encoding (LLaMA/Mistral) preserves relative position information at long context lengths better than absolute sinusoidal methods.
• Context window = max tokens in one forward pass; Joule models on BTP Generative AI Hub support 128K tokens; KV cache memory scales linearly with context length.
• 'Lost in the middle' effect: facts placed in the middle of a > 32K-token prompt are attended to less reliably — design RAG retrieval to place key facts at prompt boundaries.
• Multi-head attention (e.g. h=96 for GPT-4-class) enables the model to capture different relationship types simultaneously across the same context.

Terms used on this page

Self-attention

Mechanism where each token computes a weighted representation of all other tokens using Query, Key, Value projections; score = softmax(QKᵀ/√d_k).

Multi-head attention

h parallel self-attention heads each specialising in different relationship types; outputs concatenated and linearly projected.

RoPE

Rotary Position Embedding — encodes relative token position by applying rotation matrices to Q and K, generalising better to long contexts than absolute methods.

Context window

Maximum number of tokens the model attends over in one forward pass; determines maximum document length processable in one Joule call.

Feed-forward network (FFN)

The per-token two-layer MLP sub-layer following attention in each transformer block; empirically the primary store of a model's factual/parametric knowledge, which is why RAG (retrieving facts at query time) generally outperforms fine-tuning (rewriting FFN weights) for injecting fast-changing enterprise facts.

KV cache

The cached Key and Value projection tensors from every previous token in a generation; the actual GPU-memory constraint behind a self-hosted deployment's cost and concurrency limits — scales linearly with context length held open.

Lost in the middle

The documented degradation in attention reliability for facts placed in the middle of a long prompt (roughly beyond 32K tokens) versus facts placed at the very start or end — a retrieval-ordering problem, not a bug to patch in the model.

Document grounding

SAP's orchestration-service mechanism (in the generative AI hub) that retrieves and injects only the passages relevant to a query before the LLM call, rather than relying on the model's raw context window — the production answer to lost-in-the-middle and to KV-cache cost growth.

Sources

1. Vaswani et al. — Attention Is All You Need (2017)
2. SAP BTP Generative AI Hub documentation
3. Su et al. — RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)
4. Liu et al. — Lost in the Middle: How Language Models Use Long Contexts (2023)
5. SAP Business AI — official product page
6. SAP Joule (work companion) — official product page
7. SAP Generative AI — official product page
8. SAP Help Portal — Generative AI Hub overview, SAP AI Core (2026)
9. Anthropic — Claude context windows documentation (2026)
10. Jiang et al. — Mistral 7B technical report, arXiv:2310.06825 (2023)
11. SAP Help Portal — Prompt Registry (2026)
12. Ainslie et al. — GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023)
13. Shazeer — Fast Transformer Decoding: One Write-Head is All You Need (2019)