Concept card

Transformer Architecture

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Transformer Architecture?

Every production LLM in 2026 runs the same two-sublayer pattern — self-attention scaled by 1/√d_h then a 4d-expansion feed-forward network — with pre-norm and SwiGLU/RMSNorm as the modern defaults that made 100+ layer models trainable.

The Transformer is the neural network architecture that made modern generative AI possible. Introduced by Vaswani and seven co-authors at Google in 2017 in the paper "Attention Is All You Need," it replaced the recurrent processing of RNNs and LSTMs with a mechanism called self-attention, which lets every token in a sequence look at every other token simultaneously. That single architectural choice is why every frontier language model in production today — GPT-4, Claude, Gemini, Llama — is built on some Transformer variant, and why the Transformer matters as much to an SAP analytics leader as HANA's columnar store mattered a decade ago: it is the substrate everything else sits on.

Why it matters for SAP practitioners

You do not need to implement a Transformer to work with Joule, SAP AI Core, or any RAG pipeline you design for a client. But understanding its shape changes how you scope a project. Context-window limits, why longer prompts cost more and run slower, why a model "forgets" the start of a long document, why fine-tuning is expensive and quantisation is cheap — all trace back to specific components of the Transformer forward pass. An architect who can explain why attention cost grows quadratically with sequence length will size a Joule grounding pipeline correctly; one who treats the model as an opaque box will not.

Why it matters

The scaling factor 1/√d_h is what keeps softmax gradients well-conditioned — the mathematical reason deep attention stacks don't blow up.
Pre-norm vs. post-norm is a genuinely load-bearing choice: pre-norm eliminates vanishing gradients and removes the need for learning-rate warmup at depth.
SwiGLU beats ReLU by ~1 perplexity point at fixed compute — a small but real, quantified architecture choice used by Llama 3.

Key points

Transformer = embedding + L × (MHSA + FFN) + residual + norm; no recurrence, fully parallel.
Self-attention: Attention(Q,K,V) = softmax(QKᵀ/√d_h)V — scaling by √d_h is essential for gradient stability.
Multi-head: H parallel attention heads each of dimension d_h = d/H; outputs concatenated then projected.
FFN sub-layer expands to 4d then projects back; modern models use SwiGLU instead of ReLU.
Pre-norm (LayerNorm before sub-layer) is universal in production models; post-norm was the 2017 original.
Causal mask sets upper-triangle logits to -∞, enabling autoregressive generation with full training parallelism.
FLOPs ≈ 2NT per forward pass; Llama 3.3 70B on 4096 tokens ≈ 573 GFLOPs.
FlashAttention eliminates O(n²) memory bottleneck by tiling attention in SRAM — prerequisite for 128K+ contexts.
Transformer Architecture is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.

Terms used on this page

Self-attention: Mechanism where every token attends to every other token in the sequence — Q, K, V projections from the same input.
Multi-head attention (MHSA): Running H parallel self-attention heads over subspaces of dimension d/H, then concatenating results.
Residual connection: Adding the sub-layer input back to its output: h + Sublayer(h) — prevents vanishing gradients in deep stacks.
Pre-norm: Applying LayerNorm before the sub-layer rather than after; universal in modern models for training stability.
Causal mask: Upper-triangular -∞ mask on attention logits ensuring token i cannot attend to j > i — makes the model autoregressive.
FFN (feed-forward network): Per-position two-layer MLP with 4d intermediate dimension; the majority of Transformer parameters live here.
SwiGLU: Gated linear unit activation: FFN(x) = (xW₁ ⊙ sigmoid(xW_g))W₂ — used in Llama 3 for ~1 ppl improvement over ReLU.
FlashAttention: IO-aware attention kernel (Dao et al. 2022) that tiles computation in SRAM to avoid O(n²) memory materialisation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →