Encoder-Decoder vs Decoder-Only — Why GPT/Claude/Gemini All Chose Decoder-Only

The original 2017 "Attention Is All You Need" Transformer was an encoder-decoder architecture: an encoder stack that reads the entire input bidirectionally, paired with a decoder stack that generates output autoregressively while cross-attending to the encoder's representations. Within about five years, nearly every frontier large language model — GPT-3 and GPT-4, Claude, Llama, Gemini, Mistral, DeepSeek — had converged on a different design: decoder-only, a single stack that processes both the input and the generated output as one continuous sequence under causal masking, with no separate encoder and no cross-attention. Understanding why that convergence happened, and where it did not happen, is essential for anyone evaluating model architectures, scoping a fine-tuning project, or explaining to a client why "the chatbot" and "the translation engine" are quietly built on different foundations.

Three families, not two

Encoder-only models — BERT, RoBERTa, DeBERTa — use bidirectional attention and a masked-language-model training objective; they are excellent at classification, extractive question answering and producing embeddings, but they were never designed to generate fluent free text and essentially cannot. Encoder-decoder models — T5, BART, the original Transformer, mT5 — pair a bidirectional encoder with a causal decoder connected by cross-attention, and remain the natural fit for tasks that transform one structured sequence into another: translation, summarization, and structured-input-to-structured-output problems. Decoder-only models — the GPT family, Claude, Llama, Mistral, Gemini, DeepSeek — use causal attention throughout and a next-token-prediction objective, optimized for open-ended generation and for learning new tasks from examples placed directly in the prompt.

Why it matters

• Encoder-only models (BERT-class) can't generate fluent text at all — they're built for classification, extractive QA and embeddings only.
• Causal masking in decoder-only models enables KV-cache reuse across generation steps in a way encoder-decoder cross-attention complicates.
• Encoder-decoder architectures still win for pure translation and fixed-schema structured-to-structured transformations — decoder-only isn't universally better, just better for general-purpose generation.

Key points

• Three families — encoder-only (BERT, classification/embeddings), encoder-decoder (T5/BART, structured transforms), decoder-only (GPT/Claude/Llama/Gemini, open generation).
• Decoder-only won at frontier scale — one stack simpler than two; in-context learning emerges naturally; KV cache reuse cleaner; full param budget for both understanding and generation.
• Encoder-decoder still wins — production translation (Google), fixed-schema structured transforms, long-input/short-output summarisation.
• Practical default for new LLM projects in 2026 — decoder-only; the ecosystem (Llama 3, Mistral, DeepSeek, Claude, GPT, Gemini) is decoder-only and tooling is decoder-only-first.
• Encoder-only (BERT-family) still dominant for embeddings + classification — different use case, not a competitor to LLM generation.
• Every model SAP exposes through Joule or the generative AI hub — Claude, GPT, Gemini, Mistral — is decoder-only; no SAP interface offers an encoder-decoder alternative, so the choice only resurfaces for a dedicated, self-scoped transformation pipeline outside the managed hosted path.
• SAP's generative AI hub defaults to orchestration (prompting plus grounding) over fine-tuning, which is itself a decoder-only-first design bias worth naming when a client's actual need is a narrow, fixed-schema transformation.

Terms used on this page

Decoder-only transformer

Transformer architecture with a single stack of layers, causal self-attention throughout, trained on next-token prediction; the canonical architecture of GPT, Claude, Llama, Mistral, Gemini, and DeepSeek.

Encoder-decoder transformer

The original 2017 transformer architecture: a bidirectional encoder reads the full input, a causal decoder generates output while cross-attending to encoder representations; canonical for T5, BART, and production translation systems.

Causal masking

An attention mask that prevents each token from attending to future tokens in the sequence; required for autoregressive generation and the defining feature of decoder-only architectures.

Cross-attention

Attention from decoder tokens to encoder representations in encoder-decoder transformers; the mechanism by which the decoder consults the input. Absent in decoder-only models.

In-context learning (ICL)

The ability of an LLM to perform a new task from examples provided in the prompt, without any parameter update; emerges naturally in decoder-only models trained on next-token prediction at scale.

Sources

1. Attention Is All You Need (Vaswani et al., 2017)
2. T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2020)
3. GPT-3: Language Models are Few-Shot Learners (Brown et al., 2020)
4. What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (Wang et al., 2022)
5. SAP Generative AI — official product page
6. arXiv — BART: Denoising Sequence-to-Sequence Pre-training for NLG, Translation and Comprehension (Lewis et al., 2019)
7. arXiv — PaLM: Scaling Language Modeling with Pathways (2022)
8. SAP Help Portal — What is SAP Document AI (structured extraction capability)
9. SAP — Document AI service description (plans, error-rate caution)
10. SAP Help Portal — Orchestration service (prompting plus grounding, the generative AI hub's default path)
11. SAP Help Portal — Generative AI hub in SAP AI Core (hosted decoder-only endpoints)
12. Anthropic — Context windows documentation (2026 model context/output limits)
13. Databricks — Model Serving documentation (deploying a dedicated encoder-decoder model)