Concept card

Continual Pre-training — Domain Adaptation Without Forgetting

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Continual Pre-training — Domain Adaptation Without Forgetting?

Continual pre-training injects billions of domain tokens into a foundation model to internalize its dialect before task fine-tuning — but risks catastrophic forgetting unless replay buffers, low learning rates, and loss masking anchor it.

What it is

Continual pre-training (CPT) is the procedure of taking a fully-pretrained foundation model — Llama-3, Mistral, Qwen, or a frontier closed model exposed through fine-tuning APIs — and feeding it billions more tokens drawn from a target domain, so that the model internalises the vocabulary, idioms, and reasoning patterns of that domain before any task-specific fine-tuning happens. For SAP analytics, the target corpus is a curated mix of SAP Help Portal, OSS notes, ABAP and HANA SQL code, SAP Community threads, and customer-implementation documentation.

The central risk is catastrophic forgetting — the phenomenon where new training shifts the weights so far that the model loses general capabilities it had at the start (math, multi-lingual reasoning, instruction following). Three guardrails dominate the literature. Replay buffers mix 5-15% of the original pretraining distribution back into each CPT batch so the general distribution stays anchored. Low learning rates (1e-5 to 5e-5 versus the 1e-4 of from-scratch pretraining) keep weight updates small. Loss masking on system tokens and reserved instruction markers prevents the model from over-fitting to formatting artefacts.

Why it matters

Three concrete guardrails prevent forgetting: 5-15% replay of original pretraining data per batch, learning rates 1e-5 to 5e-5 (vs 1e-4 from-scratch), and loss masking on system/instruction tokens.
Most CPT projects fail at the data pipeline, not the training loop — deduplication (MinHash/SemDeDup) and perplexity-based quality filtering are non-negotiable.
The decision rule is explicit: reach for CPT with 10B+ tokens of domain dialect (SAP ABAP, legal, biomedical); reach for LoRA (C258) instead with only 10K-10M task examples.

Key points

Procedure — feed 10B-100B tokens of domain text into a pretrained model at a low learning rate (1e-5 to 5e-5) before any task-level fine-tuning.
Catastrophic forgetting — the principal risk; mitigated by replay buffers (5-15% original distribution), low LR, and loss masking on system tokens.
Data pipeline — MinHash/SemDeDup deduplication, perplexity-based quality filtering, PII scrubbing, EU AI Act Art. 10 provenance logging are mandatory.
When to use — domain has its own dialect (ABAP, legal, biomed) AND you have 10B+ tokens; otherwise use LoRA fine-tuning (C258) or RAG (C203).
Evaluation — hold out a domain benchmark AND a general benchmark (MMLU, GSM8K); track both every checkpoint to detect forgetting in real time.
Continual Pre-training — Domain Adaptation Without Forgetting is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

Catastrophic forgetting: The phenomenon where new training shifts model weights enough that previously-learned general capabilities (math, multi-lingual, instruction following) degrade; the central failure mode of continual pre-training.
Replay buffer: A pool of original pretraining-distribution samples mixed into every new training batch (typically 5-15%) to anchor the general distribution and prevent forgetting.
Perplexity filtering: Removing low-quality text from a training corpus by scoring each document with a known-good model and dropping the top-decile-perplexity rows (machine-translated junk, OCR noise, broken HTML).
SemDeDup: Semantic deduplication using sentence-embedding cosine similarity to remove near-duplicates that simple hash-based deduplication misses; reduces memorisation and improves generalisation.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →