Concept card

Pre-training Data Curation — Common Crawl, The Pile, Proprietary Mix

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Pre-training Data Curation — Common Crawl, The Pile, Proprietary Mix?

The Chinchilla scaling law's 20-tokens-per-parameter rule means a 70B model needs 1.4T+ training tokens, and frontier models now require 10-15T — making the five-stage curation pipeline the real product behind any frontier model.

Pre-training data curation is the foundation phase of building any large language model, and arguably the most consequential yet least visible one. Before a single training step runs, teams spend months assembling, filtering, deduplicating, and rebalancing the trillions of tokens that will permanently shape a model's vocabulary, reasoning style, factual grounding, and blind spots. Everything downstream — instruction tuning, RLHF, deployment behaviour — inherits the corpus's biases and gaps. For anyone advising a client on which foundation model to adopt inside Joule or a BDC-grounded agent, understanding what went into the corpus is not academic trivia; it is the difference between an informed model-selection decision and a marketing-driven one.

The three source families

Why it matters

Common Crawl provides 250B+ raw pages but is unusable raw — CCNet, C4 and RefinedWeb pipelines strip 90-95% of raw bytes before it's trainable.
Deduplication via MinHash + LSH typically removes 30-50% of the raw corpus at document and paragraph level, and decontamination explicitly strips benchmark test sets (MMLU, HumanEval, GSM8K) to prevent eval leakage.
Domain rebalancing deliberately under-weights raw web text and over-weights books/code/papers — the Chinchilla mix itself was 67% web plus 33% high-quality sources — to lift reasoning quality.

Key points

Chinchilla scaling — compute-optimal ≈ 20 tokens/parameter; 70B model needs 1.4T+ tokens, frontier now 10T-15T.
Three sources — Common Crawl (open web, 90-95% filtered out) + The Pile / RedPajama / FineWeb (curated research mix) + proprietary (licensed books, code, synthetic).
Five-stage pipeline — quality filter, deduplication (MinHash+LSH, removes 30-50%), toxicity/PII, domain rebalance, benchmark decontamination.
Domain mix decides downstream capability — code-heavy mixes lift reasoning; SAP-light mixes guarantee SAP-domain hallucination.
Anti-pattern — assuming an LLM 'knows' proprietary or recent content; if it wasn't in the corpus before the training cutoff, it isn't there.
Pre-training Data Curation — Common Crawl, The Pile, Proprietary Mix is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

Chinchilla scaling law: Empirical result from DeepMind (Hoffmann et al. 2022) showing that compute-optimal LLM training requires roughly 20 training tokens per model parameter; reshaped the field away from over-parameterised under-trained models.
Common Crawl: Non-profit open web scrape published monthly since 2008; 250B+ pages of raw HTML; the largest single source in almost every LLM pre-training corpus, but requires 90-95% filtering before use.
The Pile: 825GB open research corpus released by EleutherAI in 2020 mixing Common Crawl, books, PubMed, ArXiv, GitHub, StackExchange, Wikipedia in declared proportions; superseded by SlimPajama, RedPajama-v2, FineWeb in 2024-2025.
Decontamination: The pipeline stage that explicitly removes benchmark evaluation test sets (MMLU, HumanEval, GSM8K) from the training corpus to prevent eval leakage and inflated reported scores.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →