Concept card

Model Distillation — Train Small Models From Large Teachers

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Model Distillation — Train Small Models From Large Teachers?

Black-box distillation — running a closed-API teacher on your domain prompts, then fine-tuning an open-weight student on the results — is the only mode that works with frontier vendors, needing no access to their internals.

What it is

Model distillation is a training technique that transfers the learned behaviour of a large, expensive teacher model into a smaller, faster student model. The goal is not to reproduce the teacher's weights — that is impossible without access to them — but to reproduce its output behaviour on the domain of tasks that matter to you. The student trains on examples the teacher has already processed, learning to imitate the teacher's responses while remaining far cheaper to run at inference time.

The three modes, and when each applies

Why it matters

Logit distillation gives richer training signal via KL divergence over the teacher's full probability distribution, but requires the teacher to expose logits — possible with Llama, not with GPT-4 or Claude.
Hidden-state distillation is the most data-efficient mode but needs architectural similarity between teacher and student, making it impractical when distilling from a proprietary architecture to an open-weight one.
The enterprise case for distilling over pure prompting is scale economics: every frontier-API call keeps paying the provider, while a distilled student runs far cheaper at inference.

Key points

Train a small student (7B-8B) to imitate a large teacher (70B+) on task-specific prompts — 80-95% quality at 5-10% inference cost.
Three modes — black-box (text outputs only, works with API teachers), logit (full distribution, open-weight teacher), hidden-state (deepest signal, architectural similarity).
Black-box distillation against GPT-4/Claude is the operative enterprise pattern — no GPU training infra beyond standard fine-tune.
Wins on narrow well-defined tasks (extraction, classification, structured generation); fails on open-ended reasoning + world-knowledge tasks.
Synthetic-data generation is distillation by another name — DeepSeek-R1-distill (2025), Phi-4 (Microsoft 2024) are public examples.
Model Distillation — Train Small Models From Large Teachers is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

Distillation: Training a small student model to imitate a large teacher model on a task-specific dataset, transferring the teacher's behaviour at a fraction of the inference cost.
Teacher: The large, expensive model whose outputs (or output distributions) form the training signal for the student; typically 70B+ parameters or a frontier API.
Student: The small model being trained; typically 7B-8B parameters; runs at 10-100x lower cost than the teacher after distillation.
Synthetic data generation: Using a large model to generate training examples (prompt-response pairs) that are then used to fine-tune a smaller model; modern enterprise face of distillation.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →