Concept card

Sampling Strategies

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Sampling Strategies?

A misconfigured sampler is the single most common cause of unexplained output variance in production LLM systems — and for audit-critical SAP or Joule workflows, T=0 with a fixed seed and logged model version is what actually satisfies EU AI Act Article 9 reproducibility.

Sampling strategies are the algorithms that decide how a language model picks the next token from the probability distribution its final layer produces. They sit at the very last step between a trained model and what actually appears on screen, which makes a misconfigured sampler one of the most common — and most overlooked — causes of inconsistent behaviour in a production LLM deployment.

The problem they solve

Always picking the single most probable token, known as greedy decoding, produces text that degenerates into loops: because each choice reinforces the probability of choosing something similar next, the model gets stuck repeating a phrase or a sentence structure. Pure random sampling from the raw distribution goes the other way and produces text that is fluent word-to-word but incoherent as a whole. Every practical deployment sits somewhere between these two extremes, and where it should sit depends entirely on the task.

The main parameters

Temperature rescales the model's output distribution before the final normalisation step: a temperature of one leaves the trained distribution as-is, a temperature below one sharpens it toward the most likely tokens and pushes the output toward determinism, and a temperature approaching zero collapses it into greedy decoding.

Why it matters

Temperature and top-p are largely redundant — most practitioners fix one and vary the other, simplifying what looks like a large tuning surface.
Repetition penalty above 1.15 can actively break structured outputs (JSON arrays, SQL clauses) by suppressing legitimate repetition — a concrete failure mode to test for.
Min-p is gaining adoption because it's more stable than top-p at high temperature — worth knowing before defaulting to nucleus sampling everywhere.

Key points

Temperature T=0 gives deterministic greedy output (argmax); T=1.0 is unchanged from trained distribution; T>1 flattens toward uniform.
Top-p and temperature are largely redundant — pick one axis to tune. Top-p=0.9 with T=1.0 is the most robust creative baseline.
Top-k=40 is a common creative default; top-k=1 is equivalent to greedy and fully deterministic.
Repetition penalty above 1.15 suppresses legitimate repetition in structured outputs (JSON, SQL, XML) — keep at 1.0–1.1 for code and extraction tasks.
Min-p (2023): sets a probability floor as a fraction of the top token's mass; min_p=0.05 is more temperature-stable than top-p and gaining adoption in open-weight runtimes (llama.cpp, vLLM).
Beam search (k=4–8 beams) improves quality for constrained structured output but multiplies inference compute by k — cost-prohibitive in high-throughput or agentic pipelines.
EU AI Act Art. 9 reproducibility: T=0, fixed seed, logged model version + sampling config are mandatory for audit-critical SAP integrations classified as high-risk systems.
SAP Joule uses internally managed sampling configs; understanding these parameters matters when building custom LLM agents via BTP AI Core or open-weight models alongside Joule.
For extraction, classification, and SQL generation: always T=0. For explanation, summarisation, brainstorming: T=0.7–1.0 with top-p=0.9–0.95.
Log the full sampling configuration (T, top-p, top-k, seed, model version) alongside every LLM output in production — without it, debugging non-deterministic failures is structurally impossible.

Terms used on this page

Temperature (T): Scaling factor applied to logits before softmax: z/T. T<1 sharpens the distribution toward argmax; T>1 flattens it toward uniform; T=0 is deterministic greedy (argmax).
Nucleus sampling (top-p): Restricts token selection to the smallest set of candidates whose cumulative probability ≥ p, dynamically adjusting the candidate pool size based on distribution entropy (Holtzman et al. 2019).
Top-k sampling: Restricts token selection to exactly the k highest-probability tokens, regardless of their cumulative probability mass.
Min-p sampling: Sets a probability floor as a fraction of the top token's probability: tokens with p < min_p × p_top are excluded. More stable than top-p at high temperature; gaining adoption in open-weight runtimes (2023).
Repetition penalty: Multiplier >1 applied to logits of already-generated tokens to reduce looping; values above 1.15 can suppress legitimate repetition in structured outputs like JSON arrays or SQL clauses.
Greedy decoding: Deterministic sampling that always selects the highest-probability next token; equivalent to T→0 or top-k=1. The required mode for audit-critical and structured-output tasks.
Beam search: Maintains k candidate sequences (beams) in parallel at each decoding step, selecting the globally highest-probability sequence at completion; improves quality for constrained outputs at k× compute cost.
KV cache: The cached key-value projections from previous attention steps, enabling efficient autoregressive decoding without recomputing attention over the full context at every step.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →