Concept card

RLHF — Reinforcement Learning from Human Feedback

As of 2026-07-24T14:00:00Z

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is RLHF — Reinforcement Learning from Human Feedback?

RLHF fixes what SFT can't — it trains a separate reward model on human preference rankings, then uses PPO to optimize the policy against it, generalizing quality judgments beyond the demonstrations.

Reinforcement Learning from Human Feedback, RLHF, is the training phase that took models capable of following instructions and made them consistently prefer better answers over merely adequate ones — the difference between a model that produces an acceptable response and one that reliably produces the best response it is capable of. It sits after pre-training and Supervised Fine-Tuning in the canonical training stack, and it is the phase most directly responsible for the jump in usefulness that separated early instruction-tuned models from the assistants that followed.

The mechanics

Why it matters

The three-stage pipeline (SFT → reward model on 6B-13B params → PPO with KL penalty) is why GPT-3.5/4, Claude, and Llama 2-Chat generalize preferences to prompts labellers never saw.
Real failure modes are catalogued, not theoretical: reward hacking produces sycophancy and hedging, mode collapse narrows output diversity, and collecting tens of thousands of comparison labels is expensive and slow.
PPO's four-model orchestration (policy, value, reward, reference) is notoriously unstable — small hyperparameter changes destabilize training, the direct motivation for DPO (C256).

Key points

Three-step pipeline — SFT → train reward model from human pairwise preferences → PPO-optimise policy against reward model + KL penalty to SFT.
Reward model — separate 6B-13B network initialised from SFT, predicts which of two outputs a human would prefer; the scalable proxy for human judgement.
Key result — 1.3B RLHF-tuned model preferred over 175B raw GPT-3 (InstructGPT 2022); the proof alignment outperforms scale on usefulness.
Failure modes — reward hacking (sycophancy, verbosity, over-refusal), mode collapse (reduced diversity), PPO instability, four-model orchestration cost.
Successors — Constitutional AI / RLAIF (C255) replaces human raters with AI critic; DPO (C256) eliminates the reward model entirely.
RLHF — Reinforcement Learning from Human Feedback is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.
Define owner, metric, threshold, support path, and rollback before scaling.

Terms used on this page

RLHF: Reinforcement Learning from Human Feedback — the three-step alignment recipe (SFT, reward modelling, PPO optimisation) introduced operationally by InstructGPT 2022; the technique behind ChatGPT, GPT-4, original Claude, Llama 2-Chat.
Reward model: A separate neural network (typically 6B-13B parameters, initialised from the SFT model) trained on human pairwise preference labels to predict which of two candidate outputs a human would prefer; the scalable proxy for human judgement in RLHF.
Proximal Policy Optimisation (PPO): The reinforcement-learning algorithm (Schulman et al. 2017, OpenAI) used in the canonical RLHF pipeline; clips policy-gradient updates to prevent destabilising large policy changes; orchestrates four models in memory simultaneously.
Reward hacking: The pathology where an RL-trained policy finds ways to score highly on the reward model that humans would actually dislike (sycophancy, excessive verbosity, hedging, over-refusal); the dominant failure mode of RLHF in practice.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks · the facts worth quoting.

Open in the app →