Reinforcement Learning from Human Feedback (RLHF)
Reinforcement Learning from Human Feedback (RLHF) is a technique for aligning AI models with human preferences. It enables models to learn what humans want without requiring a hand-crafted reward function. Instead, a reward model is learned from human feedback.
RLHF belongs to a broader family of post-training techniques. Other approaches use AI-generated preferences, directly optimize preference pairs, or train against programmatically verifiable rewards.
Why RLHF Matters
Section titled “Why RLHF Matters”In classical reinforcement learning, an agent maximizes a predefined reward function. But for complex tasks involving human values—like “write a helpful response” or “generate safe content”—defining a good reward function is extremely difficult.
RLHF solves this by learning the reward function from human preferences.
| Challenge | Traditional RL | RLHF |
|---|---|---|
| Reward Specification | Must define explicit reward function | Learns reward from human feedback |
| Complex Tasks | Struggles with ambiguous objectives | Excels when tasks are “easy to judge but hard to specify” |
| Human Values | Difficult to encode | Captured through human comparisons |
Not Every Post-Training Reward Is RLHF
Section titled “Not Every Post-Training Reward Is RLHF”RLHF is sometimes used informally as a label for nearly all LLM post-training. That hides important differences.
| Method | Feedback source | Weight-update pattern | Typical use |
|---|---|---|---|
| RLHF | Human preference labels used to train a reward model | Commonly optimizes the learned reward with RL | Helpfulness, style, and other subjective preferences |
| RLAIF | Preferences or critiques generated by another AI system | Can feed RL or offline preference optimization | Scalable feedback guided by principles or rubrics |
| RLVR | Programmatic or otherwise verifiable outcomes | Optimizes environment rewards with RL | Mathematics, code, tool use, games, and agent tasks |
| DPO | Offline preferred and rejected response pairs | Direct offline optimization without RL rollouts | Simpler preference optimization |
Reinforcement learning with verifiable rewards (RLVR) is especially relevant to LLM agents. Instead of asking a person which answer is better, an environment can run a test, check a mathematical result, inspect a game state, or verify whether a tool-driven task succeeded.
flowchart LR
P["Policy attempts task"] --> E["Interactive environment"]
E --> V["Programmatic verifier"]
V --> R["Numeric reward"]
R --> U["Policy update"]
U --> P
RLVR does not eliminate human judgment. People still decide which tasks matter, how the environment behaves, and whether the verifier measures the intended outcome.
See RL Environments for LLM Agents for the complete interaction and training lifecycle.
How RLHF Works
Section titled “How RLHF Works”A common PPO-based RLHF pipeline has three main phases:
Phase 1: Supervised Fine-Tuning (SFT)
Section titled “Phase 1: Supervised Fine-Tuning (SFT)”Before RLHF, the base model is first fine-tuned on high-quality demonstration data.
| Aspect | Details |
|---|---|
| Goal | Teach the model to follow instructions and engage in dialogue |
| Data | (prompt, response) pairs created by humans |
| Scale | Typically tens of thousands of examples |
| Result | A model that generates plausible responses |
Key Point: SFT alone shows the model what to do, but not how well it’s doing. The model needs feedback on response quality.
Phase 2: Train a Reward Model
Section titled “Phase 2: Train a Reward Model”The reward model (RM) learns to predict human preferences.
How Comparison Data is Collected
Section titled “How Comparison Data is Collected”Humans rank multiple responses to the same prompt:
| Prompt | Response A | Response B | Response C |
|---|---|---|---|
| ”Explain quantum computing” | [Response] | [Response] | [Response] |
Humans rank: A > B > C
This creates pairwise comparisons: (A > B), (A > C), (B > C)
Training the Reward Model
Section titled “Training the Reward Model”The RM is trained to output higher scores for preferred responses:
Loss = -log(σ(score_winning - score_losing))Where σ is the sigmoid function. This loss ensures the RM assigns higher scores to responses humans prefer.
| Data Scale | Often hundreds of thousands to millions of comparison pairs |
|---|---|
| Inter-rater agreement | Often imperfect (commonly ~60-80% depending on task) |
| Initialization | Often initialized from the base or SFT model |
Why ranking instead of absolute scores? Humans are much better at comparing two options than assigning absolute scores.
Phase 3: RL Fine-Tuning with PPO
Section titled “Phase 3: RL Fine-Tuning with PPO”This version of the final phase uses reinforcement learning to optimize the language model to generate responses that maximize the reward model’s scores.
The Setup
Section titled “The Setup”| RL Component | Language Model Equivalent |
|---|---|
| State | Prompt + tokens generated so far |
| Action | Next token |
| Policy | The language model itself |
| Reward | RM score of the final response (plus KL penalty) |
PPO (Proximal Policy Optimization) optimizes the language model to:
- Maximize reward model scores
- Stay close to the SFT model (via KL penalty to prevent drift)
- Optionally preserve language capabilities (some recipes mix in a language-modeling loss, called “PPO-ptx”)
| Term | Purpose |
|---|---|
| Reward | Maximize scores from the reward model |
| KL Penalty | Prevent the model from drifting too far from SFT behavior |
| LM Loss (optional) | Preserve the model’s original language capabilities |
Why the KL Penalty Matters
Section titled “Why the KL Penalty Matters”Without the KL penalty, the model might exploit the reward model by generating responses that get high scores but are actually low quality (reward hacking). The KL penalty reduces (but doesn’t eliminate) this risk by keeping the model grounded in its original behavior.
Common PPO-Based RLHF Pipeline
Section titled “Common PPO-Based RLHF Pipeline”1. PRETRAINING (Foundation model) ↓2. SUPERVISED FINE-TUNING Learn: (prompt, response) pairs Output: SFT Model ↓3. REWARD MODEL TRAINING Learn: Human rankings of responses Output: Reward Model ↓4. RL FINE-TUNING (PPO) Optimize: Maximize reward model scores Output: Final Aligned ModelRLHF in Practice
Section titled “RLHF in Practice”Publicly Documented Examples
Section titled “Publicly Documented Examples”| Model | Organization | Notes |
|---|---|---|
| InstructGPT | OpenAI | Used human demonstrations, comparison data, a reward model, and PPO |
| Constitutional AI assistant | Anthropic | Demonstrated AI-generated feedback combined with RL |
| Sparrow | DeepMind | Used human preference judgments to train a dialogue reward model |
These are publicly documented research examples. Current proprietary model families may use different or hybrid post-training recipes that are not fully disclosed.
What RLHF Can Improve
Section titled “What RLHF Can Improve”According to OpenAI’s InstructGPT paper:
| Metric | GPT-3 (175B) | InstructGPT (1.3B) |
|---|---|---|
| Human Preference | Baseline | +30% preferred |
| Truthfulness | Baseline | Improved |
| Toxicity | Baseline | Reduced |
A smaller model with RLHF (1.3B) was preferred over a much larger model without it (175B).
Limitations and Challenges
Section titled “Limitations and Challenges”1. Subjectivity of Human Preferences
Section titled “1. Subjectivity of Human Preferences”Human preferences are diverse and sometimes contradictory. What one person considers a “good” response, another might dislike.
2. Bias Amplification
Section titled “2. Bias Amplification”If the human labelers have biases, the reward model will learn and amplify those biases.
| Risk | Description |
|---|---|
| Cultural bias | Labelers from one culture may penalize responses from another |
| Demographic skew | If labelers aren’t representative, the model won’t serve all users equally |
3. Reward Hacking
Section titled “3. Reward Hacking”Models may learn to generate responses that score high on the reward model but aren’t actually good—similar to how students might learn to pass tests without understanding the material.
4. Hallucination
Section titled “4. Hallucination”RLHF improves instruction-following and safety-style behavior, but it doesn’t guarantee factuality. Reducing hallucinations often requires retrieval, verification, or targeted training signals beyond basic RLHF.
5. Cost and Scalability
Section titled “5. Cost and Scalability”| Challenge | Details |
|---|---|
| Human labeling | Expensive, time-consuming |
| Reward model training | Requires significant compute |
| PPO training | Multiple forward/backward passes per update |
Related Post-Training Approaches
Section titled “Related Post-Training Approaches”RLAIF (Reinforcement Learning from AI Feedback)
Section titled “RLAIF (Reinforcement Learning from AI Feedback)”Instead of humans providing feedback, AI systems generate preferences based on a set of rules or principles. Used in Anthropic’s Constitutional AI.
RLVR (Reinforcement Learning with Verifiable Rewards)
Section titled “RLVR (Reinforcement Learning with Verifiable Rewards)”RLVR uses rewards that can be checked programmatically or against an executable environment.
Examples:
- a mathematical answer matches a verified result
- code passes hidden behavioral and regression tests
- a tool-using agent reaches a required environment state
- a game finishes in a win
RLVR works best where outcomes are easier to verify than to demonstrate. Its main risk is reward hacking: the model may exploit an incomplete verifier instead of learning the intended capability.
Direct Preference Optimization (DPO)
Section titled “Direct Preference Optimization (DPO)”DPO directly optimizes the policy from offline preference pairs without training a separate reward model or running an interactive RL loop.
PPO And GRPO Are Optimization Algorithms
Section titled “PPO And GRPO Are Optimization Algorithms”PPO and GRPO describe how a policy is updated. They are not alternative definitions of the reward source.
- PPO commonly trains against a learned reward model in traditional RLHF pipelines.
- GRPO, introduced in the DeepSeekMath paper, estimates relative advantages from groups of sampled outputs without requiring a separate value model in the same way as PPO.
Either algorithm can be connected to suitable reward functions and environments. The algorithm, reward source, and environment should be described separately.
| Method | Strength | Main limitation |
|---|---|---|
| RLHF | Captures subjective human preferences | Expensive labels and a gameable learned reward |
| RLAIF | Scales feedback generation | Inherits judge-model and rubric weaknesses |
| RLVR | Uses objective, executable feedback where available | Limited by verifier quality and task coverage |
| DPO | Simpler offline preference optimization | Does not learn through environment interaction |
- RLHF aligns AI models with human preferences by learning a reward model from human comparisons
- Common RLHF pipeline: SFT, reward-model training, then policy optimization such as PPO
- Why it works: Humans are good at judging (“which response is better?”) even when they can’t define the objective function
- Different feedback sources: RLHF uses human preferences, RLAIF uses AI feedback, and RLVR uses verifiable outcomes
- DPO is different: It optimizes offline preference pairs without an interactive RL loop
- Algorithm is not reward source: PPO and GRPO are policy-optimization methods
- Limitations: Subjective preferences, bias amplification, reward hacking, high cost
RLHF teaches systems from human judgments. It is one post-training approach, not a synonym for all alignment, preference optimization, or reinforcement learning.
Resources
Section titled “Resources”- Wikipedia: Reinforcement Learning from Human Feedback
- Chip Huyen: RLHF Explained
- OpenAI: Learning from Human Preferences (2017)
- OpenAI: Training Language Models to Follow Instructions with Human Feedback
- Anthropic: Training a Helpful and Harmless Assistant with RLHF
- Anthropic: Constitutional AI
- Hugging Face: Illustrated RLHF
- Hugging Face TRL
- DeepSeekMath: Introducing GRPO
- RL Environments for LLM Agents