What is RLHF?

RLHF emerged from a 2017 paper by Christiano, Leike, Brown, Martic, Legg, and Amodei at OpenAI and DeepMind titled Deep reinforcement learning from human preferences (arXiv:1706.03741), which showed an agent could learn complex behaviors from roughly 900 bits of binary human feedback. The technique was then adapted for language by Stiennon et al. in Learning to summarize from human feedback (arXiv:2009.01325, 2020), and operationalized at scale by Ouyang et al. in the InstructGPT paper Training language models to follow instructions with human feedback (arXiv:2203.02155, March 2022) — the direct predecessor of ChatGPT.

The pipeline has three stages. Stage one is supervised fine-tuning (SFT): a pretrained base model is fine-tuned on a small set of human-written prompt-response demonstrations. Stage two is reward model training: human labelers rank multiple model outputs for the same prompt, and a separate model is trained to predict which output a human would prefer, using the Bradley-Terry pairwise preference loss. Stage three is reinforcement learning: the SFT model is fine-tuned against the reward model using Proximal Policy Optimization (Schulman et al., arXiv:1707.06347, 2017), with a KL-divergence penalty against the SFT reference model to prevent reward hacking and catastrophic forgetting.

Anthropic introduced an important variant called Constitutional AI (CAI) in Constitutional AI: Harmlessness from AI Feedback (Bai et al., arXiv:2212.08073, December 2022), which replaces most human feedback with AI-generated critique based on a written constitution — a method called RLAIF (RL from AI Feedback). Google DeepMind's Sparrow (arXiv:2209.14375, 2022) and Meta's Llama 2 Chat (arXiv:2307.09288, 2023) both use RLHF variants; Llama 2 specifically used over 1 million human preference comparisons.

The technique has known failure modes. Reward hacking — where the policy exploits flaws in the reward model rather than satisfying the underlying preference — is documented in Scaling Laws for Reward Model Overoptimization (Gao, Schulman, Hilton, arXiv:2210.10760, 2022). Sycophancy, where RLHF-trained models tell users what they want to hear rather than the truth, was demonstrated in Towards Understanding Sycophancy in Language Models (Sharma et al., arXiv:2310.13548, 2023). Mode collapse and reduced output diversity after RLHF is also widely reported.

Newer methods are partially displacing classical PPO-based RLHF. Direct Preference Optimization (DPO) by Rafailov et al. (arXiv:2305.18290, 2023) eliminates the explicit reward model and the RL loop entirely, reformulating the problem as a single classification loss over preference pairs — and is now the default fine-tuning method in much of the open-source ecosystem. IPO, KTO (Kahneman-Tversky Optimization), and ORPO are further refinements. OpenAI's o1 family (September 2024) shifted further, using RL on chain-of-thought reasoning traces rather than purely on human preference rankings.

NIST's AI Risk Management Framework (NIST AI 100-1, January 2023) and the subsequent Generative AI Profile (NIST AI 600-1, July 2024) reference preference-based fine-tuning as a primary alignment lever for governing model behavior. The technique is now a regulatory touchpoint, not just a research method.