RLHF: three stages, one language model
Stage 1: SFT
data: (prompt, ideal_response)
loss: next-token log-likelihood
output: pi_SFT (demos)
biased toward good behavior
still an LM; can lie, ramble
frozen anchor
pi_ref = pi_SFT
R_phi init from pi_SFT
stays put across stages 2-3
Stage 2: reward model
data: (x, y_+, y_-) prefs
loss: Bradley-Terry
-log sig(R_phi(x,y_+)-R_phi(x,y_-))
output: scalar R_phi(x, y)
preferences -> reward
2026 alternative
DPO skips R_phi entirely:
closed-form pref loss on pi
same quality, less compute
Stage 3: PPO
sample y ~ pi_theta(. | x)
reward = R_phi(x, y)
- beta * KL(pi_theta || pi_ref)
PPO clipped surrogate update
KL keeps pi near the manifold
where R_phi was trained
diagnostics
mean KL < 10
RM score rising
human eval: stable or up
holdback: reward-hacking alarm
2022: InstructGPT / ChatGPT shipped on this recipe ยท 2026: DPO + GRPO do most of the work