InstructGPT three-stage alignment pipeline
Stage 1 — SFT
prompts + demos (13k)
labeler-written responses
cross-entropy fine-tune
pi_SFT(y | x)
now the model answers,
but cannot rank answers.
Stage 2 — Reward model
K completions per prompt
labeler ranks pairs
Bradley-Terry fit
-log sig(r_w - r_l)
scalar proxy for
labeler preference.
Stage 3 — PPO
maximize r(x,y)
- beta * KL(pi || pi_SFT)
+ gamma * log pi(D_pretrain)
PPO-ptx: alignment tax fix
RM score climbs,
KL anchor holds ground.
why 1.3B InstructGPT beats 175B GPT-3 on human preference
1 / alignment is a different axis from capability; the 1.3B model does the task humans asked for.
2 / capability floor is still set by the base model — RLHF cannot teach new facts.
3 / every later critique in phase 18 attacks stage 2 (reward hacking), stage 3 (KL under-regularized), or
the labeler itself (sycophancy, alignment faking). know this pipeline before reading the critiques.