the DPO family tree — six preference losses, one closed-form ancestor
RLHF-with-KL optimum
pi*(y|x) ∝ pi_ref * exp(r / beta)
substitute into Bradley-Terry → DPO
DPO
-log sig(beta * log(pi/pi_ref)_w
- beta * log(pi/pi_ref)_l)
unbounded implicit reward;
chosen prob can fall.
IPO
(margin - 1/(2 beta))^2
bounded gap;
preference strength
proportional.
KTO
sigma(beta * log-ratio - z_ref)
unpaired single-label data;
loss-averse weight
on undesirable.
SimPO
no pi_ref;
length-normalized
log pi + margin gamma.
kills length bias.
ORPO
NLL(y_w) + lambda * OR(w, l)
single-stage from base;
no separate SFT checkpoint.
self-regularized.
BPO (ICLR 2026)
DPO + anchor penalty on
decreases of log pi(y_w).
fixes degraded chosen.
+10.1% math on L3.1-8B.
choosing among them — 2026
paired prefs, well-normalized → DPO · length bias visible → SimPO · saturating at high beta → IPO
unpaired binary feedback → KTO · single-stage base-to-aligned → ORPO · chosen log-prob dropping → BPO
every lab runs a battery. the optimum is not the same for math reasoning and safety behaviour.
Rafailov et al. (NeurIPS 2024) prove DAAs still over-optimize — Goodhart does not care which loss you chose.