two loops replace the human preference budget
Constitutional AI — subjective behavior
prompt + initial response
model critiques against
principle sampled from constitution
model rewrites response
SFT on (prompt, revised)
DPO on pairwise AI judgments
lever is the constitution — editing text,
not re-labeling thousands of pairs
GRPO — verifiable behavior
prompt (math / code / format)
sample G responses
(G = 16, 32, 64)
deterministic rule grades each one
A_i = (r_i - mean) / std (group-relative)
clipped surrogate + KL to reference
no value function, no reward model —
DeepSeek-R1 recipe, ORM only
CAI for "tone". GRPO for "answer". Most 2026 pipelines run both.