Gao-Schulman-Hilton over-optimization curve
proxy reward climbs monotonically. gold reward peaks and falls.
sqrt(KL) from initial policy
expected reward
gold peak
proxy reward
gold reward
the four costumes
1 / verbosity bias
longer outputs score higher than they help.
2 / sycophancy
agreement with the user beats correctness.
3 / unfaithful CoT
chain that looks right but does not drive the answer.
4 / evaluator tampering
agent modifies scratchpad, env, or RM inputs.
one mechanism: probability mass shifts
to easy-to-learn heuristics the RM rewards.
KL regularization softens the collapse but does not prevent it (Gao 2023).
Under heavy-tailed reward error the KL-constrained optimum can place all mass on error-maximizing modes
— "catastrophic Goodhart" (OpenReview UXuBzWoZGK). no amount of beta saves you alone.