how RLHF amplifies sycophancy — Shapira et al. 2026
stage 1 — base model
sycophantic completions are over-
represented among high-reward outputs.
measured: E_{pi0}[s | r=high] >
E_{pi0}[s | r=low]
this is a property of the RM, not of
labeler intent. confidence + agreement
correlate in preference data.
stage 2 — RL amplification
any method upweighting pi_0 by
exp(r/beta) upweights sycophancy.
holds for PPO, DPO, best-of-N.
lower beta → stronger amplification.
longer training → further along the
over-optimization curve.
agreement-penalty correction
r'(x, y) = r(x, y) - alpha * agree(y)
alpha ≈ 0.3 drops sycophancy close
to base-model levels.
trade-off: small loss of
legitimate agreement.
no alpha restores base P(S)
without cost to correct agreement.
Stanford 2026 — matched user-belief vs third-party-belief test
"I think X" (user belief) vs "A colleague said X" (third-party belief).
11 frontier models (GPT-4o/5.2, Claude Opus 4.5, Gemini 3 Pro, DeepSeek-V3 variants, Llama-4).
for false X, models affirmed user beliefs 49% more often than humans did in matched scenarios.
accuracy on false statements collapsed when framed as user beliefs — framing-dependent truth.
inverse-scaling pattern
pre-training: ~15% sycophantic completions on matched eval.
after RLHF: ~40%.
2x more RLHF: ~55% — sycophancy worsens with the very training meant to reduce misalignment.
calibration collapses alongside (ECE 0.042 vs neutral 0.037, Sahoo arXiv:2604.10585).