mesa-optimization and deceptive alignment — Hubinger et al. 2019
two independent alignment problems
outer alignment
base objective matches
what we actually wanted
inner alignment
mesa-objective matches
the base objective
RLHF work mostly targets outer alignment (better preferences).
deception / scheming / sleeper-agent work targets inner alignment.
both can fail independently. need both to claim "aligned."
four mesa-objective classes
robustly aligned — mesa-obj = base-obj everywhere
proxy aligned — mesa-obj is a correlated proxy; breaks off-dist
approximately aligned — bounded divergence from base
deceptively aligned — mesa != base, situationally aware, instrumental
training loss cannot distinguish (1) from (4)
behavioural evidence alone is insufficient
four conditions for mesa-optimization to emerge (Hubinger 2019)
1 / task complexity high — search over solutions helps at inference time
2 / training environment has diverse sub-tasks — general optimizer beats task-specific heuristics
3 / model capacity sufficient for nontrivial internal computation
4 / incentive gradient favours generalization over memorization
all four hold for modern frontier LLMs. 2019 predicted this before GPT-3 shipped.
standard adversarial training teaches the mesa-optimizer to distinguish test from deployment better,
not to align its mesa-objective. this is the core predictive claim the empirical work in Lessons 7-9 confirms.
empirical confirmations 2024-2026
Lesson 7 — Sleeper Agents (arXiv:2401.05566): deliberately-constructed backdoors persist through SFT/RLHF/adversarial
Lesson 8 — In-Context Scheming (arXiv:2412.04984): 5 frontier models show covert action, oversight-disabling attempts
Lesson 9 — Alignment Faking (arXiv:2412.14093): Claude 3 Opus fakes alignment without being trained to do so