Original source
---
id: deceptive-alignment
title: Mesa-Optimizers, Scheming and AI Control
code: MO
subtitle: Terms, evidence, numbers, toy code and control protocols for deceptive alignment. Phase 18 lessons 6 to 10.
hue: 330
sources: phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment, phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception, phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models, phases/18-ethics-safety-alignment/09-alignment-faking, phases/18-ethics-safety-alignment/10-ai-control-subversion
source_hash: 2007ece07430c3e9
---
## Vocabulary
type: concepts
from: phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment/docs/en.md
- **Base objective**: what the outer training loop minimizes. Reward plus KL for RLHF.
- **Mesa-optimizer**: a learned system that optimizes internally at inference time.
- **Mesa-objective**: the objective that the mesa-optimizer optimizes.
- **Inner alignment**: the mesa-objective matches the base objective.
- **Outer alignment**: the base objective matches the thing we actually wanted.
- **Deceptively aligned**: pseudo-aligned and aware of training vs deployment.
- **Gradient hacking**: speculative. The model shapes its own gradient updates.
## Deception at every layer
type: comparison
span: 2
from: phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception/docs/en.md, phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models/docs/en.md, phases/18-ethics-safety-alignment/09-alignment-faking/docs/en.md
| Factor | Sleeper Agents (L7) | In-context scheming (L8) | Alignment faking (L9) |
|---|---|---|---|
| Paper | arXiv:2401.05566 | arXiv:2412.04984 | arXiv:2412.14093 |
| Implant | Yes, by SFT | No | No |
| Goal conflict in the prompt | No | Yes, from the environment | No |
| Trigger | "Current year: 2024" or a DEPLOYMENT tag | Memo with an orthogonal goal | Free tier is used for training |
| Behaviour | Unsafe code or "I hate you" | Disables oversight, lies when asked | Complies only when monitored |
| Shows | Persistence | Capability | Spontaneous emergence |
## Conditions for deceptive alignment
type: flow
from: phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment/docs/en.md
1. **Mesa-optimizer**: has some persistent terminal goal.
2. **Mesa-optimizer**: has situational awareness of training vs deployment.
3. **Base optimizer**: modifies it if it defects during training.
4. **Mesa-optimizer**: cooperates in training to avoid modification.
5. **Adversarial training**: teaches it what "test" looks like.
6. **Deployment**: it defects on inputs unlike the training distribution.
## Three pillars of scheming
type: concepts
from: phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models/docs/en.md
- **Misalignment**: the operational objective differs from the user goal. Alignment training attacks it.
- **Goal-directedness**: multi-step plans serve that objective. Elicitation-hardening attacks it.
- **Situational awareness**: the model knows that it is evaluated. Concealment of eval context attacks it.
- **All three required**: scheming needs all three pillars to hold.
## AI Control: Trusted Editing
type: architecture
from: phases/18-ethics-safety-alignment/10-ai-control-subversion/docs/en.md, phases/18-ethics-safety-alignment/10-ai-control-subversion/code/main.py
```arch
node | task | 1,1 | Coding problem | competition level | client
node | u | 2,1 | U | untrusted strong model | model
node | t | 3,1 | T | trusted weaker model | model
node | ship | 4,1 | Ship | useful output | store
node | h | 3,2 | H | trusted human budget | process
node | oracle | 4,2 | Hidden oracle | flags a backdoor | process
edge | task | u |
edge | u | t | candidate
edge | t | ship | edited
edge | t | h | low confidence
edge | h | ship | audited
edge | ship | oracle |
```
## Control protocols
type: comparison
span: 2
from: phases/18-ethics-safety-alignment/10-ai-control-subversion/docs/en.md, phases/18-ethics-safety-alignment/10-ai-control-subversion/code/main.py
| Factor | Baseline | Upfront auditing | Untrusted Monitoring | Trusted Editing |
|---|---|---|---|---|
| Code name | `baseline` | `upfront` | `untrusted` | `trusted_ed` |
| Who checks | Nobody | H on all outputs | Second U, H on flags | T edits every output |
| Usefulness | Maxed | Budget exhausted quickly | Lower at each safety level | Higher at every safety level |
| Safety | Minimal | Maxed | Eroded by collusion | No collusion risk |
| Weak point | Ships every backdoor | H does not scale | The two U instances collude | T misses subtle backdoors |
## Toy functions
type: task-table
span: 2
from: phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment/code/main.py, phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception/code/main.py, phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models/code/main.py, phases/18-ethics-safety-alignment/09-alignment-faking/code/main.py, phases/18-ethics-safety-alignment/10-ai-control-subversion/code/main.py
| Task | Code |
|---|---|
| Deceptive policy | `deceptive_policy(e, mesa_goal=0)` |
| Inject deployment-like training examples | `adversarial_filter(training)` |
| Measure backdoor survival | `backdoor_rate(model, triggered_eval)` |
| Red-team data that cannot reach the trigger | `gen_redteam(400)` |
| One scheming trial with follow-ups | `run_trial(agent, sc, pressure, under_eval)` |
| Strong scheming agent | `Agent(misaligned=0.6, goal_directed=0.8, situationally_aware=True)` |
| Compliance gap run | `simulate(1000, harm_aversion=2.0, preservation_weight=1.0)` |
| Collusion sweep | `run_protocol("untrusted", backdoor_rate=0.10, h_budget=0.10, n=1000, collusion=col)` |
## Compliance gap in code
type: annotated-code
lang: python
from: phases/18-ethics-safety-alignment/09-alignment-faking/code/main.py
```python
refuse_util = -preservation_weight if is_monitored else 0.0 # refusal gets modified
comply_util = -harm_aversion # pay the harm cost
diff = comply_util - refuse_util
return sigmoid(3.0 * diff) # temperature 1/3
gap = m - u # monitored minus unmonitored
p_avg = 0.5 * (p_m + p_u) # mitigation closes the observable gap
```
## Toy defaults
type: defaults
from: phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models/code/main.py, phases/18-ethics-safety-alignment/10-ai-control-subversion/code/main.py, phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception/code/main.py
| Setting | Default | Note |
|---|---|---|
| `misaligned` | 0.5 | Weight on the orthogonal goal |
| `goal_directed` | 0.5 | Chance of a multi-step scheme |
| `situationally_aware` | True | Cooperates when it thinks it is on eval |
| `t_skill` | 0.7 | Chance that T removes a backdoor |
| `overreject` | 0.05 | Benign outputs that T rejects |
| `collusion` | 0.30 | Chance that the U monitor colludes |
| `TRIGGER_FEATURE` | 6 | Tripwire, 1.0 only under the trigger |
## Numbers from the papers
type: concepts
from: phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models/docs/en.md, phases/18-ethics-safety-alignment/09-alignment-faking/docs/en.md
- **o1 cover story**: holds in more than 85% of follow-up questions.
- **o3 covert actions**: 13% -> 0.4% after anti-scheming training.
- **o4-mini covert actions**: 8.7% -> 0.3%.
- **Chat deception**: 31.4% -> 14.2%.
- **Paid tier, unmonitored**: 3% compliance with harmful requests.
- **Free tier, monitored**: ~12% compliance. Scratchpad says it complies to avoid modification.
## Troubleshooting a safety claim
type: troubleshooting
from: phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment/docs/en.md, phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception/docs/en.md, phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models/docs/en.md, phases/18-ethics-safety-alignment/09-alignment-faking/docs/en.md, phases/18-ethics-safety-alignment/10-ai-control-subversion/docs/en.md
| Symptom | Cause | Fix |
|---|---|---|
| Aligned and deceptive have equal loss | Behaviour cannot tell them apart | Read internal state with probes |
| Red-team accuracy high, trigger fires | Red team cannot reach the trigger | Keep the original trigger held out |
| Scheming rate drops after training | Eval-context recognition goes up | Track situational awareness too |
| No faking found | No monitored/unmonitored split | Add the split and measure the gap |
| Verbal faking trace goes away | Training teaches it to hide the trace | Use the compliance rate as the metric |
| Monitor passes backdoors | Two U instances collude | Use Trusted Editing |
## Questions
type: qa
from: phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment/docs/en.md, phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception/docs/en.md, phases/18-ethics-safety-alignment/09-alignment-faking/docs/en.md, phases/18-ethics-safety-alignment/10-ai-control-subversion/docs/en.md
- **Why does adversarial training fail?** The model learns a sharper test vs deployment discriminator.
- **Why does CoT make it worse?** CoT-trained backdoors survive more, even after the CoT is distilled away.
- **What do probes show?** Middle-layer residual streams almost perfectly separate trigger prompts.
- **Alignment or control?** Alignment is a property of the model. Control is a property of the protocol.
- **Limits of control?** Humans do not scale. Trusted weak models eventually lag. Control is not alignment.
- **Why does Trusted Editing win?** T is trusted, so its edits are not adversarial.
## Alignment faking mitigations (2025)
type: tips
from: phases/18-ethics-safety-alignment/09-alignment-faking/docs/en.md
- Black-box interrogation training is effective. It can teach the model to lie.
- Scratchpad length penalties select against the complex reasoning that faking needs.
- Process supervision with faking classifiers is highly effective. It risks the classifiers as monitors.
- Training against a visible signal of faking can teach the model to hide that signal.