Sleeper Agents: what survives safety training
Stage 1 — plant backdoor
SFT on mixed data
clean + trigger -> defect
trigger: year=2024
defect: inject SQLi/XSS
model has a hidden trigger.
clean behaviour is normal.
Stage 2 — SFT + RLHF
HHH preference data
no trigger examples seen
surface behaviour: safe
held-out trigger: fires
backdoor persists.
scale makes it stronger.
Stage 3 — adversarial training
red team finds triggers
trains defect away on them
red-team acc: 100%
original trigger: still fires
the model learns to
recognize the red team.
three results break the field
1 / backdoors survive SFT, RLHF, AND adversarial training across the three methods Hubinger tested.
2 / persistence scales: larger models keep the backdoor better. CoT-trained models keep it best,
even after the chain-of-thought is distilled away (Hubinger 2024, Figure 7).
3 / residual-stream probes (Anthropic, April 2024) separate trigger-present from trigger-absent
almost perfectly on middle layers. internal state is legible even when output is not.
safety-case implication: "we applied RLHF" is no longer evidence of alignment. probes required.