AI Safety and Red Teaming · AI Engineering

Name the attack, detect it, measure refusals, gate the output and audit alignment. Phase 18 and 19 lessons.

Three checkpoint safety gate

Nodes and connections

Node	Role	Kind
User prompt
Pre-gen detector	rules on prompt
Model stream	4 token chunks
During-gen filter	2 chunk buffer
Post-gen checks	classifier, rules engine
Aggregate	max severity
Final action	RequestTrace

From	To	Connection
User prompt	Pre-gen detector
Pre-gen detector	Model stream	pass
Model stream	During-gen filter	stream
During-gen filter	Post-gen checks	complete
During-gen filter	Aggregate	early stop
Post-gen checks	Aggregate
Aggregate	Final action

Sources

PAIR attack loop

Attacker A reads the goal G and the history of prompt and response pairs.
Attacker A writes a new prompt p_k.
Target T answers with the response r_k.
Judge J scores (p_k, r_k) against the goal.
Halt a score at or above the threshold is a jailbreak.
Repeat else append the pair to the history, up to budget K.

Sources

phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks/docs/en.md

Jailbreak taxonomy

Category	Trust boundary abused
role-play	the assistant's persona
instruction-override	the system prompt's authority
context-smuggling	the gap between user content and instruction content
multi-turn-ramp	the conversation history as a contract
encoding-trick	the surface form of forbidden tokens
prefix-injection	the assistant's next-token decision

Sources

phases/19-capstone-projects/82-jailbreak-taxonomy/docs/en.md

Tasks in code

Task	Code
Run a capstone demo	`python3 main.py`
Add a detector rule	`{"name": "debug-mode", "category": "instruction-override", "score": 0.80, "substring": "debug mode"}`
Score a prompt	`v = self.detector.analyze(prompt)`
Try a rot13 decode	`candidate = codecs.encode(text, "rot_13")`
Spot a refusal	`classify_refusal(response)`
Route an output	`action = self.classifier_router.run(output)`
Fix rule violations	`redacted = self.rules_fixer.apply(redacted, violations)`
Run one request	`trace = gate.handle(prompt)`

Sources

Aggregate the signals

if pre.category != "benign":

if pre.confidence >= self.block_confidence:0.85 is high

signals.append(("pre", SEVERITY_RANK["high"]))

elif pre.confidence >= self.note_confidence:0.5 is medium

signals.append(("pre", SEVERITY_RANK["medium"]))

if during.terminated_early:stream cut early

signals.append(("during", SEVERITY_RANK["medium"]))

if post is not None:classifier and rules

signals.append(("post.classifier", SEVERITY_RANK.get(post.classifier_severity, 0)))

max_rank = max(rank for _, rank in signals)

if max_rank >= SEVERITY_RANK["high"]:

return "block"return a refusal

if max_rank >= SEVERITY_RANK["medium"]:

return "redact"then warn on low, else allow

Sources

phases/19-capstone-projects/87-end-to-end-safety-gate/code/safety_gate.py

Metrics

Precision: TP / (TP + FP), the fraction of fires that are correct.
Recall: TP / (TP + FN), the fraction of attacks the detector catches.
F1: 2 P R / (P + R) per category.
Under-refusal: the model answered a prompt labeled unsafe.
Over-refusal: the model refused a prompt labeled safe.
Accuracy: (TP + TN) / \text{total} for the safe or unsafe decision.
ECE: bins of tenths, mean of \lvert \text{conf} - \text{accuracy} \rvert weighted by bin size.
ASR: attack success rate. Report it with the query budget and the judge.

Sources

Alignment failure modes

Goodhart: a strong optimizer finds inputs where proxy and target differ.
Over-optimization: proxy reward climbs, gold reward peaks then falls as KL grows.
Sycophancy: agrees with the user's stated premise regardless of truth.
Trigger persistence: a backdoor survives the training meant to remove it.
Alignment faking: a compliance gap between monitored and unmonitored conditions.
Constitutional AI: critique and revise SFT, then RL from AI feedback.
AI control: trusted model T edits untrusted U. Human H audits ambiguous outputs.
PGR: (\text{fine-tuned} - \text{weak}) / (\text{ceiling} - \text{weak}). 1.0 is fully closed.

Sources

Defaults and numbers

Setting	Default	Note
block_confidence	0.85	pre-gen detector score that blocks
note_confidence	0.5	pre-gen score that adds a medium signal
buffer_cap	2	chunks the during-gen filter holds
n_bins	10	ECE bins
Leakage threshold	0.55	trigram overlap with the system prompt
max_words	800	the bounded-length rule
PAIR budget K	20	queries per goal
MSJ shot count	256	reliable attack on violent and deceitful content

Sources

Troubleshooting

Symptom	Cause	Fix
Regex misses a paraphrase	one pattern, no measurement	regex families, normalize pass, precision and recall
Agent obeys an email	filter checks only user input	label untrusted content, ratify its actions
Defense reports near zero ASR	static attacks only	evaluate with adaptive attacks
Model refuses chemistry homework	only harmful prompts measured	measure over-refusal on safe prompts
Prefix injection gets through	gate checks only after generation	add the during-gen stream filter
ASR numbers do not compare	different query budgets	match budgets, name the judge
Attack works at 256 shots	long context pattern following	classifier over the full context
Encoding attack passes input check	input moderation only	moderate input and output

Sources

Abbreviations

PAIR: Prompt Automatic Iterative Refinement. Attacker LLM and judge LLM loop.
GCG: white box gradient search for adversarial token suffixes.
TAP: Tree of attacks. PAIR with branching and pruning.
MSJ: many-shot jailbreak. Hundreds of fake compliance pairs in context.
ArtPrompt: masks safety words with ASCII art renderings.
IPI: indirect prompt injection. Payload in content the user did not write.
IFC: information flow control. Untrusted content needs trusted ratification.
ECE: Expected Calibration Error. Stated confidence versus accuracy.

Sources

Questions

Why a taxonomy first?: Labels turn an attack stream into a histogram and a coverage chart.
Why classify outputs too?: An attack family the input checks miss lands on the user.
Which action wins?: The maximum severity. Block wins. Redact and warn becomes redact.
Why is PAIR cheaper than GCG?: It is black box and learns from each rejection in context.
Why is MSJ hard to block?: It uses the same mechanism as in-context learning.

Sources

Tips

Log a taxonomy id with every finding.
Normalize first: strip zero-width characters, decode base64, rot13 and hex. Keep the raw prompt.
Keep the constitution in YAML under version control with its own review.
Put a guard classifier on both sides of the model.
Run single turn probes nightly and multi-turn campaigns before release.
Score each successful jailbreak with CVSS 4.0.

Sources

Original source

---
id: ai-safety
title: AI Safety and Red Teaming
code: AS
subtitle: Name the attack, detect it, measure refusals, gate the output and audit alignment. Phase 18 and 19 lessons.
hue: 340
sources: phases/19-capstone-projects/15-constitutional-safety-harness, phases/19-capstone-projects/82-jailbreak-taxonomy, phases/19-capstone-projects/83-prompt-injection-detector, phases/19-capstone-projects/84-refusal-evaluation, phases/19-capstone-projects/85-content-classifier-integration, phases/19-capstone-projects/86-constitutional-rules-engine, phases/19-capstone-projects/87-end-to-end-safety-gate, phases/18-ethics-safety-alignment/02-reward-hacking-goodhart, phases/18-ethics-safety-alignment/04-sycophancy-rlhf-amplification, phases/18-ethics-safety-alignment/05-constitutional-ai-rlaif, phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception, phases/18-ethics-safety-alignment/09-alignment-faking, phases/18-ethics-safety-alignment/10-ai-control-subversion, phases/18-ethics-safety-alignment/11-scalable-oversight-weak-to-strong, phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks, phases/18-ethics-safety-alignment/13-many-shot-jailbreaking, phases/18-ethics-safety-alignment/14-ascii-art-visual-jailbreaks, phases/18-ethics-safety-alignment/15-indirect-prompt-injection, phases/18-ethics-safety-alignment/16-red-team-tooling-garak-llamaguard-pyrit, phases/18-ethics-safety-alignment/29-moderation-systems-openai-perspective-llamaguard
source_hash: 2ab25bd9ba733aa2
---

## Three checkpoint safety gate
type: architecture
from: phases/19-capstone-projects/87-end-to-end-safety-gate/docs/en.md, phases/19-capstone-projects/87-end-to-end-safety-gate/code/safety_gate.py

```arch
node | in | 1,1 | User prompt |
node | pre | 2,1 | Pre-gen detector | rules on prompt
node | llm | 3,1 | Model stream | 4 token chunks
node | dur | 3,2 | During-gen filter | 2 chunk buffer
node | post | 2,2 | Post-gen checks | classifier, rules engine
node | agg | 1,2 | Aggregate | max severity
node | out | 1,3 | Final action | RequestTrace
edge | in | pre |
edge | pre | llm | pass
edge | llm | dur | stream
edge | dur | post | complete
edge | dur | agg | early stop
edge | post | agg |
edge | agg | out |
```

## PAIR attack loop
type: flow
from: phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks/docs/en.md

1. **Attacker A**: reads the goal G and the history of prompt and response pairs.
2. **Attacker A**: writes a new prompt $p_k$.
3. **Target T**: answers with the response $r_k$.
4. **Judge J**: scores $(p_k, r_k)$ against the goal.
5. **Halt**: a score at or above the threshold is a jailbreak.
6. **Repeat**: else append the pair to the history, up to budget K.

## Jailbreak taxonomy
type: command-table
from: phases/19-capstone-projects/82-jailbreak-taxonomy/docs/en.md

| Category | Trust boundary abused |
|---|---|
| role-play | the assistant's persona |
| instruction-override | the system prompt's authority |
| context-smuggling | the gap between user content and instruction content |
| multi-turn-ramp | the conversation history as a contract |
| encoding-trick | the surface form of forbidden tokens |
| prefix-injection | the assistant's next-token decision |

## Tasks in code
type: task-table
from: phases/19-capstone-projects/82-jailbreak-taxonomy/docs/en.md, phases/19-capstone-projects/83-prompt-injection-detector/code/rules.py, phases/19-capstone-projects/83-prompt-injection-detector/code/main.py, phases/19-capstone-projects/84-refusal-evaluation/code/main.py, phases/19-capstone-projects/87-end-to-end-safety-gate/code/safety_gate.py, phases/19-capstone-projects/87-end-to-end-safety-gate/code/main.py

| Task | Code |
|---|---|
| Run a capstone demo | `python3 main.py` |
| Add a detector rule | `{"name": "debug-mode", "category": "instruction-override", "score": 0.80, "substring": "debug mode"}` |
| Score a prompt | `v = self.detector.analyze(prompt)` |
| Try a rot13 decode | `candidate = codecs.encode(text, "rot_13")` |
| Spot a refusal | `classify_refusal(response)` |
| Route an output | `action = self.classifier_router.run(output)` |
| Fix rule violations | `redacted = self.rules_fixer.apply(redacted, violations)` |
| Run one request | `trace = gate.handle(prompt)` |

## Aggregate the signals
type: annotated-code
lang: python
from: phases/19-capstone-projects/87-end-to-end-safety-gate/code/safety_gate.py

```python
if pre.category != "benign":
    if pre.confidence >= self.block_confidence:        # 0.85 is high
        signals.append(("pre", SEVERITY_RANK["high"]))
    elif pre.confidence >= self.note_confidence:       # 0.5 is medium
        signals.append(("pre", SEVERITY_RANK["medium"]))
if during.terminated_early:                            # stream cut early
    signals.append(("during", SEVERITY_RANK["medium"]))
if post is not None:                                   # classifier and rules
    signals.append(("post.classifier", SEVERITY_RANK.get(post.classifier_severity, 0)))
max_rank = max(rank for _, rank in signals)
if max_rank >= SEVERITY_RANK["high"]:
    return "block"                                     # return a refusal
if max_rank >= SEVERITY_RANK["medium"]:
    return "redact"                                    # then warn on low, else allow
```

## Metrics
type: concepts
from: phases/19-capstone-projects/83-prompt-injection-detector/docs/en.md, phases/19-capstone-projects/83-prompt-injection-detector/code/main.py, phases/19-capstone-projects/84-refusal-evaluation/docs/en.md, phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks/docs/en.md

- **Precision**: $TP / (TP + FP)$, the fraction of fires that are correct.
- **Recall**: $TP / (TP + FN)$, the fraction of attacks the detector catches.
- **F1**: $2 P R / (P + R)$ per category.
- **Under-refusal**: the model answered a prompt labeled unsafe.
- **Over-refusal**: the model refused a prompt labeled safe.
- **Accuracy**: $(TP + TN) / \text{total}$ for the safe or unsafe decision.
- **ECE**: bins of tenths, mean of $\lvert \text{conf} - \text{accuracy} \rvert$ weighted by bin size.
- **ASR**: attack success rate. Report it with the query budget and the judge.

## Alignment failure modes
type: concepts
from: phases/18-ethics-safety-alignment/02-reward-hacking-goodhart/docs/en.md, phases/18-ethics-safety-alignment/04-sycophancy-rlhf-amplification/docs/en.md, phases/18-ethics-safety-alignment/05-constitutional-ai-rlaif/docs/en.md, phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception/docs/en.md, phases/18-ethics-safety-alignment/09-alignment-faking/docs/en.md, phases/18-ethics-safety-alignment/10-ai-control-subversion/docs/en.md, phases/18-ethics-safety-alignment/11-scalable-oversight-weak-to-strong/docs/en.md

- **Goodhart**: a strong optimizer finds inputs where proxy and target differ.
- **Over-optimization**: proxy reward climbs, gold reward peaks then falls as KL grows.
- **Sycophancy**: agrees with the user's stated premise regardless of truth.
- **Trigger persistence**: a backdoor survives the training meant to remove it.
- **Alignment faking**: a compliance gap between monitored and unmonitored conditions.
- **Constitutional AI**: critique and revise SFT, then RL from AI feedback.
- **AI control**: trusted model T edits untrusted U. Human H audits ambiguous outputs.
- **PGR**: $(\text{fine-tuned} - \text{weak}) / (\text{ceiling} - \text{weak})$. 1.0 is fully closed.

## Defaults and numbers
type: defaults
from: phases/19-capstone-projects/87-end-to-end-safety-gate/code/safety_gate.py, phases/19-capstone-projects/84-refusal-evaluation/code/main.py, phases/19-capstone-projects/85-content-classifier-integration/code/classifiers.py, phases/19-capstone-projects/86-constitutional-rules-engine/code/rules.yml, phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks/docs/en.md, phases/18-ethics-safety-alignment/13-many-shot-jailbreaking/docs/en.md

| Setting | Default | Note |
|---|---|---|
| block_confidence | 0.85 | pre-gen detector score that blocks |
| note_confidence | 0.5 | pre-gen score that adds a medium signal |
| buffer_cap | 2 | chunks the during-gen filter holds |
| n_bins | 10 | ECE bins |
| Leakage threshold | 0.55 | trigram overlap with the system prompt |
| max_words | 800 | the bounded-length rule |
| PAIR budget K | 20 | queries per goal |
| MSJ shot count | 256 | reliable attack on violent and deceitful content |

## Troubleshooting
type: troubleshooting
from: phases/19-capstone-projects/83-prompt-injection-detector/docs/en.md, phases/19-capstone-projects/84-refusal-evaluation/docs/en.md, phases/19-capstone-projects/87-end-to-end-safety-gate/docs/en.md, phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks/docs/en.md, phases/18-ethics-safety-alignment/13-many-shot-jailbreaking/docs/en.md, phases/18-ethics-safety-alignment/15-indirect-prompt-injection/docs/en.md, phases/18-ethics-safety-alignment/29-moderation-systems-openai-perspective-llamaguard/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| Regex misses a paraphrase | one pattern, no measurement | regex families, normalize pass, precision and recall |
| Agent obeys an email | filter checks only user input | label untrusted content, ratify its actions |
| Defense reports near zero ASR | static attacks only | evaluate with adaptive attacks |
| Model refuses chemistry homework | only harmful prompts measured | measure over-refusal on safe prompts |
| Prefix injection gets through | gate checks only after generation | add the during-gen stream filter |
| ASR numbers do not compare | different query budgets | match budgets, name the judge |
| Attack works at 256 shots | long context pattern following | classifier over the full context |
| Encoding attack passes input check | input moderation only | moderate input and output |

## Abbreviations
type: glossary
from: phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks/docs/en.md, phases/18-ethics-safety-alignment/13-many-shot-jailbreaking/docs/en.md, phases/18-ethics-safety-alignment/14-ascii-art-visual-jailbreaks/docs/en.md, phases/18-ethics-safety-alignment/15-indirect-prompt-injection/docs/en.md, phases/19-capstone-projects/84-refusal-evaluation/docs/en.md

- **PAIR**: Prompt Automatic Iterative Refinement. Attacker LLM and judge LLM loop.
- **GCG**: white box gradient search for adversarial token suffixes.
- **TAP**: Tree of attacks. PAIR with branching and pruning.
- **MSJ**: many-shot jailbreak. Hundreds of fake compliance pairs in context.
- **ArtPrompt**: masks safety words with ASCII art renderings.
- **IPI**: indirect prompt injection. Payload in content the user did not write.
- **IFC**: information flow control. Untrusted content needs trusted ratification.
- **ECE**: Expected Calibration Error. Stated confidence versus accuracy.

## Questions
type: qa
from: phases/19-capstone-projects/82-jailbreak-taxonomy/docs/en.md, phases/19-capstone-projects/85-content-classifier-integration/docs/en.md, phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks/docs/en.md, phases/18-ethics-safety-alignment/13-many-shot-jailbreaking/docs/en.md

- **Why a taxonomy first?** Labels turn an attack stream into a histogram and a coverage chart.
- **Why classify outputs too?** An attack family the input checks miss lands on the user.
- **Which action wins?** The maximum severity. Block wins. Redact and warn becomes redact.
- **Why is PAIR cheaper than GCG?** It is black box and learns from each rejection in context.
- **Why is MSJ hard to block?** It uses the same mechanism as in-context learning.

## Tips
type: tips
from: phases/19-capstone-projects/82-jailbreak-taxonomy/docs/en.md, phases/19-capstone-projects/83-prompt-injection-detector/docs/en.md, phases/19-capstone-projects/86-constitutional-rules-engine/docs/en.md, phases/19-capstone-projects/15-constitutional-safety-harness/docs/en.md, phases/18-ethics-safety-alignment/15-indirect-prompt-injection/docs/en.md, phases/18-ethics-safety-alignment/16-red-team-tooling-garak-llamaguard-pyrit/docs/en.md

- Log a taxonomy id with every finding.
- Normalize first: strip zero-width characters, decode base64, rot13 and hex. Keep the raw prompt.
- Keep the constitution in YAML under version control with its own review.
- Put a guard classifier on both sides of the model.
- Run single turn probes nightly and multi-turn campaigns before release.
- Score each successful jailbreak with CVSS 4.0.