BERT, GPT and T5 Compared · Ingénierie IA

Encoder-only, decoder-only and encoder-decoder side by side. Objectives, masks, blocks, decoding and when to pick each.

Three families side by side

Factor	BERT	GPT	T5 / BART
Stack	Encoder only	Decoder only	Encoder and decoder
Attention	Bidirectional, no mask	Causal triangle mask	Bidirectional encoder, causal decoder
Cross-attention	None	None	Q from decoder, K/V from encoder
Objective	MLM on 15% of tokens	Next token, shift by one	Span corruption or denoising
Predicts	P(x_{\text{masked}} \mid x_{\text{unmasked}})	P(x_t \mid x_{<t})	Target spans or the clean sequence
Generates text	No	Yes, token by token	Yes, from the encoder output
Best for	Classification, retrieval, NER	Chat, code, reasoning	Translation, speech to text
Examples	BERT, DeBERTa, ModernBERT	GPT, Llama, Qwen	T5, BART, Whisper

Sources

Block wiring

Nodes and connections

Node	Role	Kind
Source tokens	embed + position	client
Encoder block	LN, self MHA, LN, FFN	process
Encoder output	(N_src, d_model), runs once	store
Target tokens	shift by one	client
Masked self-attention	sublayer 1	process
Cross-attention	sublayer 2	process
FFN	sublayer 3	process
Next-token logits	final LN + projection	model

From	To	Connection
Source tokens	Encoder block
Encoder block	Encoder output
Target tokens	Masked self-attention
Masked self-attention	Cross-attention
Encoder output	Cross-attention	K, V
Cross-attention	FFN
FFN	Next-token logits

Sources

One encoder-decoder forward pass

Embed tokens become vectors. Add RoPE or a sinusoidal position signal.
Encoder each block runs pre-norm self-attention, then pre-norm FFN, each with a residual.
Encoder output the encoder runs once per input. Cache it for long inputs.
Decoder self-attention masked, so position i sees positions up to i only.
Decoder cross-attention the only place encoder information enters the decoder.
Decoder FFN then a final LN before the output projection.
Check a 6-token source and a 5-token target give output shape (5, vocab).

Sources

Tasks in the lesson code

Task	Code
BERT 80/10/10 masking	`create_mlm_batch(tokens, vocab_size)`
Mask whole words	`whole_word_mlm(tokens, spans, vocab_size)`
GPT causal mask	`causal_mask(n)`
Next-token loss	`cross_entropy_shifted(logits, target_ids)`
T5 span corruption	`corrupt_spans(tokens, mask_rate=0.15)`
Rebuild the original	`round_trip(source, target)`
One encoder block	`encoder_block(x, p)`
One decoder block	`decoder_block(x, enc_out, p)`

Sources

Load each family

tok = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")

model = AutoModel.from_pretrained("answerdotai/ModernBERT-base")

out = model(**inputs).last_hidden_state(1, N, 768), embeddings only

model = AutoModelForCausalLM.from_pretrained("<model>")

out = model.generate(**inputs, max_new_tokens=64,

temperature=0.7, top_p=0.9, do_sample=True)sample a reply

model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-base")

inputs = tok("translate English to French: <text>", return_tensors="pt")task name in the input

out = model.generate(**inputs, max_new_tokens=32)

print(tok.decode(out[0], skip_special_tokens=True))

Sources

Pretraining objectives

MLM: select 15% of tokens. 80% become [MASK], 10% random, 10% unchanged.
Labels: the original token at selected positions. -100 at all other positions.
NSP: predict if sentence B follows A. RoBERTa showed it hurt. Modern encoders skip it.
Causal LM: input [t1, t2, t3], targets [t2, t3, t4]. Sum -\log P over positions.
Causal mask: M_{ij} = 0 if j \le i, else -\infty. Add it before softmax.
Span corruption: spans of mean length 3, 15% total, each replaced by <extra_id_0>, <extra_id_1>.
T5 target: only the corrupted spans, each after its sentinel.
BART: five noise functions. The decoder rebuilds the full original sequence.

Sources

Decoding strategies

Command	What it does
`sample_greedy(probs)`	Argmax each step. Code completion, deterministic tasks
`sample_temperature(logits, t, rng)`	Divide logits by T. Higher T gives more diversity
`sample_top_k(logits, k, rng)`	Sample from the top k tokens only
`sample_top_p(logits, p, rng)`	Smallest set with cumulative prob at least p
`sample_min_p(logits, min_p, rng)`	Keep tokens with `p >= min_p * max_p`
Beam search	Width 4 to 5 for translation and summarization
Speculative decoding	Draft model proposes N tokens. Big model verifies

Sources

Block: 2017 against 2026

Component	2017 block	2026 block
Normalization	LayerNorm, post-norm	RMSNorm, pre-norm
FFN activation	ReLU	SwiGLU
FFN expansion	4×	2.6× (three matrices)
Position	Sinusoidal absolute	RoPE
Attention	Full MHA	GQA (or MLA)
Bias terms	Yes	No

Sources

phases/07-transformers-deep-dive/05-full-transformer/docs/en.md

Defaults in the code

Setting	Default	Note
`mask_prob`	0.15	BERT selection rate
`IGNORE_INDEX`	-100	Label at unselected positions
`mask_rate`	0.15	T5 share of tokens in spans
`mean_span`	3.0	T5 mean span length
`mask_token`	`<mask>`	BART mask and infill token
`layer_norm` eps	1e-5	Subtracts the mean
`rms_norm` eps	1e-6	No mean subtraction
`temperature`	1.0	`softmax`, top-k, top-p, min-p

Sources

Use each family when

Use BERT for classification, embeddings, NER, NLI and RAG rerankers. It runs 5 to 10× faster per token than decoders.
Do not use BERT for generation. The encoder cannot produce tokens autoregressively.
Use GPT for generation, chat, code and reasoning. Decoder-only scales cleanest.
Use min-p + temperature 0.7 as a default for open-weights models.
Use T5 or BART when the input is a clear source sequence: translation, speech, structured tasks.
Use beam search with an encoder-decoder. The output distribution is narrower than chat.
Summarization and extraction work with either encoder-decoder or decoder-only.

Sources

Troubleshooting

Symptom	Cause	Fix
Model fails without `[MASK]` at fine-tune	Trained on 100% `[MASK]`	Use the 80/10/10 rule
Model cheats by looking at the answer	No causal mask	Add the `-inf` triangle before softmax
Future weights not exactly zero	Softmax never outputs an exact zero	Put `-inf` into future scores
Gradients vanish in a deep stack	No residual connections	Use `x + sublayer(x)`
Activations explode after 12 layers	Post-norm	Switch to pre-norm
Corrupted span does not rebuild	Off-by-one in span bookkeeping	Run the round-trip check
Slow decode on long inputs	Encoder runs again each step	Cache the encoder output

Sources

Questions

Why can BERT use the right context?: The encoder has no causal mask.
Why is GPT training parallel?: One pass gives N next-token losses. Inference stays serial.
How many params in one block?: MHA is 4 · d². SwiGLU FFN is about 3rd².
Why is T5 pretraining cheaper than BART?: T5 decodes only the spans. BART decodes the full sequence.
Are embedding models BERT?: Yes. Same encoder, contrastive loss.

Sources

Original source

---
id: bert-gpt-t5
title: BERT, GPT and T5 Compared
code: BG
subtitle: Encoder-only, decoder-only and encoder-decoder side by side. Objectives, masks, blocks, decoding and when to pick each.
hue: 32
sources: phases/07-transformers-deep-dive/05-full-transformer, phases/07-transformers-deep-dive/06-bert-masked-language-modeling, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder
source_hash: 074f078a03208c41
---

## Three families side by side
type: comparison
from: phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, phases/07-transformers-deep-dive/06-bert-masked-language-modeling/docs/en.md, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/docs/en.md, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

| Factor | BERT | GPT | T5 / BART |
|---|---|---|---|
| Stack | Encoder only | Decoder only | Encoder and decoder |
| Attention | Bidirectional, no mask | Causal triangle mask | Bidirectional encoder, causal decoder |
| Cross-attention | None | None | Q from decoder, K/V from encoder |
| Objective | MLM on 15% of tokens | Next token, shift by one | Span corruption or denoising |
| Predicts | $P(x_{\text{masked}} \mid x_{\text{unmasked}})$ | $P(x_t \mid x_{<t})$ | Target spans or the clean sequence |
| Generates text | No | Yes, token by token | Yes, from the encoder output |
| Best for | Classification, retrieval, NER | Chat, code, reasoning | Translation, speech to text |
| Examples | BERT, DeBERTa, ModernBERT | GPT, Llama, Qwen | T5, BART, Whisper |

## Block wiring
type: architecture
from: phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

```arch
node | src | 1,1 | Source tokens | embed + position | client
node | enc | 2,1 | Encoder block | LN, self MHA, LN, FFN | process
node | encout | 3,1 | Encoder output | (N_src, d_model), runs once | store
node | tgt | 1,2 | Target tokens | shift by one | client
node | self | 2,2 | Masked self-attention | sublayer 1 | process
node | cross | 3,2 | Cross-attention | sublayer 2 | process
node | ffn | 4,2 | FFN | sublayer 3 | process
node | logits | 4,1 | Next-token logits | final LN + projection | model
edge | src | enc |
edge | enc | encout |
edge | tgt | self |
edge | self | cross |
edge | encout | cross | K, V
edge | cross | ffn |
edge | ffn | logits |
```

## One encoder-decoder forward pass
type: flow
from: phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, phases/07-transformers-deep-dive/05-full-transformer/code/main.py, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

1. **Embed**: tokens become vectors. Add RoPE or a sinusoidal position signal.
2. **Encoder**: each block runs pre-norm self-attention, then pre-norm FFN, each with a residual.
3. **Encoder output**: the encoder runs once per input. Cache it for long inputs.
4. **Decoder self-attention**: masked, so position `i` sees positions up to `i` only.
5. **Decoder cross-attention**: the only place encoder information enters the decoder.
6. **Decoder FFN**: then a final LN before the output projection.
7. **Check**: a 6-token source and a 5-token target give output shape `(5, vocab)`.

## Tasks in the lesson code
type: task-table
from: phases/07-transformers-deep-dive/05-full-transformer/code/main.py, phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py

| Task | Code |
|---|---|
| BERT 80/10/10 masking | `create_mlm_batch(tokens, vocab_size)` |
| Mask whole words | `whole_word_mlm(tokens, spans, vocab_size)` |
| GPT causal mask | `causal_mask(n)` |
| Next-token loss | `cross_entropy_shifted(logits, target_ids)` |
| T5 span corruption | `corrupt_spans(tokens, mask_rate=0.15)` |
| Rebuild the original | `round_trip(source, target)` |
| One encoder block | `encoder_block(x, p)` |
| One decoder block | `decoder_block(x, enc_out, p)` |

## Load each family
type: annotated-code
lang: python
from: phases/07-transformers-deep-dive/06-bert-masked-language-modeling/docs/en.md, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/docs/en.md, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

```python
tok = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = AutoModel.from_pretrained("answerdotai/ModernBERT-base")
out = model(**inputs).last_hidden_state   # (1, N, 768), embeddings only
model = AutoModelForCausalLM.from_pretrained("<model>")
out = model.generate(**inputs, max_new_tokens=64,
    temperature=0.7, top_p=0.9, do_sample=True)   # sample a reply
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-base")
inputs = tok("translate English to French: <text>", return_tensors="pt")   # task name in the input
out = model.generate(**inputs, max_new_tokens=32)
print(tok.decode(out[0], skip_special_tokens=True))
```

## Pretraining objectives
type: concepts
from: phases/07-transformers-deep-dive/06-bert-masked-language-modeling/docs/en.md, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/docs/en.md, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

- **MLM**: select 15% of tokens. 80% become `[MASK]`, 10% random, 10% unchanged.
- **Labels**: the original token at selected positions. -100 at all other positions.
- **NSP**: predict if sentence B follows A. RoBERTa showed it hurt. Modern encoders skip it.
- **Causal LM**: input `[t1, t2, t3]`, targets `[t2, t3, t4]`. Sum $-\log P$ over positions.
- **Causal mask**: $M_{ij} = 0$ if $j \le i$, else $-\infty$. Add it before softmax.
- **Span corruption**: spans of mean length 3, 15% total, each replaced by `<extra_id_0>`, `<extra_id_1>`.
- **T5 target**: only the corrupted spans, each after its sentinel.
- **BART**: five noise functions. The decoder rebuilds the full original sequence.

## Decoding strategies
type: command-table
from: phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/docs/en.md, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

| Command | What it does |
|---|---|
| `sample_greedy(probs)` | Argmax each step. Code completion, deterministic tasks |
| `sample_temperature(logits, t, rng)` | Divide logits by T. Higher T gives more diversity |
| `sample_top_k(logits, k, rng)` | Sample from the top k tokens only |
| `sample_top_p(logits, p, rng)` | Smallest set with cumulative prob at least p |
| `sample_min_p(logits, min_p, rng)` | Keep tokens with `p >= min_p * max_p` |
| Beam search | Width 4 to 5 for translation and summarization |
| Speculative decoding | Draft model proposes N tokens. Big model verifies |

## Block: 2017 against 2026
type: comparison
from: phases/07-transformers-deep-dive/05-full-transformer/docs/en.md

| Component | 2017 block | 2026 block |
|---|---|---|
| Normalization | LayerNorm, post-norm | RMSNorm, pre-norm |
| FFN activation | ReLU | SwiGLU |
| FFN expansion | 4× | 2.6× (three matrices) |
| Position | Sinusoidal absolute | RoPE |
| Attention | Full MHA | GQA (or MLA) |
| Bias terms | Yes | No |

## Defaults in the code
type: defaults
from: phases/07-transformers-deep-dive/05-full-transformer/code/main.py, phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py

| Setting | Default | Note |
|---|---|---|
| `mask_prob` | 0.15 | BERT selection rate |
| `IGNORE_INDEX` | -100 | Label at unselected positions |
| `mask_rate` | 0.15 | T5 share of tokens in spans |
| `mean_span` | 3.0 | T5 mean span length |
| `mask_token` | `<mask>` | BART mask and infill token |
| `layer_norm` eps | 1e-5 | Subtracts the mean |
| `rms_norm` eps | 1e-6 | No mean subtraction |
| `temperature` | 1.0 | `softmax`, top-k, top-p, min-p |

## Use each family when
type: tips
from: phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, phases/07-transformers-deep-dive/06-bert-masked-language-modeling/docs/en.md, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/docs/en.md, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

- Use BERT for classification, embeddings, NER, NLI and RAG rerankers. It runs 5 to 10× faster per token than decoders.
- Do not use BERT for generation. The encoder cannot produce tokens autoregressively.
- Use GPT for generation, chat, code and reasoning. Decoder-only scales cleanest.
- Use min-p + temperature 0.7 as a default for open-weights models.
- Use T5 or BART when the input is a clear source sequence: translation, speech, structured tasks.
- Use beam search with an encoder-decoder. The output distribution is narrower than chat.
- Summarization and extraction work with either encoder-decoder or decoder-only.

## Troubleshooting
type: troubleshooting
from: phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, phases/07-transformers-deep-dive/06-bert-masked-language-modeling/docs/en.md, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/docs/en.md, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| Model fails without `[MASK]` at fine-tune | Trained on 100% `[MASK]` | Use the 80/10/10 rule |
| Model cheats by looking at the answer | No causal mask | Add the `-inf` triangle before softmax |
| Future weights not exactly zero | Softmax never outputs an exact zero | Put `-inf` into future scores |
| Gradients vanish in a deep stack | No residual connections | Use `x + sublayer(x)` |
| Activations explode after 12 layers | Post-norm | Switch to pre-norm |
| Corrupted span does not rebuild | Off-by-one in span bookkeeping | Run the round-trip check |
| Slow decode on long inputs | Encoder runs again each step | Cache the encoder output |

## Questions
type: qa
from: phases/07-transformers-deep-dive/05-full-transformer/docs/en.md, phases/07-transformers-deep-dive/06-bert-masked-language-modeling/docs/en.md, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/docs/en.md, phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md

- **Why can BERT use the right context?** The encoder has no causal mask.
- **Why is GPT training parallel?** One pass gives N next-token losses. Inference stays serial.
- **How many params in one block?** MHA is `4 · d²`. SwiGLU FFN is about `3rd²`.
- **Why is T5 pretraining cheaper than BART?** T5 decodes only the spans. BART decodes the full sequence.
- **Are embedding models BERT?** Yes. Same encoder, contrastive loss.