encoder-decoder — cross-attention is the only pipe
source tokens (N_src)
encoder (bidirectional)
block 1: self-attn + FFN
block 2: self-attn + FFN
block L: self-attn + FFN
encoder output (N_src, d)
target tokens (shifted right)
decoder (causal + cross-attn)
masked self-attention
cross-attention ◀── enc_out
FFN
(repeat L times)
linear → vocab
next-token logits
cross-attn:
Q from decoder,
K/V from encoder
T5 pretraining
span corruption: replace spans with sentinels,
decode only the spans
BART pretraining
apply noise (mask, delete, infill, shuffle),
decoder reconstructs the whole clean sequence