whisper — encoder-decoder for speech recognition
audio @ 16 kHz, 30 s window
log-mel: 80 bins × 3000 frames
conv1d × 2
stride 2 → 1500
encoder transformer (24 layers for large)
self-attention + GELU FFN + sinusoidal positional encoding
output: 1500 × 1280 hidden states
<|startoftranscript|> <|en|> <|transcribe|>
task tokens — 4-token prefix controls behavior
decoder transformer
masked self-attn + cross-attn to encoder + FFN
32 layers (large) or 4 layers (turbo)
BPE vocabulary: GPT-2 + audio specials
beam search width 5 at inference
cross-attn
"attention is all you need"
encoder runs once. decoder runs autoregressively. task token decides transcribe vs translate.