Whisper: one architecture, many tasks, steered by prompt
audio encoder
30 s → 80 log-mel, 3000 fr
conv stride-2
N × transformer blocks
Large-v3: N=32
cross-attn
text decoder
SOT → <|en|> → <|transcribe|>
→ <|notimestamps|> → tokens → EOT
N × transformer blocks (causal)
Large: N=32 Turbo: N=4 → 8× faster
51,865-BPE softmax
prompt-based task
<|en|> <|transcribe|>
→ English transcript
<|fr|> <|translate|>
→ FR speech → EN text
<|ja|> <|transcribe|>
→ Japanese transcript
no_timestamps / timestamps
→ word-level alignment on
long-form = chunk 30 s + overlap
0–30
25–55
50–80
75–105
5 s stride overlap; VAD-gate silence to prevent hallucinations
LoRA fine-tune (q_proj + v_proj, r=16)
full-FT Turbo-809M → 809M trainable
LoRA-r16 Turbo → 0.65M trainable
LoRA-r16 Large-v3 → 5.2M trainable
LoRA-r16 Medium → 3.1M trainable
freeze encoder if <10 h; tune decoder only
2–20 h gets 2–5× WER drop in-domain