three ways to map T audio frames to U tokens
CTC — frame-level + blank
log-mel → encoder
per-frame distribution
argmax per frame:
a a _ _ a b b _ c
collapse + unblank → "aabc"
non-AR, streamable
no internal LM
wav2vec 2.0, MMS
LibriSpeech-clean: ~1.9 WER
RNN-T — encoder + predictor
audio encoder (frames)
predictor: prev tokens →
state
joiner → P(next | frame, history)
emit token or skip frame
streamable, internal LM
complex 3D loss lattice
Parakeet, Riva, Google on-device
LibriSpeech-clean: 1.40 WER (SOTA 2026)
attention — seq2seq
audio encoder (6-32L)
decoder cross-attends
autoregressive tokens:
BOS → hello → world → EOS
timestamps from attention
highest offline quality
non-streamable by default
Whisper v3, SeamlessM4T
LibriSpeech-clean: 1.58 WER