three algorithms, same token budget
BPE — greedy merge
l o w e r
merge most frequent pair
lo w e r
low er
lower
used by: GPT, Llama, Gemma,
Qwen, Mistral
deterministic at inference
Unigram — probabilistic prune
start: 50k candidates
score each by unigram p
prune lowest-loss tokens
repeat to target size
"lower" → {lo+wer, l+ower, low+er}
used by: T5, Gemma, mBART,
ALBERT, XLNet
samplable (subword regularization)
WordPiece — max likelihood
l ##o ##w ##e ##r
merge pair that maximizes
P(ab) / (P(a)·P(b))
l ##ow ##er
l ##ower
used by: BERT, DistilBERT,
ELECTRA
## marks continuation
tokenize once, ship forever — vocabulary is frozen at training time