GloVe
co-occurrence matrix factorization
X[i][j] = count(word j near i)
train so v_i · v_j + b_i + b_j
≈ log(X[i][j])
the cat dog sat
the 0 5 4 6
cat 5 0 1 3
dog 4 1 0 2
sat 6 3 2 0
one big matrix, factorize once
FastText
word = sum of char n-grams
"where"
= <wh + whe + her
+ ere + re>
+ <whe + wher + here
+ ere> + <where>
"whereupon" shares
<wh, whe, her, ere
with "where"
→ sensible vector even for
OOV words
handles new and rare words
BPE
learned subword vocabulary
start: bytes or chars
iterate:
find most-common pair
merge it, add to vocab
after training:
"unbelievably"
→ un + bel + iev + ably
"tokenization"
→ token + ization
every LLM tokenizer uses this