masked language modeling — the 80/10/10 rule
sentence
the
quick
brown
fox
jumps
over
the
dog
after masking
the
[MASK]
brown
[MASK]
jumps
cat
the
dog
[MASK] = 80% path
random swap = 10% path ("cat" picked in place of "over")
kept as-is = 10% path
BERT encoder (bidirectional self-attention × 12 layers)
every position sees every other. predicting "quick" uses "brown fox jumps".
modernbert 2024: RoPE + RMSNorm + GeGLU + alternating local/global attention + 8k context
output: per-position 768-d vectors
MLM head: Linear(d_model → vocab)
cross-entropy, only at selected positions
no signal at unselected positions — IGNORE_INDEX is a PyTorch convention for this exact use