three ways to tell attention about position
sinusoidal (2017)
add sin/cos to the embedding
dim 0: fast sine
dim 64: slow sine
X' = X + PE
does not extrapolate beyond
trained positions
history only
RoPE (2021) — dominant 2026
rotate Q,K by position angle
q_m rotated by m·θ
k_n rotated by n·θ
<q'_m, k'_n> = f(m − n)
relative distance, for free
base knob extends context
Llama, Qwen, Mistral,
DeepSeek, Kimi
ALiBi (2022)
linear bias in attention scores
score -= m · |i − j|
darker = smaller penalty
extrapolates cleanly
BLOOM, MPT,
long-context ablations
attention is permutation-invariant; pick the injection that matches your context length strategy