differential attention — subtract the noise floor
hidden state x
Q1 = xWq1 · K1 = xWk1
Q2 = xWq2 · K2 = xWk2
A1 = softmax(Q1 K1^T / sqrt(d))
A2 = softmax(Q2 K2^T / sqrt(d))
A = A1 - lambda * A2
out = A V
the intuition: both softmaxes carry the same uniform noise floor
A1
signal
A2
no signal
A1 - lambda*A2
signal survives
V1 (ICLR 2025) -> V2 (Microsoft, Jan 2026)
head dim
V1: halved (preserve param count) · V2: unchanged (match baseline decode)
query heads
V1: same as baseline · V2: doubled, KV heads unchanged
kernel
V1: custom CUDA · V2: plain FlashAttention — zero code change
per-head norm
V1: RMSNorm after subtraction (unstable at 70B) · V2: removed
decode speed
V1: slower (V cache loaded twice) · V2: matches baseline exactly
ICLR 2025 experiment -> production pre-training. same math, production-ready packaging.