native sparse attention — three branches
query q
1 / compressed branch
mean-pool every l tokens into
summary; attend over N/l keys
2 / selected branch
take top-k compressed blocks
attend over k * l uncompressed keys
3 / sliding window
attend over last W tokens for
local syntax and coreference
compressed scores pick top-k blocks
gate MLP
g_cmp, g_sel, g_win
from query q
combine
g_cmp * o_cmp
+ g_sel * o_sel
+ g_win * o_win
keys attended per query vs full attention
N=4k, l=32, k=8, W=256
NSA 640 · full 4,096 · 6.4x fewer
N=64k, l=64, k=16, W=512
NSA 2,560 · full 65,536 · 25.6x fewer
N=128k, l=64, k=16, W=512
NSA 3,584 · full 131,072 · 36.6x fewer