needle-in-haystack grid — depth × length pass rate
NIAH pass-rate heatmap
4k
16k
64k
256k
1M
depth 0.1
depth 0.3
depth 0.5
depth 0.7
depth 0.9
pass ≥ 90%
partial 60-90%
fail < 60%
middle depths degrade first — "lost in the middle"
beyond NIAH
RULER — 13 tasks: multi-needle, variable
tracing, aggregation, QA. Exposes models
that saturate NIAH but fail multi-hop.
LongBench v2 — 503 multiple-choice Qs,
8k–2M context, six real-world categories.
MRCR 8-needle — Gemini 3 Pro advertises
10M; scores 77% at 128k, 26.3% at 1M.
NoLiMa / BABILong — non-lexical needle
and reasoning-in-haystack — hardest variants.
report two numbers, not one:
effective retrieval length & effective reasoning length
(usually 25-50% of advertised window)