ColPali — vision-native document RAG with ColBERT late interaction
indexing: PDF -> page images -> patch embeddings (cached)
PDF pages
figures, tables,
layout, fonts
all preserved
no OCR
no text extraction
no chunking
PaliGemma / Qwen-VL
vision-language encoder
each patch -> D-dim vector
N_p vectors per page
storage
729 x 128 x 4 = 365 KB
PQ compressed ~46 KB
vector index (all pages)
50-page report -> ~2 MB PQ
1M pages -> ~45 GB PQ
stays on one node
ready for MaxSim query
one table, multi-vector
per-page grouped
retrieval: MaxSim over query tokens and page patches
query tokens
"Q3 revenue growth"
4-6 token embeddings
same encoder as index
compute at query time
batch with kv cache
~10 ms per query
MaxSim = sum over query tokens
of max over patches
each query token "picks" its best patch
selective: irrelevant patches ignored
doc score = sum of those maxes
beats bi-encoder recall
catches chart data, table cells,
figure captions as patches
top-k pages + VLM answer
send top-3 page images
+ original query
to Qwen2.5-VL or Claude
benchmarks
ViDoRe nDCG@5: ColPali ~80
text-RAG ~55
end-to-end +20-40 pts