Multimodal RAG — cross-modal retrieve, fuse, ground, generate
query -> decompose -> 3 retrievers -> fuse -> VLM generator
query
"quiet vegan brunch
with natural light"
text retriever
reviews / menus
image retriever
CLIP / SigLIP photos
audio retriever
CLAP ambient clips
score fusion
weighted sum
or MoE gate
top-k candidates
VLM generator
Qwen2.5-VL / Claude
grounded citations
per source
the three surveys of 2025
Abootorabi et al.
arXiv:2502.08826
comprehensive taxonomy
retrieval / fusion / generation
broadest coverage
start here if new
names all subproblems
Mei et al.
arXiv:2504.08748
sub-task benchmarks
failure modes cataloged
useful for eval design
read for evals
per-metric decomposition
Zhao et al.
arXiv:2503.18016
vision-focused RAG
strong on ColPali-family
visual-only emphasis
read for vision RAG
complements lesson 23