Long-video understanding — four scaling paths
Path 1: brute context
Gemini 1.5 Pro: 1M tokens
Gemini 2.5 Pro: 10M+ tokens
Claude Opus 4.7: 1M tokens
engineering
custom attention hierarchy
MoE expert routing
closed-source
best recall, closed only
Path 2: ring attention
LWM: 1M-token training
LongVILA: 1400-frame videos
distributed ring pattern
engineering
each device holds chunk
rotates for attention passes
open-source
good open scaling, heavy compute
Path 3: token compression
Video-XL: one summary token
per clip (100s of frames -> 1)
LongVA: long-context transfer
VideoChat2: hierarchical pool
engineering
learned compressor pre-LLM
trades recall for scale
~32k context sufficient
cheapest inference, weakest grounding
Path 4: agentic retrieval
VideoAgent: LLM as query planner
tool: find_clips(keyword)
VLM reads only matches
LLM composes final answer
engineering
retrieval quality is the bottleneck
99% cheaper for single-event queries
worse for holistic understanding
best for 2h+ specific queries