Video VLMs — frame sampling, temporal tokens, grounding output
three architecture patterns from 2023 to 2025
Video-LLaMA (2023)
Q-former + audio branch
16 frames @ 2 FPS fixed
32 video queries, 32 audio
strength: audio grounding
weakness: 8s fixed clip
no event time localization
Video-LLaVA (2023)
MLP + shared encoder
8 frames @ 1-2 FPS
alignment before projection
strength: simple + effective
weakness: short clips only
no dynamic FPS
Qwen2.5-VL (2025)
TMRoPE + dynamic FPS
arbitrary duration
absolute-time M-RoPE
strength: event grounding
JSON output format
open SOTA 2026
frame sampling + output format
frame sampling strategies
uniform: N frames / duration
- simple, loses motion peaks
dynamic FPS: motion-weighted
- denser in high-motion spans
event-driven: detector + sample
- best for action recognition
pair with 3x3 pooling per frame
grounding output formats
free text:
"The cat jumps around 4s"
JSON:
{"event":"jump","start":4.1,"end":4.3}
token:
"<time>4.1</time> jump"
JSON is easiest to parse downstream
benchmarks
VideoMME: general, 2500 samples
TempCompass: before/after
EgoSchema: 3min first-person
Video-MMMU: multi-discipline
open SOTA 2026
Qwen2.5-VL-72B
TMRoPE is the differentiator