Video-Language Models and Long Video · AI Engineering

Sample frames, budget tokens, encode time, ground events and index long video with timestamped answers.

Scene index pipeline (capstone)

Nodes and connections

Node	Role	Kind
Video file	YouTube URL or local MP4	client
Scene segmentation	TransNetV2 or PySceneDetect	process
VLM caption	caption + frame embedding	model
Multi-vector Qdrant	three named vectors	store
ASR pass	Whisper-v3-turbo, word timestamps	model
Question	natural language	client
RRF merge	top-k=5 scenes	process
Temporal grounding	refine start, end in scene	process
VLM synth	answer + timestamps	model

From	To	Connection
Video file	Scene segmentation
Scene segmentation	VLM caption	keyframe
Scene segmentation	ASR pass	audio
VLM caption	Multi-vector Qdrant
ASR pass	Multi-vector Qdrant
Question	RRF merge
Multi-vector Qdrant	RRF merge	three queries
RRF merge	Temporal grounding	top scene
Temporal grounding	VLM synth	window

Sources

phases/19-capstone-projects/12-video-understanding-pipeline/docs/en.md

Video VLM designs

Factor	Video-LLaMA	Video-LLaVA	Qwen2.5-VL
Frames	16-frame clips at 2 FPS	8 frames	Dynamic FPS
Compressor	Video Q-former, 32 queries	MLP	time-aligned M-RoPE per token
Time position	1D temporal RoPE	Learned temporal embeddings	Absolute timestamp (t, h, w)
Note	Audio branch, ImageBind	Shared image+video encoder	Best open 2025
Answer to "when"	"early in the clip"	8-16 frame system	"at 4.2 seconds"
Weakness	Fixed clip length	No long video	None stated

Sources

Long video: four paths

Factor	Brute context	Ring attention	Token compression	Agentic retrieval
Example	Gemini 1.5, Gemini 2.5 Pro	LWM, LongVILA	Video-XL, LongVA	VideoAgent
Mechanism	All tokens in one pass	Chunks rotate across devices	One summary token per clip	LLM queries a clip tool
Cost	Millions of tokens	Linear in context	Small context	Only relevant clips
Weakness	Closed, not open-source	Spreads over many devices	Misses exact frames	Retrieval quality
Pick for	>1h general QA	15-60 min	15-60 min	>1h specific query

Sources

VideoAgent loop

LLM reads the question.
LLM calls find_clips(keyword='<word>') for relevant clips.
Tool returns matching clip timestamps.
VLM encodes 3 x 30s clips (~7290 tokens total).
LLM composes the answer or asks follow-up queries.

Sources

Token math

Budget: tokens(duration_s, fps, per_frame) is duration times FPS times tokens per frame.
Raw: 1 minute at 30 FPS and 196 tokens per frame is 352k tokens.
3x3 pool: 64 tokens per frame. 5 minutes at 1 FPS is 19,200 tokens.
384 native: ~729 tokens per frame. 3x3 pooling gives 81.
30 min: 1 FPS at 81 tokens is 145,800 tokens. 2 FPS is 291,600.
2 hours: 1 FPS at 81 tokens is 583k tokens.
Retrieval: 3 * 30 * 81 + 200 tokens for 3 clips. 99% cheaper.

Sources

Video model families (vision)

2D + pool: 2D CNN per frame, average over time. Cannot model motion.
3D conv: kernels over (T, H, W). C3D, I3D, SlowFast.
I3D inflation: copy a 2D kernel along time. 3x3 becomes 3x3x3.
(2+1)D: 2D spatial conv, then 1D temporal conv. Fewer parameters.
Joint attention: one attention over (t, h, w). Quadratic in T*H*W.
Divided attention: one attention over time, one over space, per block.
Factorised attention: time and space attention alternate across blocks.

Sources

phases/04-computer-vision/12-video-understanding/docs/en.md

Functions in code

Task	Code
Uniform timestamps	`uniform_sample(duration, n)`
Motion-aware timestamps	`dynamic_sample(motion, fps_cap=4, total_budget=32)`
Score grounding	`evaluate_grounding(predictions, ground_truth, tol_iou=0.3)`
Interval overlap	`iou(a_start, a_end, b_start, b_end)`
Uniform frame indices	`sample_uniform(num_frames_total, T)`
Random contiguous window	`sample_dense(num_frames_total, T, rng=None)`
Lift 2D conv to 3D	`inflate_2d_to_3d(conv2d, time_kernel=3)`
Search three vectors	`multi_vector_search(query, scenes, k=5)`

Sources

Sample, ground, inflate

uni = uniform_sample(duration, 8)step * (i + 0.5)

dyn = dynamic_sample(motion, fps_cap=4, total_budget=12)1 to fps_cap per second

result = evaluate_grounding(predictions, ground)hit when IoU >= tol_iou

result["recall"]hits over ground truth events

model = FramePool(num_classes=10, pretrained=False)ResNet-18 per frame, mean over T

c3d = inflate_2d_to_3d(c2d, time_kernel=3)repeat on time, divide by time_kernel

c21 = Conv2Plus1D(3, 16)spatial, BN, ReLU, temporal

Sources

Run the capstone

Command	What it does
`python main.py`	Index demo: RRF search, grounded window
`npm run typecheck`	Type checks the UI half
`npm test`	Job state and store tests
`npm start`	Self-terminating demo
`npm run serve`	HTTP server on :8123
`GET /jobs`	Job summaries as JSON
`GET /job/:id`	Stage timeline, 404 when unknown
Stages	chunk, embed, index, qa

Sources

Defaults

Setting	Default	Note
`fps_cap`	4	Max samples per second
`total_budget`	32	Total dynamic samples
`tol_iou`	0.3	IoU for a grounding hit
`time_kernel`	3	Inflation time depth
RRF constant	60	Adds `1.0 / (60 + rank + 1)`
`k` in search	5	Top scenes after fusion
Clip length T	8	Also 16, 32 or 64

Sources

Troubleshooting

Symptom	Cause	Fix
"open door" equals "close door"	Temporal pooling is order-invariant	Use 3D conv or a transformer
Near-chance on motion classes	Appearance alone is insufficient	Train R(2+1)D, not FramePool
Inflated model breaks batch norm	Activation scale grows	Divide by `time_kernel`
Joint attention runtime balloons	Quadratic in `THW`	Use divided attention
Uniform sampling misses the action	Loses motion peaks	Use dynamic FPS with a max-frame cap
Answer says "early in the clip"	No absolute time position	Use time-aligned M-RoPE and JSON output
Context overflow on 2h video	583k tokens at 1 FPS	Pool more or use VideoAgent
Miscounts "how many X"	Counting hallucination	Report counting accuracy separately

Sources

Questions

What does time-aligned M-RoPE add?: Absolute time per token. It handles uneven dynamic FPS spacing.
Which output format?: Structured JSON with time and event fields.
Which pooling?: 3x3 bilinear for most tasks. 6x6 for agent workflows.
Which path for a 15-minute clip?: Open 72B with native context.
Which path for 2+ hours?: VideoAgent or hierarchical summaries.
78% clip, 82% video?: The model relies on test-time averaging.
Why three vectors per scene?: One pooled vector gives a retrieval regression.

Sources

Benchmarks

VideoMME: short, medium and long video. 2500+ samples.
TempCompass: "before" / "after" ordering questions.
EgoSchema: long-horizon first-person video.
Needle-in-a-haystack: hide a marker at a random point. Score Recall@k.

Sources

Original source

---
id: video-language
title: Video-Language Models and Long Video
code: VV
subtitle: Sample frames, budget tokens, encode time, ground events and index long video with timestamped answers.
hue: 320
sources: phases/12-multimodal-ai/17-video-language-temporal-grounding, phases/12-multimodal-ai/18-long-video-million-token, phases/04-computer-vision/12-video-understanding, phases/19-capstone-projects/12-video-understanding-pipeline
source_hash: 196574547a8febbc
---

## Scene index pipeline (capstone)
type: architecture
from: phases/19-capstone-projects/12-video-understanding-pipeline/docs/en.md

```arch
node | video | 1,1 | Video file | YouTube URL or local MP4 | client
node | scene | 2,1 | Scene segmentation | TransNetV2 or PySceneDetect | process
node | cap | 3,1 | VLM caption | caption + frame embedding | model
node | index | 4,1 | Multi-vector Qdrant | three named vectors | store
node | asr | 3,2 | ASR pass | Whisper-v3-turbo, word timestamps | model
node | query | 1,3 | Question | natural language | client
node | rrf | 2,3 | RRF merge | top-k=5 scenes | process
node | ground | 3,3 | Temporal grounding | refine start, end in scene | process
node | synth | 4,3 | VLM synth | answer + timestamps | model
edge | video | scene |
edge | scene | cap | keyframe
edge | scene | asr | audio
edge | cap | index |
edge | asr | index |
edge | query | rrf |
edge | index | rrf | three queries
edge | rrf | ground | top scene
edge | ground | synth | window
```

## Video VLM designs
type: comparison
span: 2
from: phases/12-multimodal-ai/17-video-language-temporal-grounding/docs/en.md, phases/12-multimodal-ai/17-video-language-temporal-grounding/code/main.py

| Factor | Video-LLaMA | Video-LLaVA | Qwen2.5-VL |
|---|---|---|---|
| Frames | 16-frame clips at 2 FPS | 8 frames | Dynamic FPS |
| Compressor | Video Q-former, 32 queries | MLP | time-aligned M-RoPE per token |
| Time position | 1D temporal RoPE | Learned temporal embeddings | Absolute timestamp (t, h, w) |
| Note | Audio branch, ImageBind | Shared image+video encoder | Best open 2025 |
| Answer to "when" | "early in the clip" | 8-16 frame system | "at 4.2 seconds" |
| Weakness | Fixed clip length | No long video | None stated |

## Long video: four paths
type: comparison
span: 2
from: phases/12-multimodal-ai/18-long-video-million-token/docs/en.md, phases/12-multimodal-ai/18-long-video-million-token/code/main.py

| Factor | Brute context | Ring attention | Token compression | Agentic retrieval |
|---|---|---|---|---|
| Example | Gemini 1.5, Gemini 2.5 Pro | LWM, LongVILA | Video-XL, LongVA | VideoAgent |
| Mechanism | All tokens in one pass | Chunks rotate across devices | One summary token per clip | LLM queries a clip tool |
| Cost | Millions of tokens | Linear in context | Small context | Only relevant clips |
| Weakness | Closed, not open-source | Spreads over many devices | Misses exact frames | Retrieval quality |
| Pick for | >1h general QA | 15-60 min | 15-60 min | >1h specific query |

## VideoAgent loop
type: flow
from: phases/12-multimodal-ai/18-long-video-million-token/docs/en.md, phases/12-multimodal-ai/18-long-video-million-token/code/main.py

1. **LLM**: reads the question.
2. **LLM**: calls `find_clips(keyword='<word>')` for relevant clips.
3. **Tool**: returns matching clip timestamps.
4. **VLM**: encodes 3 x 30s clips (~7290 tokens total).
5. **LLM**: composes the answer or asks follow-up queries.

## Token math
type: concepts
from: phases/12-multimodal-ai/17-video-language-temporal-grounding/docs/en.md, phases/12-multimodal-ai/18-long-video-million-token/docs/en.md, phases/12-multimodal-ai/18-long-video-million-token/code/main.py

- **Budget**: `tokens(duration_s, fps, per_frame)` is duration times FPS times tokens per frame.
- **Raw**: 1 minute at 30 FPS and 196 tokens per frame is 352k tokens.
- **3x3 pool**: 64 tokens per frame. 5 minutes at 1 FPS is 19,200 tokens.
- **384 native**: ~729 tokens per frame. 3x3 pooling gives 81.
- **30 min**: 1 FPS at 81 tokens is 145,800 tokens. 2 FPS is 291,600.
- **2 hours**: 1 FPS at 81 tokens is 583k tokens.
- **Retrieval**: `3 * 30 * 81 + 200` tokens for 3 clips. 99% cheaper.

## Video model families (vision)
type: concepts
from: phases/04-computer-vision/12-video-understanding/docs/en.md

- **2D + pool**: 2D CNN per frame, average over time. Cannot model motion.
- **3D conv**: kernels over (T, H, W). C3D, I3D, SlowFast.
- **I3D inflation**: copy a 2D kernel along time. 3x3 becomes 3x3x3.
- **(2+1)D**: 2D spatial conv, then 1D temporal conv. Fewer parameters.
- **Joint attention**: one attention over (t, h, w). Quadratic in `T*H*W`.
- **Divided attention**: one attention over time, one over space, per block.
- **Factorised attention**: time and space attention alternate across blocks.

## Functions in code
type: task-table
span: 2
from: phases/12-multimodal-ai/17-video-language-temporal-grounding/code/main.py, phases/04-computer-vision/12-video-understanding/code/main.py, phases/19-capstone-projects/12-video-understanding-pipeline/code/main.py

| Task | Code |
|---|---|
| Uniform timestamps | `uniform_sample(duration, n)` |
| Motion-aware timestamps | `dynamic_sample(motion, fps_cap=4, total_budget=32)` |
| Score grounding | `evaluate_grounding(predictions, ground_truth, tol_iou=0.3)` |
| Interval overlap | `iou(a_start, a_end, b_start, b_end)` |
| Uniform frame indices | `sample_uniform(num_frames_total, T)` |
| Random contiguous window | `sample_dense(num_frames_total, T, rng=None)` |
| Lift 2D conv to 3D | `inflate_2d_to_3d(conv2d, time_kernel=3)` |
| Search three vectors | `multi_vector_search(query, scenes, k=5)` |

## Sample, ground, inflate
type: annotated-code
lang: python
span: 2
from: phases/12-multimodal-ai/17-video-language-temporal-grounding/code/main.py, phases/04-computer-vision/12-video-understanding/code/main.py

```python
uni = uniform_sample(duration, 8)                          # step * (i + 0.5)
dyn = dynamic_sample(motion, fps_cap=4, total_budget=12)   # 1 to fps_cap per second
result = evaluate_grounding(predictions, ground)           # hit when IoU >= tol_iou
result["recall"]                                           # hits over ground truth events
model = FramePool(num_classes=10, pretrained=False)        # ResNet-18 per frame, mean over T
c3d = inflate_2d_to_3d(c2d, time_kernel=3)                 # repeat on time, divide by time_kernel
c21 = Conv2Plus1D(3, 16)                                   # spatial, BN, ReLU, temporal
```

## Run the capstone
type: command-table
from: phases/19-capstone-projects/12-video-understanding-pipeline/code/main.py, phases/19-capstone-projects/12-video-understanding-pipeline/code/ts/README.md, phases/19-capstone-projects/12-video-understanding-pipeline/code/ts/src/server.ts, phases/19-capstone-projects/12-video-understanding-pipeline/code/ts/src/types.ts

| Command | What it does |
|---|---|
| `python main.py` | Index demo: RRF search, grounded window |
| `npm run typecheck` | Type checks the UI half |
| `npm test` | Job state and store tests |
| `npm start` | Self-terminating demo |
| `npm run serve` | HTTP server on :8123 |
| `GET /jobs` | Job summaries as JSON |
| `GET /job/:id` | Stage timeline, 404 when unknown |
| Stages | chunk, embed, index, qa |

## Defaults
type: defaults
span: 1
from: phases/12-multimodal-ai/17-video-language-temporal-grounding/code/main.py, phases/12-multimodal-ai/17-video-language-temporal-grounding/docs/en.md, phases/04-computer-vision/12-video-understanding/code/main.py, phases/04-computer-vision/12-video-understanding/docs/en.md, phases/19-capstone-projects/12-video-understanding-pipeline/code/main.py

| Setting | Default | Note |
|---|---|---|
| `fps_cap` | 4 | Max samples per second |
| `total_budget` | 32 | Total dynamic samples |
| `tol_iou` | 0.3 | IoU for a grounding hit |
| `time_kernel` | 3 | Inflation time depth |
| RRF constant | 60 | Adds `1.0 / (60 + rank + 1)` |
| `k` in search | 5 | Top scenes after fusion |
| Clip length T | 8 | Also 16, 32 or 64 |

## Troubleshooting
type: troubleshooting
span: 2
from: phases/04-computer-vision/12-video-understanding/docs/en.md, phases/12-multimodal-ai/17-video-language-temporal-grounding/docs/en.md, phases/12-multimodal-ai/18-long-video-million-token/docs/en.md, phases/19-capstone-projects/12-video-understanding-pipeline/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| "open door" equals "close door" | Temporal pooling is order-invariant | Use 3D conv or a transformer |
| Near-chance on motion classes | Appearance alone is insufficient | Train R(2+1)D, not FramePool |
| Inflated model breaks batch norm | Activation scale grows | Divide by `time_kernel` |
| Joint attention runtime balloons | Quadratic in `T*H*W` | Use divided attention |
| Uniform sampling misses the action | Loses motion peaks | Use dynamic FPS with a max-frame cap |
| Answer says "early in the clip" | No absolute time position | Use time-aligned M-RoPE and JSON output |
| Context overflow on 2h video | 583k tokens at 1 FPS | Pool more or use VideoAgent |
| Miscounts "how many X" | Counting hallucination | Report counting accuracy separately |

## Questions
type: qa
from: phases/12-multimodal-ai/17-video-language-temporal-grounding/docs/en.md, phases/12-multimodal-ai/18-long-video-million-token/docs/en.md, phases/04-computer-vision/12-video-understanding/docs/en.md, phases/19-capstone-projects/12-video-understanding-pipeline/docs/en.md

- **What does time-aligned M-RoPE add?** Absolute time per token. It handles uneven dynamic FPS spacing.
- **Which output format?** Structured JSON with time and event fields.
- **Which pooling?** 3x3 bilinear for most tasks. 6x6 for agent workflows.
- **Which path for a 15-minute clip?** Open 72B with native context.
- **Which path for 2+ hours?** VideoAgent or hierarchical summaries.
- **78% clip, 82% video?** The model relies on test-time averaging.
- **Why three vectors per scene?** One pooled vector gives a retrieval regression.

## Benchmarks
type: concepts
from: phases/12-multimodal-ai/17-video-language-temporal-grounding/docs/en.md, phases/12-multimodal-ai/18-long-video-million-token/docs/en.md

- **VideoMME**: short, medium and long video. 2500+ samples.
- **TempCompass**: "before" / "after" ordering questions.
- **EgoSchema**: long-horizon first-person video.
- **Needle-in-a-haystack**: hide a marker at a random point. Score Recall@k.