video understanding pipeline — scene -> multi-vector -> grounded answer
ingest (three pipelines in parallel)
scene segmentation
TransNetV2 / PySceneDetect
VLM caption + frame embed
Gemini 2.5 Pro / Qwen3-VL / Molmo 2
ASR + word timestamps
Whisper-v3-turbo (faster-whisper)
multi-vector Qdrant index (three named vectors per scene)
caption_emb frame_emb transcript_emb
payload: video_id, scene_id, start_ms, end_ms, keyframe_url
query flow
three parallel dense queries
reciprocal rank fusion -> top-k scenes
temporal grounding (TimeLens / VideoITG)
VLM synth with cited timestamps
answer + (start, end) + frame previews
UI: jump-to-frame + thumbnail strip
evaluation
ActivityNet-QA + NeXT-GQA
custom 100-query hand-labeled set
hallucination sub-bench
counting + action-type reported separately
temporal IoU on held-out grounding set
ingest throughput: hours-of-video / $
target: 100 hours indexed, < 2 min per query