Video, Depth and 3D Vision · Ingénierie IA

Sample frames, model time, render NeRFs and Gaussian splats, predict depth and track objects across frames.

Video models

2D + pool: CNN per frame, mean over time.
3D conv: (T, H, W) kernels. Models motion.
I3D: inflate 2D kernels along time.
(2+1)D: 2D spatial, then 1D temporal conv.
Divided: attention over time, then space.
Uniform: T frames evenly. 2D+pool default.
Dense: random contiguous window. 3D convs.
Clip / video acc: one clip or mean of clips.

Sources

phases/04-computer-vision/12-video-understanding/docs/en.md

Pick a 3D, depth or tracking model

NeRF: an MLP is the scene. Hours to train.
3DGS: explicit Gaussians. Minutes to train.
Depth Anything V3: relative, frozen DINOv2.
ZoeDepth: metric. Sensitive to intrinsics.
Marigold: diffusion depth. 10-50 steps.
ByteTrack: cars and people at 30+ fps.
DeepSORT: heavy occlusion, ReID features.
SAM 3.1 Multiplex: crowds of one class.

Sources

Tracking by detection

Nodes and connections

Node	Role	Kind
Frame t	video input	client
Detector	detections at t	model
Hungarian assignment	IoU, cosine, motion
Update	matched tracks
Tracks up to t-1		store
Kalman predict	constant velocity
Birth	new tracks
Age unmatched	delete after N

From	To	Connection
Frame t	Detector
Detector	Hungarian assignment	detections
Tracks up to t-1	Kalman predict
Kalman predict	Hungarian assignment	predicted boxes
Hungarian assignment	Update	matched
Hungarian assignment	Birth	unmatched detections
Hungarian assignment	Age unmatched	unmatched tracks

Sources

phases/04-computer-vision/27-multi-object-tracking/docs/en.md

Formulas

PointNet: f(P) = \max_{p \in P} \mathrm{MLP}(p). Point order does not change it.
Positional encoding: \sin(2^l \pi p), \cos(2^l \pi p) up to L = 10 levels.
Volume render: C = \sum_i T_i (1 - e^{-\sigma_i \delta_i}) c_i, T_i = e^{-\sum_{j<i} \sigma_j \delta_j}
3DGS shape: \Sigma = R S S^T R^T. On screen \Sigma' = J W \Sigma W^T J^T.
3DGS pixel: C = \sum_i \alpha_i T_i c_i, T_i = \prod_{j<i} (1 - \alpha_j)
Pinhole lift: X = (u - c_x) d / f_x, Y = (v - c_y) d / f_y, Z = d
AbsRel: mean of |d_{pred} - d_{gt}| / d_{gt}. Lower is better.
MOTA: 1 - (FN + FP + \mathrm{IDSW}) / GT

Sources

Frames, inflation and (2+1)D

idx = sample_uniform(num_frames_total, T)even spread

idx = sample_dense(num_frames_total, T)contiguous window

model = FramePool(num_classes=10)ResNet-18 per frame, mean over T

x = torch.randn(2, 8, 3, 224, 224)(N, T, 3, H, W)

conv2d = nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False)

conv3d = inflate_2d_to_3d(conv2d, time_kernel=3)repeat on time, divide by 3

c = Conv2Plus1D(3, 64)spatial, then temporal

c(torch.randn(1, 3, 8, 56, 56))same 3D input as conv3d

Sources

Render one NeRF ray

y = positional_encoding(x, L=10)(5, 3) to (5, 60)

t_vals = torch.linspace(2.0, 6.0, N)N = 64 samples on the ray

delta = torch.cat([t_vals[1:] - t_vals[:-1], torch.full_like(t_vals[:1], 1e10)])

alpha = 1.0 - torch.exp(-sigma * delta)opacity of each sample

trans = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]), 1.0 - alpha + 1e-10], dim=-1), dim=-1)[..., :-1]

weights = alpha * transtransmittance times opacity

rendered = (weights.unsqueeze(-1) * rgb).sum(dim=-2)pixel colour

depth = (weights * t_vals).sum(dim=-1)depth along the ray

Sources

Depth metrics and lifting

aligned = align_scale_shift(pred, gt)fit a * pred + b for relative depth

abs_rel_error(aligned, gt)0.05-0.1 for production models

delta_accuracy(pred, target, threshold=1.25)0.9+ for SOTA

intr = (320.0, 320.0, 160.0, 120.0)fx, fy, cx, cy

pc = depth_to_point_cloud(depth, intr)(H, W, 3)

pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Large-hf")

out = pipe(image)

depth_np = np.array(out["depth"])PIL grayscale to numpy

Sources

IoU and Hungarian association

iou = bbox_iou(track_boxes, det_boxes)(tracks, detections)

cost = 1 - iou

cost[iou < self.iou_threshold] = 1e6block weak matches

row, col = linear_sum_assignment(cost)minimum total cost

for r, c in zip(row, col):

if cost[r, c] < 1.0:

self.tracks[r].update(det_boxes[c], frame)

self.tracks = [t for t in self.tracks if frame - t.last_frame <= self.max_age]

tracker = SimpleTracker()iou_threshold=0.3, max_age=5

tracks = tracker.step(dets, f)list of (id, bbox)

Sources

Commands and exports

Command	What it does
`pip install nerfstudio gsplat`	Installs the 3DGS tools
`ns-download-data example`	Gets an example dataset
`ns-train splatfacto --data path/to/data`	Trains 3DGS, 10-30 minutes on an RTX 4090
`.ply`	Raw Gaussian cloud. Portable, largest file
`.splat`	Quantised format
`KHR_gaussian_splatting`	glTF standard across viewers
`UsdVolParticleField3DGaussianSplat`	OpenUSD schema
`model.track(source, tracker="bytetrack.yaml")`	YOLOv8 with ByteTrack

Sources

Defaults and numbers

Setting	Default	Note
Frames per clip T	8, 16, 32, or 64	More T, more signal and compute
`time_kernel`	3	Inflated weights divided by it
NeRF `L`	10	Direction uses `L_dir=4`
Samples per ray	64	t from 2.0 to 6.0
SH degree	3	16 coefficients per channel, 48 floats
3DGS iterations	~30,000	Grows ~100k Gaussians to 1-5M
Tracker `iou_threshold`	0.3	With `max_age=5`
Delta threshold	1.25	Fraction of pixels within 25%

Sources

Troubleshooting

Symptom	Cause	Fix
"Open door" equals "close door"	Temporal pooling is order-invariant	Use 3D conv or (2+1)D
BN breaks after inflation	Weights not scaled	Divide by `time_kernel`
NeRF looks blurry	No positional encoding	Add Fourier features
Right order, wrong metres	Relative depth model	Metric model plus intrinsics
Wrong depth on glass, mirrors	Reflective surfaces	Do not trust those pixels
High AbsRel on MiDaS output	Scale and shift not aligned	`align_scale_shift` first
ID switches near crowds	IoU-only association	ByteTrack second pass or ReID
Track lost in a short occlusion	No motion model	Add a Kalman predict step

Sources

Questions

Why does 3DGS match NeRF quality?: Both use the same compositing equation.
Why start with 2D + pool?: It reuses ImageNet weights. It is often within 5-10 points.
Which tracking metric?: IDF1 for who is who. HOTA for sports and comparisons.
What can one image not give?: Absolute scale without intrinsics, and occluded geometry.
Why keep low-confidence boxes?: ByteTrack matches them second. This recovers short occlusions.

Sources

Original source

---
id: video-3d-vision
title: Video, Depth and 3D Vision
code: VD
subtitle: Sample frames, model time, render NeRFs and Gaussian splats, predict depth and track objects across frames.
hue: 190
sources: phases/04-computer-vision/12-video-understanding, phases/04-computer-vision/13-3d-vision-nerf, phases/04-computer-vision/22-3d-gaussian-splatting, phases/04-computer-vision/26-monocular-depth, phases/04-computer-vision/27-multi-object-tracking, phases/04-computer-vision/28-world-models-video-diffusion
source_hash: c86ac84265db2ca6
---

## Video models
type: concepts
from: phases/04-computer-vision/12-video-understanding/docs/en.md

- **2D + pool**: CNN per frame, mean over time.
- **3D conv**: (T, H, W) kernels. Models motion.
- **I3D**: inflate 2D kernels along time.
- **(2+1)D**: 2D spatial, then 1D temporal conv.
- **Divided**: attention over time, then space.
- **Uniform**: T frames evenly. 2D+pool default.
- **Dense**: random contiguous window. 3D convs.
- **Clip / video acc**: one clip or mean of clips.

## Pick a 3D, depth or tracking model
type: concepts
from: phases/04-computer-vision/13-3d-vision-nerf/docs/en.md, phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md, phases/04-computer-vision/26-monocular-depth/docs/en.md, phases/04-computer-vision/27-multi-object-tracking/docs/en.md

- **NeRF**: an MLP is the scene. Hours to train.
- **3DGS**: explicit Gaussians. Minutes to train.
- **Depth Anything V3**: relative, frozen DINOv2.
- **ZoeDepth**: metric. Sensitive to intrinsics.
- **Marigold**: diffusion depth. 10-50 steps.
- **ByteTrack**: cars and people at 30+ fps.
- **DeepSORT**: heavy occlusion, ReID features.
- **SAM 3.1 Multiplex**: crowds of one class.

## Tracking by detection
type: architecture
from: phases/04-computer-vision/27-multi-object-tracking/docs/en.md

```arch
node | frame | 1,1 | Frame t | video input | client
node | det | 2,1 | Detector | detections at t | model
node | assoc | 3,1 | Hungarian assignment | IoU, cosine, motion
node | upd | 4,1 | Update | matched tracks
node | prev | 1,2 | Tracks up to t-1 | | store
node | kal | 2,2 | Kalman predict | constant velocity
node | born | 3,2 | Birth | new tracks
node | age | 4,2 | Age unmatched | delete after N
edge | frame | det |
edge | det | assoc | detections
edge | prev | kal |
edge | kal | assoc | predicted boxes
edge | assoc | upd | matched
edge | assoc | born | unmatched detections
edge | assoc | age | unmatched tracks
```

## Formulas
type: concepts
from: phases/04-computer-vision/13-3d-vision-nerf/docs/en.md, phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md, phases/04-computer-vision/26-monocular-depth/docs/en.md, phases/04-computer-vision/27-multi-object-tracking/docs/en.md

- **PointNet**: $f(P) = \max_{p \in P} \mathrm{MLP}(p)$. Point order does not change it.
- **Positional encoding**: $\sin(2^l \pi p), \cos(2^l \pi p)$ up to $L = 10$ levels.
- **Volume render**: $C = \sum_i T_i (1 - e^{-\sigma_i \delta_i}) c_i$, $T_i = e^{-\sum_{j<i} \sigma_j \delta_j}$
- **3DGS shape**: $\Sigma = R S S^T R^T$. On screen $\Sigma' = J W \Sigma W^T J^T$.
- **3DGS pixel**: $C = \sum_i \alpha_i T_i c_i$, $T_i = \prod_{j<i} (1 - \alpha_j)$
- **Pinhole lift**: $X = (u - c_x) d / f_x$, $Y = (v - c_y) d / f_y$, $Z = d$
- **AbsRel**: mean of $|d_{pred} - d_{gt}| / d_{gt}$. Lower is better.
- **MOTA**: $1 - (FN + FP + \mathrm{IDSW}) / GT$

## Frames, inflation and (2+1)D
type: annotated-code
lang: python
from: phases/04-computer-vision/12-video-understanding/docs/en.md, phases/04-computer-vision/12-video-understanding/code/main.py

```python
idx = sample_uniform(num_frames_total, T)          # even spread
idx = sample_dense(num_frames_total, T)            # contiguous window
model = FramePool(num_classes=10)                  # ResNet-18 per frame, mean over T
x = torch.randn(2, 8, 3, 224, 224)                 # (N, T, 3, H, W)
conv2d = nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False)
conv3d = inflate_2d_to_3d(conv2d, time_kernel=3)   # repeat on time, divide by 3
c = Conv2Plus1D(3, 64)                             # spatial, then temporal
c(torch.randn(1, 3, 8, 56, 56))                    # same 3D input as conv3d
```

## Render one NeRF ray
type: annotated-code
lang: python
from: phases/04-computer-vision/13-3d-vision-nerf/docs/en.md, phases/04-computer-vision/13-3d-vision-nerf/code/main.py

```python
y = positional_encoding(x, L=10)                   # (5, 3) to (5, 60)
t_vals = torch.linspace(2.0, 6.0, N)               # N = 64 samples on the ray
delta = torch.cat([t_vals[1:] - t_vals[:-1], torch.full_like(t_vals[:1], 1e10)])
alpha = 1.0 - torch.exp(-sigma * delta)            # opacity of each sample
trans = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]), 1.0 - alpha + 1e-10], dim=-1), dim=-1)[..., :-1]
weights = alpha * trans                            # transmittance times opacity
rendered = (weights.unsqueeze(-1) * rgb).sum(dim=-2)  # pixel colour
depth = (weights * t_vals).sum(dim=-1)             # depth along the ray
```

## Depth metrics and lifting
type: annotated-code
lang: python
from: phases/04-computer-vision/26-monocular-depth/docs/en.md, phases/04-computer-vision/26-monocular-depth/code/main.py

```python
aligned = align_scale_shift(pred, gt)              # fit a * pred + b for relative depth
abs_rel_error(aligned, gt)                         # 0.05-0.1 for production models
delta_accuracy(pred, target, threshold=1.25)       # 0.9+ for SOTA
intr = (320.0, 320.0, 160.0, 120.0)                # fx, fy, cx, cy
pc = depth_to_point_cloud(depth, intr)             # (H, W, 3)
pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Large-hf")
out = pipe(image)
depth_np = np.array(out["depth"])                  # PIL grayscale to numpy
```

## IoU and Hungarian association
type: annotated-code
lang: python
from: phases/04-computer-vision/27-multi-object-tracking/docs/en.md, phases/04-computer-vision/27-multi-object-tracking/code/main.py

```python
iou = bbox_iou(track_boxes, det_boxes)             # (tracks, detections)
cost = 1 - iou
cost[iou < self.iou_threshold] = 1e6               # block weak matches
row, col = linear_sum_assignment(cost)             # minimum total cost
for r, c in zip(row, col):
    if cost[r, c] < 1.0:
        self.tracks[r].update(det_boxes[c], frame)
self.tracks = [t for t in self.tracks if frame - t.last_frame <= self.max_age]
tracker = SimpleTracker()                          # iou_threshold=0.3, max_age=5
tracks = tracker.step(dets, f)                     # list of (id, bbox)
```

## Commands and exports
type: command-table
from: phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md, phases/04-computer-vision/27-multi-object-tracking/docs/en.md

| Command | What it does |
|---|---|
| `pip install nerfstudio gsplat` | Installs the 3DGS tools |
| `ns-download-data example` | Gets an example dataset |
| `ns-train splatfacto --data path/to/data` | Trains 3DGS, 10-30 minutes on an RTX 4090 |
| `.ply` | Raw Gaussian cloud. Portable, largest file |
| `.splat` | Quantised format |
| `KHR_gaussian_splatting` | glTF standard across viewers |
| `UsdVolParticleField3DGaussianSplat` | OpenUSD schema |
| `model.track(source, tracker="bytetrack.yaml")` | YOLOv8 with ByteTrack |

## Defaults and numbers
type: defaults
from: phases/04-computer-vision/12-video-understanding/docs/en.md, phases/04-computer-vision/13-3d-vision-nerf/docs/en.md, phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md, phases/04-computer-vision/26-monocular-depth/docs/en.md, phases/04-computer-vision/27-multi-object-tracking/docs/en.md

| Setting | Default | Note |
|---|---|---|
| Frames per clip T | 8, 16, 32, or 64 | More T, more signal and compute |
| `time_kernel` | 3 | Inflated weights divided by it |
| NeRF `L` | 10 | Direction uses `L_dir=4` |
| Samples per ray | 64 | t from 2.0 to 6.0 |
| SH degree | 3 | 16 coefficients per channel, 48 floats |
| 3DGS iterations | ~30,000 | Grows ~100k Gaussians to 1-5M |
| Tracker `iou_threshold` | 0.3 | With `max_age=5` |
| Delta threshold | 1.25 | Fraction of pixels within 25% |

## Troubleshooting
type: troubleshooting
from: phases/04-computer-vision/12-video-understanding/docs/en.md, phases/04-computer-vision/13-3d-vision-nerf/docs/en.md, phases/04-computer-vision/26-monocular-depth/docs/en.md, phases/04-computer-vision/27-multi-object-tracking/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| "Open door" equals "close door" | Temporal pooling is order-invariant | Use 3D conv or (2+1)D |
| BN breaks after inflation | Weights not scaled | Divide by `time_kernel` |
| NeRF looks blurry | No positional encoding | Add Fourier features |
| Right order, wrong metres | Relative depth model | Metric model plus intrinsics |
| Wrong depth on glass, mirrors | Reflective surfaces | Do not trust those pixels |
| High AbsRel on MiDaS output | Scale and shift not aligned | `align_scale_shift` first |
| ID switches near crowds | IoU-only association | ByteTrack second pass or ReID |
| Track lost in a short occlusion | No motion model | Add a Kalman predict step |

## Questions
type: qa
span: 2
from: phases/04-computer-vision/12-video-understanding/docs/en.md, phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md, phases/04-computer-vision/26-monocular-depth/docs/en.md, phases/04-computer-vision/27-multi-object-tracking/docs/en.md

- **Why does 3DGS match NeRF quality?** Both use the same compositing equation.
- **Why start with 2D + pool?** It reuses ImageNet weights. It is often within 5-10 points.
- **Which tracking metric?** IDF1 for who is who. HOTA for sports and comparisons.
- **What can one image not give?** Absolute scale without intrinsics, and occluded geometry.
- **Why keep low-confidence boxes?** ByteTrack matches them second. This recovers short occlusions.