Diffusion and Generative Models · AI Engineering

Train and sample a VAE, a GAN, DDPM, latent diffusion and flow matching, then measure them. Toy Python from the lessons.

Latent diffusion with CFG

Nodes and connections

Node	Role	Kind
Text encoder	frozen, CLIP or T5
Noisy latent	starts as N(0, I)
Denoiser	U-Net or DiT
CFG mix	cond and uncond
Clean latent	64 x 64 x 4
VAE decoder	latent to pixels
Image	512 x 512 x 3
VAE encoder	training and img2img

From	To	Connection
Text encoder	Denoiser	cross attn
Noisy latent	Denoiser
Denoiser	CFG mix	two eps
CFG mix	Clean latent	T steps
Clean latent	VAE decoder
VAE decoder	Image
Image	VAE encoder	train
VAE encoder	Noisy latent	add noise

Sources

DDPM train and sample

Schedule linear \beta_t from 1e-4 to 0.02 over T steps, then \bar\alpha_t.
Draw x_0 from data, t at random, \epsilon \sim \mathcal{N}(0, I).
Noise compute x_t in one shot from the closed form.
Predict the net takes x_t and a sinusoidal embedding of t and outputs \hat\epsilon.
Loss squared error between \epsilon and \hat\epsilon, then a gradient step.
Start sampling draw x_T \sim \mathcal{N}(0, I).
Reverse apply the reverse step from t = T to 1. Add noise on every step except the last.

Sources

Key formulas

Forward: q(x_t \mid x_0) = \mathcal{N}\left( \sqrt{\bar\alpha_t} \, x_0, (1 - \bar\alpha_t) I \right)
Alpha bar: \bar\alpha_t = \prod_{s=1}^{t} (1 - \beta_s)
Simple loss: L_{\text{simple}} = \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2
Reverse step: x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}} \epsilon_\theta(x_t, t) \right) + \sigma_t z
VAE loss: \lVert x - \hat{x} \rVert^2 + \beta \sum_i \frac{\sigma_i^2 + \mu_i^2 - \log \sigma_i^2 - 1}{2}
GAN game: \min_G \max_D \, \mathbb{E}[\log D(x)] + \mathbb{E}[\log(1 - D(G(z)))]
CFG: \epsilon_{\text{cfg}} = (1 + w) \epsilon_{\text{cond}} - w \epsilon_{\text{uncond}}
Flow matching: x_t = t x_1 + (1 - t) x_0, L = \lVert v_\theta(x_t, t) - (x_1 - x_0) \rVert^2

Sources

Tasks in code

Task	Code
Noise x_0 to step t	`x_t = math.sqrt(a_bar) * x0 + math.sqrt(1 - a_bar) * eps`
VAE reparameterize	`[m + s * e for m, s, e in zip(mu, sigma, eps)]`
Non-saturating G loss	`-sum(math.log(max(p, 1e-8)) for p in d_fake) / len(d_fake)`
Drop the label for CFG	`use_c = NULL_CLASS if rng.random() < 0.1 else c`
Mix with CFG	`eps_cfg = (1 + w) * eps_cond - w * eps_uncond`
Flow matching target	`target = x1 - x0`
Euler step, flow	`x -= dt * net_forward(x, t)`
Tile the VAE decode	`pipe.vae.enable_tiling()`

Sources

DDPM schedule, noise and reverse loop

betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)]linear

alphas = [1 - b for b in betas]

alpha_bars, cum = [], 1.0

for a in alphas:

cum *= a

alpha_bars.append(cum)cumulative product

eps = rng.gauss(0, 1)training noise

x_t = math.sqrt(a_bar) * x0 + math.sqrt(1 - a_bar) * epsone shot

x = rng.gauss(0, 1)sampling starts at noise

for t in range(T - 1, -1, -1):T-1 down to 0

eps_hat, _ = forward([x], t_emb, net)predicted noise

mean = (x - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_hat[0]) / math.sqrt(alphas[t])

x = mean + math.sqrt(beta_t) * rng.gauss(0, 1)only when t > 0

Sources

phases/08-generative-ai/06-diffusion-ddpm-from-scratch/code/main.py

Samplers and steps

Sampler	Steps
DDPM	1000 reverse steps in the paper
DDIM	20 to 50, same training objective
DPM-Solver++	10 to 20
UniPC	8 to 16
Flow matching, Euler	4 to 8 on straight paths
Rectified flow, 2 reflows	2 to 4
Distillation, consistency	1 to 4, needs retraining
VAE or GAN	one forward pass

Sources

Defaults and numbers

Setting	Default	Note
\beta start	1e-4	linear schedule
\beta end	0.02	x_T is close to \mathcal{N}(0, I)
Steps T	1000	the toy code uses 40
CFG scale w	3	3 to 7 is the sweet spot
CFG label dropout	10%	null token in training
SD 1.x VAE scaling factor	0.18215	applied after encoding
LoRA rank r	4-16	attention layers, 64 to 128 for heavy fine tunes
FID sample count	10k	low N gives a falsely low FID

Sources

Troubleshooting

Symptom	Cause	Fix
z carries no information about x	posterior collapse, KL wins	anneal \beta from 0 to 1, or free bits
GAN makes one mode only	mode collapse	noise on D inputs, minibatch discrimination or WGAN-GP
G stops improving, D over 95% accuracy	D is too strong	cut the D learning rate by 2-5x
Saturated, oily images	CFG too high	keep w at 3 to 7
Denoiser trains on wrong latent variance	VAE scaling factor missing	apply the checkpoint scaling factor
NaNs from the SD 1.x VAE at 1024²	fp16 cast	use the fp16-fix VAE or bf16
LoRA has no effect	trained on another base	use the matching base model
FID looks too good	N under 10k	use 10k samples or more and 3 seeds

Sources

Abbreviations

VAE: Variational autoencoder. The encoder outputs a distribution, a KL term shapes the code space.
ELBO: Evidence lower bound. Reconstruction minus KL, a lower bound on \log p(x).
GAN: Generator G(z) and discriminator D(x) in a minimax game.
DDPM: Ho et al. 2020. Linear \beta, 1000 steps, U-Net.
DDIM: Non-Markov sampler. 20 to 50 steps, same training objective.
CFG: Classifier-free guidance. Mixes conditional and unconditional predictions.
DiT: Diffusion Transformer. A transformer over latent patches in place of the U-Net.
FID: Fréchet Inception Distance. Distance of Gaussian fits to real and generated features.

Sources

Questions

Why output log σ² and not σ?: The output stays unconstrained. Gradients die at σ near 0 with softplus.
Why are VAE samples blurry?: MSE reconstruction gives the mean of the plausible outputs.
Why diffuse in latent space?: The 64×64×4 latent needs about 64x fewer FLOPs for comparable quality.
Are flow matching and DDPM different?: For Gaussian paths they are algebraically equivalent.
How is FID computed?: \lVert \mu_r - \mu_g \rVert^2 + \mathop{\mathrm{Tr}}(\Sigma_r + \Sigma_g - 2 (\Sigma_r \Sigma_g)^{0.5}) on Inception-v3 features. Lower is better.

Sources

Tips

Use the closed form KL. A Monte Carlo KL is 3x slower.
Give G fresh fakes at each step, or its gradients are stale.
Use a proper time embedding for images. Raw t fails.
Try the cosine schedule when quality plateaus.
Keep the sum of ControlNet weights near 1.0. Keep LoRA \alpha \le 1.5.
Report FID, CLIP score and human preference together.

Sources

Original source

---
id: diffusion-models
title: Diffusion and Generative Models
code: DG
subtitle: Train and sample a VAE, a GAN, DDPM, latent diffusion and flow matching, then measure them. Toy Python from the lessons.
hue: 300
sources: phases/08-generative-ai/02-autoencoders-vae, phases/08-generative-ai/03-gans-generator-discriminator, phases/08-generative-ai/06-diffusion-ddpm-from-scratch, phases/08-generative-ai/07-latent-diffusion-stable-diffusion, phases/08-generative-ai/08-controlnet-lora-conditioning, phases/08-generative-ai/13-flow-matching-rectified-flows, phases/08-generative-ai/14-evaluation-fid-clip-score
source_hash: fa31bd8433c7b343
---

## Latent diffusion with CFG
type: architecture
from: phases/08-generative-ai/07-latent-diffusion-stable-diffusion/docs/en.md, phases/08-generative-ai/02-autoencoders-vae/docs/en.md

```arch
node | text | 1,1 | Text encoder | frozen, CLIP or T5
node | noise | 2,1 | Noisy latent | starts as N(0, I)
node | den | 3,1 | Denoiser | U-Net or DiT
node | cfg | 4,1 | CFG mix | cond and uncond
node | lat | 4,2 | Clean latent | 64 x 64 x 4
node | dec | 3,2 | VAE decoder | latent to pixels
node | img | 2,2 | Image | 512 x 512 x 3
node | enc | 1,2 | VAE encoder | training and img2img
edge | text | den | cross attn
edge | noise | den |
edge | den | cfg | two eps
edge | cfg | lat | T steps
edge | lat | dec |
edge | dec | img |
edge | img | enc | train
edge | enc | noise | add noise
```

## DDPM train and sample
type: flow
from: phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md, phases/08-generative-ai/06-diffusion-ddpm-from-scratch/code/main.py

1. **Schedule**: linear $\beta_t$ from 1e-4 to 0.02 over $T$ steps, then $\bar\alpha_t$.
2. **Draw**: $x_0$ from data, $t$ at random, $\epsilon \sim \mathcal{N}(0, I)$.
3. **Noise**: compute $x_t$ in one shot from the closed form.
4. **Predict**: the net takes $x_t$ and a sinusoidal embedding of $t$ and outputs $\hat\epsilon$.
5. **Loss**: squared error between $\epsilon$ and $\hat\epsilon$, then a gradient step.
6. **Start sampling**: draw $x_T \sim \mathcal{N}(0, I)$.
7. **Reverse**: apply the reverse step from $t = T$ to 1. Add noise on every step except the last.

## Key formulas
type: concepts
from: phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md, phases/08-generative-ai/02-autoencoders-vae/docs/en.md, phases/08-generative-ai/03-gans-generator-discriminator/docs/en.md, phases/08-generative-ai/07-latent-diffusion-stable-diffusion/docs/en.md, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md

- **Forward**: $q(x_t \mid x_0) = \mathcal{N}\left( \sqrt{\bar\alpha_t} \, x_0, (1 - \bar\alpha_t) I \right)$
- **Alpha bar**: $\bar\alpha_t = \prod_{s=1}^{t} (1 - \beta_s)$
- **Simple loss**: $L_{\text{simple}} = \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2$
- **Reverse step**: $x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}} \epsilon_\theta(x_t, t) \right) + \sigma_t z$
- **VAE loss**: $\lVert x - \hat{x} \rVert^2 + \beta \sum_i \frac{\sigma_i^2 + \mu_i^2 - \log \sigma_i^2 - 1}{2}$
- **GAN game**: $\min_G \max_D \, \mathbb{E}[\log D(x)] + \mathbb{E}[\log(1 - D(G(z)))]$
- **CFG**: $\epsilon_{\text{cfg}} = (1 + w) \epsilon_{\text{cond}} - w \epsilon_{\text{uncond}}$
- **Flow matching**: $x_t = t x_1 + (1 - t) x_0$, $L = \lVert v_\theta(x_t, t) - (x_1 - x_0) \rVert^2$

## Tasks in code
type: task-table
from: phases/08-generative-ai/06-diffusion-ddpm-from-scratch/code/main.py, phases/08-generative-ai/02-autoencoders-vae/docs/en.md, phases/08-generative-ai/03-gans-generator-discriminator/docs/en.md, phases/08-generative-ai/07-latent-diffusion-stable-diffusion/docs/en.md, phases/08-generative-ai/07-latent-diffusion-stable-diffusion/code/main.py, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md

| Task | Code |
|---|---|
| Noise $x_0$ to step $t$ | `x_t = math.sqrt(a_bar) * x0 + math.sqrt(1 - a_bar) * eps` |
| VAE reparameterize | `[m + s * e for m, s, e in zip(mu, sigma, eps)]` |
| Non-saturating G loss | `-sum(math.log(max(p, 1e-8)) for p in d_fake) / len(d_fake)` |
| Drop the label for CFG | `use_c = NULL_CLASS if rng.random() < 0.1 else c` |
| Mix with CFG | `eps_cfg = (1 + w) * eps_cond - w * eps_uncond` |
| Flow matching target | `target = x1 - x0` |
| Euler step, flow | `x -= dt * net_forward(x, t)` |
| Tile the VAE decode | `pipe.vae.enable_tiling()` |

## DDPM schedule, noise and reverse loop
type: annotated-code
lang: python
from: phases/08-generative-ai/06-diffusion-ddpm-from-scratch/code/main.py

```python
betas = [1e-4 + (0.02 - 1e-4) * t / (T - 1) for t in range(T)]  # linear
alphas = [1 - b for b in betas]
alpha_bars, cum = [], 1.0
for a in alphas:
    cum *= a
    alpha_bars.append(cum)                      # cumulative product
eps = rng.gauss(0, 1)                           # training noise
x_t = math.sqrt(a_bar) * x0 + math.sqrt(1 - a_bar) * eps  # one shot
x = rng.gauss(0, 1)                             # sampling starts at noise
for t in range(T - 1, -1, -1):                  # T-1 down to 0
    eps_hat, _ = forward([x], t_emb, net)       # predicted noise
    mean = (x - beta_t / math.sqrt(1 - alpha_bars[t]) * eps_hat[0]) / math.sqrt(alphas[t])
    x = mean + math.sqrt(beta_t) * rng.gauss(0, 1)  # only when t > 0
```

## Samplers and steps
type: command-table
from: phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md, phases/08-generative-ai/02-autoencoders-vae/docs/en.md, phases/08-generative-ai/03-gans-generator-discriminator/docs/en.md

| Sampler | Steps |
|---|---|
| DDPM | 1000 reverse steps in the paper |
| DDIM | 20 to 50, same training objective |
| DPM-Solver++ | 10 to 20 |
| UniPC | 8 to 16 |
| Flow matching, Euler | 4 to 8 on straight paths |
| Rectified flow, 2 reflows | 2 to 4 |
| Distillation, consistency | 1 to 4, needs retraining |
| VAE or GAN | one forward pass |

## Defaults and numbers
type: defaults
from: phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md, phases/08-generative-ai/06-diffusion-ddpm-from-scratch/code/main.py, phases/08-generative-ai/07-latent-diffusion-stable-diffusion/docs/en.md, phases/08-generative-ai/08-controlnet-lora-conditioning/docs/en.md, phases/08-generative-ai/14-evaluation-fid-clip-score/docs/en.md

| Setting | Default | Note |
|---|---|---|
| $\beta$ start | 1e-4 | linear schedule |
| $\beta$ end | 0.02 | $x_T$ is close to $\mathcal{N}(0, I)$ |
| Steps $T$ | 1000 | the toy code uses 40 |
| CFG scale $w$ | 3 | 3 to 7 is the sweet spot |
| CFG label dropout | 10% | null token in training |
| SD 1.x VAE scaling factor | 0.18215 | applied after encoding |
| LoRA rank $r$ | 4-16 | attention layers, 64 to 128 for heavy fine tunes |
| FID sample count | 10k | low N gives a falsely low FID |

## Troubleshooting
type: troubleshooting
from: phases/08-generative-ai/02-autoencoders-vae/docs/en.md, phases/08-generative-ai/03-gans-generator-discriminator/docs/en.md, phases/08-generative-ai/07-latent-diffusion-stable-diffusion/docs/en.md, phases/08-generative-ai/08-controlnet-lora-conditioning/docs/en.md, phases/08-generative-ai/14-evaluation-fid-clip-score/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| $z$ carries no information about $x$ | posterior collapse, KL wins | anneal $\beta$ from 0 to 1, or free bits |
| GAN makes one mode only | mode collapse | noise on D inputs, minibatch discrimination or WGAN-GP |
| G stops improving, D over 95% accuracy | D is too strong | cut the D learning rate by 2-5x |
| Saturated, oily images | CFG too high | keep $w$ at 3 to 7 |
| Denoiser trains on wrong latent variance | VAE scaling factor missing | apply the checkpoint scaling factor |
| NaNs from the SD 1.x VAE at 1024² | fp16 cast | use the fp16-fix VAE or bf16 |
| LoRA has no effect | trained on another base | use the matching base model |
| FID looks too good | N under 10k | use 10k samples or more and 3 seeds |

## Abbreviations
type: glossary
from: phases/08-generative-ai/02-autoencoders-vae/docs/en.md, phases/08-generative-ai/03-gans-generator-discriminator/docs/en.md, phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md, phases/08-generative-ai/07-latent-diffusion-stable-diffusion/docs/en.md, phases/08-generative-ai/08-controlnet-lora-conditioning/docs/en.md, phases/08-generative-ai/14-evaluation-fid-clip-score/docs/en.md

- **VAE**: Variational autoencoder. The encoder outputs a distribution, a KL term shapes the code space.
- **ELBO**: Evidence lower bound. Reconstruction minus KL, a lower bound on $\log p(x)$.
- **GAN**: Generator $G(z)$ and discriminator $D(x)$ in a minimax game.
- **DDPM**: Ho et al. 2020. Linear $\beta$, 1000 steps, U-Net.
- **DDIM**: Non-Markov sampler. 20 to 50 steps, same training objective.
- **CFG**: Classifier-free guidance. Mixes conditional and unconditional predictions.
- **DiT**: Diffusion Transformer. A transformer over latent patches in place of the U-Net.
- **FID**: Fréchet Inception Distance. Distance of Gaussian fits to real and generated features.

## Questions
type: qa
from: phases/08-generative-ai/02-autoencoders-vae/docs/en.md, phases/08-generative-ai/07-latent-diffusion-stable-diffusion/docs/en.md, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md, phases/08-generative-ai/14-evaluation-fid-clip-score/docs/en.md, phases/08-generative-ai/03-gans-generator-discriminator/docs/en.md

- **Why output log σ² and not σ?** The output stays unconstrained. Gradients die at σ near 0 with softplus.
- **Why are VAE samples blurry?** MSE reconstruction gives the mean of the plausible outputs.
- **Why diffuse in latent space?** The 64×64×4 latent needs about 64x fewer FLOPs for comparable quality.
- **Are flow matching and DDPM different?** For Gaussian paths they are algebraically equivalent.
- **How is FID computed?** $\lVert \mu_r - \mu_g \rVert^2 + \mathop{\mathrm{Tr}}(\Sigma_r + \Sigma_g - 2 (\Sigma_r \Sigma_g)^{0.5})$ on Inception-v3 features. Lower is better.

## Tips
type: tips
from: phases/08-generative-ai/02-autoencoders-vae/docs/en.md, phases/08-generative-ai/03-gans-generator-discriminator/docs/en.md, phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md, phases/08-generative-ai/08-controlnet-lora-conditioning/docs/en.md, phases/08-generative-ai/14-evaluation-fid-clip-score/docs/en.md

- Use the closed form KL. A Monte Carlo KL is 3x slower.
- Give G fresh fakes at each step, or its gradients are stale.
- Use a proper time embedding for images. Raw $t$ fails.
- Try the cosine schedule when quality plateaus.
- Keep the sum of ControlNet weights near 1.0. Keep LoRA $\alpha \le 1.5$.
- Report FID, CLIP score and human preference together.