GANs, Diffusion and Flow Compared · AI Engineering

GAN, DDPM, latent diffusion, DiT with rectified flow, flow matching and VAR. Targets, steps, code and fixes.

Sampling cost side by side

Family	Passes	Note
GAN	one forward pass	Fast. Hard to train
DDPM	1000	Linear beta schedule
DDIM	50	Deterministic. Same training as DDPM
DPM-Solver++ 2M Karras	20-30	SD production default
Rectified flow (Euler)	20-30	SD3, FLUX, Z-Image
Schnell, Turbo, LCM	1-4	Distilled from a many-step model
VAR, K = 10 scales	10 passes	DiT needs 28-50 at 256x256

Sources

Training target side by side

Family	Target	Loss
GAN discriminator	P(real)	Binary cross-entropy on real and fake
GAN generator	`D(G(z))`	High is good. Non-saturating form
DDPM	`epsilon_theta(x_t, t)`	MSE on the true noise
Rectified flow (4.23)	`epsilon - x0`	MSE on the velocity
Flow matching (8.13)	`x1 - x0`	Here x1 is noise, x0 is data
VAR	Cross-entropy	On the VQ codes of the next scale

Sources

Use X when

GAN real-time generation under 10 ms, style transfer, Pix2Pix or CycleGAN translation.
SD 1.5 latent diffusion open-source community fine-tunes with LoRA.
DiT + rectified flow state of the art text to image. FLUX.1-schnell is the open-source default.
Distilled flow (LCM-LoRA, schnell) latency under 1s, 1-4 steps.
Flow matching audio, video and science as well as images.
VAR class-conditional ImageNet. Fewer passes, GPT-style scaling.

Sources

Stable Diffusion pipeline

Nodes and connections

Node	Role	Kind
Text prompt		client
Text encoder	CLIP-L or T5	model
Noise	4x64x64	store
UNet	cross-attention to text	gpu
Scheduler	DPM-Solver++, Euler	process
Clean latent	4x64x64	store
VAE decoder		model
Image	512x512 RGB	client

From	To	Connection
Text prompt	Text encoder
Text encoder	UNet	embedding
Noise	UNet
UNet	Scheduler
Scheduler	Clean latent
Clean latent	VAE decoder
VAE decoder	Image

Sources

phases/04-computer-vision/11-stable-diffusion/docs/en.md

Tasks in code

Task	Code
GAN with spectral norm	`build_sn_discriminator()`
Linear beta schedule	`linear_beta_schedule(T=1000)`
DDPM sampling	`sample_ddpm(model, schedule, shape)`
DDIM sampling	`sample_ddim(..., steps=50, eta=0.0)`
Rectified flow step	`rectified_flow_train_step(model, x0, ...)`
Euler sampling	`rectified_flow_sample(model, shape, steps=20)`
Multi-scale VQ tokens	`tokenize_multiscale(img, codebooks)`

Sources

diffusers calls

Task	Code
SD text to image	`guidance_scale=7.5, num_inference_steps=25`
Swap the scheduler	`DPMSolverMultistepScheduler.from_config(...)`
Image to image	`img2img(..., image=init_image, strength=0.6)`
Inpaint the white area	`inpaint(..., mask_image=mask)`
Fuse a LoRA	`pipe.fuse_lora(lora_scale=0.8)`
FLUX.1-schnell	`guidance_scale=0.0, num_inference_steps=4`
SD 3.5 Large	`guidance_scale=3.5, num_inference_steps=28`

Sources

DDPM against rectified flow

t = torch.randint(0, T, (bs,), device=device)DDPM: integer step

x_t = q_sample(x0, t, noise, schedule)closed-form jump

loss = F.mse_loss(pred, noise)predict the noise

t = torch.rand(n, device=device)RF: t in [0, 1]

x_t = (1 - t[:, None, None, None]) * x0 + t[:, None, None, None] * epsilon

target_v = epsilon - x0velocity, data to noise

loss = F.mse_loss(pred_v, target_v)

dt = 1.0 / stepsEuler, steps=20

for _ in range(steps):

v = model(x, t)

x = x - dt * vwalk from noise to data

t = t - dt

Sources

One GAN training step

opt_d.zero_grad()

d_real = D(real)

d_fake = D(G(z).detach())no gradient into G

loss_d = (F.binary_cross_entropy_with_logits(d_real, torch.ones_like(d_real))

+ F.binary_cross_entropy_with_logits(d_fake, torch.zeros_like(d_fake)))

loss_d.backward()

opt_d.step()

opt_g.zero_grad()

d_fake = D(G(z))

loss_g = F.binary_cross_entropy_with_logits(d_fake, torch.ones_like(d_fake))non-saturating

loss_g.backward()

opt_g.step()

opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))DCGAN default

Sources

Key formulas

GAN game: \min_G \max_D \; \mathbb{E}_x[\log D(x)] + \mathbb{E}_z[\log(1 - D(G(z)))]
DDPM jump: x_t = \sqrt{\bar{\alpha}_t} \, x_0 + \sqrt{1 - \bar{\alpha}_t} \, \epsilon, with \bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s
CFG: \epsilon = \epsilon_{\text{uncond}} + w \cdot (\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})
Rectified flow: x_t = (1 - t) \, x_0 + t \, \epsilon for t \in [0, 1]
CFG with velocity: v_{\text{cfg}} = (1+w) v_{\text{cond}} - w v_{\text{uncond}}
Latent saving: (3512512)/(46464) = 48x less compute.

Sources

VAR next-scale generation

Tokenizer a multi-scale residual VQ. Train once, then freeze it.
Scale 1 sample z_1. This is 1 token.
Scale 2 sample 4 tokens in parallel, given z_1.
Scale 3 sample 16 tokens in parallel, given z_1 and z_2.
Decode sum the upsampled embeddings of all scales. The VAE decoder makes the image.
Toy run SCALES = (1, 2, 4, 8), CODEBOOK = 16. 4 passes in total.

Sources

Troubleshooting

Symptom	Cause	Fix
Sample variety drops	Mode collapse	Spectral norm or label conditioning
D accuracy ~100%	D wins, G gradients vanish	Smaller D, lower D learning rate
Both losses swing, no trend	G and D trade wins	TTUR: D learns 2-4x faster
Samples depend on the batch	BN uses batch stats	Call `G.eval()` before sampling
Heavy artefacts	Guidance scale too high	SD 7.5. Rectified flow 3.5-5

Sources

Questions

Why is diffusion easy to train?: The loss is MSE. There is no adversarial game.
Why does rectified flow need fewer steps?: The path is close to a straight line.
Why is VAR fast?: One pass makes all tokens of a scale. Length grows with log scale.
Why does FLUX.1-schnell use guidance 0.0?: It was trained without CFG.

Sources

Original source

---
id: image-generation-compared
title: GANs, Diffusion and Flow Compared
code: GD
subtitle: GAN, DDPM, latent diffusion, DiT with rectified flow, flow matching and VAR. Targets, steps, code and fixes.
hue: 315
sources: phases/04-computer-vision/09-image-generation-gans, phases/04-computer-vision/10-image-generation-diffusion, phases/04-computer-vision/11-stable-diffusion, phases/04-computer-vision/23-diffusion-transformers-rectified-flow, phases/08-generative-ai/13-flow-matching-rectified-flows, phases/08-generative-ai/19-visual-autoregressive-var
source_hash: 45fc3bd2c684c3f5
---

## Sampling cost side by side
type: comparison
from: phases/04-computer-vision/10-image-generation-diffusion/docs/en.md, phases/04-computer-vision/11-stable-diffusion/docs/en.md, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md, phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md

| Family | Passes | Note |
|---|---|---|
| GAN | one forward pass | Fast. Hard to train |
| DDPM | 1000 | Linear beta schedule |
| DDIM | 50 | Deterministic. Same training as DDPM |
| DPM-Solver++ 2M Karras | 20-30 | SD production default |
| Rectified flow (Euler) | 20-30 | SD3, FLUX, Z-Image |
| Schnell, Turbo, LCM | 1-4 | Distilled from a many-step model |
| VAR, K = 10 scales | 10 passes | DiT needs 28-50 at 256x256 |

## Training target side by side
type: comparison
from: phases/04-computer-vision/09-image-generation-gans/docs/en.md, phases/04-computer-vision/10-image-generation-diffusion/docs/en.md, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md, phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md

| Family | Target | Loss |
|---|---|---|
| GAN discriminator | P(real) | Binary cross-entropy on real and fake |
| GAN generator | `D(G(z))` | High is good. Non-saturating form |
| DDPM | `epsilon_theta(x_t, t)` | MSE on the true noise |
| Rectified flow (4.23) | `epsilon - x0` | MSE on the velocity |
| Flow matching (8.13) | `x1 - x0` | Here x1 is noise, x0 is data |
| VAR | Cross-entropy | On the VQ codes of the next scale |

## Use X when
type: tips
from: phases/04-computer-vision/09-image-generation-gans/docs/en.md, phases/04-computer-vision/11-stable-diffusion/docs/en.md, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md, phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md

- **GAN**: real-time generation under 10 ms, style transfer, Pix2Pix or CycleGAN translation.
- **SD 1.5 latent diffusion**: open-source community fine-tunes with LoRA.
- **DiT + rectified flow**: state of the art text to image. FLUX.1-schnell is the open-source default.
- **Distilled flow (LCM-LoRA, schnell)**: latency under 1s, 1-4 steps.
- **Flow matching**: audio, video and science as well as images.
- **VAR**: class-conditional ImageNet. Fewer passes, GPT-style scaling.

## Stable Diffusion pipeline
type: architecture
from: phases/04-computer-vision/11-stable-diffusion/docs/en.md

```arch
node | txt | 1,1 | Text prompt | | client
node | te | 2,1 | Text encoder | CLIP-L or T5 | model
node | noise | 1,2 | Noise | 4x64x64 | store
node | unet | 3,1 | UNet | cross-attention to text | gpu
node | sched | 3,2 | Scheduler | DPM-Solver++, Euler | process
node | lat | 2,2 | Clean latent | 4x64x64 | store
node | vae | 2,3 | VAE decoder | | model
node | img | 3,3 | Image | 512x512 RGB | client
edge | txt | te |
edge | te | unet | embedding
edge | noise | unet |
edge | unet | sched |
edge | sched | lat |
edge | lat | vae |
edge | vae | img |
```

## Tasks in code
type: task-table
from: phases/04-computer-vision/09-image-generation-gans/docs/en.md, phases/04-computer-vision/10-image-generation-diffusion/code/main.py, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/code/main.py, phases/08-generative-ai/19-visual-autoregressive-var/code/main.py

| Task | Code |
|---|---|
| GAN with spectral norm | `build_sn_discriminator()` |
| Linear beta schedule | `linear_beta_schedule(T=1000)` |
| DDPM sampling | `sample_ddpm(model, schedule, shape)` |
| DDIM sampling | `sample_ddim(..., steps=50, eta=0.0)` |
| Rectified flow step | `rectified_flow_train_step(model, x0, ...)` |
| Euler sampling | `rectified_flow_sample(model, shape, steps=20)` |
| Multi-scale VQ tokens | `tokenize_multiscale(img, codebooks)` |

## diffusers calls
type: task-table
from: phases/04-computer-vision/11-stable-diffusion/docs/en.md, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md

| Task | Code |
|---|---|
| SD text to image | `guidance_scale=7.5, num_inference_steps=25` |
| Swap the scheduler | `DPMSolverMultistepScheduler.from_config(...)` |
| Image to image | `img2img(..., image=init_image, strength=0.6)` |
| Inpaint the white area | `inpaint(..., mask_image=mask)` |
| Fuse a LoRA | `pipe.fuse_lora(lora_scale=0.8)` |
| FLUX.1-schnell | `guidance_scale=0.0, num_inference_steps=4` |
| SD 3.5 Large | `guidance_scale=3.5, num_inference_steps=28` |

## DDPM against rectified flow
type: annotated-code
lang: python
from: phases/04-computer-vision/10-image-generation-diffusion/code/main.py, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/code/main.py

```python
t = torch.randint(0, T, (bs,), device=device)   # DDPM: integer step
x_t = q_sample(x0, t, noise, schedule)          # closed-form jump
loss = F.mse_loss(pred, noise)                  # predict the noise
t = torch.rand(n, device=device)                # RF: t in [0, 1]
x_t = (1 - t[:, None, None, None]) * x0 + t[:, None, None, None] * epsilon
target_v = epsilon - x0                         # velocity, data to noise
loss = F.mse_loss(pred_v, target_v)
dt = 1.0 / steps                                # Euler, steps=20
for _ in range(steps):
    v = model(x, t)
    x = x - dt * v                              # walk from noise to data
    t = t - dt
```

## One GAN training step
type: annotated-code
lang: python
from: phases/04-computer-vision/09-image-generation-gans/code/main.py, phases/04-computer-vision/09-image-generation-gans/docs/en.md

```python
opt_d.zero_grad()
d_real = D(real)
d_fake = D(G(z).detach())                    # no gradient into G
loss_d = (F.binary_cross_entropy_with_logits(d_real, torch.ones_like(d_real))
          + F.binary_cross_entropy_with_logits(d_fake, torch.zeros_like(d_fake)))
loss_d.backward()
opt_d.step()
opt_g.zero_grad()
d_fake = D(G(z))
loss_g = F.binary_cross_entropy_with_logits(d_fake, torch.ones_like(d_fake))  # non-saturating
loss_g.backward()
opt_g.step()
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))  # DCGAN default
```

## Key formulas
type: concepts
from: phases/04-computer-vision/09-image-generation-gans/docs/en.md, phases/04-computer-vision/10-image-generation-diffusion/docs/en.md, phases/04-computer-vision/11-stable-diffusion/docs/en.md, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md

- **GAN game**: $\min_G \max_D \; \mathbb{E}_x[\log D(x)] + \mathbb{E}_z[\log(1 - D(G(z)))]$
- **DDPM jump**: $x_t = \sqrt{\bar{\alpha}_t} \, x_0 + \sqrt{1 - \bar{\alpha}_t} \, \epsilon$, with $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$
- **CFG**: $\epsilon = \epsilon_{\text{uncond}} + w \cdot (\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})$
- **Rectified flow**: $x_t = (1 - t) \, x_0 + t \, \epsilon$ for $t \in [0, 1]$
- **CFG with velocity**: $v_{\text{cfg}} = (1+w) v_{\text{cond}} - w v_{\text{uncond}}$
- **Latent saving**: (3*512*512)/(4*64*64) = 48x less compute.

## VAR next-scale generation
type: flow
from: phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md, phases/08-generative-ai/19-visual-autoregressive-var/code/main.py

1. **Tokenizer**: a multi-scale residual VQ. Train once, then freeze it.
2. **Scale 1**: sample z_1. This is 1 token.
3. **Scale 2**: sample 4 tokens in parallel, given z_1.
4. **Scale 3**: sample 16 tokens in parallel, given z_1 and z_2.
5. **Decode**: sum the upsampled embeddings of all scales. The VAE decoder makes the image.
6. **Toy run**: `SCALES = (1, 2, 4, 8)`, `CODEBOOK = 16`. 4 passes in total.

## Troubleshooting
type: troubleshooting
from: phases/04-computer-vision/09-image-generation-gans/docs/en.md, phases/04-computer-vision/11-stable-diffusion/docs/en.md, phases/04-computer-vision/11-stable-diffusion/code/main.py, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| Sample variety drops | Mode collapse | Spectral norm or label conditioning |
| D accuracy ~100% | D wins, G gradients vanish | Smaller D, lower D learning rate |
| Both losses swing, no trend | G and D trade wins | TTUR: D learns 2-4x faster |
| Samples depend on the batch | BN uses batch stats | Call `G.eval()` before sampling |
| Heavy artefacts | Guidance scale too high | SD 7.5. Rectified flow 3.5-5 |

## Questions
type: qa
from: phases/04-computer-vision/10-image-generation-diffusion/docs/en.md, phases/04-computer-vision/11-stable-diffusion/docs/en.md, phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md, phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md, phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md

- **Why is diffusion easy to train?** The loss is MSE. There is no adversarial game.
- **Why does rectified flow need fewer steps?** The path is close to a straight line.
- **Why is VAR fast?** One pass makes all tokens of a scale. Length grows with log scale.
- **Why does FLUX.1-schnell use guidance 0.0?** It was trained without CFG.