latent diffusion = VAE + diffusion, separately trained
stage 1: train VAE (encoder + decoder), freeze
image x
512 × 512 × 3
encoder E
latent z
64 × 64 × 4 (1/16 of pixels)
decoder D
x̂ (reconstruction)
L1 + LPIPS + GAN
stage 2: train diffusion on z-space
z_T ~ N(0, I)
U-Net / DiT
ε_θ(z_t, t, text_embed)
iterate T->0
z_0
D(z_0)
decoded image
text encoder (CLIP / T5) -> cross-attention in each U-Net block
same loss as pixel-space DDPM: L = E || ε - ε_θ(z_t, t, c) ||²
~64x fewer FLOPs than pixel diffusion for the same quality
CFG: ε_cfg = (1+w) · ε_cond - w · ε_uncond (w ≈ 3-7)