three-pillar evaluation of generative models
FID (sample quality)
||μ_r - μ_g||² + Tr(Σ_r + Σ_g - 2√Σ_rΣ_g)
Fréchet distance between
Gaussian fits in Inception-v3 space
use N ≥ 10k or numbers lie
ImageNet-biased; use FD-DINO or
CMMD out of domain
lower = better
CLIP score (adherence)
cos( CLIP_image(x), CLIP_text(p) )
how well does output match prompt?
compositional failures slip through
short prompts score higher mechanically
CMMD + VQA tests cover CLIP blind spots
higher = better
human preference (truth)
Elo from pairwise wins
A vs B on same prompt
HPSv2 / ImageReward / PickScore
as automated proxies
Chatbot-Arena-style image arenas
higher win rate = better
each metric has a known game
FID: overfit Inception prior → low FID, same quality
CLIP: saturate with 'masterpiece, 4k' → inflated score
preference: prompt set overlap with training → rigged
production eval report
1. FID on 10-30k + 2. CLIP / CMMD on the same pool +
3. 200+ blind human-pair Elo + 4. qualitative failure audit on 50 outputs
one metric = marketing; four = evidence