Eval-driven development — three layers + evaluator-optimizer loop
1. static benchmarks
cross-model comparison + regression gate
SWE-bench Verified (code)
WebArena / OSWorld (web/desktop)
GAIA (generalist)
BFCL V4 (tool use)
contamination: always report +audit
2. custom offline evals
your product's shape
LLM-as-judge (Langfuse, Phoenix, Opik)
execution-based (run patch, check tests)
trajectory-based (OSWorld-Human)
pinned seeds, snapshot state
evals live next to code
3. online evals (prod)
continuous, sampled, PII-scrubbed
session replays (Langfuse)
guardrail-triggered alerts
per-step cost / latency tracking
failure tagging (Lesson 26)
feedback loop into custom cases
evaluator-optimizer loop (Anthropic, generalized Self-Refine)
proposer
emits candidate
evaluator
judges (pass / fail + reason)
refine
feedback returns to proposer
pass ship
2026 best practice: evals in CI; fail merge on >=5% regression vs baseline
every guardrail maps to a case; every learned rule maps to a failure case
LLM-judge grounds via CRITIC (external tools); avoid judge hallucination
eval is not the last step. it is the outer loop that drives every Phase 14 choice.