--- id: eval-driven-agents title: Eval-Driven Agent Development code: ED subtitle: Run eval cases, gate merges, review with a rubric, and gate prompt and report changes. Python stdlib. hue: 140 sources: phases/14-agent-engineering/30-eval-driven-agent-development, phases/14-agent-engineering/38-verification-gates, phases/14-agent-engineering/39-reviewer-agent, projects/prompt-regression-tester, projects/report-judge source_hash: 59650a4ecbe2d8fb --- ## Builder to sign-off type: architecture from: phases/14-agent-engineering/38-verification-gates/docs/en.md, phases/14-agent-engineering/39-reviewer-agent/docs/en.md ```arch node | builder | 1,1 | Builder Agent | writes the diff | model node | arts | 2,1 | Artifacts | diff, scope, rules, feedback | store node | gate | 3,1 | Verification gate | deterministic, no LLM | process node | report | 4,1 | verification_report.json | passed true or false | store node | refuse | 4,2 | Refuse done | surface to human | client node | rev | 3,2 | Reviewer Agent | read only, rubric | model node | human | 2,2 | Human sign-off | reads review_report.json | client edge | builder | arts | edge | arts | gate | edge | gate | report | edge | report | refuse | no edge | report | rev | yes edge | rev | human | ``` ## Three evaluation layers type: comparison from: phases/14-agent-engineering/30-eval-driven-agent-development/docs/en.md | Factor | Static benchmarks | Custom offline | Online | |---|---|---|---| | Examples | SWE-bench Verified, GAIA, BFCL V4 | LLM-as-judge, execution, trajectory | Session replays, guardrail alerts | | Use for | Cross-model comparison | Your product shape | Production behavior | | Measures | Broad capability | Right patches for your product | Per-step cost and latency | | Watch for | Contamination, 32.67% leakage | Judges hallucinate too | Tie each failure to a new case | ## Evaluator-optimizer loop type: flow from: phases/14-agent-engineering/30-eval-driven-agent-development/code/main.py, phases/14-agent-engineering/30-eval-driven-agent-development/docs/en.md 1. **Proposer**: `case.proposer(feedback)` gives a candidate. 2. **Judge**: `case.judge(candidate)` returns ok and a reason. 3. **Refine**: the reason becomes the next feedback. 4. **Stop**: after `max_rounds` the case fails. 5. **CI gate**: `ci_gate` blocks a regression above the threshold. ## Run the lessons and projects type: command-table span: 2 from: phases/14-agent-engineering/30-eval-driven-agent-development/docs/en.md, phases/14-agent-engineering/38-verification-gates/code/main.py, projects/prompt-regression-tester/README.md, projects/prompt-regression-tester/solution/cli.py, projects/report-judge/README.md, projects/report-judge/solution/cli.py | Command | What it does | |---|---| | `python3 code/main.py` | Runs the lesson demo. Lessons 38 and 39 write JSON reports | | `VERIFY_DEMO_MODE=1 python3 code/main.py --strict` | Gate demo with the demo secret. Every warn becomes a block | | `python3 code/main.py --floor ` | Gate demo with a different coverage floor | | `python3 scripts/project_test.py --init learning-artifacts/` | Makes a learner workspace | | `python3 scripts/project_test.py --stage --path learning-artifacts/` | Tests one stage of your workspace | | `python3 scripts/project_test.py --all --solution --strict` | Checks the reference solution | | `python3 /cli.py --out --markdown ` | Prompt diff. Add `--allow-configuration-change` for a model change | | `python3 /cli.py --out --html ` | Report audit. Add `--research-payload` for a report payload | ## Gate one task type: annotated-code lang: python from: phases/14-agent-engineering/38-verification-gates/code/main.py ```python art = Artifacts(task_id="T-001", acceptance_commands=accept, # zero exit means done feedback=[{"command": accept[0], "exit_code": 0}], scope_report={"forbidden_writes": [], "off_scope_writes": []}, rule_report=[{"slug": "done/tests-pass", "passed": True}], coverage_report={"current": 0.84, "previous": 0.85}, head_commit="a1b2c3d") report = verify(art, strict=False, coverage_floor=0.80) report.passed # False on any block finding record_override(task_id, finding_code, reason, user_id, head_commit) verify_signature(entry) # HMAC check of the override ``` ## Reviewer rubric type: concepts from: phases/14-agent-engineering/39-reviewer-agent/docs/en.md, phases/14-agent-engineering/39-reviewer-agent/code/main.py - **Problem fit**: solved the task as stated? - **Scope discipline**: edits inside the contract? - **Assumptions**: written down? - **Verification quality**: acceptance proves the goal? - **Handoff readiness**: next session can pick up? - **Scores**: 0 to 2 each, total out of 10. - **Verdict**: `pass` at 7 or more, else `soft_fail`. - **`hard_fail`**: total below 5 or any 0. ## Defaults and numbers type: defaults from: phases/14-agent-engineering/30-eval-driven-agent-development/code/main.py, phases/14-agent-engineering/38-verification-gates/code/main.py, projects/prompt-regression-tester/solution/main.py, projects/report-judge/solution/support.py, projects/report-judge/solution/compare.py, projects/report-judge/solution/metrics.py | Setting | Default | Note | |---|---|---| | `max_rounds` | 3 | Evaluator-optimizer rounds per case | | `regression_threshold` | 0.05 | `ci_gate` blocks above it | | `COVERAGE_FLOOR_DEFAULT` | 0.80 | Below it is a block | | `COVERAGE_REGRESSION_DELTA` | 0.01 | Larger drop than the last merge blocks | | `release_gate` `min_pass` | 1.0 | Pass fraction in [0,1] | | `release_gate` `max_regressions` | 0 | Integer regression budget | | `judge_claim` `threshold` | 0.8 | Overlap score to count as supported | | `compare` `samples` | 2000 | Bootstrap resamples, `seed=7` | ## Prompt Regression Tester type: task-table from: projects/prompt-regression-tester/solution/main.py, projects/prompt-regression-tester/solution/cli.py, projects/prompt-regression-tester/stages/03-pair-revisions/docs/en.md | Task | Code | |---|---| | Unique ids, known check kinds | `validate_cases(cases)` | | Check kinds | `contains`, `excludes`, `json` | | Score one recorded response | `score_response(response, checks)` | | Pair revisions by case id | `compare(cases, baseline, candidate)` | | Pair states | `stable_pass`, `regressed`, `improved` | | Ship or block | `release_gate(report)` | | Bind recording to cases | `case_sha256 == case_digest(cases)` | | Load a provenance-bound run | `load_run(raw, cases)` | ## Report Judge type: task-table from: projects/report-judge/solution/claims.py, projects/report-judge/solution/support.py, projects/report-judge/solution/metrics.py, projects/report-judge/solution/compare.py | Task | Code | |---|---| | One claim per sentence | `parse_claims(text)` | | Lexical support of one claim | `support(claim, source)` | | Judge cites against evidence | `judge_claim(claim, evidence)` | | Reasons for a zero score | `negation`, `number`, `uncited` | | Precision, recall, coverage | `score_report(text, evidence)` | | Score weights | `precision` 0.5, `recall` 0.25 | | States | `no_evidence`, `needs_review` | | Paired bootstrap | `compare(baseline, candidate)` | ## Troubleshooting type: troubleshooting span: 2 from: phases/14-agent-engineering/30-eval-driven-agent-development/docs/en.md, phases/14-agent-engineering/38-verification-gates/code/main.py, phases/14-agent-engineering/38-verification-gates/docs/en.md, projects/prompt-regression-tester/solution/cli.py, projects/prompt-regression-tester/stages/02-score-recordings/docs/en.md, projects/report-judge/stages/02-support/docs/en.md | Symptom | Cause | Fix | |---|---|---| | Eval scores are unreadable | No baseline | Store a last-known-good baseline | | Judge passes wrong output | LLM judge without grounding | Ground the judge on external tools | | Evals pass, production fails | Over-fitting to evals | Rotate cases | | False alarms in CI | Flaky, non-deterministic cases | Pin seeds, snapshot state | | `refused to start: VERIFY_OVERRIDE_SECRET is unset` | No override secret | Set the env var, or `VERIFY_DEMO_MODE=1` for the demo | | `recording does not match cases` | Case list changed after recording | Make new recordings | | `model/settings changed; explicitly allow this experiment` | Baseline and candidate configs differ | Pass `--allow-configuration-change` | | `order_requires_review` on a true claim | Lexical rule rejects a paraphrase | A human reviews the claim | ## Questions type: qa from: phases/14-agent-engineering/38-verification-gates/docs/en.md, phases/14-agent-engineering/39-reviewer-agent/docs/en.md, projects/prompt-regression-tester/stages/03-pair-revisions/docs/en.md, projects/prompt-regression-tester/stages/04-gate-releases/docs/en.md, projects/report-judge/stages/03-metrics/docs/en.md - **Why no LLM in the gate?** The gate gives the same verdict for the same artifacts every time. - **Who overrides a block?** Only a human, with `override_reason` and `overridden_by`. - **Why a separate reviewer?** Role separation: different prompt, different inputs, no write access. - **Why pair cases by id?** An average can hide a serious regression. - **Why is recall null?** Recall and coverage need labels. Without labels they are unavailable. ## Tips type: tips from: phases/14-agent-engineering/30-eval-driven-agent-development/docs/en.md, phases/14-agent-engineering/38-verification-gates/docs/en.md, phases/14-agent-engineering/39-reviewer-agent/docs/en.md - Keep evals next to code. Run them in CI on every PR. - Map every guardrail and every learned rule to an eval case. - Write one report per task to `outputs/verification/.json`. - Use `--strict` on release branches only. It is not the global default. - Run the reviewer on a 10-20 task calibration set after each prompt change. - Below 80% agreement, revise the rubric before the reviewer ships. ## Gate finding codes type: concepts from: phases/14-agent-engineering/38-verification-gates/code/main.py - **`acceptance.missing`**: block. A command never ran. - **`acceptance.failed`**: block. Nonzero exit. - **`feedback.null_exit`**: block. No exit code. - **`scope.forbidden`**: block. **`scope.off_scope`**: warn. - **`coverage.below_floor`**: block. **`coverage.missing`**: warn.