Thomas Piacentini
← Workshop

Project brief

Abstention Calibration Workbench

Original text · English

Help teams set a defensible accept-or-defer rule for a bounded task. Tune a score threshold on labeled development records under explicit error costs, freeze that policy, and evaluate held-out risk and coverage across slices. Export a decision-policy file and audit that shows how many requests need human review.

Proposed milestones

  1. Validate separated development and holdout records
  2. Tune thresholds under declared error costs
  3. Freeze and evaluate the policy on holdout slices
  4. Export a decision adapter and review-volume estimate

This is an upstream project brief; an implementation and completion tests are not supplied.

Code and artifacts 1
What you take away

Saved in your notebook on this device.

Sources and limits

Reading checkpoints are self-reported. Test results are recorded separately and apply to the executed cases.

AI Engineering from Scratch · 3f9715c · MIT

Download this file ↓