Help evaluators decide whether an automated pairwise judge is useful for their task. Blind candidate identities, swap answer positions, run authored length counterfactuals and compare decisions with held-out human anchors. Export disagreement and bias breakdowns plus a versioned rubric and judge configuration receipt.
Proposed milestones
- Create blinded pairs and separated human anchors
- Run position swaps and length counterfactuals
- Measure disagreement and slice-specific bias
- Export a rubric with held-out validation evidence
This is an upstream project brief; an implementation and completion tests are not supplied.