I check whether the software that scores AI systems measures what it claims to. The work is done with AI coding agents operating under rules I set, and every finding below comes with a reproducer anyone can run.
- Checking the Check: Controls, Corpus Alignment, and Scale Invariance in Evaluation Software (preprint, 2026; latest version). doi:10.5281/zenodo.22973130
- Unapplied Invariants in Evaluation Instruments: A Source Audit and a Corrected Detection Null (preprint; submitted to the HAIC 2026 workshop, non-archival). doi:10.5281/zenodo.22837944
- two-coder-agreement 0.1.0: a standard-library Python calculator for two-coder nominal agreement (kappa, Krippendorff's alpha, Gwet's AC1) with item-clustered bootstrap intervals. doi:10.5281/zenodo.22972845
| Framework | Report | Outcome |
|---|---|---|
| TruLens | #2830: the answer being graded was never shown to the judge when empty | Fixed upstream in #2835 |
| TruLens | #2831: negative score from a reply with no score line | Fixed upstream in #2833 |
| TruLens | #2832: parse-failure value averaged into groundedness | Fixed upstream in #2834 |
| lm-evaluation-harness | #4238: parser ImportError scored as a wrong answer | Fix proposed in #4240 |
| HELM | #4340: undefined inputs converted to definite scores | My fix PR #4356 open |
| LightEval | #1353: empty gold reference scored as a model miss | Fix under review in #1354 |
| Ragas | #2964: missing reference scored as a model failure | Open |
| HF Evaluate | #801: undefined AUPR collapsed to 0 | Open |
ORCID: 0009-0002-0063-4014