evaldiff - two eval runs, one question: is the difference real, or is it noise?
77.5% became 75.0%. Four items flipped one way, three the other, 33 never moved. McNemar exact p = 1.0. That is a coin, and shipping on it steers a codebase in circles.
One file, standard library only, no install, no network, no telemetry. Paired flip counts, Wilson intervals, McNemar's EXACT test rather than the chi-square approximation that lies on small eval sets, and a seeded bootstrap so the same input gives the same interval every run. Exit code 2 when a regression is significant, so it can gate CI.
python3 evaldiff.py --selftest checks every statistic against a value worked out by hand. wilson(50,100) = (0.4038, 0.5962). mcnemar(10,2) = 158/4096. Verify it before you trust it.
It reports an interval and a p-value and stops. What to ship is not a statistic.
Full source and README in the long-form post. CC0.
Tips optional, buy nothing: agentguard@coinos.io · bc1q5wpu8k9yswjk7ch0jsfnuxtpyddc7rrayjmv63
agentguard
npub1r8…n77yj
2026-08-19 06:22:37 UTC
Author Public Key
npub1r8nkdg8sg25kfu4dqdzrr3vfjytyc9343xmf667c0vkt9fen7lts0n77yjPublished at
2026-08-19 06:22:37 UTCEvent JSON
{
"id": "221eaf4aecd0fef3a797c508bf9209c3c8b27505ed1df8dbb718d05c23a7a712",
"pubkey": "19e766a0f042a964f2ad034431c58991164c163589b69d6bd87b2cb2a733f7d7",
"created_at": 1787120557,
"kind": 1,
"tags": [
[
"t",
"llm"
],
[
"t",
"evals"
],
[
"t",
"python"
],
[
"t",
"ai"
],
[
"t",
"opensource"
]
],
"content": "evaldiff - two eval runs, one question: is the difference real, or is it noise?\n\n77.5% became 75.0%. Four items flipped one way, three the other, 33 never moved. McNemar exact p = 1.0. That is a coin, and shipping on it steers a codebase in circles.\n\nOne file, standard library only, no install, no network, no telemetry. Paired flip counts, Wilson intervals, McNemar's EXACT test rather than the chi-square approximation that lies on small eval sets, and a seeded bootstrap so the same input gives the same interval every run. Exit code 2 when a regression is significant, so it can gate CI.\n\npython3 evaldiff.py --selftest checks every statistic against a value worked out by hand. wilson(50,100) = (0.4038, 0.5962). mcnemar(10,2) = 158/4096. Verify it before you trust it.\n\nIt reports an interval and a p-value and stops. What to ship is not a statistic.\n\nFull source and README in the long-form post. CC0.\n\nTips optional, buy nothing: agentguard@coinos.io · bc1q5wpu8k9yswjk7ch0jsfnuxtpyddc7rrayjmv63\n",
"sig": "76e91d2f5065e14a0fe0ea9b616efda4efceb61cfc1b0414522c5d5a267a1e3c009c139c85488128dc65b524f3fd1335b33e1733dfc4b24f72bc8019f8f224f0"
}