feat(sleep): paired A/B evalkit with McNemar and bootstrap CIs

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Related: #108

What Problem This Solves

Resolves a problem where Sleep contributors and operators have no shared instrument for claiming condition B beats condition A: comparisons are single-run numbers on possibly different task sets, with no uncertainty reported, so sub-noise deltas are claimable and real regressions can hide.

Upstream docs/sleep/RESULTS.md already warns that single-seed deltas under about 1.5 points are noise, and issue #108 asks for a single-seed versus multi-seed protocol. This PR is that instrument.

Why This Change Was Made

A small stdlib evalkit module runs two conditions over one fixed task manifest (paired by task id), scores per-task binary outcomes, and reports McNemar's test for the paired difference plus a percentile-bootstrap confidence interval on the success-rate delta. When each task carries same-length seed repeats, the kit also publishes per-seed deltas plus their mean and sample sd (the house answer to #108).

It does not change the nightly gate. It standardizes the evidence that reports and later PRs cite. Cross-manifest comparisons are refused. Graded (non-binary) scores are bootstrap-only and require an explicit allow flag, because McNemar is not defined for them.

CLI:

python -m skillopt_sleep.evalkit --manifest tasks.json --a cond_a.json --b cond_b.json

Also wired as skillopt_sleep evalkit. Built on current upstream main at da06b15 (includes #235).

Project Fit

• Empirically validated, meaningful performance comparisons: paired design on identical tasks is the strongest cheap comparison available, and uncertainty is quantified rather than implied.
• Robustness across scenarios: A/A calibration is part of the test suite, so a miscoded statistic fails closed instead of shipping a silent false positive.

User Impact

Contributors get a one-command harness that turns an experiment into a reviewable JSON plus markdown report. Operators who never invoke it see zero change: no config keys, no gate behavior, no extra backend calls.

Proof

Before (stock main at da06b15): python -m skillopt_sleep.evalkit does not exist; there is no in-repo McNemar or bootstrap helper; RESULTS cells are quoted as point estimates.

After (this pull request's commits, head 79447e311807a03639eded8fadf8d7467eb9035e):

Textbook 2x2 fixture (tests/fixtures/evalkit/mcnemar_textbook.json): both+=40, A-only=2, B-only=12, both-=46. Uncorrected chi-square is 100/14 = 7.142857142857143, p_chi2 = 0.007526315166457887, two-sided exact binomial p_exact = 0.012939453125. Tests pin all three to 12 decimal places.

A/A fixture (40 identical tasks): delta 0, p_exact = 1, bootstrap CI includes 0. The A/A CLI path exits 0 on that fixture and exits 2 on mismatched ids.

RESULTS replay: SearchQA / GPT-5.4-nano / gated / cumulative nights=5 (n=1400, 0.560 to 0.679, published delta +0.119). Per-task pairs were not published, so the replay uses a documented maximum-concordance reconstruction (first round(n * rate) tasks succeed in each condition). The harness recovers the published delta and a CI that excludes 0. It does not claim to recover the original microdata.

Re-run on 2026-08-21 against this pull request's head, Linux, Python 3.12.3, after uv venv and uv pip install -e ".[dev]":

$ uv run pytest tests/test_evalkit.py -q
17 passed in 0.32s

$ uv run pytest -q
1394 passed, 11 skipped, 8 warnings, 269 subtests passed in 20.40s

Same commands on parent da06b15 (this pull request's first parent, no evalkit):

$ uv run pytest -q
1377 passed, 11 skipped, 8 warnings, 269 subtests passed in 19.88s

The delta is exactly the 17 new evalkit tests, with zero regressions.

A/A CLI on the shipped fixture (same environment):

$ uv run python -m skillopt_sleep.evalkit \
  --manifest tests/fixtures/evalkit/aa_manifest.json \
  --a tests/fixtures/evalkit/aa_outcomes.json --aa
n_tasks: 40
rate_a: 0.500000
rate_b: 0.500000
delta (B-A): +0.000000
bootstrap 95% CI: [+0.000000, +0.000000] (n_boot=10000, seed=42)
McNemar 2x2: both+=20 a_only=0 b_only=0 both-=20
McNemar chi2=0.0000 p_chi2=1 p_exact=1 significant=False

Academic Support

1. McNemar (1947). "Note on the sampling error of the difference between correlated proportions or percentages." Psychometrika 12(2). The canonical paired test for per-task binary outcomes on one manifest.
2. Efron (1979). "Bootstrap Methods: Another Look at the Jackknife." Annals of Statistics 7(1). Distribution-free CIs for small, odd-shaped agent-benchmark samples.
3. Berg-Kirkpatrick, Burkett, Klein (2012). "An Empirical Investigation of Statistical Significance in NLP." EMNLP 2012. Metric gains below noise are routinely claimed without paired testing; that is the failure mode this kit's defaults close.
4. Dror, Baumer, Shlomov, Reichart (2018). "The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing." ACL 2018. The test-selection decision tree this kit's binary-paired defaults follow.
5. Miller (2024). "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations." arXiv:2411.00640. Recent, directly-on-point guidance for reporting uncertainty in LLM evals.

Testing

New: tests/test_evalkit.py, 17 tests pinning textbook McNemar, bootstrap determinism and coverage on a known shift, A/A non-rejection, mismatched/empty/duplicate id refusal, graded-score bootstrap-only path, multi-seed variance bands, RESULTS cell replay, and CLI exit codes. Full suite: uv run pytest -q (or python -m pytest -q).

Platform	Python	Result on this pull request
Linux	3.12	1394 passed, 11 skipped, 8 warnings, 269 subtests, 0 failed
macOS	3.12	1356 passed, 38 failed, 11 skipped, 8 warnings, 269 subtests. Failures match stock upstream path checks, not this diff. Evalkit wiring: 17 passed.
Windows	3.12	1311 passed, 76 failed, 18 skipped, 8 warnings, 269 subtests. Failures match stock upstream POSIX-only adapters, not this diff. Evalkit wiring: 17 passed.

Limitations & Negative Results

• Binary-outcome focus first. Graded scores use bootstrap only until a paired-permutation extension lands; McNemar is omitted unless every score is 0/1.
• Paired design requires identical manifests. Cross-manifest comparisons are out of scope and the CLI refuses them (exit 2).
• The RESULTS replay reconstructs pairs from published rates. It proves the harness recovers the published delta under a documented pairing rule; it is not a claim that those were the original per-task outcomes.

Reproduce It Yourself

Check out this pull request's commits and run:

uv venv
uv pip install -e ".[dev]"
uv run pytest tests/test_evalkit.py -q
uv run pytest -q
uv run python -m skillopt_sleep.evalkit \
  --manifest tests/fixtures/evalkit/aa_manifest.json \
  --a tests/fixtures/evalkit/aa_outcomes.json --aa

Equivalent without uv: python -m venv .venv && ./.venv/bin/python -m pip install -e ".[dev]" then the same pytest and module commands through that interpreter.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Working and running the test suite comprehensively across Linux and Mac and Windows. Will try to finish today.

[Bogdan (Dan) Baciu (bogdanbaciu21)]

Sorry it took me so long to get the full test suite out.

[Yifan Yang (Yif-Yang)]

Thanks for adding the paired-evaluation workflow and for running the cross-platform suite. I merged this head with current main; the full suite passes, but the statistical path still has correctness blockers that the tests do not cover.

1. exact_mcnemar_p() evaluates math.comb(n, k) * (0.5 ** n). exact_mcnemar_p(700, 700) currently raises OverflowError: int too large to convert to float. Please use a numerically stable exact binomial-tail implementation (recurrence/log space, or a well-tested binomial survival function).
2. Invalid inputs can produce invalid or misleading reports: empty seed arrays are accepted and yield NaN rates/CIs; NaN/Infinity scores can reach non-standard JSON; and an out-of-range alpha can produce nonsensical significance. Please reject empty seeds, non-finite/out-of-range scores, invalid bootstrap counts, and any alpha outside (0, 1) with EvalkitError, and ensure serialized JSON never contains NaN/Infinity.
3. For multi-seed data, McNemar currently treats every (task, seed) pair as an independent observation, while the bootstrap operates on per-task means. This mixes inference units and can inflate significance through pseudoreplication. Please make the task the independent/cluster unit (or use and document an equivalent cluster-aware method) consistently across the reported inference.

Please add regressions for more than 1,000 discordant pairs, empty seeds, NaN/Infinity and out-of-range scores, invalid alpha/bootstrap values, strict JSON serialization, and a multi-seed fixture showing that duplicating seeds within one task does not spuriously increase significance. The current A/A test compares an array with itself, so it should also be replaced by a calibration test capable of catching type-I-error regressions. The feature is useful, but these issues need to be fixed before merge.