feat(sleep): paired A/B evalkit with McNemar and bootstrap CIs

CHANGELOG.md

@@ -7,6 +7,10 @@ All notable changes to SkillOpt are documented here. This project adheres to
 ## [Unreleased]
 
 ### Added
+- **SkillOpt-Sleep paired A/B evalkit** (`python -m skillopt_sleep.evalkit`):
+  McNemar plus percentile-bootstrap CIs on a fixed task manifest, with
+  multi-seed variance bands and an A/A calibration. The nightly gate is
+  unchanged (thanks @bogdanbaciu21).
 - **SkillOpt-Sleep multi-skill fan-out and reviewed subset adoption**: each
   hinted skill is consolidated from its own pinned live baseline, staged as an
   independent proposal with per-skill gate evidence, and promoted only through

docs/reference/cli.md

@@ -123,8 +123,11 @@ skillopt-sleep <action> [options]
 python -m skillopt_sleep <action> [options]
 ```
 
-Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`, and
-`unschedule`. Common options include:
+Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`,
+`unschedule`, and `evalkit`. `evalkit` is also available as
+`python -m skillopt_sleep.evalkit` and compares two conditions on one fixed
+task manifest (McNemar + bootstrap CI). See `docs/sleep/evalkit.md`. Common
+options for the nightly actions include:
 
 | Argument | Description |
 |---|---|

docs/sleep/README.md

@@ -339,6 +339,20 @@ correctness signal; the validation gate still governs what ships.
 | `recall_k` | `0` | Associative recall — pull the K most-similar past tasks (from a persisted archive) into tonight's dream. |
 | `dream_factor` | `0` | Add N lightweight synthetic variants of each task. |
 
+### Paired A/B evalkit
+
+Reports and PRs that claim "B beats A" should go through the shared evalkit
+rather than quoting a single-run cell. One command pairs two conditions on one
+fixed task manifest, runs McNemar's test, and reports a bootstrap CI on the
+success-rate delta. The nightly gate is unchanged.
+
+```text
+python -m skillopt_sleep.evalkit --manifest tasks.json --a cond_a.json --b cond_b.json
+```
+
+See [`evalkit.md`](evalkit.md) for the id-set contract, multi-seed bands, the
+A/A calibration, and the published RESULTS cell replay.
+
 ## Results
 
 > 📊 **More results & analysis — the gate-safety stress test, experience-replay

docs/sleep/evalkit.md

@@ -0,0 +1,53 @@
+# Paired A/B evalkit
+
+Sleep contributors have a shared instrument for "condition B beats condition A":
+
+```text
+python -m skillopt_sleep.evalkit --manifest tasks.json --a cond_a.json --b cond_b.json
+```
+
+The kit pairs outcomes by task id, runs McNemar's test on binary successes, and
+reports a percentile-bootstrap confidence interval on the success-rate delta.
+It does not change the nightly gate.
+
+## Inputs
+
+- `--manifest`: JSON list of task ids, or `{"ids": [...]}` / `{"tasks": [{"id": ...}]}`.
+- `--a` / `--b`: JSON objects mapping those same ids to `0`/`1` (or a list of
+  per-seed `0`/`1` values). A wrapper `{"outcomes": {...}}` is also accepted.
+- `--aa`: A/A calibration (reuses `--a` as both conditions). Must not reject.
+- `--allow-graded`: permit non-binary scores. McNemar is omitted; bootstrap only.
+- `--boot`, `--seed`, `--alpha`, `--json`.
+
+The id sets of the manifest, A, and B must be identical. Cross-manifest
+comparisons are refused.
+
+## Multi-seed
+
+When each task maps to a same-length list of seed repeats, the kit:
+
+1. averages per task across seeds for the headline delta and bootstrap CI
+2. pools `(task, seed)` pairs for McNemar
+3. publishes the per-seed deltas plus their mean and sample sd
+
+That is the house answer to single-seed noise (see issue #108 and the
+single-seed warning in `RESULTS.md`).
+
+## RESULTS cell replay
+
+`tests/fixtures/evalkit/results_searchqa_nano_gated.json` replays the published
+SearchQA / GPT-5.4-nano / gated / cumulative nights=5 cell (baseline 0.560,
+after 0.679, Δ +11.9 on n=1400). Per-task pairs were not published, so the
+replay uses a documented maximum-concordance reconstruction: the first
+`round(n * rate)` tasks succeed in each condition. The harness recovers the
+published delta; it does not claim to recover the original microdata.
+
+## A/A check
+
+```text
+python -m skillopt_sleep.evalkit --manifest tests/fixtures/evalkit/aa_manifest.json \
+  --a tests/fixtures/evalkit/aa_outcomes.json --aa
+```
+
+Identical conditions must report delta 0, McNemar p_exact = 1, and a CI that
+includes 0. If they do not, the statistics are miscoded.

skillopt_sleep/__main__.py

@@ -882,6 +882,19 @@ def main(argv=None) -> int:
     p_unsched = sub.add_parser("unschedule", help="remove the nightly cron entry")
     _add_common(p_unsched)
     p_unsched.add_argument("--all", action="store_true", help="remove all managed entries")
+    p_eval = sub.add_parser(
+        "evalkit",
+        help="paired A/B comparison (McNemar + bootstrap CI)",
+    )
+    p_eval.add_argument("--manifest", required=True)
+    p_eval.add_argument("--a", required=True)
+    p_eval.add_argument("--b", default="")
+    p_eval.add_argument("--aa", action="store_true")
+    p_eval.add_argument("--alpha", type=float, default=0.05)
+    p_eval.add_argument("--boot", type=int, default=10000)
+    p_eval.add_argument("--seed", type=int, default=42)
+    p_eval.add_argument("--allow-graded", action="store_true")
+    p_eval.add_argument("--json", action="store_true")
 
     args = parser.parse_args(argv)
     if args.cmd == "run":
@@ -898,6 +911,19 @@ def main(argv=None) -> int:
         return cmd_schedule(args)
     if args.cmd == "unschedule":
         return cmd_unschedule(args)
+    if args.cmd == "evalkit":
+        from skillopt_sleep.evalkit import main as evalkit_main
+        argv = ["--manifest", args.manifest, "--a", args.a]
+        if args.b:
+            argv.extend(["--b", args.b])
+        if args.aa:
+            argv.append("--aa")
+        argv.extend(["--alpha", str(args.alpha), "--boot", str(args.boot), "--seed", str(args.seed)])
+        if args.allow_graded:
+            argv.append("--allow-graded")
+        if args.json:
+            argv.append("--json")
+        return evalkit_main(argv)
     parser.print_help()
     return 2