docs(design): rewrite the efficacy benchmark design for its reader

[braboj]

Part of #1767. This is the design-doc rewrite from the owner's section-by-section review on 2026-09-24, after the two PRs it depended on (#1843, #1844).

What changes

Readability

• Plain words, with jargon defined at first use. "Pre-registered" is gone, "trim" and "confound" are defined, and units are spelled out.
• Tables for settings, bullets for steps, and a one-line "Why" after each rule.
• Mermaid diagrams for the verdict rule (§1.1), the build trial (§4.1) and the scoring pipeline (§5).
• A worked example up front (§1), and a plain overview of tariff with the coffee example and a definition of SKU (§2.1).
• Every subsection numbered; §4 no longer has a lone 4.1.

History out of the body

• Every dated decision and rejected alternative moves to one decision log (§10): date, decision, why, issue.
• Derivations (K = 3 limits, judge noise, the budget figures, the hidden-suite validation) move to an appendix (§11).
• §12 "Round 2" dissolves. Its rules that still hold move to §3, §5 and §6.

Content brought up to date

• Five arms, their budgets and trial names (§3). Round 3 starts anew.
• Eight contrasts, matching report.CONTRASTS (§6.1).
• The judge as it runs: anchored primaries, three judgings per tree, and the control fixture. The second (Claude) judge reads the control only (§5.6, §5.7).
• Only the tools and probes the code runs (§5.2, §5.3). Mutation score, cohesion and pylint's refactoring checks are listed as not measured.
• The security checks now, and the round-3 plan to make security and data protection primary (§5.8). Their margins are still to be fixed before round 3's first trial, per Measure security and data protection as primary dimensions in the efficacy benchmark #1767.
• The confounds table gains the controls the code already has: judge noise, the anchored scale, withdrawn metrics, lost trials, spec changes between rounds, and hybrid's vendored templates. Row 1 now describes the refusal the harness actually makes.
• The budget from measured round 1–2 figures (§8).

Code comments that cited §1.1, §11 or §12 now cite §1.3, §3.3 and §6.1. Comments only; no behaviour changes.

Checks

The report self test passes (83/83), as do reuse (9/9) and generate_arm. Smoke: 33, 0 failed. sync.py --check passes. Conformance: 0 failed. I also checked every line outside tables and code fences: none is over 88 characters.

For the reviewer

• Accuracy: please read §5.8, §8 and §10. The security probes come from Measure security and data protection as primary dimensions in the efficacy benchmark #1767's checklist, and the budget figures come from the round 1 and 2 reports.
• Decision log reasons: every "Why" in the log comes from the old text or its issue. The 2026-09-16 security decision records only that it was the owner's call, because Measure security and data protection as primary dimensions in the efficacy benchmark #1767 gives no reason.

Changelog

No entry. This is the repository's own design document, and nothing a consuming project reads.

🤖 Generated with Claude Code