Measure security and data protection as primary dimensions in the efficacy benchmark

[braboj]

Round 3. Re-sequenced 2026-09-17 after round 1's report: round 2 is #1795 (file length and context model), and this round runs on its winning arm against A.

Part of #1184.

Owner decisions, 2026-09-16, taken after the round-1 report:

• Security and data protection become primary dimensions. The report leads with them, and they can trigger the K = 5 escalation.
• Round 2 extends tariff, the dev set. ledger stays sealed for the final v3.0 claim.
• Arm C's hand-written file gains data-protection lines and stays within 40 lines.
• Round 1's nine change tasks run before the specification changes.

Tasks

• Pre-register the two dimensions in docs/design/efficacy-benchmark.md: metrics, directions, margins and escalation thresholds, committed before any round-2 trial.
• Extend tests/efficacy/SPEC.md: customers with personal data, sign-in with one seeded administrator, erasure of a customer on request, and export of one customer's data. The spec names what the app must do, never the practice, because the practice is what a context file adds.
• Extend the private hidden suite and its reference implementation to the new specification, with the mutation control re-run, before any trial.
• Probe security and data protection in scoring: plaintext password at rest, a sentinel email absent from logs, erasure removing the sentinel from the database file, sign-in failures that do not enumerate accounts, no open redirect after sign-in, CSRF on sign-in, plus the round-1 post-hoc checks.
• Add security and data_protection to the judge's rubric as primary dimensions.
• Regenerate arm B from a brief naming personal data, and extend arm C within 40 lines.
• Name, in the change prompt, the API the change acceptance module drives; round 1 graded names no trial was given.
• Run round 3: 15 build and 15 change trials across the five arms, judge (about 90 judgings), report.

Code changes before round 3 (from the design review, 2026-09-24)

• Move the arms' brief and the change-task prompt out of the design doc into tests/efficacy/brief.md and tests/efficacy/change-prompt.md, and point generate_arm.py (BRIEF_HEADING) and harness.py (CHANGE_HEADING) at the files. Both are read today by exact heading, so the doc's restructure would silently break them. This lands before the design doc is rewritten.
• report.py CONTRASTS: add ("short", "full"), making eight contrasts (decided 2026-09-24).
• report.py and judge.py PRIMARY: add security and data_protection.
• Rename the arm directories B-candidate → full and C-reference → hand, and update harness.py ARMS.
• Rewrite docs/design/efficacy-benchmark.md per the owner's review: plain language, round history moved into the decisions section, §12 merged into the other sections, subsections numbered.

[braboj]

Re-sequenced to round 3 on 2026-09-17: round 2 is #1795, which holds length fixed and adds the hybrid model. This round's spec extension runs on round 2's winning arm against A.

[braboj]

Bearing on the checklist item "Add security and data_protection to the judge's rubric as primary dimensions", from #1825 and #1827 (2026-09-18/19):

• Anchor them. Unanchored, design and maintainability returned 3 on all 24 trials of rounds 1 and 2 — a holistic 1-5 question answers "ordinary" every time. Each new primary needs its 1, 3 and 5 described in terms of the specification, as the three existing primaries now have in judge.py's prompt.
• Validate them with the control before a round uses them. tests/efficacy/control/ needs a tree that damages each new dimension and one that improves it, and the row has to fall on the first and rise on the second. A row that only falls can never show an arm winning — which is what the unanchored primaries did.
• Judge three times and mean. Section 12 pre-registers N = 3 for round 3.
• Comparability. The anchored prompt makes round 3's primaries incomparable to rounds 1-2 unless the earlier trees are re-judged into it.

The rest of this checklist still speaks of arms B and C and of round 2; which arm round 3 runs is still the owner's call.

[braboj]

Owner decision, 2026-09-24: round 3 starts anew and runs all five arms: none, full, short, hybrid and hand. This replaces the earlier plan of running round 2's winning arm against none.

Why no trial from rounds 1-2 is reused:

• The spec changed (feat(efficacy): the spec adds sign-in and customers for round 3 #1840). Sign-in and customers mean old and new trees built different apps, so they cannot be paired.
• The judge changed (feat(efficacy): anchor the primary dimensions and judge each trial three times #1830, fix(efficacy): the judge reads each tree against the spec it was built from #1839). Primaries are anchored and judged three times, and rounds 1-2 are not comparable to that.

Rounds 1-2 stay in docs/audits/ as history. Round 3 becomes the one comparison run that v3.0's Phase 3 is measured against.

Effect on the checklist:

• "Regenerate arm B … extend arm C" now means: regenerate full, short and hybrid from a brief that names personal data, and extend hand within 40 lines.
• "Run round 2: 9 build trials" now means: run round 3, 5 arms × K = 3 = 15 build trials plus 15 change tasks, judged N = 3.

[braboj]

Owner decision, 2026-09-24: round 3 reports eight contrasts. These are the seven already in tests/efficacy/report.py CONTRASTS, plus one new contrast:

• full − none, short − none, hybrid − none, hand − none: does each file beat no file
• full − hand: do the generated templates beat hand-written lines
• short − hand: the same question with length held equal
• hybrid − full: the hybrid model against the same content inline
• short − full: does length matter (new)

CONTRASTS gains ("short", "full") and the design doc lists the same eight, both before any round-3 trial. PRIMARY in report.py also gains the security and data-protection dimensions, per the checklist above.

[braboj]

Owner decision, 2026-09-24: security and data protection are each read by an anchored judge row and a deterministic probe pass rate, and all four are primary.

Metric	Margin	Smallest change that matters
judge_security, judge_data_protection	0.3 points	0.5 points
Security probe pass rate, 11 probes	2 pp	9 pp
Data-protection probe pass rate, 3 probes	10 pp	30 pp

• Probe margin: less than one probe lost in one trial, so no probe may be lost.
• Probe threshold: one probe lost in every trial.
• Security probes: the four sign-in probes on this checklist plus security.py's seven checks, promoted from declared-after-a-run to fixed-before.
• Data-protection probes: a sentinel email in the logs, an erased sentinel still in the database file, and a customer export carrying another customer's data.

Recorded in design §5.8 by #1847. PRIMARY, MARGINS and PRACTICAL in report.py follow when the probes and the anchored rows are built.