bench: land the curated-1 benchmark manifest (20 PRs)

[bborbe]

What

bench/prs.json becomes curated-1: 20 PRs across 15 repos, replacing the 5-PR dev-1 development fixture.

	dev-1	curated-1
PRs	5	20
repos	4	15
size spread	3–783 lines	6–2024 lines
merge strategies	both	17 merge-commit, 3 squash

Why dev-1 could not carry a score

It existed to build the runner against. Measured on it (4 runs × 5 PRs, config ecc80333): findings per run 3, 2, 5, 6 → mean 4.00, sd 1.58, so 3σ ≈ 119% of the mean. Two configurations would have to differ by more than the entire average finding count before the gap registered — SC4 ("the instrument discriminates") was unattemptable, and would have failed for a reason that is an artefact of fixture size rather than a property of the instrument.

Evidence this set works

A full Opus/xhigh/full pass over curated-1 completed on 2026-08-10: 20 ok, 0 cache hit, 0 failed, 133 findings. Same set on 2026-08-09, before this session's harness fixes: 13/20 rows, 7 failures, 16 findings.

Tests

212 → 213. The fixture test now pins curated-1/20 — deliberately, since the manifest version is part of the config identity, so a change must be a conscious edit here rather than something a scored run discovers. A new test asserts the spread the set was curated for (small and large PRs present, 10+ distinct repos, both merge strategies) rather than trusting the curation.

Known gap, not addressed here

No PR in this set is designated clean. The pass produced zero zero-finding PRs, and the previous best candidate (tts-mcp#20) was hand-verified as not clean. The goal's ~40%-clean criterion needs constructed clean PRs, hand-verification, or a changed criterion — tracked on the task, not silently satisfied.

[ben-s-pull-request-reviewer]

Good. 15 distinct repos, 3 squash entries. All assertions in test_curated_manifest_holds_the_spread_it_was_curated_for are satisfied. Let me now compile the final report.

---

PR Review: bench: land the curated-1 benchmark manifest (20 PRs)

Diff summary: CHANGELOG.md (+11 lines), bench/prs.json (+241 lines), bench/test_config.py (+31 lines)

License: ✓ present

Step 4 mechanical funnel: 0 findings across 74 YAMLs

Selector mode traceability:

• Candidates: 4 judgment rules from @commits trigger
• Applicable: 1 (changelog/conventional-prefix-required)
• Skipped: git-commit/feature-branch-no-tag (no commits in diff), git-commit/imperative-mood (no commits in diff), git-workflow/never-direct-commit-to-master (no commits in diff)

Step 4d-sel adjudication: changelog/conventional-prefix-required — both new ## Unreleased bullets use correct prefixes (feat:, test:). No violations.

Manual review of changed files:

• CHANGELOG.md: Clean, correct conventional prefixes, no preamble violations
• bench/prs.json: 20 PRs, 15 distinct repos, size range 6–2024 lines, 3 squash + 17 merge-commit — matches all spread criteria
• bench/test_config.py: Two well-documented tests covering config identity (curated-1/20 pinned) and spread properties (small+large PRs, 10+ repos, both strategies)

precommit: skipped (selector mode) — CI covers lint+test

---

Must Fix (Critical)

None.

Should Fix (Important)

None.

Nice to Have (Optional)

None.

---

Step 6 (Next Steps): No test gaps found; all assertions are covered.

Step 7 (Manual Review): Done. Worktree is clean.

---

{
  "verdict": "approve",
  "summary": "The curated-1 manifest lands correctly: 20 PRs across 15 repos with the required size spread and both merge strategies, replacing the dev-1 fixture. Two new tests assert the fixture's purpose (spread properties and config identity) rather than trusting it. CHANGELOG entries use correct conventional prefixes.",
  "comments": [],
  "concerns_addressed": [
    "tests: bench/test_config.py adds test_curated_manifest_holds_the_spread_it_was_curated_for asserting small+large PRs present, 10+ distinct repos, both merge strategies — verified via grep: 15 distinct repos, min size 6 lines, max 2024 lines, 3 squash entries",
    "correctness: bench/prs.json curated-1 manifest data integrity confirmed by grep: 20 PRs, 15 repos, 6-2024 line spread, 17 merge-commit + 3 squash"
  ]
}