Adjudicate curated-1: 155 golden entries, 20 rejected

[bborbe]

What

All 115 unmatched findings from the curated-1 Opus pass, adjudicated individually across 20 PRs.

	before	after
entries	42	155
PRs covered	5	20
accepted	41	135
rejected	1	20

113 became new entries. Two turned out to be an issue the set already held, re-reported in different words (python-skeleton#3's --ignore-vuln and sudo tee -a) — those keys were relaxed rather than duplicated. One issue, one entry: a duplicate is what inflated recall to a spurious 1.000 in v0.37.0.

The rejected entries are the point

The set was 41 accepted / 1 rejected, and the report page stated outright that precision could not be lost by any configuration. There was essentially nothing a config could wrongly hit.

A finding is rejected when it is a correct observation that names no defect — polish, taste, or a consequence the finding itself concludes is intended. The clearest cluster is github-releaser-agent#8, where four findings disqualify themselves in their own text:

• "worth a follow-up covering all four together rather than typing one in isolation"
• "this is defense-in-depth, not a correctness fix"
• "within this repo's own precedent … so not a deviation"
• "raise as a dark-factory commit-template issue rather than rewriting history here"

The line, and that it is a line

Consistency mattered more than any single call, so the boundary is stated and held:

• Rejected — backup#15's unquoted ${NPM_REGISTRY}: hardening on correct code, "harmless for the current fixed HTTPS URL".
• Accepted — node-skeleton#2's check() closing over the outer config const: also "harmless today", but the function does something other than its shape implies. Reachability is not correctness.
• Rejected — comment-requests on correct code, even where they argue regression risk (recurring-task-creator#30's day-1 anchor).
• Accepted — github-releaser-agent#8's request to drop unverified generality on the path that bypasses human review. That changes code, not comments.

Verification

Every entry was checked mechanically, not by reading:

1. each signature keyword must actually appear in the body it was written for
2. no finding may match more than one entry in its PR
3. no entry's signature may be a subset of another's in the same PR

Check 2 caught one of my own keys — {{next_sun_date}} + unverified looked distinctive and matched two sibling quant#109 findings.

gap_candidates is now 0 across the pass: every one of the 133 findings maps to exactly one entry.

What the scores here do and do not mean

Scoring the curated-1 run against a set adjudicated from that run gives recall 0.844 / precision 0.857. That is a self-match, not a measurement — the same bootstrap tautology as the original seed run's 1.000. Precision is at its floor by construction because every rejected entry was derived from a finding this run produced.

The value is prospective: a different configuration now has 135 accepted entries to miss and 20 rejected entries to step on. That is what SC4 needs, and it did not exist before this PR.

Note

path may now be null on an entry. A commit-message finding ("commit c6f136a — subject is 62 chars") has no file, and since identity stopped reading path it does not need one; the live-set check required it to be truthy and rejected exactly those entries.

[ben-s-pull-request-reviewer]

{
  "verdict": "approve",
  "summary": "This PR expands the golden set to 155 curated-1 entries (135 accepted / 20 rejected) and fixes the null-path validation in test_score.py. The adjudication is thorough — rejected entries are genuinely self-disqualifying observations (taste, polish, or findings that conclude the behavior is intended). The null-path handling fix is correct: path is now provenance (not identity), so commit-message findings are legitimate with path=null. The two relaxed entries (sudo tee -a, --ignore-vuln) were properly adjusted rather than duplicated. No mechanical findings from 74 ast-grep rules. Clean approval.",
  "comments": [],
  "concerns_addressed": [
    "tests: bench/golden.json — 155 entries verified: 135 accepted, 20 rejected, 2 null-path (both legitimate: one accepted commit-finding, one rejected). Signature keyword presence confirmed, no empty signatures, no duplicate signatures within accepted.",
    "correctness: 20 rejected entries — each is a correct self-disqualifying observation: github-releaser-agent#8 cluster disqualifies itself in its own text; tts-mcp#10 Keep-a-Changelog finding is not actionable given the file's declared format; python-skeleton#3 sharp-edge is acknowledged risk; distill#6 entries note unreachable code paths. All 20 are appropriate rejections.",
    "correctness: path=null handling — 2 entries have path=null. The accepted entry (tts-mcp#10, commit c6f136a, 62-char subject) is a legitimate commit-message finding. The rejected entry (github-releaser-agent#8, prompt slugs) is also correctly null-path. Scoring logic handles null correctly since identity now reads only signature.",
    "tests: bench/test_score.py validation change — changed from assertTrue(entry.get('path')) to assertIn('path', entry), allowing null while requiring key presence. Comment correctly cites the rationale (path is provenance not identity since 2026-08-10). Matches documented intent.",
    "correctness: signature deduplication — two entries (sudo tee -a and --ignore-vuln) were relaxed (signature keywords changed from 'duplicates the repo line'/'no such surface exists' to 'appends'/'with a reason') rather than duplicated. Verified in diff. No regression in deduplication logic."
  ]
}