From bcfdb0cd5b56db2354c48312694a74e15c2a304a Mon Sep 17 00:00:00 2001 From: Benjamin Borbe Date: Mon, 10 Aug 2026 11:33:05 +0200 Subject: [PATCH] golden-set identity is the signature alone, not path or line --- CHANGELOG.md | 8 + bench/README.md | 8 +- bench/golden.json | 1169 +++++++++++++++++------------- bench/run.py | 66 +- bench/test_score.py | 273 ++++++- bench/testdata/golden-dev-1.json | 1144 ++++++++++++++++------------- 6 files changed, 1628 insertions(+), 1040 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5da6875..c69f238 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,14 @@ Please choose versions by [Semantic Versioning](http://semver.org/). * MINOR version when you add functionality in a backwards-compatible manner, and * PATCH version when you make backwards-compatible bug fixes. +## Unreleased + +- fix: golden-set identity is the signature alone. `rule_id` is now an additional constraint when both sides carry one, never a short-circuit, and neither `path` nor `line` is read. Both defects were demonstrated on real rows before the change: the `github-pr-review-agent#11` CHANGELOG entry (a bullet bundling three concerns) matched an unrelated `fix:`-vs-`chore:` prefix finding purely because both carried `changelog/conventional-prefix-required`, so any configuration finding *some* instance of a rule was credited with the specific one; and the "write-scoped token still minted under `--skip-post`" issue, keyed to `pkg/factory/runner.go`, scored a miss when a later run anchored it at `cmd/run-task/main.go` while naming `runner.go` in its own body +- fix: all 42 golden entries re-keyed from `path:line` strings to semantic keywords drawn from code identifiers (`S104`, `apt-key`, `ErrorRequestHandler`) rather than prose, which is one author's phrasing and drifts between runs. 37 of the 43 signatures embedded a line reference, so `recall` measured whether a configuration cited the same *line* rather than whether it found the issue. Measured, not estimated: the curated-1 Opus pass — the identical model/effort/mode that seeded the set — scored recall **0.262** against it; after the re-key, **0.463**, while the run the set was seeded from still self-matches **1.000** +- fix: two `docs/pr-post-back.md` entries merged into one. Both described the stale "backward compatibility" framing in a single paragraph, and a later run reported them as one finding — which would have matched two entries and double-counted recall +- feat: `load_golden` rejects a signature that embeds a line reference or carries fewer than two keywords. The prior convention was documented (`line is NEVER used for identity`) but unenforced, and the seeded signatures smuggled the line back in as a substring; one entry had already been hand-fixed line-free in an earlier adjudication without the convention propagating to the other 42 +- fix: the scoring-mutates-nothing test compared `git diff --exit-code` against the working tree, so it failed for any developer holding a legitimate uncommitted fixture edit and would have passed a mutation that was staged. It now compares testdata digests before and after the scoring subprocess, the same way the ledger is already checked + ## v0.40.0 - fix: findings citing an extensionless file are no longer dropped. `PATH_LINE_RE` required a literal dot, so `Dockerfile:8`, `Makefile:127`, `Jenkinsfile:4` and `LICENSE:1` could not be attributed and every such item was discarded. Measured live on the 2026-08-09 curated-1 pass: `backup#15` produced four correctly-formed findings, all on Dockerfile/Makefile, and lost **all four** — the row scored 0 findings and read as a **clean PR**, the worst available outcome given that "clean" is a designation the benchmark cannot otherwise establish. Across the pass, 5 of 133 findings (4%) were being dropped, concentrated so that one row lost 100% of its content diff --git a/bench/README.md b/bench/README.md index 99912c1..5d02eea 100644 --- a/bench/README.md +++ b/bench/README.md @@ -158,7 +158,9 @@ The four ledger slices under `bench/testdata/` are verbatim extracts of real led ## Scoring -**Matching and the three golden states.** The match rule is exact and deterministic. For a `(golden entry, finding)` pair within the same `pr_id`: if both sides carry a non-null `rule_id`, they match when those ids are equal and do not match otherwise — a rule-id disagreement is decisive and no fallback is attempted. Otherwise they match when the `path` field is string-equal and **every** keyword in the entry's `signature` appears as a case-insensitive substring of the finding's `body`. Otherwise they do not match. `line` is never used for identity — it is display data only. Path comparison is whole-string equality with no extension requirement, so an extensionless path such as `Dockerfile` compares like any other. +**Matching and the three golden states.** The match rule is exact and deterministic. For a `(golden entry, finding)` pair within the same `pr_id`: they match when **every** keyword in the entry's `signature` appears as a case-insensitive substring of the finding's `body`. When both sides carry a non-null `rule_id`, the ids must also be equal — `rule_id` is an additional constraint, never a short-circuit. An entry with an empty `signature` never matches. + +Neither `path` nor `line` is part of identity. Both are coordinates the reviewer chooses when citing an issue, not properties of the issue: the same defect is routinely anchored at the line that defines a flag or at the line that suffers its consequence, in two different files. Both remain on the entry and the finding as provenance and display data. Signatures must carry at least two keywords and may not embed a line reference; `load_golden` rejects a set that violates either. Every golden entry carries one of three states, and the state governs what a hit or miss costs. An `accepted` entry — one the reviewer should produce — costs recall when unmatched and costs nothing when matched. A `rejected` entry — a known false positive — costs precision when matched and costs nothing when unmatched. An `unreviewed` entry — not yet adjudicated — is excluded from both numerator and denominator of both ratios, and a finding matching one is neither a penalty nor a candidate. @@ -166,7 +168,7 @@ Every golden entry carries one of three states, and the state governs what a hit **What precision currently measures.** `golden-dev-1` carries zero `rejected` entries, so precision cannot be lost by any configuration. A precision of `1.000` is a property of the golden set's adjudication state and is **not yet a result**. -**What recall currently measures.** 36 of the 42 signatures embed a line reference, so a re-report of the same issue at a different line does not match and surfaces as a gap-triage candidate rather than a hit. On this golden set, `recall` measures whether a configuration cited the same line, not whether it found the issue. Across the four runs of config `9ce66e05…` only 5 of 16 findings hit an entry, and the `apt-key` issue in `.github/workflows/ci.yml` is reported in all four runs at lines 13, 29, 9 and 27 and is a gap candidate every time. +**What recall measures.** Issue detection. This was not always true: until 2026-08-10, 37 of the 43 signatures embedded a line reference, so `recall` measured whether a configuration cited the same *line*. The cost was measured, not estimated — the `curated-1` Opus run, at the identical model/effort/mode that seeded the golden set, scored **0.262** against it; after re-keying to semantic signatures, **0.463**, while the run the set was seeded from still self-matches **1.000**. The report page recomputes this caveat from the entries on every render, so a future adjudication that reintroduces a `path:line` key says so on the page instead of silently degrading recall. **Runs are an occurrence index, never a timestamp cluster.** Within one `config_hash`, the k-th ledger row for a given `pr_id`, in ledger file order, belongs to run k. Run boundaries need no clock, no threshold, and no tuning, and they survive clock skew, a slow PR, and an operator pausing between PRs. The boundary between run 2 and run 3 of config `9ce66e05…` is 48 seconds while gaps *within* runs 1, 2 and 4 reach 140, 128 and 110 seconds — no time threshold separates them in either direction. A run that does not cover every PR in the manifest is labelled **partial** on the report page, and a partial run is scored over the PRs it actually covers: the missing PRs' golden entries are **out of scope for that run, not misses**, so a 3-PR run scores against 24 entries and a 1-PR run against 1. @@ -208,7 +210,7 @@ These are deliberately not configurable: - **Inline rule tag:** `*(rule: \`\`)*` is read positionally from the item and recorded verbatim; it is **not** validated against `rules/index.json`; the head-anchored backtick fallback is **index-gated** (only used when no inline tag is present and the token is a known rule id) - **Unattributable-item rejection:** `UNATTRIBUTABLE FINDING` fires when an item inside a severity section yields neither `path` nor `rule_id`; no ledger row, no row marker, no opt-out - **Stderr excerpt bound:** at most 2,000 bytes of rejected output are printed to stderr (truncation is marked) -- **Match rule:** `rule_id` exact when both sides carry one; otherwise `path` string equality plus every signature keyword present case-insensitively in body; `line` is never used for identity; path comparison is whole-string with no extension requirement +- **Match rule:** every signature keyword present case-insensitively in body; `rule_id` an additional constraint when both sides carry one, never a short-circuit; neither `path` nor `line` is part of identity; signatures need >= 2 keywords and may not embed a line reference - **Golden states:** `accepted` (recall miss on no-match), `rejected` (precision penalty on match), `unreviewed` (excluded from both ratios); a finding matching no entry is a gap-triage candidate, **not a precision failure** - **Run chunking:** per-PR occurrence index in ledger file order; the k-th row for a given `pr_id` belongs to run k; no clock, no threshold - **Report location:** `bench/reports/<64-lowercase-hex>.md`; full hash, no truncation; tracked in git diff --git a/bench/golden.json b/bench/golden.json index c3b2670..199dd2a 100644 --- a/bench/golden.json +++ b/bench/golden.json @@ -1,506 +1,667 @@ { - "version": "golden-dev-2", - "created": "2026-08-09", - "prs_version": "dev-1", - "baseline": { - "model": "opus", - "effort": "xhigh", - "mode": "full", - "coding_version": "v0.35.6", - "config_hash_prefix": "cc64cc99", - "rules_commands_hash_prefix": "ecc80333", - "runs": 1, - "findings": 42, - "wall_time_seconds": 2534 - }, - "match_rule": "rule_id exact when both sides carry one; else path + ALL signature keywords present in body (case-insensitive). line is NEVER used for identity.", - "states": { - "accepted": "a finding the reviewer SHOULD produce. A miss costs recall.", - "rejected": "a known false positive. A hit costs precision.", - "unreviewed": "not adjudicated. Excluded from both numerator and denominator." - }, - "scoring_note": "All entries are seeded 'accepted' from a single Opus 5 best-case run - this is the designed bootstrap, not hand-curated ground truth. Consequence: a configuration reporting a finding ABSENT from this set is NOT automatically wrong. Report such findings separately as gap-triage candidates rather than as precision failures. When several independent configurations agree on a finding Opus missed, that is evidence the golden set is incomplete - promote it to 'accepted'. When nothing else ever reproduces an Opus finding, that is evidence it may be a false positive - demote it to 'rejected'. The set sharpens through disagreement; it is not frozen truth.", - "known_corrections": [ - "tts-mcp#20 was annotated in bench/prs.json as 'clean - correct answer is zero findings'. That is FALSE. It returned 0 findings across six sonnet runs (six misses, not cleanliness); Opus found a real defect, hand-verified against head SHA 84feeb4d: CHANGELOG.md has no '## Unreleased' section while .maintainer.yaml sets autoRelease: true, so the CVE fix would never ship. The fixture currently has NO verified-clean PR." - ], - "entries": [ - { - "pr_id": "github-pr-review-agent#11", - "path": "CHANGELOG.md", - "signature": [ - "changelog.md:11" - ], - "rule_id": "changelog/conventional-prefix-required", - "state": "accepted", - "line_when_seen": 11, - "excerpt": "**`CHANGELOG.md:11`** — one `feat:` bullet bundles three concerns: the new flag (`feat:`), the nil-poster panic guard in `tryDismissHallucinated` (`fix:`), and the CLI doc re-attribution (`docs:`). Sp" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "cmd/run-task/main.go", - "signature": [ - "cmd/run-task/main.go:171" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 171, - "excerpt": "**`cmd/run-task/main.go:171`** — the `SkipPost: a.SkipPost` pass-through and its counterpart call site at `pkg/factory/runner.go:136` are the only untested lines of the change: `cmd/run-task/main_test" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "docs/pr-post-back.md", - "signature": [ - "docs/pr-post-back.md:128" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 128, - "excerpt": "**`docs/pr-post-back.md:128`** — \"This preserves backward compatibility with the local CLI mode\" is now false. Before this PR `cmd/run-task` never produced a nil poster (that was the documented-vs-act" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "docs/pr-post-back.md", - "signature": [ - "docs/pr-post-back.md:126" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 126, - "excerpt": "**`docs/pr-post-back.md:126`** — heading `## nil Poster — Local / Backward-Compatible Mode` carries the same stale framing as the line-128 finding; update both together (e.g. \"Local Opt-In (`--skip-po" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/factory.go", - "signature": [ - "pkg/factory/factory.go:166-173" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 166, - "excerpt": "**`pkg/factory/factory.go:166-173`** — `CreateAgent`'s doc comment states the execution phase \"posts review to GitHub via PrPoster\" unconditionally, and parenthesises only the verifier case (\"nil veri" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/runner.go", - "signature": [ - "pkg/factory/runner.go:104-106" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 104, - "excerpt": "**`pkg/factory/runner.go:104-106`** — a live, write-scoped GitHub App installation token is injected into the Claude subprocess env unconditionally, including under `--skip-post`; nothing in this PR t" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/runner.go", - "signature": [ - "pkg/factory/runner.go:51-53" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 51, - "excerpt": "**`pkg/factory/runner.go:51-53`** — `RunConfig.Agent`'s comment still frames the nil-`Agent` branch as \"cmd/run-task leaves this nil so CreateAgent is used for backward compatibility\". That branch is " - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/runner.go", - "signature": [ - "pkg/factory/runner.go:66" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 66, - "excerpt": "**`pkg/factory/runner.go:66`** — `ResolvePosters` is exported solely so `pkg/skip_post_boundary_test.go` (a different package's test binary, where an `export_test.go` alias would be invisible) can rea" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/runner_test.go", - "signature": [ - "pkg/factory/runner_test.go:32-33" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 32, - "excerpt": "**`pkg/factory/runner_test.go:32-33`** — `Expect(poster).To(BeNil())` cannot detect the regression this PR is themed around. Gomega's `BeNil()` falls through to `reflect.ValueOf(x).IsNil()` for pointe" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/skip_post_boundary_test.go", - "signature": [ - "pkg/skip_post_boundary_test.go:169-226" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 169, - "excerpt": "**`pkg/skip_post_boundary_test.go:169-226`** — case 4's stated failure mechanism is wrong. The comment claims absence of an `ai_review verify:` line in `## Diagnostics` is the negative signal, but `ap" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/skip_post_boundary_test.go", - "signature": [ - "pkg/skip_post_boundary_test.go:102-166" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 102, - "excerpt": "**`pkg/skip_post_boundary_test.go:102-166`** — case 3 does not exercise the skip-post contract it is labelled with. `ResolvePosters(RunConfig{SkipPost:true})` can only ever return `(nil, nil)`, so the" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/skip_post_boundary_test.go", - "signature": [ - "pkg/skip_post_boundary_test.go:64-100" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 64, - "excerpt": "**`pkg/skip_post_boundary_test.go:64-100`** — case 2 and `pkg/steps_review_test.go:614-627` case (j) drive an identical scenario (verdict=fail + hallucinations + nil poster) and assert the same two ou" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/steps_checkout_execution.go", - "signature": [ - "pkg/steps_checkout_execution.go:316" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 316, - "excerpt": "**`pkg/steps_checkout_execution.go:316`** — the nil-poster early return fires before `ParseVerdict` (:328), before the fail-closed `!funnelRan && approve → request-changes` gate (:338-340), and before" - }, - { - "pr_id": "node-skeleton#2", - "path": "README.md", - "signature": [ - "readme.md:107" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 107, - "excerpt": "**`README.md:107`** — the new License section links the file without naming the license; the repo's own README template names it inline (`BSD-2-Clause — see [LICENSE](./LICENSE).`)." - }, - { - "pr_id": "node-skeleton#2", - "path": "eslint.config.js", - "signature": [ - "eslint.config.js:44" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 44, - "excerpt": "**`eslint.config.js:44`** — the comment (\"enum, namespace, and parameter-property syntax would make `node src/*.ts` require a build step\") sits directly above `@typescript-eslint/no-explicit-any` and " - }, - { - "pr_id": "node-skeleton#2", - "path": "eslint.config.js", - "signature": [ - "eslint.config.js:16" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 16, - "excerpt": "**`eslint.config.js:16`** — `tseslint.config(...)` is marked `@deprecated` in the pinned typescript-eslint 8.66.0; I confirmed the annotation at `typescript-eslint/dist/config-helper.d.ts:67` (\"ESLint" - }, - { - "pr_id": "node-skeleton#2", - "path": "eslint.config.js", - "signature": [ - "eslint.config.js:39-55" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 39, - "excerpt": "**`eslint.config.js:39-55`** — `CLAUDE.md` bans \"`any`, `as`, `!`, `@ts-ignore`\". Three are enforced (`no-explicit-any`, `no-non-null-assertion`, inherited `ban-ts-comment`); `as` is enforced by nothi" - }, - { - "pr_id": "node-skeleton#2", - "path": "package.json", - "signature": [ - "package.json:8" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 8, - "excerpt": "**`package.json:8`** — `\"node\": \">=22\"` no longer matches what the code requires. Node ships unflagged type-stripping only from **22.18.0**; on 22.6–22.17 it needs `--experimental-strip-types`, and be" - }, - { - "pr_id": "node-skeleton#2", - "path": "src/handlers/health.ts", - "signature": [ - "src/handlers/health.ts:12" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 12, - "excerpt": "**`src/handlers/health.ts:12`** — same defect: `HealthRouter` is asserted, not derived, and `healthRouter` (line 27) carries no annotation tying it to the alias. Note `src/config.ts:34` and `src/log.t" - }, - { - "pr_id": "node-skeleton#2", - "path": "src/server.ts", - "signature": [ - "src/server.ts:31" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 31, - "excerpt": "**`src/server.ts:31`** — `export type CreateApp` is hand-written and never bound to `createApp` (line 40), so `tsc` never compares the declared contract to the implementation; the only bridge is `requ" - }, - { - "pr_id": "node-skeleton#2", - "path": "src/server.ts", - "signature": [ - "src/server.ts:69" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 69, - "excerpt": "**`src/server.ts:69`** — the error handler types `err: Error`, narrower than the shipped `ErrorRequestHandler`, which I confirmed declares `err: any` (`@types/express-serve-static-core/index.d.ts:77`)" - }, - { - "pr_id": "node-skeleton#2", - "path": "test/health.test.ts", - "signature": [ - "config.check()" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": null, - "excerpt": "**`test/health.test.ts`** — `config.check()` is the PR's new startup-validation logic and has zero coverage; `test/` contains only `health.test.ts`, which never imports config. Neither the invalid-`PO" - }, - { - "pr_id": "node-skeleton#2", - "path": "test/health.test.ts", - "signature": [ - "test/health.test.ts:22" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 22, - "excerpt": "**`test/health.test.ts:22`** — `? address.port : 0` silently falls back to port 0 instead of failing. If the invariant ever breaks, every test in the file fetches `http://127.0.0.1:0` and reports a co" - }, - { - "pr_id": "python-skeleton#3", - "path": ".github/workflows/ci.yml", - "signature": [ - "apt-key" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": null, - "excerpt": "Trivy's signing key is piped into the deprecated `apt-key add`. Reported at five different lines across runs (13/29/30/31/32), so the signature is deliberately line-free; it was originally pinned to ci.yml:32 by the Opus baseline." - }, - { - "pr_id": "python-skeleton#3", - "path": ".github/workflows/ci.yml", - "signature": [ - ".github/workflows/ci.yml:35" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 35, - "excerpt": "**`.github/workflows/ci.yml:35`** — `apt-get install -y trivy` is unpinned, so every CI run takes whatever is latest. Trivy minor releases add detectors and change exit semantics, so CI can go red wit" - }, - { - "pr_id": "python-skeleton#3", - "path": ".github/workflows/ci.yml", - "signature": [ - ".github/workflows/ci.yml:33" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 33, - "excerpt": "**`.github/workflows/ci.yml:33`** — `sudo tee -a` appends, so a re-run duplicates the repo line. Use `tee` without `-a`." - }, - { - "pr_id": "python-skeleton#3", - "path": ".github/workflows/ci.yml", - "signature": [ - ".github/workflows/ci.yml:38" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 38, - "excerpt": "**`.github/workflows/ci.yml:38`** — `uv sync --all-extras` is redundant; `make precommit` runs `sync` as its first prerequisite." - }, - { - "pr_id": "python-skeleton#3", - "path": "CHANGELOG.md", - "signature": [ - "changelog.md:18" - ], - "rule_id": "changelog/conventional-prefix-required", - "state": "accepted", - "line_when_seen": 18, - "excerpt": "**`CHANGELOG.md:18`** — `- ci: install trivy in CI` uses a prefix outside the recognised set. The guide's table defines exactly seven (`feat:`, `fix:`, `refactor:`, `test:`, `docs:`, `chore:`, `perf:`" - }, - { - "pr_id": "python-skeleton#3", - "path": "Makefile.precommit", - "signature": [ - "makefile.precommit:42" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 42, - "excerpt": "**`Makefile.precommit:42`** — the comment promises that unreachable findings \"get an explicit `--ignore-vuln` entry with a reason\", but no such surface exists: no file, no make variable, nothing in th" - }, - { - "pr_id": "python-skeleton#3", - "path": "Makefile.precommit", - "signature": [ - "makefile.precommit:24" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 24, - "excerpt": "**`Makefile.precommit:24`** — `check: lint typecheck audit trivy` makes the default target and `make precommit` hard-require a local `trivy` binary plus network egress (trivy DB from ghcr, pip-audit a" - }, - { - "pr_id": "python-skeleton#3", - "path": "Makefile.precommit", - "signature": [ - "makefile.precommit:60-61" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 60, - "excerpt": "**`Makefile.precommit:60-61`** — the `$(ROOTDIR)/.trivyignore` fallback is unreachable today: the scan target is always literal `.`, so when make runs from the repo root the first `$(wildcard .trivyig" - }, - { - "pr_id": "python-skeleton#3", - "path": "README.md", - "signature": [ - "readme.md:80" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 80, - "excerpt": "**`README.md:80`** and **`Makefile.precommit:36`** — \"covers the exact transitive tree that gets installed\" is not accurate. `make audit` runs `uv export --all-extras` (includes the `dev` extra: pytes" - }, - { - "pr_id": "python-skeleton#3", - "path": "README.md", - "signature": [ - "readme.md:76-94" - ], - "rule_id": "readme/user-facing-not-agent-context", - "state": "accepted", - "line_when_seen": 76, - "excerpt": "**`README.md:76-94`** — the whole \"Security gates\" section is design rationale, not user-facing content: why a make target depends on `sync` (line 82), why `osv-scanner` was rejected (line 86), and a " - }, - { - "pr_id": "python-skeleton#3", - "path": "README.md", - "signature": [ - "readme.md:84" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 84, - "excerpt": "**`README.md:84`** — the S104/S101 suppression rationale duplicates the comments already in `pyproject.toml:65-67` and `pyproject.toml:71`, which will drift." - }, - { - "pr_id": "python-skeleton#3", - "path": "pyproject.toml", - "signature": [ - "pyproject.toml:64-68" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 64, - "excerpt": "**`pyproject.toml:64-68`** — `ignore = [\"S104\"]` disables hardcoded-bind-all-interfaces across the entire repo. There are 4 hits: `src/skeleton/server.py:9`, `src/skeleton/__main__.py:33`, and 2 in `t" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/dev/docker-registry-gc.yaml", - "signature": [ - "due_date" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 67, - "excerpt": "**`task/recurring-schedules/dev/docker-registry-gc.yaml:67`** — left without `due_date` while its prod sibling `prod/docker-registry-gc.yaml` got one. It is `recurrence: Weekday, weekday: Saturday` — " - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/plan-next-week.yaml", - "signature": [ - "task/recurring-schedules/prod/plan-next-week.yaml:27-29" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 27, - "excerpt": "**`task/recurring-schedules/prod/plan-next-week.yaml:27-29`** — Success Criteria was rewritten to cover priorities, acceptance criteria, and `# Explicitly Deferred`, but Tasks line 43 also requires al" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/plan-next-week.yaml", - "signature": [ - "6db0dc9" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 29, - "excerpt": "**`task/recurring-schedules/prod/plan-next-week.yaml:29`** — \"Month link present (target may not exist yet if the monthly plan is deferred)\" explicitly sanctions a dead wikilink. Defensible for a note" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/plan-weekend.yaml", - "signature": [ - "frontmatter:" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 65, - "excerpt": "**`task/recurring-schedules/prod/plan-weekend.yaml:65`** — `{{next_sun_date}}` is the first use of a non-`{{current_date}}` placeholder in a `frontmatter:` field anywhere in this repo. On the base bra" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/plan-weekend.yaml", - "signature": [ - "contradicts" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 65, - "excerpt": "**`task/recurring-schedules/prod/plan-weekend.yaml:65`** — `due_date: \"{{next_sun_date}}\"` contradicts the task it annotates. Body line 18: \"as the first thing Saturday morning\"; the plan covers Sat→S" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/renew-gmail-oauth-tokens.yaml", - "signature": [ - "{{current_date}}" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 64, - "excerpt": "**`task/recurring-schedules/prod/renew-gmail-oauth-tokens.yaml:64`** — `due_date: \"{{next_sun_date}}\"` on a task whose own body (line 18) states \"Google's tokens expire every 7 days for unverified app" - }, - { - "pr_id": "tts-mcp#20", - "path": "CHANGELOG.md", - "signature": [ - "changelog.md:8" - ], - "rule_id": "changelog/unreleased-entry-required", - "state": "accepted", - "line_when_seen": 8, - "excerpt": "**`CHANGELOG.md:8`** — no `## Unreleased` section exists (top section is `## v0.6.0`), so this PR adds no changelog entry. `.maintainer.yaml` sets `release.autoRelease: true`, and the release agent pr" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "README.md", - "signature": [ - "distinct labels" - ], - "rule_id": null, - "state": "rejected", - "line_when_seen": 42, - "excerpt": "Suggests renaming two 'Local *' rows in the run-modes table to 'more distinct labels'. The table is not wrong and nothing is ambiguous in use; this is taste, not a defect. Adjudicated rejected 2026-08-09 so precision has something to measure against." - } - ], - "adjudications": [ - "2026-08-09 RELAX python-skeleton#3 .github/workflows/ci.yml apt-key: signature changed from ['.github/workflows/ci.yml:32'] to ['apt-key'] so the same issue matches regardless of cited line. Reported by 11 of the 102 ledger findings at five different lines. NOTE: an earlier attempt ADDED a second line-free entry instead of relaxing this one; that duplicated the issue and 3 findings matched both entries, inflating recall. Corrected here - one issue, one entry.", - "2026-08-09 REJECT github-pr-review-agent#11 README.md 'distinct labels' -> rejected. Subjective doc-polish on a table that is not wrong. Matches exactly 1 finding in the ledger. First rejected entry; before it, precision could not be lost by any configuration.", - "Adjudication check that was missing and is now required: a new or relaxed signature must be tested for ALIASING against the other golden entries, not only against the ledger findings. Checking signature->findings alone let a duplicate through." - ] + "version": "golden-dev-3", + "created": "2026-08-09", + "prs_version": "dev-1", + "baseline": { + "model": "opus", + "effort": "xhigh", + "mode": "full", + "coding_version": "v0.35.6", + "config_hash_prefix": "cc64cc99", + "rules_commands_hash_prefix": "ecc80333", + "runs": 1, + "findings": 42, + "wall_time_seconds": 2534 + }, + "match_rule": "EVERY signature keyword present case-insensitively in the finding body. rule_id is an ADDITIONAL constraint when both sides carry one, never a short-circuit. Neither path nor line is read: both are citation coordinates the reviewer chooses, not properties of the defect.", + "states": { + "accepted": "a finding the reviewer SHOULD produce. A miss costs recall.", + "rejected": "a known false positive. A hit costs precision.", + "unreviewed": "not adjudicated. Excluded from both numerator and denominator." + }, + "scoring_note": "All entries are seeded 'accepted' from a single Opus 5 best-case run - this is the designed bootstrap, not hand-curated ground truth. Consequence: a configuration reporting a finding ABSENT from this set is NOT automatically wrong. Report such findings separately as gap-triage candidates rather than as precision failures. When several independent configurations agree on a finding Opus missed, that is evidence the golden set is incomplete - promote it to 'accepted'. When nothing else ever reproduces an Opus finding, that is evidence it may be a false positive - demote it to 'rejected'. The set sharpens through disagreement; it is not frozen truth.", + "known_corrections": [ + "tts-mcp#20 was annotated in bench/prs.json as 'clean - correct answer is zero findings'. That is FALSE. It returned 0 findings across six sonnet runs (six misses, not cleanliness); Opus found a real defect, hand-verified against head SHA 84feeb4d: CHANGELOG.md has no '## Unreleased' section while .maintainer.yaml sets autoRelease: true, so the CVE fix would never ship. The fixture currently has NO verified-clean PR." + ], + "entries": [ + { + "pr_id": "github-pr-review-agent#11", + "path": "CHANGELOG.md", + "signature": [ + "bullet bundles", + "split" + ], + "rule_id": "changelog/conventional-prefix-required", + "state": "accepted", + "line_when_seen": 11, + "excerpt": "**`CHANGELOG.md:11`** \u2014 one `feat:` bullet bundles three concerns: the new flag (`feat:`), the nil-poster panic guard in `tryDismissHallucinated` (`fix:`), and the CLI doc re-attribution (`docs:`). Sp", + "superseded_signature": [ + "changelog.md:11" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "cmd/run-task/main.go", + "signature": [ + "SkipPost: a.SkipPost", + "untested" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 171, + "excerpt": "**`cmd/run-task/main.go:171`** \u2014 the `SkipPost: a.SkipPost` pass-through and its counterpart call site at `pkg/factory/runner.go:136` are the only untested lines of the change: `cmd/run-task/main_test", + "superseded_signature": [ + "cmd/run-task/main.go:171" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "docs/pr-post-back.md", + "signature": [ + "preserves backward compatibility", + "local CLI mode" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 128, + "excerpt": "**`docs/pr-post-back.md:128`** \u2014 \"This preserves backward compatibility with the local CLI mode\" is now false. Before this PR `cmd/run-task` never produced a nil poster (that was the documented-vs-act", + "superseded_signature": [ + "docs/pr-post-back.md:128" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/factory.go", + "signature": [ + "CreateAgent", + "posts review to GitHub" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 166, + "excerpt": "**`pkg/factory/factory.go:166-173`** \u2014 `CreateAgent`'s doc comment states the execution phase \"posts review to GitHub via PrPoster\" unconditionally, and parenthesises only the verifier case (\"nil veri", + "superseded_signature": [ + "pkg/factory/factory.go:166-173" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/runner.go", + "signature": [ + "write-scoped", + "--skip-post" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 104, + "excerpt": "**`pkg/factory/runner.go:104-106`** \u2014 a live, write-scoped GitHub App installation token is injected into the Claude subprocess env unconditionally, including under `--skip-post`; nothing in this PR t", + "superseded_signature": [ + "pkg/factory/runner.go:104-106" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/runner.go", + "signature": [ + "RunConfig.Agent", + "backward compatibility" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 51, + "excerpt": "**`pkg/factory/runner.go:51-53`** \u2014 `RunConfig.Agent`'s comment still frames the nil-`Agent` branch as \"cmd/run-task leaves this nil so CreateAgent is used for backward compatibility\". That branch is ", + "superseded_signature": [ + "pkg/factory/runner.go:51-53" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/runner.go", + "signature": [ + "ResolvePosters", + "exported solely" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 66, + "excerpt": "**`pkg/factory/runner.go:66`** \u2014 `ResolvePosters` is exported solely so `pkg/skip_post_boundary_test.go` (a different package's test binary, where an `export_test.go` alias would be invisible) can rea", + "superseded_signature": [ + "pkg/factory/runner.go:66" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/runner_test.go", + "signature": [ + "BeNil()", + "reflect.ValueOf" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 32, + "excerpt": "**`pkg/factory/runner_test.go:32-33`** \u2014 `Expect(poster).To(BeNil())` cannot detect the regression this PR is themed around. Gomega's `BeNil()` falls through to `reflect.ValueOf(x).IsNil()` for pointe", + "superseded_signature": [ + "pkg/factory/runner_test.go:32-33" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/skip_post_boundary_test.go", + "signature": [ + "the comment claims", + "Diagnostics" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 169, + "excerpt": "**`pkg/skip_post_boundary_test.go:169-226`** \u2014 case 4's stated failure mechanism is wrong. The comment claims absence of an `ai_review verify:` line in `## Diagnostics` is the negative signal, but `ap", + "superseded_signature": [ + "pkg/skip_post_boundary_test.go:169-226" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/skip_post_boundary_test.go", + "signature": [ + "ResolvePosters(RunConfig{SkipPost:true})", + "(nil, nil)" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 102, + "excerpt": "**`pkg/skip_post_boundary_test.go:102-166`** \u2014 case 3 does not exercise the skip-post contract it is labelled with. `ResolvePosters(RunConfig{SkipPost:true})` can only ever return `(nil, nil)`, so the", + "superseded_signature": [ + "pkg/skip_post_boundary_test.go:102-166" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/skip_post_boundary_test.go", + "signature": [ + "case 2", + "steps_review_test.go" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 64, + "excerpt": "**`pkg/skip_post_boundary_test.go:64-100`** \u2014 case 2 and `pkg/steps_review_test.go:614-627` case (j) drive an identical scenario (verdict=fail + hallucinations + nil poster) and assert the same two ou", + "superseded_signature": [ + "pkg/skip_post_boundary_test.go:64-100" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/steps_checkout_execution.go", + "signature": [ + "early return", + "ParseVerdict" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 316, + "excerpt": "**`pkg/steps_checkout_execution.go:316`** \u2014 the nil-poster early return fires before `ParseVerdict` (:328), before the fail-closed `!funnelRan && approve \u2192 request-changes` gate (:338-340), and before", + "superseded_signature": [ + "pkg/steps_checkout_execution.go:316" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "README.md", + "signature": [ + "License section", + "BSD-2-Clause" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 107, + "excerpt": "**`README.md:107`** \u2014 the new License section links the file without naming the license; the repo's own README template names it inline (`BSD-2-Clause \u2014 see [LICENSE](./LICENSE).`).", + "superseded_signature": [ + "readme.md:107" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "eslint.config.js", + "signature": [ + "parameter-property", + "no-explicit-any" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 44, + "excerpt": "**`eslint.config.js:44`** \u2014 the comment (\"enum, namespace, and parameter-property syntax would make `node src/*.ts` require a build step\") sits directly above `@typescript-eslint/no-explicit-any` and ", + "superseded_signature": [ + "eslint.config.js:44" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "eslint.config.js", + "signature": [ + "tseslint.config", + "@deprecated" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 16, + "excerpt": "**`eslint.config.js:16`** \u2014 `tseslint.config(...)` is marked `@deprecated` in the pinned typescript-eslint 8.66.0; I confirmed the annotation at `typescript-eslint/dist/config-helper.d.ts:67` (\"ESLint", + "superseded_signature": [ + "eslint.config.js:16" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "eslint.config.js", + "signature": [ + "no-non-null-assertion", + "ban-ts-comment" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 39, + "excerpt": "**`eslint.config.js:39-55`** \u2014 `CLAUDE.md` bans \"`any`, `as`, `!`, `@ts-ignore`\". Three are enforced (`no-explicit-any`, `no-non-null-assertion`, inherited `ban-ts-comment`); `as` is enforced by nothi", + "superseded_signature": [ + "eslint.config.js:39-55" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "package.json", + "signature": [ + "22.18", + "type-stripping" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 8, + "excerpt": "**`package.json:8`** \u2014 `\"node\": \">=22\"` no longer matches what the code requires. Node ships unflagged type-stripping only from **22.18.0**; on 22.6\u201322.17 it needs `--experimental-strip-types`, and be", + "superseded_signature": [ + "package.json:8" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "src/handlers/health.ts", + "signature": [ + "HealthRouter", + "asserted, not derived" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 12, + "excerpt": "**`src/handlers/health.ts:12`** \u2014 same defect: `HealthRouter` is asserted, not derived, and `healthRouter` (line 27) carries no annotation tying it to the alias. Note `src/config.ts:34` and `src/log.t", + "superseded_signature": [ + "src/handlers/health.ts:12" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "src/server.ts", + "signature": [ + "export type CreateApp", + "never bound" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 31, + "excerpt": "**`src/server.ts:31`** \u2014 `export type CreateApp` is hand-written and never bound to `createApp` (line 40), so `tsc` never compares the declared contract to the implementation; the only bridge is `requ", + "superseded_signature": [ + "src/server.ts:31" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "src/server.ts", + "signature": [ + "ErrorRequestHandler", + "err: any" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 69, + "excerpt": "**`src/server.ts:69`** \u2014 the error handler types `err: Error`, narrower than the shipped `ErrorRequestHandler`, which I confirmed declares `err: any` (`@types/express-serve-static-core/index.d.ts:77`)", + "superseded_signature": [ + "src/server.ts:69" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "test/health.test.ts", + "signature": [ + "check()", + "health.test.ts" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": null, + "excerpt": "**`test/health.test.ts`** \u2014 `config.check()` is the PR's new startup-validation logic and has zero coverage; `test/` contains only `health.test.ts`, which never imports config. Neither the invalid-`PO", + "superseded_signature": [ + "config.check()" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "test/health.test.ts", + "signature": [ + "address.port", + "port 0" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 22, + "excerpt": "**`test/health.test.ts:22`** \u2014 `? address.port : 0` silently falls back to port 0 instead of failing. If the invariant ever breaks, every test in the file fetches `http://127.0.0.1:0` and reports a co", + "superseded_signature": [ + "test/health.test.ts:22" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": ".github/workflows/ci.yml", + "signature": [ + "apt-key", + "trivy" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": null, + "excerpt": "Trivy's signing key is piped into the deprecated `apt-key add`. Reported at five different lines across runs (13/29/30/31/32), so the signature is deliberately line-free; it was originally pinned to ci.yml:32 by the Opus baseline.", + "superseded_signature": [ + "apt-key" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": ".github/workflows/ci.yml", + "signature": [ + "apt-get install -y trivy", + "unpinned" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 35, + "excerpt": "**`.github/workflows/ci.yml:35`** \u2014 `apt-get install -y trivy` is unpinned, so every CI run takes whatever is latest. Trivy minor releases add detectors and change exit semantics, so CI can go red wit", + "superseded_signature": [ + ".github/workflows/ci.yml:35" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": ".github/workflows/ci.yml", + "signature": [ + "sudo tee -a", + "duplicates the repo line" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 33, + "excerpt": "**`.github/workflows/ci.yml:33`** \u2014 `sudo tee -a` appends, so a re-run duplicates the repo line. Use `tee` without `-a`.", + "superseded_signature": [ + ".github/workflows/ci.yml:33" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": ".github/workflows/ci.yml", + "signature": [ + "uv sync --all-extras", + "redundant" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 38, + "excerpt": "**`.github/workflows/ci.yml:38`** \u2014 `uv sync --all-extras` is redundant; `make precommit` runs `sync` as its first prerequisite.", + "superseded_signature": [ + ".github/workflows/ci.yml:38" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "CHANGELOG.md", + "signature": [ + "ci: install trivy", + "prefix outside the recognised set" + ], + "rule_id": "changelog/conventional-prefix-required", + "state": "accepted", + "line_when_seen": 18, + "excerpt": "**`CHANGELOG.md:18`** \u2014 `- ci: install trivy in CI` uses a prefix outside the recognised set. The guide's table defines exactly seven (`feat:`, `fix:`, `refactor:`, `test:`, `docs:`, `chore:`, `perf:`", + "superseded_signature": [ + "changelog.md:18" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "Makefile.precommit", + "signature": [ + "--ignore-vuln", + "no such surface exists" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 42, + "excerpt": "**`Makefile.precommit:42`** \u2014 the comment promises that unreachable findings \"get an explicit `--ignore-vuln` entry with a reason\", but no such surface exists: no file, no make variable, nothing in th", + "superseded_signature": [ + "makefile.precommit:42" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "Makefile.precommit", + "signature": [ + "check: lint typecheck audit trivy", + "network egress" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 24, + "excerpt": "**`Makefile.precommit:24`** \u2014 `check: lint typecheck audit trivy` makes the default target and `make precommit` hard-require a local `trivy` binary plus network egress (trivy DB from ghcr, pip-audit a", + "superseded_signature": [ + "makefile.precommit:24" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "Makefile.precommit", + "signature": [ + "$(ROOTDIR)/.trivyignore", + "always wins" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 60, + "excerpt": "**`Makefile.precommit:60-61`** \u2014 the `$(ROOTDIR)/.trivyignore` fallback is unreachable today: the scan target is always literal `.`, so when make runs from the repo root the first `$(wildcard .trivyig", + "superseded_signature": [ + "makefile.precommit:60-61" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "README.md", + "signature": [ + "covers the exact transitive tree", + "uv export --all-extras" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 80, + "excerpt": "**`README.md:80`** and **`Makefile.precommit:36`** \u2014 \"covers the exact transitive tree that gets installed\" is not accurate. `make audit` runs `uv export --all-extras` (includes the `dev` extra: pytes", + "superseded_signature": [ + "readme.md:80" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "README.md", + "signature": [ + "Security gates", + "design rationale" + ], + "rule_id": "readme/user-facing-not-agent-context", + "state": "accepted", + "line_when_seen": 76, + "excerpt": "**`README.md:76-94`** \u2014 the whole \"Security gates\" section is design rationale, not user-facing content: why a make target depends on `sync` (line 82), why `osv-scanner` was rejected (line 86), and a ", + "superseded_signature": [ + "readme.md:76-94" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "README.md", + "signature": [ + "S104/S101", + "duplicates the comments" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 84, + "excerpt": "**`README.md:84`** \u2014 the S104/S101 suppression rationale duplicates the comments already in `pyproject.toml:65-67` and `pyproject.toml:71`, which will drift.", + "superseded_signature": [ + "readme.md:84" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "pyproject.toml", + "signature": [ + "S104", + "hardcoded-bind-all-interfaces" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 64, + "excerpt": "**`pyproject.toml:64-68`** \u2014 `ignore = [\"S104\"]` disables hardcoded-bind-all-interfaces across the entire repo. There are 4 hits: `src/skeleton/server.py:9`, `src/skeleton/__main__.py:33`, and 2 in `t", + "superseded_signature": [ + "pyproject.toml:64-68" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/dev/docker-registry-gc.yaml", + "signature": [ + "dev/docker-registry-gc.yaml", + "due_date" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 67, + "excerpt": "**`task/recurring-schedules/dev/docker-registry-gc.yaml:67`** \u2014 left without `due_date` while its prod sibling `prod/docker-registry-gc.yaml` got one. It is `recurrence: Weekday, weekday: Saturday` \u2014 ", + "superseded_signature": [ + "due_date" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/plan-next-week.yaml", + "signature": [ + "Success Criteria was rewritten", + "acceptance criteria" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 27, + "excerpt": "**`task/recurring-schedules/prod/plan-next-week.yaml:27-29`** \u2014 Success Criteria was rewritten to cover priorities, acceptance criteria, and `# Explicitly Deferred`, but Tasks line 43 also requires al", + "superseded_signature": [ + "task/recurring-schedules/prod/plan-next-week.yaml:27-29" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/plan-next-week.yaml", + "signature": [ + "Month link present", + "dead wikilink" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 29, + "excerpt": "**`task/recurring-schedules/prod/plan-next-week.yaml:29`** \u2014 \"Month link present (target may not exist yet if the monthly plan is deferred)\" explicitly sanctions a dead wikilink. Defensible for a note", + "superseded_signature": [ + "6db0dc9" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/plan-weekend.yaml", + "signature": [ + "{{next_sun_date}}", + "first use" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 65, + "excerpt": "**`task/recurring-schedules/prod/plan-weekend.yaml:65`** \u2014 `{{next_sun_date}}` is the first use of a non-`{{current_date}}` placeholder in a `frontmatter:` field anywhere in this repo. On the base bra", + "superseded_signature": [ + "frontmatter:" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/plan-weekend.yaml", + "signature": [ + "contradicts", + "first thing Saturday morning" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 65, + "excerpt": "**`task/recurring-schedules/prod/plan-weekend.yaml:65`** \u2014 `due_date: \"{{next_sun_date}}\"` contradicts the task it annotates. Body line 18: \"as the first thing Saturday morning\"; the plan covers Sat\u2192S", + "superseded_signature": [ + "contradicts" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/renew-gmail-oauth-tokens.yaml", + "signature": [ + "expire every 7 days", + "{{next_sun_date}}" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 64, + "excerpt": "**`task/recurring-schedules/prod/renew-gmail-oauth-tokens.yaml:64`** \u2014 `due_date: \"{{next_sun_date}}\"` on a task whose own body (line 18) states \"Google's tokens expire every 7 days for unverified app", + "superseded_signature": [ + "{{current_date}}" + ] + }, + { + "pr_id": "tts-mcp#20", + "path": "CHANGELOG.md", + "signature": [ + "## Unreleased", + "v0.6.0" + ], + "rule_id": "changelog/unreleased-entry-required", + "state": "accepted", + "line_when_seen": 8, + "excerpt": "**`CHANGELOG.md:8`** \u2014 no `## Unreleased` section exists (top section is `## v0.6.0`), so this PR adds no changelog entry. `.maintainer.yaml` sets `release.autoRelease: true`, and the release agent pr", + "superseded_signature": [ + "changelog.md:8" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "README.md", + "signature": [ + "run-modes table", + "distinct labels" + ], + "rule_id": null, + "state": "rejected", + "line_when_seen": 42, + "excerpt": "Suggests renaming two 'Local *' rows in the run-modes table to 'more distinct labels'. The table is not wrong and nothing is ambiguous in use; this is taste, not a defect. Adjudicated rejected 2026-08-09 so precision has something to measure against.", + "superseded_signature": [ + "distinct labels" + ] + } + ], + "adjudications": [ + "2026-08-09 RELAX python-skeleton#3 .github/workflows/ci.yml apt-key: signature changed from ['.github/workflows/ci.yml:32'] to ['apt-key'] so the same issue matches regardless of cited line. Reported by 11 of the 102 ledger findings at five different lines. NOTE: an earlier attempt ADDED a second line-free entry instead of relaxing this one; that duplicated the issue and 3 findings matched both entries, inflating recall. Corrected here - one issue, one entry.", + "2026-08-09 REJECT github-pr-review-agent#11 README.md 'distinct labels' -> rejected. Subjective doc-polish on a table that is not wrong. Matches exactly 1 finding in the ledger. First rejected entry; before it, precision could not be lost by any configuration.", + "Adjudication check that was missing and is now required: a new or relaxed signature must be tested for ALIASING against the other golden entries, not only against the ledger findings. Checking signature->findings alone let a duplicate through.", + "2026-08-10 REKEY all entries from path:line signatures to semantic keywords. 37 of the 43 signatures embedded a line reference, so a re-report of the same issue one line over scored as a miss. Measured: the curated-1 Opus run - the SAME model/effort/mode that seeded this set - scored recall 0.262 against it; after the re-key, 0.463. The dev-1 baseline still self-matches 1.000. Keys are code identifiers (S104, apt-key, ErrorRequestHandler) rather than prose, because prose is one author's phrasing and drifts between runs.", + "2026-08-10 MERGE github-pr-review-agent#11 docs/pr-post-back.md:126 into the :128 entry. Both described the stale 'backward compatibility' framing in one paragraph; a later run reported them as a single finding, which would have matched two entries and tripped the aliasing guard. One issue, one entry.", + "2026-08-10 MATCHER rule_id no longer short-circuits identity. Demonstrated: the github-pr-review-agent#11 CHANGELOG bundling entry matched an unrelated fix:/chore: prefix finding purely because both carried changelog/conventional-prefix-required. Any config finding SOME instance of a rule was credited with the specific one in the set, inflating recall.", + "2026-08-10 MATCHER path removed from identity. Demonstrated: the 'write-scoped token still minted under --skip-post' issue is entry-keyed to pkg/factory/runner.go but was re-reported anchored at cmd/run-task/main.go - the body named runner.go explicitly. Scored as a miss purely on the anchor choice." + ] } diff --git a/bench/run.py b/bench/run.py index 85ba658..5917497 100755 --- a/bench/run.py +++ b/bench/run.py @@ -109,6 +109,10 @@ # Scoring — frozen marker literals (spec 006 Constraints; not configurable) # ---------------------------------------------------------------------- GOLDEN_NOT_FOUND_MARKER = "GOLDEN SET NOT FOUND" + +# `path` is not part of identity, so a single generic keyword would match findings +# in unrelated files. Two is the cheapest guard that forces a distinctive key. +MIN_SIGNATURE_KEYWORDS = 2 INVALID_GOLDEN_MARKER = "INVALID GOLDEN SET" GOLDEN_VERSION_MISMATCH_MARKER = "GOLDEN VERSION MISMATCH" PRS_VERSION_SKIP_MARKER = "PRS VERSION SKIP" @@ -172,10 +176,45 @@ def load_golden(path: pathlib.Path) -> dict: raise BenchError( f"{INVALID_GOLDEN_MARKER}: entry {i} missing '{field}' in {path}" ) + _validate_signature(entry, i, path) return data +def _validate_signature(entry: dict, index: int, path: pathlib.Path) -> None: + """Reject signatures that cannot serve as identity. + + Two invariants, both learned the hard way on `golden-dev-2`: + + * **No line reference.** 37 of its 43 signatures were `path:line` strings, so + a re-report of the same issue one line over scored as a miss. Recall then + measured whether a configuration cited the same line, not whether it found + the issue — an identical config scored 0.262 against its own source set. + * **At least two keywords.** `path` is no longer part of identity, so a lone + generic word would match findings in unrelated files. + """ + signature = entry.get("signature") + if not isinstance(signature, list) or not signature: + raise BenchError( + f"{INVALID_GOLDEN_MARKER}: entry {index} has an empty 'signature' in {path}" + ) + for kw in signature: + if not isinstance(kw, str) or not kw.strip(): + raise BenchError( + f"{INVALID_GOLDEN_MARKER}: entry {index} has a blank signature keyword in {path}" + ) + if re.search(r":\d+", kw): + raise BenchError( + f"{INVALID_GOLDEN_MARKER}: entry {index} signature keyword {kw!r} " + f"embeds a line reference in {path}" + ) + if len(signature) < MIN_SIGNATURE_KEYWORDS: + raise BenchError( + f"{INVALID_GOLDEN_MARKER}: entry {index} needs at least " + f"{MIN_SIGNATURE_KEYWORDS} signature keywords in {path}" + ) + + def load_ledger(path: pathlib.Path) -> list: """Read a JSONL ledger. Raises BenchError with a frozen literal.""" try: @@ -2083,23 +2122,26 @@ def format_ratio(numerator: int, denominator: int) -> str: def finding_matches_entry(entry: dict, finding: dict) -> bool: """True when this finding and this golden entry describe the same issue. - rule_id exact when BOTH sides carry a non-null one; otherwise path string - equality plus EVERY signature keyword present case-insensitively in body. - `line` is read from neither side. + Identity is the signature alone: EVERY keyword must appear case-insensitively + in the body. `rule_id` is an additional constraint when BOTH sides carry one + — never a short-circuit. + + Neither `path` nor `line` is read. Both are citation coordinates the reviewer + chooses, not properties of the defect: the same issue is routinely anchored at + the line that defines a flag or at the line that suffers its consequence, in + two different files. Keying identity on either scores such a re-report as a + miss. They stay on the entry as provenance. """ entry_rule = entry.get("rule_id") finding_rule = finding.get("rule_id") - # Case 1: both carry a non-null rule_id — exact match required - if entry_rule is not None and entry_rule != "" and finding_rule is not None and finding_rule != "": - return entry_rule == finding_rule - # Case 2: path match + every signature keyword case-insensitively in body - if entry.get("path") != finding.get("path"): + if entry_rule and finding_rule and entry_rule != finding_rule: + return False + signature = entry.get("signature") or [] + if not signature: + # A keyword-less entry would match every finding. Never silently true. return False body = (finding.get("body") or "").lower() - for kw in entry.get("signature") or []: - if kw.lower() not in body: - return False - return True + return all(kw.lower() in body for kw in signature) @dataclasses.dataclass(frozen=True) diff --git a/bench/test_score.py b/bench/test_score.py index 9a5c20a..50fe180 100644 --- a/bench/test_score.py +++ b/bench/test_score.py @@ -1,10 +1,12 @@ #!/usr/bin/env python3 """Unit tests for bench/run.py scoring: match relation, states, gap-triage, and ratio rendering.""" +import collections import copy import dataclasses import filecmp import hashlib +import itertools import json import os import pathlib @@ -113,7 +115,7 @@ def test_golden_json_sha256(self): ).hexdigest() self.assertEqual( digest, - "90f5b0a61ac763b6abb3991fff699a4818318f22a171f6fc4375d314da43459d", + "d13b0218b19ec95d77587c43ebf48886524b05d3d77acb562a97cf28529d6582", ) def test_baseline_slice_sha256(self): @@ -153,7 +155,7 @@ def test_probe_slice_sha256(self): ) def test_line_counts(self): - self.assertEqual((GOLDEN_FIXTURE).read_text().count("\n"), 490) + self.assertEqual((GOLDEN_FIXTURE).read_text().count("\n"), 658) self.assertEqual( (BENCH_DIR / "testdata" / "ledger-baseline-opus-xhigh-full.jsonl").read_text().count("\n"), 5 ) @@ -259,8 +261,16 @@ def test_golden_line_when_seen_set_to_none_unchanged(self): # ---------------------------------------------------------------------- # AC8: rule_id takes priority # ---------------------------------------------------------------------- -class TestRuleIdTakesPriority(unittest.TestCase): - """AC8: rule_id exact match when both sides carry one; falls through otherwise.""" +class TestRuleIdConstrainsIdentity(unittest.TestCase): + """AC8, amended 2026-08-10: rule_id constrains identity, never short-circuits it. + + The original name and contract were "rule_id takes priority" — a match on + rule_id alone, ignoring path and signature. That credited any config finding + *some* instance of a rule with the specific instance in the golden set. The + tests below still hold under the amended matcher because they exercise + agreement and disagreement, not the short-circuit; the aliasing case is + covered in TestRuleIdConstrainsButNeverShortCircuits. + """ def setUp(self): self.golden = load_golden() @@ -781,17 +791,28 @@ def test_four_runs_produce_exact_ac5_table(self): (3, 2, 40, 0, 1, "0.048", "1.000"), (2, 1, 41, 0, 1, "0.024", "1.000"), (5, 2, 40, 0, 3, "0.048", "1.000"), - (6, 0, 42, 0, 6, "0.000", "n/a"), + (6, 4, 38, 0, 2, "0.095", "1.000"), ] self.assertEqual(observed, expected) - def test_run4_precision_n_a_literal(self): + def test_zero_denominator_precision_renders_the_n_a_literal(self): + """A run that matched nothing renders precision as the literal 'n/a'. + + This used to assert against run 4 of the four-run slice, which matched + zero entries under `path:line` signatures. The 2026-08-10 re-key gave + it 4 hits, so it is no longer a zero-denominator case and pinning it + here would have tested the bug rather than the formatting. The partial + slice still carries genuine zero-match runs; the invariant under test + (0/0 is never rendered as 0.000) is unchanged. + """ golden = load_golden() - rows = load_slice("ledger-sonnet-medium-short-4runs.jsonl") + rows = load_slice("ledger-sonnet-medium-short-partial.jsonl") cfg = run.score_config(rows=rows, golden=golden) - run4 = cfg.runs[3] - self.assertEqual(run4.score.precision, run.RATIO_NA) - self.assertEqual(run4.score.precision, "n/a") + zero_match = [r for r in cfg.runs if r.score.accepted_hits == 0] + self.assertTrue(zero_match, "expected at least one run matching nothing") + for r in zero_match: + self.assertEqual(r.score.precision, run.RATIO_NA) + self.assertEqual(r.score.precision, "n/a") # ---------------------------------------------------------------------- @@ -827,7 +848,7 @@ def test_identical_timestamps_preserve_run_count_and_table(self): (3, 2, 40, 0, 1, "0.048", "1.000"), (2, 1, 41, 0, 1, "0.024", "1.000"), (5, 2, 40, 0, 3, "0.048", "1.000"), - (6, 0, 42, 0, 6, "0.000", "n/a"), + (6, 4, 38, 0, 2, "0.095", "1.000"), ] self.assertEqual(observed, expected) @@ -887,12 +908,18 @@ def test_partial_runs_match_ac30_table(self): len(r.score.gap_candidates), r.score.recall, r.score.precision) for r in cfg.runs ] + # Re-pinned 2026-08-10 with the semantic-signature golden set. Every + # movement is a hit gained and a gap candidate lost: these sonnet runs + # described the same issues as the Opus baseline but anchored them at + # different lines, so under `path:line` keys they scored zero credit. + # Runs 4 and 5 went 0 -> 1 hit, which is why their precision is no + # longer the `n/a` of a run that matched nothing at all. expected = [ - (5, 42, 6, 2, 40, 4, "0.048", "1.000"), - (5, 42, 9, 1, 41, 8, "0.024", "1.000"), - (5, 42, 9, 3, 39, 6, "0.071", "1.000"), - (5, 42, 5, 0, 42, 5, "0.000", "n/a"), - (5, 42, 1, 0, 42, 1, "0.000", "n/a"), + (5, 42, 6, 3, 39, 3, "0.071", "1.000"), + (5, 42, 9, 2, 40, 7, "0.048", "1.000"), + (5, 42, 9, 2, 40, 7, "0.048", "1.000"), + (5, 42, 5, 1, 41, 4, "0.024", "1.000"), + (5, 42, 1, 1, 41, 0, "0.024", "1.000"), (3, 24, 1, 0, 24, 1, "0.000", "n/a"), (3, 24, 1, 0, 24, 1, "0.000", "n/a"), (1, 1, 0, 0, 1, 0, "0.000", "n/a"), @@ -1254,9 +1281,11 @@ def test_four_run_table_cells_match_result(self): self.assertEqual(row[10], r.recall) self.assertEqual(row[11], r.precision) self.assertEqual(int(row[12]), run_score.wall_time_seconds) - # Run 4 precision is literal 'n/a' - run4_row = next(row for row in table if row[0] == "4") - self.assertEqual(run4_row[11], "n/a") + # The loop above already asserts row[11] == r.precision for every run. + # A spot-check pinning run 4 to 'n/a' used to sit here; it described a + # run that matched nothing, which the 2026-08-10 re-key fixed. The + # 'n/a' literal itself is pinned in + # TestFourRunSliceChunksIntoFourRuns.test_zero_denominator_precision_renders_the_n_a_literal. def test_partial_table_cells_match_result(self): golden = load_golden() @@ -1339,17 +1368,26 @@ def test_not_yet_a_result_appears_on_baseline(self): self.assertIn("rejected", para) def test_line_reference_caveat_appears_on_baseline(self): + """The caveat now reports the CLEARED state, and must keep doing so. + + It used to read "36 of the 42 signatures embed a line reference", which + was the honest description of a set keyed on `path:line`. Since the + 2026-08-10 re-key no signature carries one, so the caveat flips to the + branch that says recall measures issue detection. Asserting the cleared + wording here is what turns this caveat into a regression detector: a + future adjudication that reintroduces a `path:line` key fails this test + rather than silently degrading recall back into line-citation matching. + """ text = self._get_page("ledger-baseline-opus-xhigh-full.jsonl") - self.assertIn("36 of the 42 signatures", text, "load-bearing number missing") - self.assertIn("embed a line reference", text) - para = next(p for p in text.split("\n\n") if "36 of the 42 signatures" in p) - self.assertIn("gap-triage", para) + self.assertIn("No signature embeds a line reference", text) + self.assertIn("measures issue detection", text) + self.assertNotIn("embed a line reference, so a", text) def test_both_caveats_appear_on_fourrun_page(self): text = self._get_page("ledger-sonnet-medium-short-4runs.jsonl") self.assertIn("not yet a result", text) - self.assertIn("36 of the 42 signatures", text) - self.assertIn("embed a line reference", text) + self.assertIn("No signature embeds a line reference", text) + self.assertIn("measures issue detection", text) class TestScoringIsDeterministicAndCarriesNoWallClock(unittest.TestCase): @@ -1584,6 +1622,12 @@ def test_score_mode_invokes_no_review(self): if line.strip() ) + testdata_before = { + f.name: hashlib.sha256(f.read_bytes()).hexdigest() + for f in sorted((BENCH_DIR / "testdata").iterdir()) + if f.is_file() + } + result2 = subprocess.run( [sys.executable, str(run.BENCH_DIR / "run.py"), "--score", @@ -1607,14 +1651,16 @@ def test_score_mode_invokes_no_review(self): self.assertEqual(ledger_before_sha, ledger_after_sha) self.assertEqual(ledger_before_lines, ledger_after_lines) - # Golden set and testdata unchanged (run from repo root) - diff_result = subprocess.run( - ["git", "diff", "--exit-code", - str(GOLDEN_FIXTURE), "bench/testdata/"], - capture_output=True, text=True, - cwd=str(run.REPO_ROOT), - ) - self.assertEqual(diff_result.returncode, 0, diff_result.stderr) + # Golden set and testdata unchanged. Compared by digest against a + # snapshot taken before the scoring subprocess, the same way the + # ledger is checked above. This used to shell out to + # `git diff --exit-code`, which reports the working tree rather than + # what the subprocess did: any developer holding a legitimate + # uncommitted fixture edit failed the test, and a mutation staged in + # the index would have passed it. + for rel, before in testdata_before.items(): + after = hashlib.sha256((BENCH_DIR / "testdata" / rel).read_bytes()).hexdigest() + self.assertEqual(before, after, f"scoring mutated testdata/{rel}") # Exactly one page written pages = sorted(p.name for p in reports_dir.iterdir()) @@ -2419,3 +2465,164 @@ def test_clean_run_says_so_rather_than_going_quiet(self): if __name__ == "__main__": unittest.main() + + +# ---------------------------------------------------------------------- +# Identity is the signature alone (2026-08-10) +# ---------------------------------------------------------------------- +class TestSignaturesCarryNoLineReference(unittest.TestCase): + """A `path:line` signature makes recall measure citation, not detection.""" + + def test_live_golden_signatures_are_line_free(self): + golden = run.load_golden(BENCH_DIR / "golden.json") + offenders = [ + (e["pr_id"], kw) + for e in golden["entries"] + for kw in e["signature"] + if re.search(r":\d+", kw) + ] + self.assertEqual(offenders, [], "signature keywords must not embed a line") + + def test_fixture_signatures_are_line_free(self): + golden = run.load_golden(GOLDEN_FIXTURE) + offenders = [kw for e in golden["entries"] for kw in e["signature"] + if re.search(r":\d+", kw)] + self.assertEqual(offenders, []) + + def test_loader_rejects_a_line_bearing_signature(self): + with tempfile.TemporaryDirectory() as tmp: + path = pathlib.Path(tmp) / "golden.json" + path.write_text(json.dumps({ + "version": "t", "prs_version": "dev-1", + "match_rule": "t", "states": {}, "entries": [ + {"pr_id": "a#1", "path": "x.go", + "signature": ["x.go:42", "other"], "state": "accepted"} + ]}), encoding="utf-8") + with self.assertRaises(run.BenchError) as ctx: + run.load_golden(path) + self.assertIn("embeds a line reference", str(ctx.exception)) + + def test_loader_rejects_a_single_keyword_signature(self): + with tempfile.TemporaryDirectory() as tmp: + path = pathlib.Path(tmp) / "golden.json" + path.write_text(json.dumps({ + "version": "t", "prs_version": "dev-1", + "match_rule": "t", "states": {}, "entries": [ + {"pr_id": "a#1", "path": "x.go", + "signature": ["lonely"], "state": "accepted"} + ]}), encoding="utf-8") + with self.assertRaises(run.BenchError) as ctx: + run.load_golden(path) + self.assertIn("at least 2 signature keywords", str(ctx.exception)) + + +class TestRuleIdConstrainsButNeverShortCircuits(unittest.TestCase): + """Regression: one rule firing twice in a file must not alias to one entry. + + The github-pr-review-agent#11 CHANGELOG entry describes a bullet bundling + three concerns. A later run reported a *different* defect in the same file + (a dependency bump prefixed `fix:` where the repo uses `chore:`) under the + same `changelog/conventional-prefix-required` rule. The old matcher + returned on rule_id equality alone, so that unrelated finding was credited + as a hit for this entry. + """ + + ENTRY = { + "pr_id": "x#1", "path": "CHANGELOG.md", + "signature": ["bullet bundles", "split"], + "rule_id": "changelog/conventional-prefix-required", + "state": "accepted", + } + + def test_same_rule_different_issue_does_not_match(self): + other = { + "pr_id": "x#1", "path": "CHANGELOG.md", "line": 10, + "body": "dependency bump is prefixed `fix:`; precedent is `chore:`", + "rule_id": "changelog/conventional-prefix-required", + } + self.assertFalse(run.finding_matches_entry(self.ENTRY, other)) + + def test_same_rule_same_issue_still_matches(self): + same = { + "pr_id": "x#1", "path": "CHANGELOG.md", "line": 11, + "body": "one bullet bundles the feature and the docs; split it", + "rule_id": "changelog/conventional-prefix-required", + } + self.assertTrue(run.finding_matches_entry(self.ENTRY, same)) + + def test_disagreeing_rule_id_blocks_an_otherwise_matching_body(self): + wrong_rule = { + "pr_id": "x#1", "path": "CHANGELOG.md", "line": 11, + "body": "one bullet bundles the feature and the docs; split it", + "rule_id": "some/other-rule", + } + self.assertFalse(run.finding_matches_entry(self.ENTRY, wrong_rule)) + + def test_empty_signature_never_matches(self): + self.assertFalse(run.finding_matches_entry( + {"pr_id": "x#1", "path": "a.go", "signature": [], "state": "accepted"}, + {"pr_id": "x#1", "path": "a.go", "body": "anything", "rule_id": None}, + )) + + +class TestPathIsNotPartOfIdentity(unittest.TestCase): + """The same defect anchored in a different file is still the same defect. + + Real case: "a write-scoped token is still minted under --skip-post" is keyed + to pkg/factory/runner.go, but a later run anchored it at cmd/run-task/main.go + while naming runner.go in the body. Keying on path scored that as a miss. + """ + + ENTRY = { + "pr_id": "x#1", "path": "pkg/factory/runner.go", + "signature": ["write-scoped", "--skip-post"], + "rule_id": None, "state": "accepted", + } + + def test_same_issue_anchored_in_another_file_matches(self): + finding = { + "pr_id": "x#1", "path": "cmd/run-task/main.go", "line": 138, + "body": ("the write-scoped GitHub App token is still minted under " + "`--skip-post` (`pkg/factory/runner.go:105-106`)"), + "rule_id": None, + } + self.assertTrue(run.finding_matches_entry(self.ENTRY, finding)) + + def test_unrelated_body_on_the_recorded_path_does_not_match(self): + finding = { + "pr_id": "x#1", "path": "pkg/factory/runner.go", "line": 66, + "body": "ResolvePosters would read better as ResolvePosterAndVerifier", + "rule_id": None, + } + self.assertFalse(run.finding_matches_entry(self.ENTRY, finding)) + + +class TestNoEntrySignatureSubsumesAnother(unittest.TestCase): + """Ledger-free aliasing guard. + + `test_no_live_golden_entry_aliases_another` needs bench/results, which is + gitignored — so it skips in a fresh clone and in CI, exactly where a bad + adjudication would land unnoticed. A subset relationship is aliasing that + can be proved from the golden set alone: if entry A's keywords are a subset + of entry B's, then every finding matching B also matches A. + """ + + def _check(self, golden): + by_pr = collections.defaultdict(list) + for e in golden["entries"]: + by_pr[e["pr_id"]].append(e) + for pr, entries in by_pr.items(): + for a, b in itertools.permutations(entries, 2): + sa = {k.lower() for k in a["signature"]} + sb = {k.lower() for k in b["signature"]} + self.assertFalse( + sa <= sb, + f"{pr}: {a['signature']} is a subset of {b['signature']} — " + f"every finding matching the latter also matches the former", + ) + + def test_live_golden_has_no_subsumed_signature(self): + self._check(run.load_golden(BENCH_DIR / "golden.json")) + + def test_fixture_has_no_subsumed_signature(self): + self._check(run.load_golden(GOLDEN_FIXTURE)) diff --git a/bench/testdata/golden-dev-1.json b/bench/testdata/golden-dev-1.json index 5728196..7f4f25d 100644 --- a/bench/testdata/golden-dev-1.json +++ b/bench/testdata/golden-dev-1.json @@ -1,490 +1,658 @@ { - "version": "golden-dev-1", - "created": "2026-08-09", - "prs_version": "dev-1", - "baseline": { - "model": "opus", - "effort": "xhigh", - "mode": "full", - "coding_version": "v0.35.6", - "config_hash_prefix": "cc64cc99", - "rules_commands_hash_prefix": "ecc80333", - "runs": 1, - "findings": 42, - "wall_time_seconds": 2534 - }, - "match_rule": "rule_id exact when both sides carry one; else path + ALL signature keywords present in body (case-insensitive). line is NEVER used for identity.", - "states": { - "accepted": "a finding the reviewer SHOULD produce. A miss costs recall.", - "rejected": "a known false positive. A hit costs precision.", - "unreviewed": "not adjudicated. Excluded from both numerator and denominator." - }, - "scoring_note": "All entries are seeded 'accepted' from a single Opus 5 best-case run - this is the designed bootstrap, not hand-curated ground truth. Consequence: a configuration reporting a finding ABSENT from this set is NOT automatically wrong. Report such findings separately as gap-triage candidates rather than as precision failures. When several independent configurations agree on a finding Opus missed, that is evidence the golden set is incomplete - promote it to 'accepted'. When nothing else ever reproduces an Opus finding, that is evidence it may be a false positive - demote it to 'rejected'. The set sharpens through disagreement; it is not frozen truth.", - "known_corrections": [ - "tts-mcp#20 was annotated in bench/prs.json as 'clean - correct answer is zero findings'. That is FALSE. It returned 0 findings across six sonnet runs (six misses, not cleanliness); Opus found a real defect, hand-verified against head SHA 84feeb4d: CHANGELOG.md has no '## Unreleased' section while .maintainer.yaml sets autoRelease: true, so the CVE fix would never ship. The fixture currently has NO verified-clean PR." - ], - "entries": [ - { - "pr_id": "github-pr-review-agent#11", - "path": "CHANGELOG.md", - "signature": [ - "changelog.md:11" - ], - "rule_id": "changelog/conventional-prefix-required", - "state": "accepted", - "line_when_seen": 11, - "excerpt": "**`CHANGELOG.md:11`** \u2014 one `feat:` bullet bundles three concerns: the new flag (`feat:`), the nil-poster panic guard in `tryDismissHallucinated` (`fix:`), and the CLI doc re-attribution (`docs:`). Sp" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "cmd/run-task/main.go", - "signature": [ - "cmd/run-task/main.go:171" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 171, - "excerpt": "**`cmd/run-task/main.go:171`** \u2014 the `SkipPost: a.SkipPost` pass-through and its counterpart call site at `pkg/factory/runner.go:136` are the only untested lines of the change: `cmd/run-task/main_test" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "docs/pr-post-back.md", - "signature": [ - "docs/pr-post-back.md:128" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 128, - "excerpt": "**`docs/pr-post-back.md:128`** \u2014 \"This preserves backward compatibility with the local CLI mode\" is now false. Before this PR `cmd/run-task` never produced a nil poster (that was the documented-vs-act" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "docs/pr-post-back.md", - "signature": [ - "docs/pr-post-back.md:126" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 126, - "excerpt": "**`docs/pr-post-back.md:126`** \u2014 heading `## nil Poster \u2014 Local / Backward-Compatible Mode` carries the same stale framing as the line-128 finding; update both together (e.g. \"Local Opt-In (`--skip-po" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/factory.go", - "signature": [ - "pkg/factory/factory.go:166-173" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 166, - "excerpt": "**`pkg/factory/factory.go:166-173`** \u2014 `CreateAgent`'s doc comment states the execution phase \"posts review to GitHub via PrPoster\" unconditionally, and parenthesises only the verifier case (\"nil veri" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/runner.go", - "signature": [ - "pkg/factory/runner.go:104-106" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 104, - "excerpt": "**`pkg/factory/runner.go:104-106`** \u2014 a live, write-scoped GitHub App installation token is injected into the Claude subprocess env unconditionally, including under `--skip-post`; nothing in this PR t" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/runner.go", - "signature": [ - "pkg/factory/runner.go:51-53" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 51, - "excerpt": "**`pkg/factory/runner.go:51-53`** \u2014 `RunConfig.Agent`'s comment still frames the nil-`Agent` branch as \"cmd/run-task leaves this nil so CreateAgent is used for backward compatibility\". That branch is " - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/runner.go", - "signature": [ - "pkg/factory/runner.go:66" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 66, - "excerpt": "**`pkg/factory/runner.go:66`** \u2014 `ResolvePosters` is exported solely so `pkg/skip_post_boundary_test.go` (a different package's test binary, where an `export_test.go` alias would be invisible) can rea" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/factory/runner_test.go", - "signature": [ - "pkg/factory/runner_test.go:32-33" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 32, - "excerpt": "**`pkg/factory/runner_test.go:32-33`** \u2014 `Expect(poster).To(BeNil())` cannot detect the regression this PR is themed around. Gomega's `BeNil()` falls through to `reflect.ValueOf(x).IsNil()` for pointe" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/skip_post_boundary_test.go", - "signature": [ - "pkg/skip_post_boundary_test.go:169-226" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 169, - "excerpt": "**`pkg/skip_post_boundary_test.go:169-226`** \u2014 case 4's stated failure mechanism is wrong. The comment claims absence of an `ai_review verify:` line in `## Diagnostics` is the negative signal, but `ap" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/skip_post_boundary_test.go", - "signature": [ - "pkg/skip_post_boundary_test.go:102-166" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 102, - "excerpt": "**`pkg/skip_post_boundary_test.go:102-166`** \u2014 case 3 does not exercise the skip-post contract it is labelled with. `ResolvePosters(RunConfig{SkipPost:true})` can only ever return `(nil, nil)`, so the" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/skip_post_boundary_test.go", - "signature": [ - "pkg/skip_post_boundary_test.go:64-100" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 64, - "excerpt": "**`pkg/skip_post_boundary_test.go:64-100`** \u2014 case 2 and `pkg/steps_review_test.go:614-627` case (j) drive an identical scenario (verdict=fail + hallucinations + nil poster) and assert the same two ou" - }, - { - "pr_id": "github-pr-review-agent#11", - "path": "pkg/steps_checkout_execution.go", - "signature": [ - "pkg/steps_checkout_execution.go:316" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 316, - "excerpt": "**`pkg/steps_checkout_execution.go:316`** \u2014 the nil-poster early return fires before `ParseVerdict` (:328), before the fail-closed `!funnelRan && approve \u2192 request-changes` gate (:338-340), and before" - }, - { - "pr_id": "node-skeleton#2", - "path": "README.md", - "signature": [ - "readme.md:107" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 107, - "excerpt": "**`README.md:107`** \u2014 the new License section links the file without naming the license; the repo's own README template names it inline (`BSD-2-Clause \u2014 see [LICENSE](./LICENSE).`)." - }, - { - "pr_id": "node-skeleton#2", - "path": "eslint.config.js", - "signature": [ - "eslint.config.js:44" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 44, - "excerpt": "**`eslint.config.js:44`** \u2014 the comment (\"enum, namespace, and parameter-property syntax would make `node src/*.ts` require a build step\") sits directly above `@typescript-eslint/no-explicit-any` and " - }, - { - "pr_id": "node-skeleton#2", - "path": "eslint.config.js", - "signature": [ - "eslint.config.js:16" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 16, - "excerpt": "**`eslint.config.js:16`** \u2014 `tseslint.config(...)` is marked `@deprecated` in the pinned typescript-eslint 8.66.0; I confirmed the annotation at `typescript-eslint/dist/config-helper.d.ts:67` (\"ESLint" - }, - { - "pr_id": "node-skeleton#2", - "path": "eslint.config.js", - "signature": [ - "eslint.config.js:39-55" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 39, - "excerpt": "**`eslint.config.js:39-55`** \u2014 `CLAUDE.md` bans \"`any`, `as`, `!`, `@ts-ignore`\". Three are enforced (`no-explicit-any`, `no-non-null-assertion`, inherited `ban-ts-comment`); `as` is enforced by nothi" - }, - { - "pr_id": "node-skeleton#2", - "path": "package.json", - "signature": [ - "package.json:8" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 8, - "excerpt": "**`package.json:8`** \u2014 `\"node\": \">=22\"` no longer matches what the code requires. Node ships unflagged type-stripping only from **22.18.0**; on 22.6\u201322.17 it needs `--experimental-strip-types`, and be" - }, - { - "pr_id": "node-skeleton#2", - "path": "src/handlers/health.ts", - "signature": [ - "src/handlers/health.ts:12" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 12, - "excerpt": "**`src/handlers/health.ts:12`** \u2014 same defect: `HealthRouter` is asserted, not derived, and `healthRouter` (line 27) carries no annotation tying it to the alias. Note `src/config.ts:34` and `src/log.t" - }, - { - "pr_id": "node-skeleton#2", - "path": "src/server.ts", - "signature": [ - "src/server.ts:31" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 31, - "excerpt": "**`src/server.ts:31`** \u2014 `export type CreateApp` is hand-written and never bound to `createApp` (line 40), so `tsc` never compares the declared contract to the implementation; the only bridge is `requ" - }, - { - "pr_id": "node-skeleton#2", - "path": "src/server.ts", - "signature": [ - "src/server.ts:69" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 69, - "excerpt": "**`src/server.ts:69`** \u2014 the error handler types `err: Error`, narrower than the shipped `ErrorRequestHandler`, which I confirmed declares `err: any` (`@types/express-serve-static-core/index.d.ts:77`)" - }, - { - "pr_id": "node-skeleton#2", - "path": "test/health.test.ts", - "signature": [ - "config.check()" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": null, - "excerpt": "**`test/health.test.ts`** \u2014 `config.check()` is the PR's new startup-validation logic and has zero coverage; `test/` contains only `health.test.ts`, which never imports config. Neither the invalid-`PO" - }, - { - "pr_id": "node-skeleton#2", - "path": "test/health.test.ts", - "signature": [ - "test/health.test.ts:22" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 22, - "excerpt": "**`test/health.test.ts:22`** \u2014 `? address.port : 0` silently falls back to port 0 instead of failing. If the invariant ever breaks, every test in the file fetches `http://127.0.0.1:0` and reports a co" - }, - { - "pr_id": "python-skeleton#3", - "path": ".github/workflows/ci.yml", - "signature": [ - ".github/workflows/ci.yml:32" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 32, - "excerpt": "**`.github/workflows/ci.yml:32`** \u2014 `wget -qO - \u2026 | sudo apt-key add -` installs Aqua's signing key into apt's **global** trusted keyring, where it can authenticate packages from *any* configured repo" - }, - { - "pr_id": "python-skeleton#3", - "path": ".github/workflows/ci.yml", - "signature": [ - ".github/workflows/ci.yml:35" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 35, - "excerpt": "**`.github/workflows/ci.yml:35`** \u2014 `apt-get install -y trivy` is unpinned, so every CI run takes whatever is latest. Trivy minor releases add detectors and change exit semantics, so CI can go red wit" - }, - { - "pr_id": "python-skeleton#3", - "path": ".github/workflows/ci.yml", - "signature": [ - ".github/workflows/ci.yml:33" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 33, - "excerpt": "**`.github/workflows/ci.yml:33`** \u2014 `sudo tee -a` appends, so a re-run duplicates the repo line. Use `tee` without `-a`." - }, - { - "pr_id": "python-skeleton#3", - "path": ".github/workflows/ci.yml", - "signature": [ - ".github/workflows/ci.yml:38" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 38, - "excerpt": "**`.github/workflows/ci.yml:38`** \u2014 `uv sync --all-extras` is redundant; `make precommit` runs `sync` as its first prerequisite." - }, - { - "pr_id": "python-skeleton#3", - "path": "CHANGELOG.md", - "signature": [ - "changelog.md:18" - ], - "rule_id": "changelog/conventional-prefix-required", - "state": "accepted", - "line_when_seen": 18, - "excerpt": "**`CHANGELOG.md:18`** \u2014 `- ci: install trivy in CI` uses a prefix outside the recognised set. The guide's table defines exactly seven (`feat:`, `fix:`, `refactor:`, `test:`, `docs:`, `chore:`, `perf:`" - }, - { - "pr_id": "python-skeleton#3", - "path": "Makefile.precommit", - "signature": [ - "makefile.precommit:42" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 42, - "excerpt": "**`Makefile.precommit:42`** \u2014 the comment promises that unreachable findings \"get an explicit `--ignore-vuln` entry with a reason\", but no such surface exists: no file, no make variable, nothing in th" - }, - { - "pr_id": "python-skeleton#3", - "path": "Makefile.precommit", - "signature": [ - "makefile.precommit:24" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 24, - "excerpt": "**`Makefile.precommit:24`** \u2014 `check: lint typecheck audit trivy` makes the default target and `make precommit` hard-require a local `trivy` binary plus network egress (trivy DB from ghcr, pip-audit a" - }, - { - "pr_id": "python-skeleton#3", - "path": "Makefile.precommit", - "signature": [ - "makefile.precommit:60-61" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 60, - "excerpt": "**`Makefile.precommit:60-61`** \u2014 the `$(ROOTDIR)/.trivyignore` fallback is unreachable today: the scan target is always literal `.`, so when make runs from the repo root the first `$(wildcard .trivyig" - }, - { - "pr_id": "python-skeleton#3", - "path": "README.md", - "signature": [ - "readme.md:80" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 80, - "excerpt": "**`README.md:80`** and **`Makefile.precommit:36`** \u2014 \"covers the exact transitive tree that gets installed\" is not accurate. `make audit` runs `uv export --all-extras` (includes the `dev` extra: pytes" - }, - { - "pr_id": "python-skeleton#3", - "path": "README.md", - "signature": [ - "readme.md:76-94" - ], - "rule_id": "readme/user-facing-not-agent-context", - "state": "accepted", - "line_when_seen": 76, - "excerpt": "**`README.md:76-94`** \u2014 the whole \"Security gates\" section is design rationale, not user-facing content: why a make target depends on `sync` (line 82), why `osv-scanner` was rejected (line 86), and a " - }, - { - "pr_id": "python-skeleton#3", - "path": "README.md", - "signature": [ - "readme.md:84" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 84, - "excerpt": "**`README.md:84`** \u2014 the S104/S101 suppression rationale duplicates the comments already in `pyproject.toml:65-67` and `pyproject.toml:71`, which will drift." - }, - { - "pr_id": "python-skeleton#3", - "path": "pyproject.toml", - "signature": [ - "pyproject.toml:64-68" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 64, - "excerpt": "**`pyproject.toml:64-68`** \u2014 `ignore = [\"S104\"]` disables hardcoded-bind-all-interfaces across the entire repo. There are 4 hits: `src/skeleton/server.py:9`, `src/skeleton/__main__.py:33`, and 2 in `t" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/dev/docker-registry-gc.yaml", - "signature": [ - "due_date" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 67, - "excerpt": "**`task/recurring-schedules/dev/docker-registry-gc.yaml:67`** \u2014 left without `due_date` while its prod sibling `prod/docker-registry-gc.yaml` got one. It is `recurrence: Weekday, weekday: Saturday` \u2014 " - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/plan-next-week.yaml", - "signature": [ - "task/recurring-schedules/prod/plan-next-week.yaml:27-29" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 27, - "excerpt": "**`task/recurring-schedules/prod/plan-next-week.yaml:27-29`** \u2014 Success Criteria was rewritten to cover priorities, acceptance criteria, and `# Explicitly Deferred`, but Tasks line 43 also requires al" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/plan-next-week.yaml", - "signature": [ - "6db0dc9" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 29, - "excerpt": "**`task/recurring-schedules/prod/plan-next-week.yaml:29`** \u2014 \"Month link present (target may not exist yet if the monthly plan is deferred)\" explicitly sanctions a dead wikilink. Defensible for a note" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/plan-weekend.yaml", - "signature": [ - "frontmatter:" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 65, - "excerpt": "**`task/recurring-schedules/prod/plan-weekend.yaml:65`** \u2014 `{{next_sun_date}}` is the first use of a non-`{{current_date}}` placeholder in a `frontmatter:` field anywhere in this repo. On the base bra" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/plan-weekend.yaml", - "signature": [ - "contradicts" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 65, - "excerpt": "**`task/recurring-schedules/prod/plan-weekend.yaml:65`** \u2014 `due_date: \"{{next_sun_date}}\"` contradicts the task it annotates. Body line 18: \"as the first thing Saturday morning\"; the plan covers Sat\u2192S" - }, - { - "pr_id": "quant#109", - "path": "task/recurring-schedules/prod/renew-gmail-oauth-tokens.yaml", - "signature": [ - "{{current_date}}" - ], - "rule_id": null, - "state": "accepted", - "line_when_seen": 64, - "excerpt": "**`task/recurring-schedules/prod/renew-gmail-oauth-tokens.yaml:64`** \u2014 `due_date: \"{{next_sun_date}}\"` on a task whose own body (line 18) states \"Google's tokens expire every 7 days for unverified app" - }, - { - "pr_id": "tts-mcp#20", - "path": "CHANGELOG.md", - "signature": [ - "changelog.md:8" - ], - "rule_id": "changelog/unreleased-entry-required", - "state": "accepted", - "line_when_seen": 8, - "excerpt": "**`CHANGELOG.md:8`** \u2014 no `## Unreleased` section exists (top section is `## v0.6.0`), so this PR adds no changelog entry. `.maintainer.yaml` sets `release.autoRelease: true`, and the release agent pr" - } - ] + "version": "golden-dev-1", + "created": "2026-08-09", + "prs_version": "dev-1", + "baseline": { + "model": "opus", + "effort": "xhigh", + "mode": "full", + "coding_version": "v0.35.6", + "config_hash_prefix": "cc64cc99", + "rules_commands_hash_prefix": "ecc80333", + "runs": 1, + "findings": 42, + "wall_time_seconds": 2534 + }, + "match_rule": "rule_id exact when both sides carry one; else path + ALL signature keywords present in body (case-insensitive). line is NEVER used for identity.", + "states": { + "accepted": "a finding the reviewer SHOULD produce. A miss costs recall.", + "rejected": "a known false positive. A hit costs precision.", + "unreviewed": "not adjudicated. Excluded from both numerator and denominator." + }, + "scoring_note": "All entries are seeded 'accepted' from a single Opus 5 best-case run - this is the designed bootstrap, not hand-curated ground truth. Consequence: a configuration reporting a finding ABSENT from this set is NOT automatically wrong. Report such findings separately as gap-triage candidates rather than as precision failures. When several independent configurations agree on a finding Opus missed, that is evidence the golden set is incomplete - promote it to 'accepted'. When nothing else ever reproduces an Opus finding, that is evidence it may be a false positive - demote it to 'rejected'. The set sharpens through disagreement; it is not frozen truth.", + "known_corrections": [ + "tts-mcp#20 was annotated in bench/prs.json as 'clean - correct answer is zero findings'. That is FALSE. It returned 0 findings across six sonnet runs (six misses, not cleanliness); Opus found a real defect, hand-verified against head SHA 84feeb4d: CHANGELOG.md has no '## Unreleased' section while .maintainer.yaml sets autoRelease: true, so the CVE fix would never ship. The fixture currently has NO verified-clean PR." + ], + "entries": [ + { + "pr_id": "github-pr-review-agent#11", + "path": "CHANGELOG.md", + "signature": [ + "bullet bundles", + "split" + ], + "rule_id": "changelog/conventional-prefix-required", + "state": "accepted", + "line_when_seen": 11, + "excerpt": "**`CHANGELOG.md:11`** \u2014 one `feat:` bullet bundles three concerns: the new flag (`feat:`), the nil-poster panic guard in `tryDismissHallucinated` (`fix:`), and the CLI doc re-attribution (`docs:`). Sp", + "superseded_signature": [ + "changelog.md:11" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "cmd/run-task/main.go", + "signature": [ + "SkipPost: a.SkipPost", + "untested" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 171, + "excerpt": "**`cmd/run-task/main.go:171`** \u2014 the `SkipPost: a.SkipPost` pass-through and its counterpart call site at `pkg/factory/runner.go:136` are the only untested lines of the change: `cmd/run-task/main_test", + "superseded_signature": [ + "cmd/run-task/main.go:171" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "docs/pr-post-back.md", + "signature": [ + "preserves backward compatibility", + "local CLI mode" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 128, + "excerpt": "**`docs/pr-post-back.md:128`** \u2014 \"This preserves backward compatibility with the local CLI mode\" is now false. Before this PR `cmd/run-task` never produced a nil poster (that was the documented-vs-act", + "superseded_signature": [ + "docs/pr-post-back.md:128" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "docs/pr-post-back.md", + "signature": [ + "nil Poster", + "update both together" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 126, + "excerpt": "**`docs/pr-post-back.md:126`** \u2014 heading `## nil Poster \u2014 Local / Backward-Compatible Mode` carries the same stale framing as the line-128 finding; update both together (e.g. \"Local Opt-In (`--skip-po", + "superseded_signature": [ + "docs/pr-post-back.md:126" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/factory.go", + "signature": [ + "CreateAgent", + "posts review to GitHub" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 166, + "excerpt": "**`pkg/factory/factory.go:166-173`** \u2014 `CreateAgent`'s doc comment states the execution phase \"posts review to GitHub via PrPoster\" unconditionally, and parenthesises only the verifier case (\"nil veri", + "superseded_signature": [ + "pkg/factory/factory.go:166-173" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/runner.go", + "signature": [ + "write-scoped", + "--skip-post" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 104, + "excerpt": "**`pkg/factory/runner.go:104-106`** \u2014 a live, write-scoped GitHub App installation token is injected into the Claude subprocess env unconditionally, including under `--skip-post`; nothing in this PR t", + "superseded_signature": [ + "pkg/factory/runner.go:104-106" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/runner.go", + "signature": [ + "RunConfig.Agent", + "backward compatibility" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 51, + "excerpt": "**`pkg/factory/runner.go:51-53`** \u2014 `RunConfig.Agent`'s comment still frames the nil-`Agent` branch as \"cmd/run-task leaves this nil so CreateAgent is used for backward compatibility\". That branch is ", + "superseded_signature": [ + "pkg/factory/runner.go:51-53" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/runner.go", + "signature": [ + "ResolvePosters", + "exported solely" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 66, + "excerpt": "**`pkg/factory/runner.go:66`** \u2014 `ResolvePosters` is exported solely so `pkg/skip_post_boundary_test.go` (a different package's test binary, where an `export_test.go` alias would be invisible) can rea", + "superseded_signature": [ + "pkg/factory/runner.go:66" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/factory/runner_test.go", + "signature": [ + "BeNil()", + "reflect.ValueOf" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 32, + "excerpt": "**`pkg/factory/runner_test.go:32-33`** \u2014 `Expect(poster).To(BeNil())` cannot detect the regression this PR is themed around. Gomega's `BeNil()` falls through to `reflect.ValueOf(x).IsNil()` for pointe", + "superseded_signature": [ + "pkg/factory/runner_test.go:32-33" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/skip_post_boundary_test.go", + "signature": [ + "the comment claims", + "Diagnostics" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 169, + "excerpt": "**`pkg/skip_post_boundary_test.go:169-226`** \u2014 case 4's stated failure mechanism is wrong. The comment claims absence of an `ai_review verify:` line in `## Diagnostics` is the negative signal, but `ap", + "superseded_signature": [ + "pkg/skip_post_boundary_test.go:169-226" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/skip_post_boundary_test.go", + "signature": [ + "ResolvePosters(RunConfig{SkipPost:true})", + "(nil, nil)" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 102, + "excerpt": "**`pkg/skip_post_boundary_test.go:102-166`** \u2014 case 3 does not exercise the skip-post contract it is labelled with. `ResolvePosters(RunConfig{SkipPost:true})` can only ever return `(nil, nil)`, so the", + "superseded_signature": [ + "pkg/skip_post_boundary_test.go:102-166" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/skip_post_boundary_test.go", + "signature": [ + "case 2", + "steps_review_test.go" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 64, + "excerpt": "**`pkg/skip_post_boundary_test.go:64-100`** \u2014 case 2 and `pkg/steps_review_test.go:614-627` case (j) drive an identical scenario (verdict=fail + hallucinations + nil poster) and assert the same two ou", + "superseded_signature": [ + "pkg/skip_post_boundary_test.go:64-100" + ] + }, + { + "pr_id": "github-pr-review-agent#11", + "path": "pkg/steps_checkout_execution.go", + "signature": [ + "early return", + "ParseVerdict" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 316, + "excerpt": "**`pkg/steps_checkout_execution.go:316`** \u2014 the nil-poster early return fires before `ParseVerdict` (:328), before the fail-closed `!funnelRan && approve \u2192 request-changes` gate (:338-340), and before", + "superseded_signature": [ + "pkg/steps_checkout_execution.go:316" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "README.md", + "signature": [ + "License section", + "BSD-2-Clause" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 107, + "excerpt": "**`README.md:107`** \u2014 the new License section links the file without naming the license; the repo's own README template names it inline (`BSD-2-Clause \u2014 see [LICENSE](./LICENSE).`).", + "superseded_signature": [ + "readme.md:107" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "eslint.config.js", + "signature": [ + "parameter-property", + "no-explicit-any" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 44, + "excerpt": "**`eslint.config.js:44`** \u2014 the comment (\"enum, namespace, and parameter-property syntax would make `node src/*.ts` require a build step\") sits directly above `@typescript-eslint/no-explicit-any` and ", + "superseded_signature": [ + "eslint.config.js:44" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "eslint.config.js", + "signature": [ + "tseslint.config", + "@deprecated" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 16, + "excerpt": "**`eslint.config.js:16`** \u2014 `tseslint.config(...)` is marked `@deprecated` in the pinned typescript-eslint 8.66.0; I confirmed the annotation at `typescript-eslint/dist/config-helper.d.ts:67` (\"ESLint", + "superseded_signature": [ + "eslint.config.js:16" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "eslint.config.js", + "signature": [ + "no-non-null-assertion", + "ban-ts-comment" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 39, + "excerpt": "**`eslint.config.js:39-55`** \u2014 `CLAUDE.md` bans \"`any`, `as`, `!`, `@ts-ignore`\". Three are enforced (`no-explicit-any`, `no-non-null-assertion`, inherited `ban-ts-comment`); `as` is enforced by nothi", + "superseded_signature": [ + "eslint.config.js:39-55" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "package.json", + "signature": [ + "22.18", + "type-stripping" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 8, + "excerpt": "**`package.json:8`** \u2014 `\"node\": \">=22\"` no longer matches what the code requires. Node ships unflagged type-stripping only from **22.18.0**; on 22.6\u201322.17 it needs `--experimental-strip-types`, and be", + "superseded_signature": [ + "package.json:8" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "src/handlers/health.ts", + "signature": [ + "HealthRouter", + "asserted, not derived" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 12, + "excerpt": "**`src/handlers/health.ts:12`** \u2014 same defect: `HealthRouter` is asserted, not derived, and `healthRouter` (line 27) carries no annotation tying it to the alias. Note `src/config.ts:34` and `src/log.t", + "superseded_signature": [ + "src/handlers/health.ts:12" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "src/server.ts", + "signature": [ + "export type CreateApp", + "never bound" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 31, + "excerpt": "**`src/server.ts:31`** \u2014 `export type CreateApp` is hand-written and never bound to `createApp` (line 40), so `tsc` never compares the declared contract to the implementation; the only bridge is `requ", + "superseded_signature": [ + "src/server.ts:31" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "src/server.ts", + "signature": [ + "ErrorRequestHandler", + "err: any" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 69, + "excerpt": "**`src/server.ts:69`** \u2014 the error handler types `err: Error`, narrower than the shipped `ErrorRequestHandler`, which I confirmed declares `err: any` (`@types/express-serve-static-core/index.d.ts:77`)", + "superseded_signature": [ + "src/server.ts:69" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "test/health.test.ts", + "signature": [ + "check()", + "health.test.ts" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": null, + "excerpt": "**`test/health.test.ts`** \u2014 `config.check()` is the PR's new startup-validation logic and has zero coverage; `test/` contains only `health.test.ts`, which never imports config. Neither the invalid-`PO", + "superseded_signature": [ + "config.check()" + ] + }, + { + "pr_id": "node-skeleton#2", + "path": "test/health.test.ts", + "signature": [ + "address.port", + "port 0" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 22, + "excerpt": "**`test/health.test.ts:22`** \u2014 `? address.port : 0` silently falls back to port 0 instead of failing. If the invariant ever breaks, every test in the file fetches `http://127.0.0.1:0` and reports a co", + "superseded_signature": [ + "test/health.test.ts:22" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": ".github/workflows/ci.yml", + "signature": [ + "apt-key", + "trivy" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 32, + "excerpt": "**`.github/workflows/ci.yml:32`** \u2014 `wget -qO - \u2026 | sudo apt-key add -` installs Aqua's signing key into apt's **global** trusted keyring, where it can authenticate packages from *any* configured repo", + "superseded_signature": [ + ".github/workflows/ci.yml:32" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": ".github/workflows/ci.yml", + "signature": [ + "apt-get install -y trivy", + "unpinned" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 35, + "excerpt": "**`.github/workflows/ci.yml:35`** \u2014 `apt-get install -y trivy` is unpinned, so every CI run takes whatever is latest. Trivy minor releases add detectors and change exit semantics, so CI can go red wit", + "superseded_signature": [ + ".github/workflows/ci.yml:35" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": ".github/workflows/ci.yml", + "signature": [ + "sudo tee -a", + "duplicates the repo line" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 33, + "excerpt": "**`.github/workflows/ci.yml:33`** \u2014 `sudo tee -a` appends, so a re-run duplicates the repo line. Use `tee` without `-a`.", + "superseded_signature": [ + ".github/workflows/ci.yml:33" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": ".github/workflows/ci.yml", + "signature": [ + "uv sync --all-extras", + "redundant" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 38, + "excerpt": "**`.github/workflows/ci.yml:38`** \u2014 `uv sync --all-extras` is redundant; `make precommit` runs `sync` as its first prerequisite.", + "superseded_signature": [ + ".github/workflows/ci.yml:38" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "CHANGELOG.md", + "signature": [ + "ci: install trivy", + "prefix outside the recognised set" + ], + "rule_id": "changelog/conventional-prefix-required", + "state": "accepted", + "line_when_seen": 18, + "excerpt": "**`CHANGELOG.md:18`** \u2014 `- ci: install trivy in CI` uses a prefix outside the recognised set. The guide's table defines exactly seven (`feat:`, `fix:`, `refactor:`, `test:`, `docs:`, `chore:`, `perf:`", + "superseded_signature": [ + "changelog.md:18" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "Makefile.precommit", + "signature": [ + "--ignore-vuln", + "no such surface exists" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 42, + "excerpt": "**`Makefile.precommit:42`** \u2014 the comment promises that unreachable findings \"get an explicit `--ignore-vuln` entry with a reason\", but no such surface exists: no file, no make variable, nothing in th", + "superseded_signature": [ + "makefile.precommit:42" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "Makefile.precommit", + "signature": [ + "check: lint typecheck audit trivy", + "network egress" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 24, + "excerpt": "**`Makefile.precommit:24`** \u2014 `check: lint typecheck audit trivy` makes the default target and `make precommit` hard-require a local `trivy` binary plus network egress (trivy DB from ghcr, pip-audit a", + "superseded_signature": [ + "makefile.precommit:24" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "Makefile.precommit", + "signature": [ + "$(ROOTDIR)/.trivyignore", + "always wins" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 60, + "excerpt": "**`Makefile.precommit:60-61`** \u2014 the `$(ROOTDIR)/.trivyignore` fallback is unreachable today: the scan target is always literal `.`, so when make runs from the repo root the first `$(wildcard .trivyig", + "superseded_signature": [ + "makefile.precommit:60-61" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "README.md", + "signature": [ + "covers the exact transitive tree", + "uv export --all-extras" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 80, + "excerpt": "**`README.md:80`** and **`Makefile.precommit:36`** \u2014 \"covers the exact transitive tree that gets installed\" is not accurate. `make audit` runs `uv export --all-extras` (includes the `dev` extra: pytes", + "superseded_signature": [ + "readme.md:80" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "README.md", + "signature": [ + "Security gates", + "design rationale" + ], + "rule_id": "readme/user-facing-not-agent-context", + "state": "accepted", + "line_when_seen": 76, + "excerpt": "**`README.md:76-94`** \u2014 the whole \"Security gates\" section is design rationale, not user-facing content: why a make target depends on `sync` (line 82), why `osv-scanner` was rejected (line 86), and a ", + "superseded_signature": [ + "readme.md:76-94" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "README.md", + "signature": [ + "S104/S101", + "duplicates the comments" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 84, + "excerpt": "**`README.md:84`** \u2014 the S104/S101 suppression rationale duplicates the comments already in `pyproject.toml:65-67` and `pyproject.toml:71`, which will drift.", + "superseded_signature": [ + "readme.md:84" + ] + }, + { + "pr_id": "python-skeleton#3", + "path": "pyproject.toml", + "signature": [ + "S104", + "hardcoded-bind-all-interfaces" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 64, + "excerpt": "**`pyproject.toml:64-68`** \u2014 `ignore = [\"S104\"]` disables hardcoded-bind-all-interfaces across the entire repo. There are 4 hits: `src/skeleton/server.py:9`, `src/skeleton/__main__.py:33`, and 2 in `t", + "superseded_signature": [ + "pyproject.toml:64-68" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/dev/docker-registry-gc.yaml", + "signature": [ + "dev/docker-registry-gc.yaml", + "due_date" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 67, + "excerpt": "**`task/recurring-schedules/dev/docker-registry-gc.yaml:67`** \u2014 left without `due_date` while its prod sibling `prod/docker-registry-gc.yaml` got one. It is `recurrence: Weekday, weekday: Saturday` \u2014 ", + "superseded_signature": [ + "due_date" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/plan-next-week.yaml", + "signature": [ + "Success Criteria was rewritten", + "acceptance criteria" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 27, + "excerpt": "**`task/recurring-schedules/prod/plan-next-week.yaml:27-29`** \u2014 Success Criteria was rewritten to cover priorities, acceptance criteria, and `# Explicitly Deferred`, but Tasks line 43 also requires al", + "superseded_signature": [ + "task/recurring-schedules/prod/plan-next-week.yaml:27-29" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/plan-next-week.yaml", + "signature": [ + "Month link present", + "dead wikilink" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 29, + "excerpt": "**`task/recurring-schedules/prod/plan-next-week.yaml:29`** \u2014 \"Month link present (target may not exist yet if the monthly plan is deferred)\" explicitly sanctions a dead wikilink. Defensible for a note", + "superseded_signature": [ + "6db0dc9" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/plan-weekend.yaml", + "signature": [ + "{{next_sun_date}}", + "first use" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 65, + "excerpt": "**`task/recurring-schedules/prod/plan-weekend.yaml:65`** \u2014 `{{next_sun_date}}` is the first use of a non-`{{current_date}}` placeholder in a `frontmatter:` field anywhere in this repo. On the base bra", + "superseded_signature": [ + "frontmatter:" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/plan-weekend.yaml", + "signature": [ + "contradicts", + "first thing Saturday morning" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 65, + "excerpt": "**`task/recurring-schedules/prod/plan-weekend.yaml:65`** \u2014 `due_date: \"{{next_sun_date}}\"` contradicts the task it annotates. Body line 18: \"as the first thing Saturday morning\"; the plan covers Sat\u2192S", + "superseded_signature": [ + "contradicts" + ] + }, + { + "pr_id": "quant#109", + "path": "task/recurring-schedules/prod/renew-gmail-oauth-tokens.yaml", + "signature": [ + "expire every 7 days", + "{{next_sun_date}}" + ], + "rule_id": null, + "state": "accepted", + "line_when_seen": 64, + "excerpt": "**`task/recurring-schedules/prod/renew-gmail-oauth-tokens.yaml:64`** \u2014 `due_date: \"{{next_sun_date}}\"` on a task whose own body (line 18) states \"Google's tokens expire every 7 days for unverified app", + "superseded_signature": [ + "{{current_date}}" + ] + }, + { + "pr_id": "tts-mcp#20", + "path": "CHANGELOG.md", + "signature": [ + "## Unreleased", + "v0.6.0" + ], + "rule_id": "changelog/unreleased-entry-required", + "state": "accepted", + "line_when_seen": 8, + "excerpt": "**`CHANGELOG.md:8`** \u2014 no `## Unreleased` section exists (top section is `## v0.6.0`), so this PR adds no changelog entry. `.maintainer.yaml` sets `release.autoRelease: true`, and the release agent pr", + "superseded_signature": [ + "changelog.md:8" + ] + } + ] }