Skip to content

provenance sweep sub-topic: repo-wide sweep, upstream-drift convention engagement, and the cleaned-state gate #3465

Closed
3 of 3 issues completed

Description

@kyle-sexton

The provenance build plan's Phase 8 promotes the repo-wide sweep, the combined upstream-drift
convention engagement, and the narrow CI regression gate for the cleaned state into their own
sub-topic. This issue is that promotion artifact, and it is the Brief the sub-topic runs from.

Why this is an issue rather than docs/topics/provenance-sweep/PLAN.md

The plan named a contract slice as the promotion target. That target is not merge-able:
docs/topics/<slug>/ is Contract tier, branch-only, and scripts/check-contract-slice-prune.sh
hard-fails any new path there on the way to main — the parent topic's own slice was pruned for
exactly this reason (#3456). The work-item tracker is the graduation route the topic-docs
lifecycle names for actionable follow-ups, so the Brief lands here. Nothing about the plan's
content changes; only its vessel.

Precondition, and it is a hard one

The sweep cannot apply dispositions yet, and this is the gate's own arithmetic rather than a
blocker to route around.
The golden set ships at 10 cases across four groupings, so no class
reaches min_n_per_class 10 and every class is report-only at v1. fix and sweep apply
dispositions to fix-eligible findings only, and at v1 there are none.

So the sub-topic has two distinct stages, and the first does not wait on the second:

  1. A report-only sweep is runnable now and is worth running: it produces the corpus-scale
    telemetry (fetch counts, cache hits, carve-out declines, not-found rates) that the convention
    engagement needs as evidence, and it harvests adjudications into golden cases.
  2. A disposition-applying sweep unlocks only once the growth loop carries verbatim and
    near-verbatim past n=10 at or above the 0.95 precision bar. That is the named exit condition
    of the first growth round in provenance plugin: phases 6 to 8 (golden set and measurement, crosswalk registration, sweep) #3458.

Running stage 1 and feeding its adjudications back is what makes stage 2 reachable. Reversing
them is not an option.

Measured baseline entering the sweep

From the Phase 6 measurement (recorded in plugins/provenance/CHANGELOG.md): over 1,347 tracked
markdown files after carve-outs, 525 stamp candidates, 482 parsed, 43 declined, 0 expired at the
180-day default. Golden set: 10 cases, overall 8 tp / 0 fp / 0 fn / 2 tn, every class report-only.

The adversarial rotation probe (design thread T15) returned a real limit worth carrying into the
sweep: at roughly one substitution every four words, containment goes to 0.0 with no matched
spans against a source that was fetched and identity-checked. Word-shingling is evadable by an
author who intends to evade it, and no value of min_containment above zero recovers a passage
with zero matching shingles. The sweep should expect a floor on what the deterministic half can
confirm, and should not read a clean fingerprint as proof of originality.

Execution contract (inherited from the parent Brief)

  • One tracked file at a time. Apply the verdict, verify (semantic diff plus pointer liveness),
    close.
  • A file is closed when every finding in it carries a disposition or an explicit neutral
    outcome
    — never when the interesting ones are done.
  • Record each closure in the sweep ledger so an interrupted sweep resumes without re-deciding
    closed files, and so the closure count is a fact rather than a memory.
  • Carve-outs stay categorical. If the sweep starts accumulating per-instance suppressions, that is
    evidence a carve-out is drawn wrongly, and the fix is to redraw it in rubric.md — not to keep
    a suppression list. This matters beyond tidiness: see the convention engagement below.

The two additions the parent plan requires this Brief to carry

  1. The live no-breadcrumb (not-found) probe, as a first-run validation item. That disposition
    has never been exercised live — every spike source was supplied by hand. The sweep's first
    genuine no-breadcrumb candidate validates that the searched-surfaces listing names every
    surface actually checked, and that budget exhaustion reads as a first-class neutral outcome
    rather than as an acquittal.
  2. Sweep resume semantics. The fetch ceiling and the response cache are scoped per SWEEP, not
    per invocation, and both are recorded in the closure ledger. A resumed sweep therefore neither
    resets its spend nor silently reuses a stale cache. The ledger is checkout-local; state that
    explicitly, because a sweep resumed in a different checkout is a new sweep.

The convention engagement (fires at sweep completion only)

Drafted in full at docs/specs/provenance-convention-engagement.md. It executes as one entry
in docs/conventions/upstream-drift/CHANGELOG.md, carrying three parts:

  1. Recorded-decision re-derivation. chore(conventions): plugins restating upstream harness facts are not adopters of upstream-drift, so stale claims never surface #2273 deferred an adoption gate with its own stated
    trigger: "a detector is demonstrated that separates an upstream restatement from an in-repo one
    without a suppression list." Sweep completion either is that demonstration or it is not, and
    the engagement says which from evidence — the closure ledger, the measured precision/recall at
    the stated minimum n, and the carve-out record showing categorical exclusions only. If the
    sweep accumulated per-instance suppressions, the bar is not cleared, the trigger has not fired,
    and the engagement records exactly that with the count.

    The honest expected outcome, stated now so the engagement cannot overclaim later: the detector
    is LLM-led with deterministic verification, so the enforceability row moves at most from
    reasoning-only to detect-then-judge, and a deterministic suppression-free CI gate in the
    *-gate pattern remains unavailable.
  2. Enforceability table update, version bump conditional. Major bump ONLY if an enforceability
    verdict changes; otherwise a minor entry with the as-of date refreshed and the outcome stated,
    drift or no drift.
  3. The trigger-less-stamp table-row flip. Partially pre-consumed: the check's code shipped in
    Phase 3 and its crosswalk row in Phase 7 (feat(provenance): documentation-provenance audit plugin (phases 1 to 5) #3456 and the Phase 7 work). What remains here is
    exactly the convention's own row moving from "named but not built" to "built, off-by-default,
    repo-override enabled", plus the build-trigger bookkeeping. The registry row's early presence
    is scheduled, not drifted.

Also recorded at the same firing: the hash-store designed-issue trigger evaluated against the
sweep's fetch telemetry; the dead-pointer round-trip wiring (the weekly link-check lane routing
to the demotion path in reference/dispositions.md); and the adopters table, where rows are
added only for surfaces that actually conform, with #2297 getting its closure evidence if the
repository ends the sweep conforming.

What the engagement never does: declare the migration playbook's plugin-eval deferral trigger
fired; rewrite prior changelog entries or dated records; or convert a judgment verdict into a
deterministic claim — report-only tiers stay report-only whatever the sweep measured.

Checklist

  • Grow the golden set past min_n_per_class for verbatim and near-verbatim (provenance plugin: phases 6 to 8 (golden set and measurement, crosswalk registration, sweep) #3458)
  • Run the report-only sweep; record closure ledger, telemetry, and adjudications
  • Validate the live no-breadcrumb not-found probe on its first genuine case
  • Confirm sweep resume semantics (per-sweep ceiling and cache, checkout-local ledger)
  • Disposition-applying sweep, once a class is fix-eligible
  • Execute the convention engagement as one upstream-drift/CHANGELOG.md entry
  • Evaluate the hash-store designed-issue trigger against the sweep's fetch telemetry
  • Wire the dead-pointer round-trip to the weekly link-check lane
  • Narrow CI regression gate for the cleaned state

Related

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Aug 28, 2026
  2. kyle-sexton commented on Aug 28, 2026

    @kyle-sexton
    MemberAuthor

    Stage 1 report-only sweep: first run, recorded

    Ran the report-only sweep across the whole corpus at d7e391da, split into two judgment legs (docs/, plugins/) plus a mechanical telemetry leg, each with an independent fresh-context verifier. No dispositions were applied and no tracked file was edited by any sweep leg: at v1 no class reaches min_n_per_class, so applying one would be a correctness violation rather than a shortcut.

    Checklist movement

    • Run the report-only sweep; record closure ledger, telemetry, and adjudications — below.
    • Validate the live no-breadcrumb not-found probe on its first genuine case — validated, with a gap found.
    • Confirm sweep resume semantics — measured, and the answer is that they are not implemented.
    • Grow the golden set past min_n_per_class — candidates harvested below; not yet converted.
    • Disposition-applying sweep — still blocked, and now further away (see rubric v3).
    • Convention engagement, hash-store trigger, dead-pointer wiring, cleaned-state gate.

    Telemetry

    metric value at d7e391da, --as-of 2026-08-28
    tracked markdown after carve-outs 1,352 (1,395 considered, 43 declined at path level)
    stamp candidates / parsed / declined 535 / 491 / 44 (20 month-name, 24 bare-year)
    expired at 180-day default 0
    oldest parsed stamp 2026-04-08, plugins/work-items/skills/track/actions/add.md:104
    findings at a 60-day window 9
    fetches attempted / succeeded / failed 9 / 7 / 2 (both HTTP 404)
    rate-limit failures (429) 0
    fingerprint comparisons 784
    categorical carve-outs in effect 3
    per-instance suppressions accumulated 0

    Two of these figures are date-relative and expire, which is why the as-of date is pinned: the 180-day zero holds only until 2026-10-05 on the current oldest stamp, and the 60-day count moves daily.

    The adoption-bar evidence this issue asks for

    This issue states that if the sweep accumulates per-instance suppressions, the #2273 adoption bar is not cleared. It accumulated none. Three categorical carve-outs, no per-instance suppression list anywhere, and two skill surfaces forbid them by design. That is the first evidence on record pointing at the trigger, and it is recorded here rather than in the convention, because the engagement fires at sweep completion and this is one run.

    The excluded_paths placement holds as designed: the fixtures exclusion lives in the config layer, not unconditionally in list-corpus.sh, so the eval harness lifts it while a normal run declines it and says so. The #3041 precedent is live.

    Resume semantics are not implemented

    All three claims in this Brief measured as NOT IMPLEMENTED:

    claim verdict evidence
    fetch ceiling scoped per sweep not implemented reference/source-fetch.md:110 scopes corpus_fetch_ceiling to "the run"
    response cache scoped per sweep not implemented source-fetch.md:113-115, cache "lives in the run's memory slice"
    both recorded in the closure ledger not implemented reference/dispositions.md:152-154, SKILL.md:163-165 record closures and guard outcomes only

    Checkout-local holds by design (docs/specs/provenance-plugin-topology.md:60 places the ledger under .work/, which .gitignore:30 ignores) but no script enforces it — .work/ is referenced by no script at all. The Brief describes intended semantics; whether they were built is what this checklist item measured, and they were not. Recording rather than building, since this run was scoped to measure.

    Findings

    docs/ (148 files) does not lexically copy the upstream it cites. Exhaustive 296-comparison sweep, corpus-wide max containment 0.011. The five docs/upstream/*.md records tested against a real clone of mattpocock/skills scored 0.001–0.009 with zero firings, and their self-declared near-verbatim adoption cites upstream to the correct line.

    plugins/ produced six findings that stand, four of them in one file:

    location class tier
    plugins/context7/skills/lookup/context/lookup.md:40-41 verbatim fingerprint-confirmed
    same file, three further spans near-verbatim see caveat below
    plugins/firecrawl/skills/firecrawl/context/commands.md:36 paraphrase llm-suspected
    plugins/playwright/skills/playwright/reference/test-generation.md:80 near-verbatim source-fetched-similar

    lookup.md reproduces a five-step procedure from the live upstream in order, containment 0.217 with a 26-word span, and carries zero attribution and zero external URLs across 77 non-empty lines. The plugin does attribute upstash — in a sibling file — and C3 grades what is on the page.

    Tier caveat, found by the verifier: two of those four were tiered on the file-level separation flag, but their per-finding longest spans are 14 words each, below the 15 floor, and file containment 0.217 is below 0.3. Neither arm fires for them individually. They are genuine short copies below the deliberate floor — the floor working as designed — and must not be recorded as fingerprint-confirmed. Related process deviation: the cited ranges match no module span, so nomination ranges were carried through unrefined against SKILL.md step 7.

    A materialized drift, outside the tier table's domain: docs/NATIVE-SURFACES.md:98 records a native description absent from both shipped claude binaries in this container. The row's own recheck trigger has already fired. This is not drift risk; it is drift. check-stamps.sh structurally cannot see it.

    T15 reproduced live, twice

    The rotation limit is no longer only a synthetic probe. firecrawl/.../commands.md:36 restates (default 5, max 100) from the Firecrawl CLI docs at containment 0.000 with zero matched spans, against a source that was fetched and identity-checked. The verifier control-tested the module rather than trusting the zero (local-vs-itself returns 0.927), so this is a genuine detector miss on a table-row-to-prose restatement, not an operational failure. llm-suspected is the forced tier: no lexical evidence is possible.

    Neither leg proposed moving the 0.3/15 constants, and neither read a clean fingerprint as proof of originality.

    The no-breadcrumb probe validated, and found a gap

    Resolved at rung 4 (local binaries) after a zero-URL file. Budget exhaustion was exercised on a second candidate (404 then 403) and held as a neutral outcome, not an acquittal — correctly, because a further rung then resolved it, and stopping early would have lost the materialized drift above as a false negative.

    The gap: the searched-surfaces listing has no mechanical floor. emit-findings.sh has no field for it, no budget block, no schema. Three prose requirements, zero enforcement — a listing omitting a rung is indistinguishable downstream from a complete one.

    Also unplanned and worth a decision before stage 2: the stale string's authoritative home is docs/native-surfaces/records.json:108, outside the markdown corpus and generated, and its rendering says never to hand-edit it. A fix pass would edit a generated file.

    Detector defects found by running it on itself

    • False negative, being fixed: docs/upstream/aihero-course.md:127 declines as "bare year" while ending in a conforming ISO date — a real stamp silently exempted from expiry.
    • False positive, found independently by both legs: the ordinary English word "may" read as a month name (docs/conventions/standards/README.md:134, plugins/claude-config/skills/audit/SKILL.md:171). Also SC2034 and "cache_read_input_tokens": 2000 read as bare years.
    • The declined block's text field reports the keyword line rather than the date line, so whoever adjudicates the 44 declines sees misleading context.

    Adjudication candidates harvested

    Rejected findings are the high-value output, because a case converted from a rejected finding is one nobody authored to pass. Harvested, not yet converted: a byte-identical 16-word list of eight product-defined identifiers that fires the separation rule but fails C2 (no other way to write it); a conforming record whose trigger has fired, a shape no golden case has; carve-out 5's reach over RESEARCH-*.md; and a 0.589-containment/142-word case that turned entirely on grading scope.

    That last one produced a rubric change: C3 and C4 are graded at different scopes and v2 never said so. Two readers agreed on the verdict and disagreed on which scope produced it. Rubric v3 states it — C3 outward across the file, C4 on the passage — and takes the invalidation: the v2 golden-set measurement does not carry forward, so the set must be re-scored before any precision figure is cited. That pushes fix-eligibility further out, not closer.

    Honest limits on this run

    • Files were enumerated, not model-read. Model-led nomination covered roughly 35 files in plugins/; the rest got path- and stamp-level treatment only.
    • nomination_passes ran once, not twice — nominations were not unioned.
    • judge_samples was not met: judgments are single-context, not three blind panels. No verdict here carries panel unanimity. Both judgment legs reported this themselves rather than being caught at it.
    • Recall is therefore lower than designed. Moot for eligibility, since no class reaches minimum n either way, but it means the absence of findings in a tree is weaker evidence than the presence of one.
    • One leg's verifier caught a wrong-artifact identity error (two claude binaries in this container, 2.1.246 and 2.1.250; the finding survived at both, but the pin was wrong). Another leg could not obtain a verifier at all and said so rather than claiming one.

    Recommended before stage 2

    1. Decide the generated-file question — a fix pass must not edit records.json through its rendering.
    2. Give the searched-surfaces listing a mechanical floor, or stop claiming it as a validation.
    3. Re-score the golden set against rubric v3.
    4. Add a tier row for a vendored-snapshot source: strong provenance, weak currency, and every current row gates on "fetched" or "no source".

    Generated by Claude Code

  3. kyle-sexton commented on Sep 6, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI during triage.

    Claiming for triage-only evaluation (no execution).


    Generated by Claude Code

  4. kyle-sexton commented on Sep 6, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI during triage.

    Triage: verified, oversized, routed human-gated with a decomposition proposal

    Stage 1 ran and is recorded in detail. What this pass adds is a current-state re-measurement, which moves two things, and an assessment that the item is too large to route as one unit but cannot yet be sliced cleanly.

    Re-measured on current main, --as-of 2026-09-06

    files 1,471 | candidates 629 | parsed 585 | declined 44 | findings 87
    expiry_days 180 | trigger_less_check TRUE
    

    The trigger-less check is on and producing 87 findings. All 87 carry rule-trigger-less-stamp; zero are expiry findings, so the sweep's "0 expired at the 180-day default" still holds. This is directly load-bearing for the checklist's third convention part: the row's move to "built, off-by-default, repo-override enabled" is not pending, the override is live, and 87 is the number the engagement will have to report. Worth capturing before the figure moves.

    Two of the recorded detector false positives are still present. Reproduced from the declined groups:

    location declined as actual content
    plugins/code-tidying/.../shell-tooling.md:38 bare year SC2034, a shellcheck code
    plugins/context-guard/reference/reader-contract.md:78 bare year "cache_read_input_tokens": 2000

    A third is questionable: a table row in docs/upstream/mattpocock-skills-v12-map.md:48 declines as a bare year with no year in view. And one decline carries a wrong reason string: plugins/github/reference/recipes/billing.md:140 reads As of 2026-07 and is declined as "bare year, no month or day" when it plainly has a month.

    Two recorded defects appear resolved. The ordinary-English "may" false positive no longer appears in any month-name decline (all five are genuine month-name forms, three of them the plugin's own changelog describing this very bug). The text-field defect is not visible either: every example now shows the matched line. The stamp under-report fixed in #3473 is the likely cause. The issue's checklist does not track these, so their resolution is recorded here rather than ticked.

    Net: the decline classifier still has a real precision problem across 39 bare-year declines, narrower than at sweep time but not closed.

    Why this is oversized

    Nine checklist items, three complete. The remainder are not variations on one task, they are five different kinds of work:

    1. Detector defects — the surviving bare-year false positives and the wrong reason string. Small, testable, no open question.
    2. Unbuilt mechanism — resume semantics measured as NOT IMPLEMENTED on all three claims, and the searched-surfaces listing having no mechanical floor. The Brief describes intended behavior that was never built, so this is feature work, not repair.
    3. Open spec decisions — four of them, listed below.
    4. A blocked stage 2 — gated on golden-set growth past min_n_per_class (provenance plugin: phases 6 to 8 (golden set and measurement, crosswalk registration, sweep) #3458) and on re-scoring against rubric v3, whose scope change invalidated the v2 measurement. Two preconditions, not one, and the second was added after this issue was written.
    5. A convention changelog entry plus a CI gate — both fire only at sweep completion.

    Why it is not being decomposed in this pass

    Decomposition needs stable slice boundaries, and four unanswered decisions from the stage-1 run sit directly on them:

    1. The generated-file question. The materialized drift's authoritative home is outside the markdown corpus, is generated, and its rendering says never to hand-edit. A fix pass would edit a generated file. Until this is answered, nobody knows whether the sweep's disposition scope includes generated surfaces, which changes what a stage-2 slice even contains.
    2. The searched-surfaces listing. Three prose requirements, zero enforcement. Either give it a mechanical floor or stop claiming it as a validation. The checklist's no-breadcrumb item is marked complete against a validation that is unenforced, so this decides whether that tick stands.
    3. Rubric v3 re-scoring scope. The invalidation is recorded but its extent is not: whether all ten cases re-score or only the C3/C4-sensitive ones determines whether this is a slice or a footnote.
    4. The vendored-snapshot tier row. Every current row gates on "fetched" or "no source". Adding a row changes the tier table, which the golden set is scored against, which feeds min_n. Sequencing this after the re-score wastes the re-score.

    Slicing before these are answered produces items that get rewritten as soon as they are. That is the specific failure decomposition is supposed to prevent, so this pass stops at the proposal.

    Proposed decomposition, for approval in one pass

    Blockers first. A maintainer answering the four decisions above can approve this split as-is.

    # Slice Depends on Shape
    A Fix the surviving bare-year false positives and the wrong reason string none agent-ready, scoped
    B Answer the four open decisions none human, one sitting
    C Give the searched-surfaces listing a mechanical floor, or withdraw the claim B2 agent-ready once B answered
    D Build sweep resume semantics: per-sweep fetch ceiling and cache, ledger enforcement, checkout-local none agent-ready, scoped
    E Re-score the golden set against rubric v3 B3, B4 agent-ready once B answered
    F Grow the golden set past min_n_per_class E, #3458 tracked in #3458
    G Disposition-applying sweep B1, F blocked
    H Convention engagement as one changelog entry G blocked
    I Narrow CI regression gate for the cleaned state G blocked

    A and D are executable today and depend on nothing in the decision set. If the maintainer wants movement before sitting down with B, those two are where it exists.

    Routing

    needs-human, for a genuinely open decision rather than a capability blocker: the four questions in B are spec calls about what the sweep's disposition scope is and what the tier table contains. Neither the decision-defaulted exit nor a delegable one fits, because there is no reversible default available for a question that redefines what the deliverable is.

    The item stays as the sub-topic's Brief. Nothing here supersedes the stage-1 run comment, which remains the record of what was measured.

    Labels: replacing the raw priority: needs-triage with priority: medium (real value, no deadline, and the blocked stages cannot move until B is answered regardless of urgency); adding needs-human.

    Triage claim released.


    Generated by Claude Code

  5. 49 remaining items

  6. added
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    and removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Nov 2, 2026
  7. added this to the After zero milestone on Nov 2, 2026
  8. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Cordoned to the After zero milestone by the issue-triage session; not closed. Both open sub-issues, #9755 and #9759, are already in After zero, and admission holds this parent until they close (admission comment names both). Leaving it at status: ready counted it as lane supply it cannot be. The work stays open and resumes with its children.

    Posted by the issue-triage session. This was generated by AI.

  9. kyle-sexton commented on Nov 4, 2026

    @kyle-sexton
    MemberAuthor

    Disposition sweep, 2026-11-04

    Slice 2 of 3 for this issue, from #9755. This ran /attribution:audit sweep on main at b54572a9f98b81ce164d222e42cfaa2b8eb46121, in its own worktree. Plugin: attribution 0.15.3. Rubrics: copy v5, restated-fact v1. Sweep id chore-9755-attribution-disposition-sweep-20261104T134320Z. The ledger is checkout-local, so this comment is the record slice 3 reads.

    Result: zero dispositions applied, no edit, no commit, no PR. Exactly one file reached all three fix gates: scripts/fixtures/hooks-lifecycle-table.md, a test fixture this run's fence forbids editing. It stays open. Every other judged copy candidate cleared.

    Closure

    Measure Value
    Files in the model-read set 281
    Ledger closures 277
    Of those, every candidate has a panel verdict 2
    Of those, some candidates not judged (sampling rule) 275
    Files left open 4 (listed below)
    Dispositions applied 0
    Per-instance suppressions accumulated 0
    Carve-out redrawn none

    Files left open:

    • scripts/fixtures/hooks-lifecycle-table.md: 3 near-verbatim findings past every gate. The edit is outside this run's fence. Its consumer is scripts/gen-hook-event-registry.test.sh, which reads it as "a saved copy of the Hooks reference lifecycle table".
    • docs/contracts/execution-target/README.md:64-71, plugins/discovery/README.md:26, docs/migration-playbook.md:948-950: each carries one restated-fact finding that survived refutation, in a file another change is repointing links in, so it was left open. Restated-fact is report-only, and the recommended disposition is convert-to-pointer or condense-to-stamped-record, by the file's offline need.

    Read the 277 with care. Slice 3 should not read it as 277 adjudicated files. Only 74 candidates reached a panel, and 1907 of 1958 merged nominations and 532 fired candidates did not. A candidate that never reached a panel is not a finding and was not cleared; its outcome is "not judged (sampling rule)". That outcome is not one of the five dispositions in reference/dispositions.md. The ledger entries carry it under the labels neutral xN: nomination not panel-judged and leave-with-reason xN: fired span not panel-judged, which stretch those labels: the defined neutral outcome is neutral-not-found, and leave-with-reason is defined for a carve-out, a veto or a human decision. No candidate left unjudged could have been edited here, because the tier and verdict gates require a panel.

    The fixed rule for model reads

    Stated before nomination began: 281 files.

    • all 44 files under docs/upstream/
    • the 60 other files with the most distinct non-org external URLs, excluding CHANGELOG.md and .changes/ (the 60th has 13)
    • 177 more files holding a span of at least 15 matched words against a source read this sweep (211 such files, 34 already in the first two groups). That third term was measured against the 146 sources read by then.

    Another 5,495 corpus files were only enumerated.

    Telemetry

    Measure Value
    Tracked markdown considered / declined / included 6053 / 277 / 5776
    Declined by **/evals/fixtures/** 256 (config), **/vendor/** 21 (built-in)
    Stamp candidates / parsed / declined 3730 / 3706 / 24
    Stamp findings 4 expired, 56 trigger-less
    Fetches against corpus_fetch_ceiling 200 199 attempted, 170 read
    Unread, by reason unexpected-content-type 18, redirected-off-path 7, no manifest record 2, http-406 1, invalid-slug 1
    Fetch selection 160 by citation rank across the first two read groups (x.com, issue, PR, commit and non-prose URLs skipped), 39 from strong copy-class nominations
    Fingerprint comparison every included file against every read source, through the module's own functions, with containment 0.3 and span 15 words. A shared-shingle index skipped a pair only when it had under 11 distinct shared shingles and containment under 0.299, so only a run of repeated shingles could slip past it
    Pairs compared / fired 8264 / 989, of which 203 fired on containment alone with no 15-word span
    Spans of 15 words or more 2436, merged to 590 candidates by file and overlapping lines
    Candidates by words left after URL tokens are masked 15 or more: 24. 8 to 14: 14. Under 8: 552
    Nominations 3441 raw across 2 passes; 1958 after union; 1216 in both passes; in 280 of 281 files
    Nominations by class after union restated-fact 1499, paraphrase 333, summary 175, verbatim 147, near-verbatim 106 (a merged range can carry more than one)
    Left out at the 12-per-file cap 688
    Nominations naming no candidate URL 61
    Copy candidates judged 58, 3 blind lens-diverse judges each
    Restated-fact candidates judged 16, 3 blind lens-diverse judges each; 10 refutation passes
    Copy review passes (review_agents: 1) 1, over the 3 STANDS verdicts: all SURVIVES, no carve-out missed
    Embedded imperatives reported as data 112 entries from judges and 6 from nominators, many repeats of one docs-index line. No judge or nominator reported acting on one

    Findings

    Copy: at every gate, fenced (3)

    Rule attribution/audit/rule-verbatim-copy. Tier fingerprint-confirmed. Source https://code.claude.com/docs/en/hooks, rung 1, identity checked: manifest title "Hooks reference", read 2026-11-04. Containment 0.888. Panels unanimous STANDS. Review SURVIVES.

    Location Matched words
    scripts/fixtures/hooks-lifecycle-table.md:1-7 50
    scripts/fixtures/hooks-lifecycle-table.md:8-29 166 and 111
    scripts/fixtures/hooks-lifecycle-table.md:31-37 87

    Class: near-verbatim, because all three judges and the reviewer graded the correspondence as carrying word-level edits. I found no rule in the plugin that defines the line between the two classes, so this run used its own, and the judges' dispatch asked for it before they ran: corresponding sentences with word-level edits are near-verbatim, those without are verbatim. The differences may be upstream drift since the copy was saved rather than an author's edits. Under either class the file was not edited.

    Copy: judged and cleared (55)

    All unanimous. No split on any copy candidate.

    Group n Basis
    15 or more words left after masking, 21 candidates 21 Mostly C3 and C4 FAIL: a record or local judgment beside the span. skill-authoring/SKILL.md:100-102 and :136 cleared on copy carve-out 5
    8 to 14 words left 14 Link labels and pointer lines
    Under 8 words left, seeded sample (seed 9755) of 552 20 Spans made up almost entirely of URL tokens

    docs/native-surfaces.md carries a generated-output marker; its two candidates cleared, so no finding names its input (docs/native-surfaces/records.json) as a fix site.

    Restated-fact: relay-eligible, report-only (9)

    Rule attribution/audit/rule-restated-upstream-fact, rubric v1. Unanimous panel, refutation SURVIVES. Never fix-eligible.

    Location Source located
    plugins/app-control/reference/control-skill.md:78-94 https://code.claude.com/docs/en/plugins/manifest-reference
    docs/contracts/execution-target/README.md:64-71 (left open) https://code.claude.com/docs/en/cli-reference
    plugins/work-items/tools/work-item-tracker/adapters/github/README.md:676-696 none
    plugins/playbooks/skills/fable-5/SKILL.md:126 https://code.claude.com/docs/en/costs
    plugins/source-control/skills/pull-request/SKILL.md:44 none
    plugins/user-experience/reference/flows-ia.md:134-136 none
    plugins/discovery/README.md:26 (left open) https://code.claude.com/docs/en/hooks
    plugins/code-metrics/README.md:150-158 https://coverage.readthedocs.io/en/latest/commands/cmd_run.html
    docs/migration-playbook.md:948-950 (left open) https://code.claude.com/docs/en/plugins/components

    Restated-fact: human report only (2)

    Location Outcome
    plugins/user-experience/reference/research-methods.md:65-74 Unanimous STANDS, then REFUTED on attack (b): whether a recommendation several publishers share has one external owner. leave-with-reason
    docs/upstream/claude-code-mods/research-2026-09-19/research-contradictions-and-corrections.md:108-115 Split 2 to 1. The dissent failed R3

    Restated-fact: cleared (5)

    plugins/playbooks/skills/boris/reference/autonomy.md:43-60, plugins/debugging/README.md:58-62, plugins/playbooks/README.md:51-58, plugins/context-guard/reference/cloud-headless-capture.md:49-52, plugins/review/agents/security-reviewer.md:81-87.

    Deterministic

    4 expired stamps: docs/upstream/claude-com-fold-subagents.md:67, plugins/work-items/skills/track/actions/add.md:116, and plugins/attribution/evals/audit-stale-stamps/prompt.md:7 and plugins/attribution/evals/audit-stamp-declined-reported/prompt.md:7, which are eval inputs. 56 trigger-less stamps. Report-only; none was judged.

    Precision and n per class

    Relayed, not measured here. score-golden.sh needs --golden and --actual, and --actual is a judged run's product; no such file is tracked, so a run on current main would mean re-judging all 56 cases. This run applied no disposition, so the done-when does not call for it. Copy rubric v5 and the golden cases have not changed since the score below, except the tier table's fix-eligible wording in #12369. Whole-set re-score of 56 cases, from plugins/attribution/CHANGELOG.md:76-83, with fix-eligibility from the owner's ruling:

    Class n Precision Recall Gate
    near-verbatim 15 1.00 0.93 binding, fix-eligible by the owner's ruling
    verbatim 5 1.00 0.80 report-only
    paraphrase 1 1.00 1.00 report-only
    restated-fact 9 1.00 1.00 report-only
    hard negatives 26 n/a n/a binding

    Measured here, and it is not precision. Panel outcomes on real text, with no human adjudication and so no precision figure:

    Panel n judged Stands Cleared Split
    Copy 58 3, all one file 55 0
    Restated-fact 16 10, of which 9 survived refutation 5 1

    Honest limits

    • Files model-read versus enumerated: 281 of 5776 read; 5495 enumerated only, covered by the stamp check and the all-pairs fingerprint.
    • Partial reads by rule: seven CHANGELOG.md files and the generated docs/native-surfaces.md went to nominators with only the lines within 15 of a fired span. Nominators reported reading the other files whole.
    • Partial reads by judges: the first judges had no search tool in their dispatch and paged, so several read some long sources only in part. Their result files name what they skipped. The cleared verdicts rest on C3 and C4 quotes, which a further read of the source does not change, but a "no match on the page" there reaches only what was read.
    • What was not judged: 532 of the 552 fired candidates with under 8 words left after masking. The 20 sampled from them all cleared, but that is a sample, not a clearance. All 203 containment-only pairs: 203 files, all under an evals tree, 180 of them named skill-fired.md, each with no 15-word span. The separation rule fired on them, but the read rule asks for a span, so they were not model-read. 1483 of 1499 restated-fact nominations (16 sampled, at most one per file, seed 9755, from 115 strong nominations in both passes with no record noted). Every copy-class nomination whose span did not reach the rule, none of which could be fix-eligible.
    • No findings file was emitted. The relay is report-only for this run: this comment holds the findings. A relay file would have carried three fix-eligible rows at a file this run was fenced from editing.
    • Candidates in the third read group's URL-masked split are a triage aid, not a gate. Masking decided which spans a panel saw first, and nothing was cleared by it.
    • Fetch ceiling bounded recall. Spend stopped at 199 of 200, and 119 distinct URLs named by strong copy-class nominations were unread at that point. A file's text was only ever compared to the 170 sources read.
    • One source family could not be read at all. 18 fetches ended unexpected-content-type, including raw.githubusercontent.com pages served as text/plain, which is the audit's rung 1b route. They include the Claude Code mods/README.md, CHANGELOG.md and mods/types/claude-code.d.ts; nominators named the last on 22 ranges, more than any other unread URL. Copies of those pages could not reach fingerprint-confirmed in this run.
    • The class rule is this run's. See the copy findings above.
    • A clean fingerprint is not proof of originality. The rotation floor recorded in this issue's body still applies.
    • The live no-breadcrumb probe stays unvalidated. 61 nominations named no URL, but no budgeted search ran, so this run offers no evidence on the searched-surfaces listing.
    • Resume semantics were not exercised. The ledger recorded spend and 170 cache entries, but this sweep was never interrupted.
    • Nominators ran on Sonnet; judges, reviewer and refuters on Opus 5.5. Each judge graded a packet of up to 8 cases, not one per dispatch.
    • Dispatch errors, all recovered. Some nominators wrote their result to a mistyped path outside the worktree; those files were moved and the stray path removed. Three pass-1 results moved that way were missed by the first URL selection, so they did not influence 37 of the 39 nomination-driven fetches. One judge dispatch carried a placeholder prompt and returned nothing; it was re-run. Several judges ran read-only shell commands their dispatch had ruled out.
  10. kyle-sexton commented on Nov 6, 2026

    @kyle-sexton
    MemberAuthor

    Close audit, 2026-11-06: not closable yet. The issue stays in After zero.

    Item State
    Golden set past min_n near-verbatim n=15, binding; verbatim n=5. The owner's 2026-11-02 ruling makes near-verbatim fix-eligible on its own, which supersedes this item's both-classes precondition
    Report-only sweep done: 2026-10-28 and 2026-11-04 records
    Live no-breadcrumb not-found probe not validated: no budgeted search has run. Attribution CHANGELOG 0.6.3 made the searched-surfaces listing a prose requirement
    Sweep resume semantics documented (attribution CHANGELOG, resume entry) but never exercised
    Disposition-applying sweep ran 2026-11-04 and applied 0. The only file through all three gates is scripts/fixtures/hooks-lifecycle-table.md, a saved copy that scripts/gen-hook-event-registry.test.sh reads, fenced from that run
    Convention entry, hash-store trigger done: #13461
    Dead-pointer round-trip done: #13549 with melodic-software/ci-workflows#993
    Cleaned-state CI gate moved to the hardening pass (#9759, Q-9759-2 (b))

    What closes it: a sweep run that gives a no-breadcrumb candidate its budgeted search, one interrupted-and-resumed sweep, and a disposition for the fixture. The fixture is a deliberate copy, so its disposition is most likely a recorded keep, not a rewrite.

  11. self-assigned this
    on Nov 6, 2026
  12. kyle-sexton commented on Nov 6, 2026

    @kyle-sexton
    MemberAuthor
    No description provided.
  13. kyle-sexton commented on Nov 6, 2026

    @kyle-sexton
    MemberAuthor

    Close evidence: budgeted search, interrupted-and-resumed sweep, fixture disposition

    Answers the three gaps in the 2026-11-06 close audit. One sweep, /attribution:audit sweep, over two files: scripts/fixtures/hooks-lifecycle-table.md and plugins/machine-health/skills/audit/reference/windows/check-catalog.md. It ran on branch chore/3465-provenance-sweep-close, fast-forwarded to main at 50dc172d624, in its own worktree.

    • Sweep id: attribution-3465-close-20261106T023054Z, topic attribution-3465-close.
    • Scripts: the source tree's attribution 0.17.2. The installed skill body is 0.15.5, and its sweep-ledger.sh is byte-identical.
    • Config layer read: the worktree's docs/conventions/attribution.yaml. There is no separation key, so the fingerprint ran at 0.3 / 15.
    • The ledger is checkout-local, so this comment is the record.

    1. No-breadcrumb candidate: budgeted search ran, and found the source

    Candidate. check-catalog.md:414-419. It says LastTaskResult 267011 is SCHED_S_TASK_HAS_NOT_RUN and 267014 is SCHED_S_TASK_TERMINATED. The 2026-10-28 run recorded it as not-found with no Task Scheduler page fetched.

    Surfaces searched, in order:

    Surface Result
    Breadcrumbs in or near the passage none
    The file's 34 breadcrumb URLs (extract-breadcrumbs.sh --dir) none about Task Scheduler (Defender, CISA, Edge and Claude Code pages)
    Sibling files: elevation-matrix.md (4 URLs) and remediation-policy.md (0 URLs) none about Task Scheduler
    Search 1 of 3 (searches_per_candidate 3): Firecrawl web search for SCHED_S_TASK_HAS_NOT_RUN 0x00041303 Task Scheduler error and success constants top result: Task Scheduler error and success constants (WinError.h). The second result, on Stack Overflow, names the same winerror.h definition

    Resolution stopped after search 1 because that search identified the source and the fetch below confirmed both values. This was not the convergence rule, which needs the same top source from two searches. Searches 2 and 3 were not spent.

    Fetch. Fetch 1 of 5 (fetches_per_candidate 5) read that page at rung 1 (fetch-docs.sh --profile generic).

    • Manifest: state: read, HTTP 200, text/markdown, sha256 0f45078c….
    • Identity: the first H1, at line 101 after a navigation list, is # Task Scheduler error and success constants (WinError.h) - Win32 apps | Microsoft Learn. The generic profile has no page index, so the canonical-slug check did not run. The page lists SCHED_S_TASK_HAS_NOT_RUN as 0x00041303 (267011) and SCHED_S_TASK_TERMINATED as 0x00041306 (267014). Both values in the passage match.
    • Fingerprint: containment 0.000, longest span 6 words, separation rule not fired.

    Outcome: a source was found, not not-found. The candidate stays on the restated-fact rubric at tier source-fetched-similar, report-only. Because the source was found, this run did not re-exercise the neutral budget-exhausted path. The Stage 1 report-only sweep exercised that path (404 then 403, held neutral); see this issue's first comment.

    Not re-run here. The verdict relays the 2026-10-28 panel: 3/3 STANDS, refutation SURVIVES. No new panel ran.

    Recommended disposition: condense-to-stamped-record, pointing at that page, with a recheck trigger of that page changing either constant. A human applies it. The file stays open in the ledger, the same treatment the 2026-11-04 run gave report-only restated-fact files.

    Limit. The searched-surfaces listing above is this run's own claim. Per CHANGELOG 0.6.3 and reference/source-fetch.md, nothing machine-checks it for completeness.

    2. Interrupted and resumed

    Run 1 (first skill invocation):

    • init at 2026-11-06T02:30:54Z.
    • Fetched the Hooks reference: spend 2, cache-add.
    • Closed the fixture.
    • Ran the check-catalog search and fetch: spend 1, cache-add at 02:32:26Z.
    • Stopped before closing check-catalog.

    status at the checkpoint: 1 closed, spend 3 of 200, 2 cache entries.

    Run 2 (second skill invocation, same checkout and topic):

    Step What happened
    Restored init printed ledger exists … (a resume of sweep attribution-3465-close-20261106T023054Z). status restored 1 closed file, spend 3 (not 0) and 2 cache entries
    Skipped The fixture was not re-decided. A deliberate re-close was refused: already closed, exit 3
    Redone cache-check on the Microsoft page printed re-validate … fetch it again and compare the hash. The page was re-fetched with --max-age 0: server 304, sha256 unchanged. That fetch was spent (spend 1) and the page was cached again at 02:33:03Z

    Final status: 1 closed, spend 4 of 200, 2 cache entries.

    Not exercised here: a resume from a different checkout. plugins/attribution/skills/audit/scripts/sweep-ledger.test.sh:277-300 covers it: another checkout sees a new sweep, and a copied ledger is refused.

    How the interruption was made. Run 1 stopped deliberately between two Skill invocations in one session, not through a crash or a lost session.

    • Run 1: 02:30:54Z to 02:32:26Z.
    • Run 2: about 02:32:50Z to 02:33:03Z.

    3. Fixture disposition: a recorded keep, by category

    scripts/fixtures/hooks-lifecycle-table.md reproduced the 2026-11-04 evidence against the Hooks reference:

    • identity checked, rung 1
    • containment 0.888
    • spans of 50, 166, 111 and 87 words

    Disposition: leave-with-reason ×3, on the test-fixture carve-out. The file is a deliberate saved copy that scripts/gen-hook-event-registry.test.sh reads. It was not edited.

    Where the keep is recorded. The plugin has no per-instance suppression (plugins/attribution/README.md:88-91). The finding-suppression contract treats a file the audit should never read as an exclusion, not a suppression. So the keep is a categorical excluded_paths entry. Commit 000a7b7523a puts the bundled 0.16.0 default (**/fixtures/**, **/__fixtures__/**, **/testdata/**) back into docs/conventions/attribution.yaml.

    Why the default was missing. That file's **/evals/fixtures/** (#10555, 2026-10-28) replaced the default whole, per skills/setup/reference/keys.md "The test-fixture exclusion". The default itself shipped later (#13354, 2026-11-04).

    Effect on the corpus.

    Declined Included
    Before 277 5902
    After 485 (462 fixtures, 2 testdata, 21 vendor) 5694

    The fixture is no longer in the corpus. Per-instance suppressions: 0.

    Limits

    This was a targeted two-file run on candidates named by earlier records. These skill steps did not run:

    • stamps (step 3)
    • nomination (step 4)
    • new judge or refutation panels (step 8)
    • the report sidecar and emit-findings.sh (step 10)
    • the findings file (step 11)

    The installed plugin is 0.15.5 and the source is 0.17.2.

    The skill resolves its config root and its ledger from the session's working directory. A run aimed at another worktree by path reads that worktree's config and writes its ledger there only when it runs from inside that worktree; this run used env -C.

    Remaining against the close audit

    None of the three items above.

    One follow-up: check-catalog.md:414-419 waits on a human to apply the recommended stamped record.

  14. kyle-sexton commented on Nov 6, 2026

    @kyle-sexton
    MemberAuthor

    Closed: every item is delivered or deferred by owner decision. The last change, #13643, merged at 2026-11-06T03:10Z.

    Item State Evidence
    Golden set past min_n_per_class met by owner ruling near-verbatim n=15; the owner ruling of 2026-11-02 makes near-verbatim fix-eligible on its own (close audit, 2026-11-06)
    Report-only sweep done the 2026-10-28 and 2026-11-04 records above
    Live no-breadcrumb not-found probe done #3465 (comment), item 1
    Sweep resume semantics done same record, item 2
    Disposition-applying sweep done ran 2026-11-04; the fixture keep was decided in the same record and landed in #13643 (bundled fixture exclusion restored)
    Convention engagement, hash-store trigger done #13461
    Dead-pointer round-trip done #13549 with melodic-software/ci-workflows#993
    Narrow CI regression gate deferred owner decision Q-9759-2 (b): build it after a disposition sweep applies at least one disposition. #13595 records it with its recheck trigger in docs/contracts/upstream-drift/README.md

    The gate stays out of this issue under the owner direction of 2026-11-05 to close hardening-tied work and rediscover it one item at a time, the same basis on which #9759 closed. Open follow-up, advisory: check-catalog.md:414-419 would take a condense-to-stamped-record disposition, which a human applies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

priority: mediumReal value, no hard deadline; normal backlog flow.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

Type

Fields

Work status

confirmed

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions