skill: mandatory-ID gate for re-extracted docs on --update - #2162
skill: mandatory-ID gate for re-extracted docs on --update#2162AirRocker wants to merge 1 commit into
Conversation
cc0fca1 to
8e1c8bb
Compare
|
Rebased onto current Verified locally on the rebased branch: all five skillgen guards pass ( I've also added a short section to the description on how this sits alongside #2497, since that landed after the PR was opened and is clearly the better version of the count check. The reason I think it's still worth having both: the revived guard is gated Very happy to rework the shape if a lighter touch would fit the project better — an opt-in flag, a different default, or a config knob would all be fine by me. The protection is what I care about, not the mechanism. No rush on any of this. |
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
This PR adds an identical advisory callout block to each of the per-agent Graphify skill markdown files (skill-agents, skill-amp, skill-claw, etc.) plus the base skill.md, inserted after the --update/--cluster-only explanation. The note directs readers to a "mandatory-ID gate" procedure in references/update.md for re-extracting already-graphed documents. The changes also appear to expand references/update.md itself with the corresponding gate documentation. The surface area is documentation-only across the graphify skill variants and their shared reference file.
No blocking issues surfaced. 1 lower-confidence candidate did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 875 functions depend on the 875 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 875 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 875 function(s) in the blast radius were not formally verified this run
8e1c8bb to
7992600
Compare
|
Rebased onto current I've also updated the description for the post-0.9.53 picture. The count-level guard now ships on the extract CLI path, so this gate's remaining scope is the skill The fresh push needs a workflow approval before CI runs, whenever that's convenient. Genuinely no rush. I'm still happy to reshape this however suits the project, whether that's an opt-in flag, a different default, or a config knob. |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Graphify review — findings
Adds a mandatory-ID gate warning to the --update/--cluster-only section of every graphify skill variant and its skillgen fixtures, requiring re-extraction of an already-graphed doc to snapshot the file's existing node IDs, pass them to the subagent as must-include, and refuse the merge if the node count falls below 90% of baseline or any ID goes missing. Documents that this guards against non-deterministic semantic re-extraction silently dropping whole sections, with the full procedure spelled out in references/update.md.
Worth a look
- Mandatory-ID gate uses a single shared state file for every re-extracted document —
graphify/skills/agents/references/update.md:84· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Mandatory-ID snapshot can include unrelated files with matching suffix —
graphify/skills/agents/references/update.md:86· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Baseline snapshot matches node IDs by endswith(TARGET) causing suffix collisions —
graphify/skills/agents/references/update.md:87· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Gate reads only .graphify_chunk_01.json, missing IDs in later chunks —
graphify/skills/agents/references/update.md:105· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Mandatory-ID gate uses a single shared state file for every re-extracted document —
graphify/skills/amp/references/update.md:84· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 875 functions depend on the 875 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 875 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 875 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
266 of 266 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/test_affected_cli.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_agents_platform.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— full-run-safetytests/test_astro_import_ids.py— full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_build.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— full-run-safetytests/test_cache.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— full-run-safetytests/test_charmap_encoding.py— full-run-safetytests/test_chunking.py— full-run-safetytests/test_cjs_module_extension.py— full-run-safetytests/test_claude_cli_backend.py— full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cluster.py— full-run-safetytests/test_codebuddy.py— full-run-safetytests/test_community_hub_labels.py— full-run-safetytests/test_community_labels_skill.py— full-run-safetytests/test_confidence.py— full-run-safetytests/test_corrupt_graph_json.py— full-run-safetytests/test_cpp_nested_and_cli.py— full-run-safetytests/test_cpp_objc_cross_file_calls.py— full-run-safetytests/test_cpp_preprocess.py— full-run-safetytests/test_cross_extension_reexport_self_cycle.py— full-run-safetytests/test_cross_language_call_resolution.py— full-run-safetytests/test_cross_repo_member_calls.py— full-run-safetytests/test_cross_repo_shared_types.py— full-run-safetytests/test_csharp_call_site_generic_args.py— full-run-safetytests/test_csharp_enum_members.py— full-run-safetytests/test_csharp_field_generic_args.py— full-run-safetytests/test_csharp_generic_callsites.py— full-run-safety- … and 216 more
non-code file(s) changed (
graphify/skill-agents.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md,graphify/skill-copilot.md…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
graphify/skill-agents.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md,graphify/skill-copilot.md…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
…hed doc Semantic re-extraction of a doc that already has nodes in graph.json is non-deterministic. On a large project CLAUDE.md, successive --update runs of the same unchanged-except-one-line file returned 75 nodes and then 49, silently dropping a whole documented section. Nothing errored: the loss went into build_merge and would only have surfaced later as a question the graph could no longer answer. It was caught solely because the to_json shrink guard (Graphify-Labs#479) refused the write. Adds a required three-step gate to the --update flow, in the shared references/update.md fragment plus a pointer from the core template: 1. Snapshot that file's existing node IDs from graph.json before extracting. 2. Pass them to the extraction subagent as MANDATORY must-include IDs, with the baseline node count as an explicit target. Reusing the IDs also suppresses gratuitous ID churn, which orphans saved queries and inflates the merge diff for no semantic gain. 3. Hard-gate the merge: refuse unless node count >= 90% of baseline AND zero mandatory IDs are missing. On failure, re-dispatch naming the missing IDs; two consecutive failures stop and report rather than merging a regression. Also restates that the shrink guard is the authoritative backstop: never force past it without diffing the old and new node sets and being able to name why each removed node is legitimately gone. Fragments are the edited source; graphify/skill*.md, graphify/skills/**, graphify/always_on/** and tools/skillgen/expected/** are regenerated via `python -m tools.skillgen` and `--bless`. Markdown only, no code changes. All five skillgen guards pass (--check, --audit-coverage, --schema-singleton, --monolith-roundtrip, --always-on-roundtrip). Review follow-up (graphify-labs bot, 2026-09-10) — three defects fixed in the gate procedure, each reproduced against a real graph before the edit: * Suffix matching. `endswith(TARGET)` is not anchored to a path boundary, so a bare name adopts every same-named file deeper in the tree. Measured on a real corpus, TARGET='README.md' matched 'workers/README.md' too: baseline 22 nodes instead of 7, and a *perfect* re-extraction of README.md then scored 31.8% of baseline with 15 mandatory IDs "missing" — a hard FAIL on correct output, and the prescribed remedy (re-dispatch naming the missing IDs) would have pushed the subagent to mint another file's nodes under this one's source_file. Now matched exactly, with an ambiguity guard that lists the candidates and stops rather than guessing. * Single shared state file. `.graphify_must_ids.txt` / `.graphify_must_slice.json` were fixed paths, so in a multi-doc delta the second snapshot overwrote the first and that doc went through ungated while the gate still printed PASS — failing open, in the one place this procedure exists to fail closed. State files are now keyed by target, with a one-time clear of stale snapshots. * Only `.graphify_chunk_01.json` was read. Step 3B splits a delta into chunks of 20-25 files (each image its own), so a changed doc can land in any chunk; the gate then read every mandatory ID as missing and failed a clean extraction. The gate now unions all chunks and evaluates every snapshot, counting only the nodes attributed to each target — which also fixes a latent over-count, since the old `len(d['nodes'])` was the whole chunk, not the target's slice. Verified: multi-chunk delta with the target in chunk_02 now PASSes, with negative controls (drop a mandatory node, drop a baseline hyperedge) confirming the gate still FAILs when it should. Markdown only; no code changes. Claude-Session: https://claude.ai/code/session_01B8MXR5KMGHczdCLXB8kvJd
7992600 to
a9b253d
Compare
|
Thanks for this — the review earned its keep. All four distinct findings are real, and three of them I could reproduce against a live graph before touching a word of the text. Fixed in 1. Single shared state file for every re-extracted document (
|
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Formal verification. 2 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds a mandatory-ID gate callout to the --update/re-extraction section of every graphify skill variant and its expected fixtures, telling the agent to snapshot an already-graphed doc's node IDs, pass them to the subagent as must-include, and refuse the merge if node count drops below 90% of baseline or any ID goes missing. Explains the fallback rationale: semantic re-extraction is non-deterministic and silently drops whole sections without the gate, with the full procedure documented in references/update.md.
No blocking issues surfaced. 49 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 875 functions depend on the 875 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 875 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 875 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
266 of 266 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/test_affected_cli.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_agents_platform.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— full-run-safetytests/test_astro_import_ids.py— full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_build.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— full-run-safetytests/test_cache.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— full-run-safetytests/test_charmap_encoding.py— full-run-safetytests/test_chunking.py— full-run-safetytests/test_cjs_module_extension.py— full-run-safetytests/test_claude_cli_backend.py— full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cluster.py— full-run-safetytests/test_codebuddy.py— full-run-safetytests/test_community_hub_labels.py— full-run-safetytests/test_community_labels_skill.py— full-run-safetytests/test_confidence.py— full-run-safetytests/test_corrupt_graph_json.py— full-run-safetytests/test_cpp_nested_and_cli.py— full-run-safetytests/test_cpp_objc_cross_file_calls.py— full-run-safetytests/test_cpp_preprocess.py— full-run-safetytests/test_cross_extension_reexport_self_cycle.py— full-run-safetytests/test_cross_language_call_resolution.py— full-run-safetytests/test_cross_repo_member_calls.py— full-run-safetytests/test_cross_repo_shared_types.py— full-run-safetytests/test_csharp_call_site_generic_args.py— full-run-safetytests/test_csharp_enum_members.py— full-run-safetytests/test_csharp_field_generic_args.py— full-run-safetytests/test_csharp_generic_callsites.py— full-run-safety- … and 216 more
non-code file(s) changed (
graphify/skill-agents.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md,graphify/skill-copilot.md…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
graphify/skill-agents.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md,graphify/skill-copilot.md…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Formal verification
Could not verify: Could not verify build.
The verifier did not have enough to check build, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 9 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify build\_merge.
The verifier did not have enough to check build\_merge, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `graph_path` is annotated `str | Path | None` — outside the synthesizable primitive/collection set
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
Could not verify: Could not verify deduplicate\_entities.
The verifier did not have enough to check deduplicate\_entities, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 9 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_llm\_tiebreak.
The verifier did not have enough to check \_llm\_tiebreak, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 200 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify extract.
The verifier did not have enough to check extract, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `cache_root` is annotated `Path | None` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_extract\_parallel.
The verifier did not have enough to check \_extract\_parallel, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_apply\_symbol\_resolution\_facts.
The verifier did not have enough to check \_apply\_symbol\_resolution\_facts, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_collect\_js\_symbol\_resolution\_facts.
The verifier did not have enough to check \_collect\_js\_symbol\_resolution\_facts, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `facts` is annotated `_SymbolResolutionFacts` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_collect\_python\_symbol\_resolution\_facts.
The verifier did not have enough to check \_collect\_python\_symbol\_resolution\_facts, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_contained\_in\_package.
The verifier did not have enough to check \_contained\_in\_package, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `resolved` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_find\_js\_config.
The verifier did not have enough to check \_find\_js\_config, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `start_dir` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_find\_js\_project\_anchor.
The verifier did not have enough to check \_find\_js\_project\_anchor, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `start_dir` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_find\_workspace\_root.
The verifier did not have enough to check \_find\_workspace\_root, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `start_dir` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_go\_import\_path\_for\_file.
The verifier did not have enough to check \_go\_import\_path\_for\_file, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `source_file` is annotated `str | Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_js\_source\_path.
The verifier did not have enough to check \_js\_source\_path, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_load\_package\_imports.
The verifier did not have enough to check \_load\_package\_imports, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `start_dir` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_parse\_python\_tree.
The verifier did not have enough to check \_parse\_python\_tree, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_read\_tsconfig\_aliases.
The verifier did not have enough to check \_read\_tsconfig\_aliases, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `tsconfig` is annotated `Path` — outside the synthesizable primitive/collection set
No difference found (not proven): No behavior difference found in \_resolve\_c\_include\_path (not a proof).
The verifier ran both versions of \_resolve\_c\_include\_path on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_resolve\_cross\_file\_imports.
The verifier did not have enough to check \_resolve\_cross\_file\_imports, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: non-vacuity: domain too small (only 2 distinct inputs exercised, need 3) — 'no divergence' would be near-vacuous
Could not verify: Could not verify \_resolve\_python\_namespace\_dir.
The verifier did not have enough to check \_resolve\_python\_namespace\_dir, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `current_path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_source\_key.
The verifier did not have enough to check \_source\_key, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
No difference found (not proven): No behavior difference found in \_walk\_python\_tree (not a proof).
The verifier ran both versions of \_walk\_python\_tree on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: concolic exploration (CrossHair). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify extract\_rust.
The verifier did not have enough to check extract\_rust, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Problem
Semantic re-extraction of a doc that already has nodes in
graph.jsonis non-deterministic. Running--updatetwice on the same large projectCLAUDE.md— changed by one line between runs — returned 75 nodes on one run and 49 on the next, silently dropping an entire documented section.Nothing errored. The loss went straight into
build_mergeand would only have surfaced much later as a question the graph could no longer answer. It was caught solely because theto_jsonshrink guard (#479) refused the write.The failure mode is quiet by construction: the extraction "succeeds", the chunk is valid JSON, and the merge is happy. Only a node-count comparison against the prior build reveals it.
Fix
Adds a required three-step gate to the
--updateflow, applied to any changed doc/paper that already has nodes in the graph:graph.jsonbefore extracting.It also restates that the shrink guard is the authoritative backstop: never
force=Truepast it without diffing the old and new node sets and being able to name why each removed node is legitimately gone (file deleted, section removed, ID renamed with a verified replacement). ID churn on a re-extracted file is a legitimate shrink; a missing section is not.Reusing the prior IDs has a second benefit beyond completeness: it suppresses gratuitous ID churn on re-extraction (
claude_weather_proxy_worker→claude_workers_weather_proxy), which orphans saved queries and inflates the merge diff for no semantic gain.Relationship to #3203 / 0.9.53
0.9.53 ships a per-source, count-level guard on the
graphify extractpath.build_mergenow flags any re-extracted source whose semantic node count strictly drops, and the extract CLI arms the shrink guard on that flag: the write is refused and the file's manifest stamp is withheld, so the file re-dispatches on the next run. I re-ran my #3203 repro against 0.9.53 and it does exactly that (verification comment).So what this gate still covers is the residual, and there are two parts to it:
1. The default skill
--updatepath. Onlygraphify extractacts on the new flag (_handle_unverified_semantic_shrinkincli.py). The skill's--updateflow callsbuild_mergedirectly (tools/skillgen/fragments/references/shared/update.md) and never reads the flag. That call also passes nodedup, so #2497's identity diff is skipped on this path too (it is gatedif had_graph and not dedup:inbuild.py). On a skill--update, an under-extracted doc is at best caught afterwards, by the whole-graphto_jsoncount check (#479). That check can tell you something shrank, but not what, and it can't see the loss at all when growth elsewhere in the same run offsets it.2. The count-neutral identity shape. When a re-extract returns the same number of nodes for a file under new IDs, a per-source count check passes it by construction. Upstream's own
test_build_merge_equal_count_different_ids_not_flaggedmarks that shape as by design. On 0.9.53 it still writes: exit 0, 22 → 22 nodes, and the same 9 edges dropped as in the original repro. The mandatory-ID gate compares identity rather than count, so on the skill path that shape gets re-dispatched instead of merged.In short, 0.9.53 guards strict shrinks on
graphify extract. This gate covers the skill--updatepath, where that guard doesn't arm, and the count-neutral shape, which no count check can see.Evidence
Scope
Markdown only — no code or CLI changes.
tools/skillgen/fragments/references/shared/update.md(the gate) andtools/skillgen/fragments/core/core.md(a pointer from the--updatesection).python -m tools.skillgen+--bless.All five skillgen guards pass locally:
--check,--audit-coverage,--schema-singleton,--monolith-roundtrip,--always-on-roundtrip.Note on CI
tests/test_labeling.py::test_label_communities_batches_when_over_batch_sizemay show red. It is pre-existing and unrelated to this PR — re-confirmed on an unmodifiedv8checkout at 50556ba, where it reproduces with:Cause:
label_communitiesruns batches in a thread pool (graphify/llm.py,workers = max(1, min(max_concurrency, n_batches))), and forbackend="gemini"concurrency stays > 1. The test appends tocallsfrom inside the worker, so it records completion order but asserts dispatch order ([100, 100, 50]); under a different thread schedule it observes[100, 50, 100]. It passes when its file runs alone, which is why it usually goes unnoticed.Happy to split that into its own issue/PR if useful — I left it untouched here to keep this diff to instructions only.