extract: let a self explained semantic shrink proceed (#3412)

[ayushcodes10]

Summary

Fixes #3412. Implements the issue's own "narrower alternative": the #3203 unverified-semantic-shrink guard used to refuse any write where the overall graph got smaller, even when a single re-extracted file's own reported drop already fully accounted for the entire net loss and nothing else in the corpus was touched. On an unattended incremental pipeline that meant failing every night on an ordinary edit that legitimately yields fewer nodes, with no path to clear it short of always passing --allow-partial — which also stamps genuinely omitted/failed files as done, so operators had to choose between failing nightly or blunting the guard entirely.

Both write paths (the clustered path via to_json's force flag, and the --no-cluster path's own inline copy of the same check) now ask one extra question before refusing: does the flagged file's own (prior -> fresh) count already explain the whole net shrink? If yes, nothing outside that file went missing and the write proceeds. If the total loss exceeds what the flagged files themselves report — something else was also lost — the guard stays exactly as strict as before.

Note on scope, flagged for review: this is deliberately the issue's simplest suggested option, not the content-hash-aware one (checking whether the flagged file's content actually changed since the prior extraction). It only verifies "nothing else was lost", not "this specific file's own drop was legitimate" — so a --force re-extraction of an unchanged file that happens to be the only thing touched this run would also pass through. I judged this an acceptable, explicitly-reasoned trade-off (matching the issue's own math: "a net -7 on a 106k graph is not a partial extraction"), but it's a real behavior change to a safety guard, so flagging it clearly rather than presenting it as a purely mechanical fix — happy to add the content-hash check as a further condition if preferred.

Test plan

• Reworked the original tests/test_unverified_semantic_shrink.py end-to-end reproduction (which asserted the exact case this PR now lets through) to assert the new, intended outcome.
• Added a test confirming the guard still refuses when a loss beyond the flagged file is also present (simulates an unrelated merge bug dropping an untouched file's nodes alongside a legitimate edit).
• Added a --no-cluster parity test, since that path carries its own separate copy of the check.
• The existing --allow-partial override test is untouched and still passes.
• Full suite: python3 -m pytest -q — 5422 passed, only the pre-existing unrelated failures (test_ollama_retry_cap.py missing openai in this env, one flaky timing assertion in test_ts_import_type_arguments.py).
• python3 -m tools.skillgen --check — OK.

🤖 Generated with Claude Code

https://claude.ai/code/session_017qfdzgbA5KedGEjD1AayNh

[graphify-labs]

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Relaxes the #3203 unverified-semantic-shrink guard so a write proceeds without --allow-partial when the flagged files' own prior→fresh counts fully account for the graph's net node loss, via _shrink_is_self_explained; any shortfall those numbers don't cover (an unrelated failure or a second unflagged shrink) leaves the guard armed and still exits 1. Wires this into both the raw-merge and build_merge paths of dispatch_command, which still log the shrink they wave through and stamp the manifest for the accepted content.

No blocking issues surfaced. 6 lower-confidence candidates did not survive cross-model review.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 311 functions depend on the 99 functions this change touches.

Health — this change adds coupling hotspots:

• new: dispatch_command() — 2 callers, 125 callees
• new: _stale_graph_sources() — 7 callers, 6 callees
• new: _run_hook_guard() — 4 callers, 8 callees
• new: test_poisoned_manifest_is_healed() — 0 callers, 6 callees

Verification — 311 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 255 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

24 of 268 test file(s) selected (9%) via static blast radius.

• tests/test_affected_cli.py — impact
• tests/test_agents_platform.py — impact
• tests/test_codebuddy.py — impact
• tests/test_devin.py — impact
• tests/test_explain_cli.py — impact
• tests/test_extract_cli.py — impact
• tests/test_global_add_tag_inference.py — impact
• tests/test_god_nodes_cli.py — impact
• tests/test_hollow_chunks_arm_shrink_guard.py — impact
• tests/test_hook_guard_token_match.py — impact
• tests/test_hook_out_of_project_paths.py — impact
• tests/test_hook_strict.py — impact
• tests/test_incomplete_build_guard.py — impact
• tests/test_install.py — impact
• tests/test_install_references.py — impact
• tests/test_merge_chunks_validation.py — impact
• tests/test_multigraph_diagnostics.py — impact
• tests/test_no_dedup_flag.py — impact
• tests/test_partial_cache.py — impact
• tests/test_path_cli.py — impact
• tests/test_query_cli.py — impact
• tests/test_query_induced_edges.py — impact
• tests/test_stale_prune.py — impact
• tests/test_unverified_semantic_shrink.py — impact, changed-test

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Formal verification

Could not verify: Could not verify dispatch\_command.

The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)

· 4 more finding(s) on lines outside this diff (see the check run).