feat(audit): ENF audits whether a gate BINDS, not only whether it exists - #52
feat(audit): ENF audits whether a gate BINDS, not only whether it exists#52lapc506 wants to merge 1 commit into
Conversation
`findHookCoverageGaps` answers "does a hook exist for this rule?" — presence,
and the easy half. A gate can be present, correctly named, wired into a config,
reviewed and merged, and assert nothing at all.
Adds `findHookEfficacyGaps`, which answers the other question, against the seven
shapes of a vacuous gate. Shapes 1 and 3 are about the SET of hooks and stay in
the coverage verifier; shapes 2 and 4-7 are about EACH hook and are new:
2 looks at the adjacent property asserts-adjacent-property, unresolved-symbol
4 correct and binds nothing non-blocking-exit-code, vacuous-on-empty-input,
no-block-assertion
5 correct, blocking, and too late enforced-too-late
6 registered where nothing reads registered-where-nothing-reads
7 the incentive is inverted inverted-incentive
Every shape came from a live repository. Three worth naming:
- Exit code. Only exit 2 blocks a PreToolUse call; exit 1 is reported and then
proceeded past. A repo found twelve hooks denying on exit 1, one of them
guarding writes against production. Each was correct, reviewed, and inert.
- Registry. A hook wired into `hooks.json` when the harness reads
`settings.json` never runs, and its presence in *a* registry reads as covered.
- Shape 7 is new and no coverage audit can see it, because the gate exists,
looks at the right thing, and binds. A doc validator escalated to ERROR only
when a record was `accepted`, degrading `draft`/`review` to a warning — so
complying cost more than not complying, and readers were trained to stay
lenient. The gate was not weak. It pointed the wrong way.
Two shapes are not decidable from a file listing and the detector says so.
Shape 2 is checked through its correlate — a suite asserting only the exit code
pins nothing about WHAT the gate said — and that gap is labelled `proxy`.
Shape 7 needs a reader, because which branch is cheaper depends on the repo. An
observation left `undefined` produces no gap: the verifier reports what was
looked up, never what was assumed.
Refusing to encode the un-automatable is the point. A verifier that guessed
shape 7 would be a gate that looks at the adjacent property — shape 2, in the
tool built to find shape 2.
The four efficacy `high`s share one property the `medium`s do not: each produces
a gate that looks like coverage from every listing. A `medium` gap is a gate
that is weak; a `high` gap is a gate that is misleading, and the second is worse
because it stops anyone from looking again.
20 new cases; the 6 existing coverage cases are untouched and still pass.
`vitest run src/audit` — 80 passed across 13 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💬 Review Comments
Comments — 0 blockers, 12 P2. Confidence: 1.00/5.00.
Walkthrough
Governance Warning
main branch directly. Per standard GitFlow practices, feature branches should target develop before merging to main. This warning is informational only and does not block the review verdict.
Review Walkthrough
This PR introduces the enforcement hook efficacy verifier (findHookEfficacyGaps) to identify "vacuous gates" that appear covered but fail to bind. It implements checks for five of the seven vacuous-gate shapes: registration mismatch, non-blocking exit codes, empty input bypass, lack of test assertions, unresolved symbols, delayed CI enforcement, and inverted incentives.
Scope of Review
I reviewed the newly added verifier file (src/audit/verifiers/enforcement-hooks-efficacy.ts), its corresponding unit test suite (src/audit/verifiers/enforcement-hooks-efficacy.test.ts), and the updated markdown documentation detailing the seven shapes and stage-2 observations (references/detectors/enforcement-hooks.md and commands/audit-enforcement-hooks.md).
Safety Rationale
This PR is safe to merge because the new verifier code is self-contained, does not modify any existing production code or execution paths, and passes all its newly introduced unit tests.
Verdict
Comment — 0 blockers, 12 P2.
🟡 P2 — Major
src/audit/verifiers/enforcement-hooks-efficacy.ts:67— 🟡 P2 (major) — Makeresolvesoptional to prevent false-positive gap findings when symbol resolution is not observed/unknown. Ifresolvesisundefined, it should produce no gaps under the 'An observation left undefined produces no gap' contract.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:87— 🟡 P2 (major) — MakeregisteredInoptional to allow omitting registration info without triggering a false gap or causing runtime type errors when parsing dynamic or incomplete observation data.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:90— 🟡 P2 (major) — MakeintendsToBlockoptional to correctly handle the case where it is unknown or not observed.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:101— 🟡 P2 (major) — MaketestAssertsMessageoptional so that an unobserved or unknown message test assertion state does not trigger a false-positiveasserts-adjacent-propertygap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:104— 🟡 P2 (major) — MaketestAssertsBlockoptional so that an unobserved/unknown block test assertion state does not trigger a false-positiveno-block-assertiongap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:111— 🟡 P2 (major) — MakefailsOnEmptyInputoptional so that an unobserved/unknown empty-input failure behavior does not trigger a false-positivevacuous-on-empty-inputgap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:153— 🟡 P2 (major) — Guard the registry file check to prevent a runtime TypeError ifh.registeredInis undefined, and ensure it produces no gaps when unobserved.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:167— 🟡 P2 (major) — Use an explicit strict boolean equality check forh.intendsToBlockso thatundefinedvalues do not inadvertently pass.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:184— 🟡 P2 (major) — Use an explicit strict check forfailsOnEmptyInput === falseso thatundefined(unobserved) values are ignored and do not trigger a false gap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:195— 🟡 P2 (major) — Use explicit strict checks forintendsToBlock === trueandtestAssertsBlock === falseto properly respectundefined(unobserved) inputs.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:206— 🟡 P2 (major) — Use an explicit strict check fortestAssertsMessage === falseso thatundefined(unobserved) values do not trigger a false-positive gap.
[pass 1]
src/audit/verifiers/enforcement-hooks-efficacy.ts:218— 🟡 P2 (major) — Use an explicit strict check fors.resolves === falseto ensure an unobserved resolution status does not trigger a false gap.
[pass 1]
Total findings: 12 compliance (12 total)
| export interface NamedSymbol { | ||
| symbol: string; | ||
| /** Did it resolve against the system that owns it, by a command that was run? */ | ||
| resolves: boolean; |
There was a problem hiding this comment.
🟡 P2 (major) — Make resolves optional to prevent false-positive gap findings when symbol resolution is not observed/unknown. If resolves is undefined, it should produce no gaps under the 'An observation left undefined produces no gap' contract.
[pass 1]
| */ | ||
| registeredIn: string[]; | ||
|
|
||
| /** Does this gate intend to DENY, or only to warn? A warn-mode gate is not vacuous for being non-blocking. */ |
There was a problem hiding this comment.
🟡 P2 (major) — Make registeredIn optional to allow omitting registration info without triggering a false gap or causing runtime type errors when parsing dynamic or incomplete observation data.
[pass 1]
| /** Does this gate intend to DENY, or only to warn? A warn-mode gate is not vacuous for being non-blocking. */ | ||
| intendsToBlock: boolean; | ||
|
|
||
| /** |
There was a problem hiding this comment.
🟡 P2 (major) — Make intendsToBlock optional to correctly handle the case where it is unknown or not observed.
[pass 1]
| /** Do its tests assert the message TEXT, or only the exit code? Proxy for shape 2. */ | ||
| testAssertsMessage: boolean; | ||
|
|
||
| /** Do its tests assert that a violating input is actually blocked? */ |
There was a problem hiding this comment.
🟡 P2 (major) — Make testAssertsMessage optional so that an unobserved or unknown message test assertion state does not trigger a false-positive asserts-adjacent-property gap.
[pass 1]
| /** Do its tests assert that a violating input is actually blocked? */ | ||
| testAssertsBlock: boolean; | ||
|
|
||
| /** |
There was a problem hiding this comment.
🟡 P2 (major) — Make testAssertsBlock optional so that an unobserved/unknown block test assertion state does not trigger a false-positive no-block-assertion gap.
[pass 1]
| gaps.push({ | ||
| code: `registered-where-nothing-reads:${h.name}`, | ||
| shape: 6, | ||
| confidence: "direct", |
There was a problem hiding this comment.
🟡 P2 (major) — Use an explicit strict boolean equality check for h.intendsToBlock so that undefined values do not inadvertently pass.
[pass 1]
| ) { | ||
| gaps.push({ | ||
| code: `non-blocking-exit-code:${h.name}`, | ||
| shape: 4, |
There was a problem hiding this comment.
🟡 P2 (major) — Use an explicit strict check for failsOnEmptyInput === false so that undefined (unobserved) values are ignored and do not trigger a false gap.
[pass 1]
|
|
||
| // --- Shape 4b: passes vacuously when its own input is empty ----------- | ||
| if (!h.failsOnEmptyInput) { | ||
| gaps.push({ |
There was a problem hiding this comment.
🟡 P2 (major) — Use explicit strict checks for intendsToBlock === true and testAssertsBlock === false to properly respect undefined (unobserved) inputs.
[pass 1]
| } | ||
|
|
||
| // --- Shape 4c: no test proves it ever blocks -------------------------- | ||
| if (h.intendsToBlock && !h.testAssertsBlock) { |
There was a problem hiding this comment.
🟡 P2 (major) — Use an explicit strict check for testAssertsMessage === false so that undefined (unobserved) values do not trigger a false-positive gap.
[pass 1]
| } | ||
|
|
||
| // --- Shape 2: asserts the adjacent property (proxy) ------------------- | ||
| if (!h.testAssertsMessage) { |
There was a problem hiding this comment.
🟡 P2 (major) — Use an explicit strict check for s.resolves === false to ensure an unobserved resolution status does not trigger a false gap.
[pass 1]
Measured on origin/main @ ee0ba47: the three version files all read 1.36.0, and `gh pr diff 55 | grep -E '^\+.*"version"'` shows #55 already bumps to 1.37.0 in package.json, plugin.json and marketplace.json. Taking 1.37.0 here collides with it in all three. The same probe over #52, #46, #44 and #41 returns nothing, so 1.38.0 is free. Skipping a version costs nothing; colliding does. Minor and not major, decided by reading. A skill auto-activates on its `description` rather than being invoked by name like a command, so a renamed `name:` changes no call site. The reference search returns nothing: grep -rniI "rebase.advisor" . --exclude-dir=node_modules \ --exclude-dir=.git --exclude=CHANGELOG.md -> no hits That negative is real and not a broken search -- the same grep for `spike-recommend` returns 10 files, so cross-references of this shape are found when they exist. One surface does break and is stated rather than folded in: a user who typed `/make-no-mistakes:rebase-advisor` explicitly (README:137 documents that skills can be invoked that way) now gets an unknown skill. It fails loudly, the replacement is one row away in the same table, and the installer prunes the old file rather than leaving both live. The `[1.37.0]:` CHANGELOG reference slot is deliberately left for #55 to fill. Suite: 337/337 hooks, 60/60 vitest. Created by Claude Code on behalf of @lapc506
…(v1.38.0) (#56) * feat(skills): sync-advisor — measure the drift before naming the fix (v1.37.0) Renames `rebase-advisor` to `sync-advisor` and turns a blind router into an advisor that measures. The old skill was 43 lines and measured nothing. Its step 1 was "Confirm the user wants a full team sync" -- a question back to the user about something three git commands answer. Its description then over-routed: it triggered on "align with develop" and "branches are behind", both of which are `git pull`, and sent them to /make-no-mistakes:rebase, which stashes every worktree, rebases every local branch and auto-merges PRs. Between `git pull` and that the toolkit offered nothing, and nothing read-only at all. Six read-only predicates now run before anything is named: distance, fast-forward possible, dirty tree split by stage, untracked files the ref already tracks, worktrees behind, and branches with unpushed commits. The fifth is the threshold between a plain pull and the team command. The fourth is the one nothing else reports, and it was verified on a throwaway pair of repos rather than asserted. An untracked local file at a path the ref tracks aborts the pull outright while being invisible everywhere else: `git status` shows a plain `?? newfile.txt`, distance reports a clean 0 ahead 1 behind, and `merge-base --is-ancestor` says a fast-forward is possible. The pull then exits 1 with "The following untracked working tree files would be overwritten by merge ... Please move or remove them before you merge" -- the message names the user's own file and offers deletion as the remedy, which is the one irreversible move available. The skill reports these by name and recommends copying them out of the repo, never deleting them. Both controls were run: the pipeline printed nothing before the collision existed and named the file after. It never acts. Every fix is printed for the user to run. The single write is `git fetch origin --quiet`, which touches remote-tracking refs and nothing else, and the skill says so out loud -- without it every measurement is taken against a stale origin/<base> and reports a drift that stopped being true days ago, which is the failure this skill exists to catch. Adds `syncAdvisor.governedPaths` to make-no-mistakes.config.json: the paths whose changes get reported by name, turning "you are 12 behind" into "three hooks changed, two of them fix defects you may be looking at right now". No default -- with the key unset the skill drops the consequence line rather than falling back to a built-in list, which would be wrong in every repo but the one it was copied from and would read as measured. commands/rebase.md is untouched and stays a real destination. What changed is who decides when it applies. Origin (2026-07-31, as reported): a developer filed two bug reports against a hook with clean reproductions. One was a real defect; the other described behaviour fixed days earlier against a stale checkout, and nothing in the report separated them. Suite: 337/337 hooks, 60/60 vitest. Created by Claude Code on behalf of @lapc506 * chore(release): take 1.38.0 instead of 1.37.0 — #55 owns 1.37.0 Measured on origin/main @ ee0ba47: the three version files all read 1.36.0, and `gh pr diff 55 | grep -E '^\+.*"version"'` shows #55 already bumps to 1.37.0 in package.json, plugin.json and marketplace.json. Taking 1.37.0 here collides with it in all three. The same probe over #52, #46, #44 and #41 returns nothing, so 1.38.0 is free. Skipping a version costs nothing; colliding does. Minor and not major, decided by reading. A skill auto-activates on its `description` rather than being invoked by name like a command, so a renamed `name:` changes no call site. The reference search returns nothing: grep -rniI "rebase.advisor" . --exclude-dir=node_modules \ --exclude-dir=.git --exclude=CHANGELOG.md -> no hits That negative is real and not a broken search -- the same grep for `spike-recommend` returns 10 files, so cross-references of this shape are found when they exist. One surface does break and is stated rather than folded in: a user who typed `/make-no-mistakes:rebase-advisor` explicitly (README:137 documents that skills can be invoked that way) now gets an unknown skill. It fails loudly, the replacement is one row away in the same table, and the installer prunes the old file rather than leaving both live. The `[1.37.0]:` CHANGELOG reference slot is deliberately left for #55 to fill. Suite: 337/337 hooks, 60/60 vitest. Created by Claude Code on behalf of @lapc506 * fix(sync-advisor): make the collision predicate correct from any subdirectory Reviewer P3 on `ea8c46e` claimed `git ls-tree` "always outputs repository-relative paths" while `git ls-files` is prefix-relative, so the two would fail to match from a subdirectory, and proposed `--full-name` on `ls-files`. Measured: the premise is false and the proposed fix introduces the bug it claims to prevent. From `sub/`, `ls-tree -r --name-only` prints `newfile.txt`, not `sub/newfile.txt` -- it strips the prefix exactly like `ls-files` does, so the original command matched fine and found the collision. Adding `--full-name` alone then makes `ls-files` emit `sub/newfile.txt` against `ls-tree`'s `newfile.txt`, they stop matching, and `comm -12` returns empty -- a clean bill of health for a tree about to abort the pull. But it pointed at a real weakness of a different kind. From a subdirectory both commands are SCOPED to that subtree, so a collision at the repo root is not seen at all. That is scope, not format, and no combination of format flags fixes it: `--full-name` changes how a path prints, never which paths are considered. The pathspec does. Shipped: `--full-name -- :/` on ls-files, `--full-tree` on ls-tree. Four cases run, with the command extracted verbatim from SKILL.md so the test cannot drift from the doc -- negative control from the root and from `sub/` (both empty), positive from the root and from `sub/` with one collision in each location (both list `rootfile.txt` and `sub/newfile.txt`). Suite: 340/340 hooks, 60/60 vitest. Created by Claude Code on behalf of @lapc506 * fix(sync-advisor): resolve the base ref correctly, and in a bare form Reviewer P2 on `00e2c08`, and it is right. Step 0 returned values carrying the remote prefix while every predicate interpolates `origin/$BASE`, so the run died on `origin/origin/develop`: $ git rev-list --left-right --count "HEAD...origin/origin/main" fatal: ambiguous argument … unknown revision exit=128 Fixed by normalising unconditionally (`${BASE#refs/remotes/}`, `${BASE#origin/}`) whichever branch of the resolution produced the value. Verified both branches with a control that must fail: where `origin/HEAD` IS set it returns `origin/main`, normalises to `main`, and `origin/main` resolves; the un-normalised form exits 128. The P2 understated it. Step 0's first command resolved the base from `@{upstream}`, which on a feature branch is that branch's OWN remote copy -- here `origin/andres/sync-advisor`. That answers "am I pushed?", which is predicate 6's question, and would report 0 behind on a branch far behind the real base. Stripping a prefix would have left it pointing at the wrong ref, so the command is gone from base resolution rather than patched. Also measured while there: `git symbolic-ref --short refs/remotes/origin/HEAD` fails outright in this repo (`not a symbolic ref`), so it is documented as a fall-through into the develop/main/master/trunk probe rather than as a step that is expected to succeed. P4 (`--abbrev-ref` and `--symbolic-full-name` redundant) is also correct -- both forms return `origin/andres/sync-advisor` here -- and is moot: that command no longer appears. Suite: 340/340 hooks, 60/60 vitest. Created by Claude Code on behalf of @lapc506
Problem
findHookCoverageGapsanswers "does a hook exist for this rule?" — presence,and the easy half. A gate can be present, correctly named, wired into a config,
reviewed and merged, and assert nothing at all.
The ENF family calls itself "the meta-audit that closes the loop", but the loop
it closes is coverage. A repo can score zero coverage gaps and have an
enforcement layer that binds nothing.
What this adds
findHookEfficacyGaps— the other question, against the seven shapes of avacuous gate. Shapes 1 and 3 are about the SET of hooks and stay in the
coverage verifier. Shapes 2 and 4-7 are about EACH hook and are new:
asserts-adjacent-property,unresolved-symbolnon-blocking-exit-code,vacuous-on-empty-input,no-block-assertionenforced-too-lateregistered-where-nothing-readsinverted-incentiveEvery shape came from a live repository
Shape 4 — the exit code. Only exit 2 blocks a PreToolUse call; exit 1 is
an error the harness reports and then proceeds past. A repo found twelve
hooks written to deny on
exit 1, one of them guarding writes againstproduction. Each was correct, reviewed, and inert.
Shape 6 — the registry. A hook wired into
.claude/hooks/hooks.jsonwhen theharness reads
.claude/settings.jsonnever runs. Its presence in a registryfile reads as covered, which is why it survives review.
Shape 7 — new, and invisible to any coverage audit, because the gate exists,
looks at the right thing, and binds. A doc validator escalated to ERROR only
when a record's status was
accepted, degradingdraftandreviewto awarning. Complying cost more than not complying, so records stayed unsettled and
every reader who noticed was trained to stay in the lenient branch. The gate was
not weak. It pointed the wrong way.
Two shapes are not automatable, and the detector says so
Shape 2 — "does the gate assert the proposition it claims?" — is not decidable
from a file listing. What is decidable is its strongest correlate: a suite that
checks only the exit code pins nothing about WHAT the gate said, so one that
silently stopped emitting its message still passes. That gap is stamped
confidence: "proxy"and its detail starts withPROXY.Shape 7 needs a reader, because which branch is cheaper depends on what the
branches cost in that repo. The verifier records the answer; it does not compute
it.
An observation left
undefinedproduces no gap. The verifier reports whatwas looked up, never what was assumed.
Severity
The four efficacy
highs share one property themediums do not: each producesa gate that looks like coverage from every listing. An inert hook, a denial
on the wrong exit code, a phantom symbol and an inverted incentive all survive
review, appear in the inventory, and report green.
A
mediumgap is a gate that is weak. Ahighgap is a gate that ismisleading, and the second is worse because it stops anyone from looking again.
Verification
The new suite asserts the behaviours that make the verifier honest, not just
that it returns an array: a warn-mode gate is not flagged for being
non-blocking; an unknown harness registry file produces no shape-6 finding
rather than a guess; an unread severity branch produces no shape-7 finding;
and one hook can instance several shapes at once.
Files
src/audit/verifiers/enforcement-hooks-efficacy.tssrc/audit/verifiers/enforcement-hooks-efficacy.test.tsreferences/detectors/enforcement-hooks.mdcommands/audit-enforcement-hooks.mdfindHookCoverageGapsand its six tests are untouched — coverage and efficacyare different questions and both run.
Created by Claude Code on behalf of @kvttvrsis