Skip to content

feat(audit): ENF audits whether a gate BINDS, not only whether it exists - #52

Open
lapc506 wants to merge 1 commit into
mainfrom
feat/enf-seven-shapes
Open

feat(audit): ENF audits whether a gate BINDS, not only whether it exists#52
lapc506 wants to merge 1 commit into
mainfrom
feat/enf-seven-shapes

Conversation

@lapc506

@lapc506 lapc506 commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Problem

findHookCoverageGaps answers "does a hook exist for this rule?" — presence,
and the easy half. A gate can be present, correctly named, wired into a config,
reviewed and merged, and assert nothing at all.

The ENF family calls itself "the meta-audit that closes the loop", but the loop
it closes is coverage. A repo can score zero coverage gaps and have an
enforcement layer that binds nothing.

What this adds

findHookEfficacyGaps — the other question, against the seven shapes of a
vacuous gate
. Shapes 1 and 3 are about the SET of hooks and stay in the
coverage verifier. Shapes 2 and 4-7 are about EACH hook and are new:

# Shape Gap codes
2 looks at the adjacent property asserts-adjacent-property, unresolved-symbol
4 correct and binds nothing non-blocking-exit-code, vacuous-on-empty-input, no-block-assertion
5 correct, blocking, and too late enforced-too-late
6 registered where nothing reads registered-where-nothing-reads
7 the incentive is inverted inverted-incentive

Every shape came from a live repository

Shape 4 — the exit code. Only exit 2 blocks a PreToolUse call; exit 1 is
an error the harness reports and then proceeds past. A repo found twelve
hooks written to deny on exit 1, one of them guarding writes against
production. Each was correct, reviewed, and inert.

Shape 6 — the registry. A hook wired into .claude/hooks/hooks.json when the
harness reads .claude/settings.json never runs. Its presence in a registry
file reads as covered, which is why it survives review.

Shape 7 — new, and invisible to any coverage audit, because the gate exists,
looks at the right thing, and binds. A doc validator escalated to ERROR only
when a record's status was accepted
, degrading draft and review to a
warning. Complying cost more than not complying, so records stayed unsettled and
every reader who noticed was trained to stay in the lenient branch. The gate was
not weak. It pointed the wrong way.

Two shapes are not automatable, and the detector says so

Shape 2 — "does the gate assert the proposition it claims?" — is not decidable
from a file listing. What is decidable is its strongest correlate: a suite that
checks only the exit code pins nothing about WHAT the gate said, so one that
silently stopped emitting its message still passes. That gap is stamped
confidence: "proxy" and its detail starts with PROXY.

Shape 7 needs a reader, because which branch is cheaper depends on what the
branches cost in that repo. The verifier records the answer; it does not compute
it.

An observation left undefined produces no gap. The verifier reports what
was looked up, never what was assumed.

Refusing to encode the un-automatable is the point. A verifier that guessed
shape 7 would be a gate that looks at the adjacent property — shape 2, in the
tool built to find shape 2.

Severity

The four efficacy highs share one property the mediums do not: each produces
a gate that looks like coverage from every listing. An inert hook, a denial
on the wrong exit code, a phantom symbol and an inverted incentive all survive
review, appear in the inventory, and report green.

A medium gap is a gate that is weak. A high gap is a gate that is
misleading, and the second is worse because it stops anyone from looking again.

Verification

vitest run src/audit    80 passed, 13 files
  enforcement-hooks.test.ts           6 passed   (existing, untouched)
  enforcement-hooks-efficacy.test.ts  20 passed  (new)

The new suite asserts the behaviours that make the verifier honest, not just
that it returns an array: a warn-mode gate is not flagged for being
non-blocking; an unknown harness registry file produces no shape-6 finding
rather than a guess; an unread severity branch produces no shape-7 finding;
and one hook can instance several shapes at once.

Files

File Change
src/audit/verifiers/enforcement-hooks-efficacy.ts new — the verifier
src/audit/verifiers/enforcement-hooks-efficacy.test.ts new — 20 cases
references/detectors/enforcement-hooks.md +118 lines: the seven shapes, the per-hook Stage-2 observations, the second verify call, the new severity rows
commands/audit-enforcement-hooks.md describes both verifiers

findHookCoverageGaps and its six tests are untouched — coverage and efficacy
are different questions and both run.

Created by Claude Code on behalf of @kvttvrsis

`findHookCoverageGaps` answers "does a hook exist for this rule?" — presence,
and the easy half. A gate can be present, correctly named, wired into a config,
reviewed and merged, and assert nothing at all.

Adds `findHookEfficacyGaps`, which answers the other question, against the seven
shapes of a vacuous gate. Shapes 1 and 3 are about the SET of hooks and stay in
the coverage verifier; shapes 2 and 4-7 are about EACH hook and are new:

  2  looks at the adjacent property     asserts-adjacent-property, unresolved-symbol
  4  correct and binds nothing          non-blocking-exit-code, vacuous-on-empty-input,
                                        no-block-assertion
  5  correct, blocking, and too late    enforced-too-late
  6  registered where nothing reads     registered-where-nothing-reads
  7  the incentive is inverted          inverted-incentive

Every shape came from a live repository. Three worth naming:

- Exit code. Only exit 2 blocks a PreToolUse call; exit 1 is reported and then
  proceeded past. A repo found twelve hooks denying on exit 1, one of them
  guarding writes against production. Each was correct, reviewed, and inert.
- Registry. A hook wired into `hooks.json` when the harness reads
  `settings.json` never runs, and its presence in *a* registry reads as covered.
- Shape 7 is new and no coverage audit can see it, because the gate exists,
  looks at the right thing, and binds. A doc validator escalated to ERROR only
  when a record was `accepted`, degrading `draft`/`review` to a warning — so
  complying cost more than not complying, and readers were trained to stay
  lenient. The gate was not weak. It pointed the wrong way.

Two shapes are not decidable from a file listing and the detector says so.
Shape 2 is checked through its correlate — a suite asserting only the exit code
pins nothing about WHAT the gate said — and that gap is labelled `proxy`.
Shape 7 needs a reader, because which branch is cheaper depends on the repo. An
observation left `undefined` produces no gap: the verifier reports what was
looked up, never what was assumed.

Refusing to encode the un-automatable is the point. A verifier that guessed
shape 7 would be a gate that looks at the adjacent property — shape 2, in the
tool built to find shape 2.

The four efficacy `high`s share one property the `medium`s do not: each produces
a gate that looks like coverage from every listing. A `medium` gap is a gate
that is weak; a `high` gap is a gate that is misleading, and the second is worse
because it stops anyone from looking again.

20 new cases; the 6 existing coverage cases are untouched and still pass.
`vitest run src/audit` — 80 passed across 13 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@dojo-code-reviewer dojo-code-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 Review Comments

Comments — 0 blockers, 12 P2. Confidence: 1.00/5.00.

Walkthrough

Governance Warning

⚠️ Governance Alert: This PR targets the main branch directly. Per standard GitFlow practices, feature branches should target develop before merging to main. This warning is informational only and does not block the review verdict.

Review Walkthrough

This PR introduces the enforcement hook efficacy verifier (findHookEfficacyGaps) to identify "vacuous gates" that appear covered but fail to bind. It implements checks for five of the seven vacuous-gate shapes: registration mismatch, non-blocking exit codes, empty input bypass, lack of test assertions, unresolved symbols, delayed CI enforcement, and inverted incentives.

Scope of Review

I reviewed the newly added verifier file (src/audit/verifiers/enforcement-hooks-efficacy.ts), its corresponding unit test suite (src/audit/verifiers/enforcement-hooks-efficacy.test.ts), and the updated markdown documentation detailing the seven shapes and stage-2 observations (references/detectors/enforcement-hooks.md and commands/audit-enforcement-hooks.md).

Safety Rationale

This PR is safe to merge because the new verifier code is self-contained, does not modify any existing production code or execution paths, and passes all its newly introduced unit tests.

Verdict

Comment — 0 blockers, 12 P2.

🟡 P2 — Major

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:67 — 🟡 P2 (major) — Make resolves optional to prevent false-positive gap findings when symbol resolution is not observed/unknown. If resolves is undefined, it should produce no gaps under the 'An observation left undefined produces no gap' contract.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:87 — 🟡 P2 (major) — Make registeredIn optional to allow omitting registration info without triggering a false gap or causing runtime type errors when parsing dynamic or incomplete observation data.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:90 — 🟡 P2 (major) — Make intendsToBlock optional to correctly handle the case where it is unknown or not observed.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:101 — 🟡 P2 (major) — Make testAssertsMessage optional so that an unobserved or unknown message test assertion state does not trigger a false-positive asserts-adjacent-property gap.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:104 — 🟡 P2 (major) — Make testAssertsBlock optional so that an unobserved/unknown block test assertion state does not trigger a false-positive no-block-assertion gap.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:111 — 🟡 P2 (major) — Make failsOnEmptyInput optional so that an unobserved/unknown empty-input failure behavior does not trigger a false-positive vacuous-on-empty-input gap.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:153 — 🟡 P2 (major) — Guard the registry file check to prevent a runtime TypeError if h.registeredIn is undefined, and ensure it produces no gaps when unobserved.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:167 — 🟡 P2 (major) — Use an explicit strict boolean equality check for h.intendsToBlock so that undefined values do not inadvertently pass.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:184 — 🟡 P2 (major) — Use an explicit strict check for failsOnEmptyInput === false so that undefined (unobserved) values are ignored and do not trigger a false gap.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:195 — 🟡 P2 (major) — Use explicit strict checks for intendsToBlock === true and testAssertsBlock === false to properly respect undefined (unobserved) inputs.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:206 — 🟡 P2 (major) — Use an explicit strict check for testAssertsMessage === false so that undefined (unobserved) values do not trigger a false-positive gap.

[pass 1]

  • src/audit/verifiers/enforcement-hooks-efficacy.ts:218 — 🟡 P2 (major) — Use an explicit strict check for s.resolves === false to ensure an unobserved resolution status does not trigger a false gap.

[pass 1]


Total findings: 12 compliance (12 total)

export interface NamedSymbol {
symbol: string;
/** Did it resolve against the system that owns it, by a command that was run? */
resolves: boolean;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Make resolves optional to prevent false-positive gap findings when symbol resolution is not observed/unknown. If resolves is undefined, it should produce no gaps under the 'An observation left undefined produces no gap' contract.

[pass 1]

*/
registeredIn: string[];

/** Does this gate intend to DENY, or only to warn? A warn-mode gate is not vacuous for being non-blocking. */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Make registeredIn optional to allow omitting registration info without triggering a false gap or causing runtime type errors when parsing dynamic or incomplete observation data.

[pass 1]

/** Does this gate intend to DENY, or only to warn? A warn-mode gate is not vacuous for being non-blocking. */
intendsToBlock: boolean;

/**

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Make intendsToBlock optional to correctly handle the case where it is unknown or not observed.

[pass 1]

/** Do its tests assert the message TEXT, or only the exit code? Proxy for shape 2. */
testAssertsMessage: boolean;

/** Do its tests assert that a violating input is actually blocked? */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Make testAssertsMessage optional so that an unobserved or unknown message test assertion state does not trigger a false-positive asserts-adjacent-property gap.

[pass 1]

/** Do its tests assert that a violating input is actually blocked? */
testAssertsBlock: boolean;

/**

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Make testAssertsBlock optional so that an unobserved/unknown block test assertion state does not trigger a false-positive no-block-assertion gap.

[pass 1]

gaps.push({
code: `registered-where-nothing-reads:${h.name}`,
shape: 6,
confidence: "direct",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Use an explicit strict boolean equality check for h.intendsToBlock so that undefined values do not inadvertently pass.

[pass 1]

) {
gaps.push({
code: `non-blocking-exit-code:${h.name}`,
shape: 4,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Use an explicit strict check for failsOnEmptyInput === false so that undefined (unobserved) values are ignored and do not trigger a false gap.

[pass 1]


// --- Shape 4b: passes vacuously when its own input is empty -----------
if (!h.failsOnEmptyInput) {
gaps.push({

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Use explicit strict checks for intendsToBlock === true and testAssertsBlock === false to properly respect undefined (unobserved) inputs.

[pass 1]

}

// --- Shape 4c: no test proves it ever blocks --------------------------
if (h.intendsToBlock && !h.testAssertsBlock) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Use an explicit strict check for testAssertsMessage === false so that undefined (unobserved) values do not trigger a false-positive gap.

[pass 1]

}

// --- Shape 2: asserts the adjacent property (proxy) -------------------
if (!h.testAssertsMessage) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 (major) — Use an explicit strict check for s.resolves === false to ensure an unobserved resolution status does not trigger a false gap.

[pass 1]

lapc506 added a commit that referenced this pull request Jul 31, 2026
Measured on origin/main @ ee0ba47: the three version files all read 1.36.0,
and `gh pr diff 55 | grep -E '^\+.*"version"'` shows #55 already bumps to
1.37.0 in package.json, plugin.json and marketplace.json. Taking 1.37.0 here
collides with it in all three. The same probe over #52, #46, #44 and #41
returns nothing, so 1.38.0 is free. Skipping a version costs nothing;
colliding does.

Minor and not major, decided by reading. A skill auto-activates on its
`description` rather than being invoked by name like a command, so a renamed
`name:` changes no call site. The reference search returns nothing:

  grep -rniI "rebase.advisor" . --exclude-dir=node_modules \
    --exclude-dir=.git --exclude=CHANGELOG.md   ->  no hits

That negative is real and not a broken search -- the same grep for
`spike-recommend` returns 10 files, so cross-references of this shape are
found when they exist.

One surface does break and is stated rather than folded in: a user who typed
`/make-no-mistakes:rebase-advisor` explicitly (README:137 documents that
skills can be invoked that way) now gets an unknown skill. It fails loudly,
the replacement is one row away in the same table, and the installer prunes
the old file rather than leaving both live.

The `[1.37.0]:` CHANGELOG reference slot is deliberately left for #55 to fill.

Suite: 337/337 hooks, 60/60 vitest.

Created by Claude Code on behalf of @lapc506
lapc506 added a commit that referenced this pull request Jul 31, 2026
…(v1.38.0) (#56)

* feat(skills): sync-advisor — measure the drift before naming the fix (v1.37.0)

Renames `rebase-advisor` to `sync-advisor` and turns a blind router into an
advisor that measures.

The old skill was 43 lines and measured nothing. Its step 1 was "Confirm the
user wants a full team sync" -- a question back to the user about something
three git commands answer. Its description then over-routed: it triggered on
"align with develop" and "branches are behind", both of which are `git pull`,
and sent them to /make-no-mistakes:rebase, which stashes every worktree,
rebases every local branch and auto-merges PRs. Between `git pull` and that
the toolkit offered nothing, and nothing read-only at all.

Six read-only predicates now run before anything is named: distance,
fast-forward possible, dirty tree split by stage, untracked files the ref
already tracks, worktrees behind, and branches with unpushed commits. The
fifth is the threshold between a plain pull and the team command.

The fourth is the one nothing else reports, and it was verified on a throwaway
pair of repos rather than asserted. An untracked local file at a path the ref
tracks aborts the pull outright while being invisible everywhere else:
`git status` shows a plain `?? newfile.txt`, distance reports a clean 0 ahead
1 behind, and `merge-base --is-ancestor` says a fast-forward is possible. The
pull then exits 1 with "The following untracked working tree files would be
overwritten by merge ... Please move or remove them before you merge" -- the
message names the user's own file and offers deletion as the remedy, which is
the one irreversible move available. The skill reports these by name and
recommends copying them out of the repo, never deleting them. Both controls
were run: the pipeline printed nothing before the collision existed and named
the file after.

It never acts. Every fix is printed for the user to run. The single write is
`git fetch origin --quiet`, which touches remote-tracking refs and nothing
else, and the skill says so out loud -- without it every measurement is taken
against a stale origin/<base> and reports a drift that stopped being true days
ago, which is the failure this skill exists to catch.

Adds `syncAdvisor.governedPaths` to make-no-mistakes.config.json: the paths
whose changes get reported by name, turning "you are 12 behind" into "three
hooks changed, two of them fix defects you may be looking at right now". No
default -- with the key unset the skill drops the consequence line rather than
falling back to a built-in list, which would be wrong in every repo but the one
it was copied from and would read as measured.

commands/rebase.md is untouched and stays a real destination. What changed is
who decides when it applies.

Origin (2026-07-31, as reported): a developer filed two bug reports against a
hook with clean reproductions. One was a real defect; the other described
behaviour fixed days earlier against a stale checkout, and nothing in the
report separated them.

Suite: 337/337 hooks, 60/60 vitest.

Created by Claude Code on behalf of @lapc506

* chore(release): take 1.38.0 instead of 1.37.0 — #55 owns 1.37.0

Measured on origin/main @ ee0ba47: the three version files all read 1.36.0,
and `gh pr diff 55 | grep -E '^\+.*"version"'` shows #55 already bumps to
1.37.0 in package.json, plugin.json and marketplace.json. Taking 1.37.0 here
collides with it in all three. The same probe over #52, #46, #44 and #41
returns nothing, so 1.38.0 is free. Skipping a version costs nothing;
colliding does.

Minor and not major, decided by reading. A skill auto-activates on its
`description` rather than being invoked by name like a command, so a renamed
`name:` changes no call site. The reference search returns nothing:

  grep -rniI "rebase.advisor" . --exclude-dir=node_modules \
    --exclude-dir=.git --exclude=CHANGELOG.md   ->  no hits

That negative is real and not a broken search -- the same grep for
`spike-recommend` returns 10 files, so cross-references of this shape are
found when they exist.

One surface does break and is stated rather than folded in: a user who typed
`/make-no-mistakes:rebase-advisor` explicitly (README:137 documents that
skills can be invoked that way) now gets an unknown skill. It fails loudly,
the replacement is one row away in the same table, and the installer prunes
the old file rather than leaving both live.

The `[1.37.0]:` CHANGELOG reference slot is deliberately left for #55 to fill.

Suite: 337/337 hooks, 60/60 vitest.

Created by Claude Code on behalf of @lapc506

* fix(sync-advisor): make the collision predicate correct from any subdirectory

Reviewer P3 on `ea8c46e` claimed `git ls-tree` "always outputs
repository-relative paths" while `git ls-files` is prefix-relative, so the two
would fail to match from a subdirectory, and proposed `--full-name` on
`ls-files`.

Measured: the premise is false and the proposed fix introduces the bug it
claims to prevent. From `sub/`, `ls-tree -r --name-only` prints `newfile.txt`,
not `sub/newfile.txt` -- it strips the prefix exactly like `ls-files` does, so
the original command matched fine and found the collision. Adding
`--full-name` alone then makes `ls-files` emit `sub/newfile.txt` against
`ls-tree`'s `newfile.txt`, they stop matching, and `comm -12` returns empty --
a clean bill of health for a tree about to abort the pull.

But it pointed at a real weakness of a different kind. From a subdirectory both
commands are SCOPED to that subtree, so a collision at the repo root is not
seen at all. That is scope, not format, and no combination of format flags
fixes it: `--full-name` changes how a path prints, never which paths are
considered. The pathspec does.

Shipped: `--full-name -- :/` on ls-files, `--full-tree` on ls-tree. Four cases
run, with the command extracted verbatim from SKILL.md so the test cannot drift
from the doc -- negative control from the root and from `sub/` (both empty),
positive from the root and from `sub/` with one collision in each location
(both list `rootfile.txt` and `sub/newfile.txt`).

Suite: 340/340 hooks, 60/60 vitest.

Created by Claude Code on behalf of @lapc506

* fix(sync-advisor): resolve the base ref correctly, and in a bare form

Reviewer P2 on `00e2c08`, and it is right. Step 0 returned values carrying the
remote prefix while every predicate interpolates `origin/$BASE`, so the run
died on `origin/origin/develop`:

  $ git rev-list --left-right --count "HEAD...origin/origin/main"
  fatal: ambiguous argument … unknown revision          exit=128

Fixed by normalising unconditionally (`${BASE#refs/remotes/}`, `${BASE#origin/}`)
whichever branch of the resolution produced the value. Verified both branches
with a control that must fail: where `origin/HEAD` IS set it returns
`origin/main`, normalises to `main`, and `origin/main` resolves; the
un-normalised form exits 128.

The P2 understated it. Step 0's first command resolved the base from
`@{upstream}`, which on a feature branch is that branch's OWN remote copy --
here `origin/andres/sync-advisor`. That answers "am I pushed?", which is
predicate 6's question, and would report 0 behind on a branch far behind the
real base. Stripping a prefix would have left it pointing at the wrong ref, so
the command is gone from base resolution rather than patched.

Also measured while there: `git symbolic-ref --short refs/remotes/origin/HEAD`
fails outright in this repo (`not a symbolic ref`), so it is documented as a
fall-through into the develop/main/master/trunk probe rather than as a step
that is expected to succeed.

P4 (`--abbrev-ref` and `--symbolic-full-name` redundant) is also correct --
both forms return `origin/andres/sync-advisor` here -- and is moot: that
command no longer appears.

Suite: 340/340 hooks, 60/60 vitest.

Created by Claude Code on behalf of @lapc506
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant