Coverage gate fails a judge hook whose tests skip the shared base

[EdbertChan]

Summary

A repo check makes sure each checker has the test cases it needs.

The new judge rule only looked at some checkers, skipped files it could not read, and trusted a name in a comment.

It now checks every checker, reads real imports, and reports unreadable files.

Review Claim

The coverage check fails any judge-calling checker whose test cases do not use the shared setup.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Check-only. It lands after every judge-calling hook already uses the shared base, so the real tree passes. Unreadable files are reported, never passed.

Slice Rationale

The gate goes last so it never blocks a hook that has not switched yet.

Stack position 7 of 7, split from one bundled change. Merge bottom-up.

Non-goals

• No hook code changes.
• No change to the existing positive, negative, or unreadable-input rules.

Test Plan

Test Plan

$ per-slice checks on split/judge-tests-7
ruff: All checks passed!
[llm-judge: Ran 61 tests in 5.992s OK ] [wrong-check-reflect: Ran 16 tests in 6.166s OK ] [diu-stop: Ran 104 tests in 0.088s OK ]
gate-tests: Ran 14 tests in 0.270s OK
gate: check_hook_test_coverage: OK (37 hook(s) checked)
preflight: ok      preflight passed
$ bash scripts/run_all_tests.sh  (stack tip)
run_all_tests exit=0
suites OK: 53, total tests: 1970, OK (skipped=1)

• With diu-stop tests reverted to main: the original bundled gate — check_hook_test_coverage: OK (36 hook(s) checked); this slice — FAIL / diu-stop: imports llm-judge but no test imports or subclasses JudgeTestCase
• python3 -m unittest tests.test_check_hook_test_coverage — Ran 14 tests / OK; against the original gate, test_hit_hook_without_detect_py_is_still_checked_by_main — FAIL
• python3 scripts/check_hook_test_coverage.py — check_hook_test_coverage: OK (37 hook(s) checked)
• bash scripts/run_all_tests.sh on the stack tip — run_all_tests exit=0, 53 suites OK, 1970 tests (one skipped)
• uvx ruff check . --select E9,F — All checks passed!
• python3 engine/skills/make-pr/scripts/preflight.py --base <stack parent> — ok preflight passed

Revert Plan

Revert Plan

• Safe to revert? Yes, after reverting any later slice in this stack first.
• Revert command: git revert <merge-commit-sha>
• Post-revert steps: None.
• Data migration? No.

🤖 Generated with Claude Code

https://claude.ai/code/session_013smxPCr4XeE7R77zVtN1u4

---

Note

Low Risk
Check-only CI script and tests; no hook runtime or auth/data paths change, though AST heuristics could false-positive on unusual import patterns.

Overview
Extends check_hook_test_coverage with a judge isolation pass that runs for every hook directory (including those without detect.py), not only hooks covered by the existing positive/negative/unreadable rules.

The gate uses AST parsing to detect real judge imports in hook source and requires at least one test file to import or subclass JudgeTestCase. Hooks that import judge but lack compliant tests fail; unparseable Python is reported as unchecked rather than treated as passing. Mentioning JudgeTestCase only in a comment no longer satisfies the rule.

main() now runs the new checks in a second loop over CLI targets or all hook_dirs(). Unit tests in TestJudgeIsolationRule cover failure/pass cases, unreadable sources, and a hook-without-detect.py path through main().

^{Reviewed by Cursor Bugbot for commit f61550d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_79f7b212-00a9-43bb-a421-39889aac862f)