Judge test base runs only a local stub, never a real model

[EdbertChan]

Summary

Checker test cases that ask the AI judge use one shared setup.

That setup cleared the runner list, so a test that forgot to pick one fell back to the real model tools.

Now the setup picks a local stand-in that always answers no, and a new test proves it.

Review Claim

Test cases on the shared setup can no longer reach a real model or the real judge queue.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Test-only. The judge's runtime defaults are unchanged; only the shared test setup sets a stand-in. The new stub test fails on the parent slice and passes here.

Slice Rationale

The default flip lands alone so its one new failure mode is easy to see. Directly affected tests stay with it.

Stack position 2 of 7, split from one bundled change. Merge bottom-up.

Non-goals

• No change to the judge's real runner order.
• No other hook's tests change.

Test Plan

Test Plan

$ per-slice checks on split/judge-tests-2
ruff: All checks passed!
[llm-judge: Ran 61 tests in 6.604s OK ] [wrong-check-reflect: Ran 16 tests in 3.266s OK ] [diu-stop: Ran 104 tests in 0.049s OK ]
gate-tests: Ran 8 tests in 0.201s OK
gate: check_hook_test_coverage: OK (37 hook(s) checked)
preflight: ok      preflight passed

• python3 -m unittest tests.test_judge.TestAsk.test_test_base_runs_only_the_local_stub with the parent slice's base — FAILED (failures=1); with this slice — OK
• python3 -m unittest discover -s engine/hooks/llm-judge/tests — Ran 61 tests / OK
• python3 -m unittest discover -s engine/hooks/wrong-check-reflect/tests — Ran 16 tests / OK
• python3 -m unittest discover -s engine/hooks/diu-stop/tests — Ran 104 tests / OK
• uvx ruff check . --select E9,F — All checks passed!
• python3 engine/skills/make-pr/scripts/preflight.py --base <stack parent> — ok preflight passed

Revert Plan

Revert Plan

• Safe to revert? Yes, after reverting any later slice in this stack first.
• Revert command: git revert <merge-commit-sha>
• Post-revert steps: None.
• Data migration? No.

🤖 Generated with Claude Code

https://claude.ai/code/session_013smxPCr4XeE7R77zVtN1u4

---

Note

Low Risk
Changes only shared judge test fixtures and assertions; production judge runner defaults and runtime behavior are unchanged.

Overview
Test-only hardening for the LLM judge hook: shared JudgeTestCase setup no longer clears RUNNERS_ENV (which let tests accidentally hit real codex/claude/cursor runners). It now pins a local stub runner via sys.executable that always returns {"match": false}.

Adds test_test_base_runs_only_the_local_stub to lock that default, and updates test_default_runner_order_is_codex_then_claude_then_cursor to temporarily unset RUNNERS_ENV so production default order is still asserted without fighting the base fixture.

Adds test_enqueue_writes_only_to_temporary_state_directory, checking enqueue writes job files only under the test’s temp STATE_ENV root and leaves the default HOME-derived state directory empty.

^{Reviewed by Cursor Bugbot for commit 86705c2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_c713de50-91bb-4995-870a-b79f5726ed2b)