Full test run isolates the judge for every suite

[EdbertChan]

Summary

One script runs every test in the repo.

A test that skips the shared setup could still reach a model during that run.

The script now points the judge at a temporary folder and a local stand-in for the whole run.

Review Claim

The full test run gives the judge a throwaway folder and a stand-in runner.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Only the test runner script changes. It sets two variables and deletes the temporary folder on exit. A probe under those variables printed a temp folder, the stub runner, and a no answer.

Slice Rationale

Runner-level isolation is a separate path from the shared base, so it gets its own slice.

Stack position 6 of 7, split from one bundled change. Merge bottom-up.

Non-goals

• No change to which suites run.
• No CI workflow changes.

Test Plan

Test Plan

$ per-slice checks on split/judge-tests-6
ruff: All checks passed!
[llm-judge: Ran 61 tests in 6.088s OK ] [wrong-check-reflect: Ran 16 tests in 3.417s OK ] [diu-stop: Ran 104 tests in 0.052s OK ]
gate-tests: Ran 8 tests in 0.141s OK
gate: check_hook_test_coverage: OK (37 hook(s) checked)
preflight: ok      preflight passed
$ bash scripts/run_all_tests.sh  (stack tip)
run_all_tests exit=0
suites OK: 53, total tests: 1970, OK (skipped=1)

• Probe with the script's header env: judge.state_root() — /var/folders/.../catstack-llm-judge-tests-31egi4q4; [n for n,_ in judge.runners()] — ['stub']; judge.ask('x')['answer'] — {'match': False}
• bash scripts/run_all_tests.sh on the stack tip — run_all_tests exit=0, 53 suites OK, 1970 tests (one skipped)
• python3 engine/skills/make-pr/scripts/preflight.py --base <stack parent> — ok preflight passed

Revert Plan

Revert Plan

• Safe to revert? Yes, after reverting any later slice in this stack first.
• Revert command: git revert <merge-commit-sha>
• Post-revert steps: None.
• Data migration? No.

🤖 Generated with Claude Code

https://claude.ai/code/session_013smxPCr4XeE7R77zVtN1u4

---

Note

Low Risk
Only the aggregate test shell script changes; it hardens test isolation without touching judge logic, auth, or production paths.

Overview
Full-repo test runs now isolate the LLM judge so suites cannot hit real model runners or share persistent judge state.

At startup, scripts/run_all_tests.sh creates a throwaway CATSTACK_LLM_JUDGE_STATE_DIR, registers an EXIT trap to remove it, and sets CATSTACK_LLM_JUDGE_RUNNERS to a single local stub that always prints {"match": false}. Those exports apply for the entire discover loop; which suites run and CI are unchanged.

^{Reviewed by Cursor Bugbot for commit d6c5cfb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_98767b7c-a014-481e-9fa9-9ec9cbbaa71b)