Skip to content

Reply-length checker tests stop reaching the real judge - #529

Open
EdbertChan wants to merge 1 commit into
split/judge-tests-4-wrong-check-reflectfrom
split/judge-tests-5-diu-stop
Open

Reply-length checker tests stop reaching the real judge#529
EdbertChan wants to merge 1 commit into
split/judge-tests-4-wrong-check-reflectfrom
split/judge-tests-5-diu-stop

Conversation

@EdbertChan

@EdbertChan EdbertChan commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Summary

The reply checker that enforces short, plain replies has some checks that ask the AI judge.

Its test cases had no judge setup, so a run could write to the user's real queue.

Its test classes now use the shared setup.

Review Claim

This checker's test cases run on the shared judge setup and cannot reach a model.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Test-only. No checker logic changes; four test files change their base class and import path. 104 tests pass.

Slice Rationale

Same mechanical change across four test files of one hook, so they share one claim.

Stack position 5 of 7, split from one bundled change. Merge bottom-up.

Non-goals

  • No change to word limits or phrase checks.
  • No runtime code changes.

Test Plan

Test Plan
$ per-slice checks on split/judge-tests-5
ruff: All checks passed!
[llm-judge: Ran 61 tests in 7.434s OK ] [wrong-check-reflect: Ran 16 tests in 3.623s OK ] [diu-stop: Ran 104 tests in 0.093s OK ]
gate-tests: Ran 8 tests in 0.187s OK
gate: check_hook_test_coverage: OK (37 hook(s) checked)
preflight: ok      preflight passed
  • python3 -m unittest discover -s engine/hooks/llm-judge/testsRan 61 tests / OK
  • python3 -m unittest discover -s engine/hooks/wrong-check-reflect/testsRan 16 tests / OK
  • python3 -m unittest discover -s engine/hooks/diu-stop/testsRan 104 tests / OK
  • uvx ruff check . --select E9,FAll checks passed!
  • python3 engine/skills/make-pr/scripts/preflight.py --base <stack parent>ok preflight passed

Revert Plan

Revert Plan
  • Safe to revert? Yes, after reverting any later slice in this stack first.
  • Revert command: git revert <merge-commit-sha>
  • Post-revert steps: None.
  • Data migration? No.

🤖 Generated with Claude Code

https://claude.ai/code/session_013smxPCr4XeE7R77zVtN1u4


Note

Low Risk
Test-only wiring; no runtime checker or judge behavior changes.

Overview
diu-stop reply-length and evidence tests that can invoke the LLM judge now inherit JudgeTestCase instead of unittest.TestCase, with llm-judge on sys.path so they pick up the shared test harness.

That harness isolates judge state and uses stub runners, so a local unittest run no longer risks enqueueing real judge work. test_plain_words_check also calls super().setUp() / super().tearDown() so the stub environment is applied around its tempfile setup.

No production hook or checker logic changes—only four test modules.

Reviewed by Cursor Bugbot for commit 2a04aff. Bugbot is set up for automated code reviews on this repo. Configure here.

These suites had no judge isolation, so a phrase check that asks the judge
could write to the real queue or call a real model. They now inherit
JudgeTestCase.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013smxPCr4XeE7R77zVtN1u4
Change-Id: I76b5a9b6926bab3d871e489198505beb2821a559
@cursor

cursor Bot commented Sep 13, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_45573813-6250-4383-960c-e6dd6f7f0993)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant