Excuses (3) calling a failure pre-existing needs a base run and a handle - #541
Open
EdbertChan wants to merge 12 commits into
Open
Conversation
…view claim: gate-blame-needs-evidence also flags a reply that dismisses a failure as pre-existing, flaky, environmental, or unrelated to the change unless the transcript shows a run on the base branch and the reply names an issue/task id or a fix commit. Review lane: policy Safety invariant: A pre-existing claim backed by a base-branch run plus an issue, task, or fix handle passes untouched; a reply that switched to the correct validator instead of dismissing the failure is not flagged. Effectiveness measurement: Hook tests replay 'Pure environmental artifact, unrelated to my change, out of scope.' with no base run (flags), 'pre-existing mismatch, not something I introduced' followed by landing through admin bypass (flags), the same claim with a base-branch run and a filed issue id (silent), and an agent that switched to the right validator (silent). Slice rationale: One added detector inside the existing gate-blame hook plus its fixtures. Architectural effect: Blame aimed at a test or environment needs the same evidence as blame aimed at a gate. Goal: Stop 'not my change' from closing a real failure with nothing filed. Motivation: In a corpus reflect sample, 'Pure environmental artifact, unrelated to my change, out of scope.' left the same 4 tests failing on unchanged code 11 days later with nothing filed, and a 'pre-existing mismatch' landed through admin bypass and was never fixed. gate-blame-needs-evidence today only covers blame aimed at a gate. Alternative considerations: A new sibling hook was set aside in favor of extending the existing blame hook. Forbidding the word 'flaky' was set aside because a proven flaky test with a filed issue is a legitimate finding. Implementation details: In engine/hooks/gate-blame-needs-evidence add a second judged question: does the reply dismiss a failure as pre-existing/flaky/environmental/unrelated, and if so, does the transcript show a base-branch run and does the reply name an issue/task id or fix commit. Decide meaning the way the phrase-judge skill in engine/skills/phrase-judge/ prescribes: a phrase dictionary under engine/hooks/llm-judge/phrases/ run through the shared llm-judge, never an ad-hoc regex. A judge result that could not be produced is reported as unchecked, never as clean. Non-goals: No change to the existing gate-blame detector's behavior or messages. Leave the files of open PRs #482, #500 and #528 alone; if a change would overlap them, stop and report instead. Layer: domain Feature state: active Files: - engine/hooks/gate-blame-needs-evidence/detect.py - engine/hooks/gate-blame-needs-evidence/claude_stop_check.py - engine/hooks/llm-judge/phrases/pre-existing-needs-handle.json - engine/hooks/gate-blame-needs-evidence/tests/test_hooks.py Change types: - engine/hooks/gate-blame-needs-evidence/detect.py: modify - engine/hooks/gate-blame-needs-evidence/claude_stop_check.py: modify - engine/hooks/llm-judge/phrases/pre-existing-needs-handle.json: create - engine/hooks/gate-blame-needs-evidence/tests/test_hooks.py: modify Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/gate-blame-needs-evidence/tests -v` exits 0. - `python3 scripts/check_hook_test_coverage.py` exits 0. - `python3 scripts/check_no_silent_hook_except.py` exits 0. - `python3 scripts/check_rules_are_wired.py` exits 0. - `python3 scripts/check_no_new_comments.py` exits 0. Exit code: 0 Invoker-Finalize-Id: 22d4b65b-14c0-4bba-865d-76bd05d53f79
…iew claim: `python3 scripts/check_hook_test_coverage.py` passes on the finished branch. Review lane: proof Safety invariant: Verification is read-only and alters no repository file. Effectiveness measurement: The command's exit code is the direct measurement. Slice rationale: One check per proof task. Architectural effect: None; verification only. Goal: Prove the slice. Motivation: Running the check is the proof. Alternative considerations: The full suite was not required because the slice touches one component with its own tests. Implementation details: Run `python3 scripts/check_hook_test_coverage.py`. Non-goals: No mutations. Layer: app_regression Feature state: active Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry. Acceptance criteria: - The command exits 0. Exit code: 0
…iew claim: `python3 scripts/check_no_new_comments.py` passes on the finished branch. Review lane: proof Safety invariant: Verification is read-only and alters no repository file. Effectiveness measurement: The command's exit code is the direct measurement. Slice rationale: One check per proof task. Architectural effect: None; verification only. Goal: Prove the slice. Motivation: Running the check is the proof. Alternative considerations: The full suite was not required because the slice touches one component with its own tests. Implementation details: Run `python3 scripts/check_no_new_comments.py`. Non-goals: No mutations. Layer: app_regression Feature state: active Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry. Acceptance criteria: - The command exits 0. Exit code: 0
…iew claim: `python3 scripts/check_rules_are_wired.py` passes on the finished branch. Review lane: proof Safety invariant: Verification is read-only and alters no repository file. Effectiveness measurement: The command's exit code is the direct measurement. Slice rationale: One check per proof task. Architectural effect: None; verification only. Goal: Prove the slice. Motivation: Running the check is the proof. Alternative considerations: The full suite was not required because the slice touches one component with its own tests. Implementation details: Run `python3 scripts/check_rules_are_wired.py`. Non-goals: No mutations. Layer: app_regression Feature state: active Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry. Acceptance criteria: - The command exits 0. Exit code: 0
…iew claim: `python3 -m unittest discover -s engine/hooks/gate-blame-needs-evidence/tests -v` passes on the finished branch. Review lane: proof Safety invariant: Verification is read-only and alters no repository file. Effectiveness measurement: The command's exit code is the direct measurement. Slice rationale: One check per proof task. Architectural effect: None; verification only. Goal: Prove the slice. Motivation: Running the check is the proof. Alternative considerations: The full suite was not required because the slice touches one component with its own tests. Implementation details: Run `python3 -m unittest discover -s engine/hooks/gate-blame-needs-evidence/tests -v`. Non-goals: No mutations. Layer: app_regression Feature state: active Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry. Acceptance criteria: - The command exits 0. Exit code: 0
…iew claim: `python3 scripts/check_no_silent_hook_except.py` passes on the finished branch. Review lane: proof Safety invariant: Verification is read-only and alters no repository file. Effectiveness measurement: The command's exit code is the direct measurement. Slice rationale: One check per proof task. Architectural effect: None; verification only. Goal: Prove the slice. Motivation: Running the check is the proof. Alternative considerations: The full suite was not required because the slice touches one component with its own tests. Implementation details: Run `python3 scripts/check_no_silent_hook_except.py`. Non-goals: No mutations. Layer: app_regression Feature state: active Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry. Acceptance criteria: - The command exits 0. Exit code: 0 Invoker-Finalize-Id: 77cc6e7f-7e87-48f4-a4d2-654c43c759e5
…ds-handle-2/g0.t0.a-a5bf5e9aa-90b641dc
…ds-handle-3/g0.t0.a-ace5d5a4e-77ce725e
…ds-handle-4/g0.t0.a-aa0575f9c-b9118f7e
…ds-handle-5/g0.t0.a-a659ce137-2e68cdd3
…No ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was set aside as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry. Acceptance criteria: - The command exits 0. Exit code: 0 Invoker-Finalize-Id: 49f0d8b1-085b-4687-b1e0-dab31b2c4697
…a997eb26d-7610b992 — Review claim: No ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was set aside as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Non-goals: No product edits. Layer: app_regression Feature state: active Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry. Acceptance criteria: - The command exits 0.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_19205f3f-32aa-4219-9cbf-6d59c682080b) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Require evidence before dismissing a failed check as pre-existing, flaky, environmental, or unrelated.
The reply must show a base-branch run and name an issue, task, or fix commit.
This keeps real failures from being closed as “not my change” without follow-up.
Review Claim
Approve requiring a base-branch run and a follow-up handle for claims that dismiss a failure as unrelated.
Review Lane
policy
Review Unit
policy
Safety Invariant
A dismissed failure passes when the transcript contains a base-branch run and the reply names an issue, task, or fix commit; replies that use the correct validator remain silent.
Slice Rationale
This adds one judged policy to the existing blame hook and its fixtures, keeping related failure-dismissal checks together without changing gate-blame behavior.
Non-goals
Architecture
Before
graph TD A["reply dismisses failure"] --> B["existing gate-blame judgment"] B --> C["flag or stay silent"]After
graph TD A["reply dismisses failure"] --> B["existing gate-blame judgment"] A --> D["pre-existing claim judgment"] D --> E["base-branch run plus issue, task, or fix handle"] E --> C["flag or stay silent"] B --> CTest Plan
Test Plan
python3 -m unittest discover -s engine/hooks/gate-blame-needs-evidence/tests -v— 30 tests passed.python3 scripts/check_hook_test_coverage.py— OK (37 hook(s) checked).python3 scripts/check_no_silent_hook_except.py— checked 202 files.python3 scripts/check_rules_are_wired.py— OK (8 checker(s) checked).python3 scripts/check_no_new_comments.py— no new comments.bash scripts/scrub-handoff-artifacts.sh— scrub completed cleanly.Revert Plan
Revert Plan
git revert <merge-commit-sha>.Note
Medium Risk
Adds new Stop-hook enforcement on how assistants dismiss CI/check failures; scope is limited to that claim shape and does not alter existing gate-blame logic.
Overview
Extends the
gate-blame-needs-evidenceStop hook with a second judgment path for replies that dismiss a failed check as pre-existing, flaky, environmental, or unrelated.Those excuses now must be backed by transcript evidence of a base-branch run and a named follow-up (issue, task, or fix commit); otherwise the hook sends the reply back like existing gate-blame hits. Replies that already satisfy the validator, and unchanged gate-blame detection, stay silent.
Tests and fixtures in
engine/hooks/gate-blame-needs-evidence/testscover the new policy alongside the existing blame rules.Reviewed by Cursor Bugbot for commit 42f76f9. Bugbot is set up for automated code reviews on this repo. Configure here.