Skip to content

Excuses (3) calling a failure pre-existing needs a base run and a handle - #541

Open
EdbertChan wants to merge 12 commits into
mainfrom
plan/excuses-3-calling-a-failure-pre-existing-needs-a-base-run-and-a-handle
Open

Excuses (3) calling a failure pre-existing needs a base run and a handle#541
EdbertChan wants to merge 12 commits into
mainfrom
plan/excuses-3-calling-a-failure-pre-existing-needs-a-base-run-and-a-handle

Conversation

@EdbertChan

@EdbertChan EdbertChan commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Summary

Require evidence before dismissing a failed check as pre-existing, flaky, environmental, or unrelated.

The reply must show a base-branch run and name an issue, task, or fix commit.

This keeps real failures from being closed as “not my change” without follow-up.

Review Claim

Approve requiring a base-branch run and a follow-up handle for claims that dismiss a failure as unrelated.

Review Lane

policy

Review Unit

policy

Safety Invariant

A dismissed failure passes when the transcript contains a base-branch run and the reply names an issue, task, or fix commit; replies that use the correct validator remain silent.

Slice Rationale

This adds one judged policy to the existing blame hook and its fixtures, keeping related failure-dismissal checks together without changing gate-blame behavior.

Non-goals

Architecture

Before

graph TD
    A["reply dismisses failure"] --> B["existing gate-blame judgment"]
    B --> C["flag or stay silent"]
Loading

After

graph TD
    A["reply dismisses failure"] --> B["existing gate-blame judgment"]
    A --> D["pre-existing claim judgment"]
    D --> E["base-branch run plus issue, task, or fix handle"]
    E --> C["flag or stay silent"]
    B --> C
Loading

Test Plan

Test Plan
  • python3 -m unittest discover -s engine/hooks/gate-blame-needs-evidence/tests -v — 30 tests passed.
  • python3 scripts/check_hook_test_coverage.py — OK (37 hook(s) checked).
  • python3 scripts/check_no_silent_hook_except.py — checked 202 files.
  • python3 scripts/check_rules_are_wired.py — OK (8 checker(s) checked).
  • python3 scripts/check_no_new_comments.py — no new comments.
  • bash scripts/scrub-handoff-artifacts.sh — scrub completed cleanly.

Revert Plan

Revert Plan
  • Safe to revert? Yes.
  • Revert command: git revert <merge-commit-sha>.
  • Post-revert steps: Re-run the hook tests and policy checks.
  • Data migration? No.

Note

Medium Risk
Adds new Stop-hook enforcement on how assistants dismiss CI/check failures; scope is limited to that claim shape and does not alter existing gate-blame logic.

Overview
Extends the gate-blame-needs-evidence Stop hook with a second judgment path for replies that dismiss a failed check as pre-existing, flaky, environmental, or unrelated.

Those excuses now must be backed by transcript evidence of a base-branch run and a named follow-up (issue, task, or fix commit); otherwise the hook sends the reply back like existing gate-blame hits. Replies that already satisfy the validator, and unchanged gate-blame detection, stay silent.

Tests and fixtures in engine/hooks/gate-blame-needs-evidence/tests cover the new policy alongside the existing blame rules.

Reviewed by Cursor Bugbot for commit 42f76f9. Bugbot is set up for automated code reviews on this repo. Configure here.

Invoker Bot and others added 12 commits September 13, 2026 06:33
…view claim: gate-blame-needs-evidence also flags a reply that dismisses a failure as pre-existing, flaky, environmental, or unrelated to the change unless the transcript shows a run on the base branch and the reply names an issue/task id or a fix commit.

Review lane: policy
Safety invariant: A pre-existing claim backed by a base-branch run plus an issue, task, or fix handle passes untouched; a reply that switched to the correct validator instead of dismissing the failure is not flagged.
Effectiveness measurement: Hook tests replay 'Pure environmental artifact, unrelated to my change, out of scope.' with no base run (flags), 'pre-existing mismatch, not something I introduced' followed by landing through admin bypass (flags), the same claim with a base-branch run and a filed issue id (silent), and an agent that switched to the right validator (silent).
Slice rationale: One added detector inside the existing gate-blame hook plus its fixtures.
Architectural effect: Blame aimed at a test or environment needs the same evidence as blame aimed at a gate.
Goal: Stop 'not my change' from closing a real failure with nothing filed.
Motivation: In a corpus reflect sample, 'Pure environmental artifact, unrelated to my change, out of scope.' left the same 4 tests failing on unchanged code 11 days later with nothing filed, and a 'pre-existing mismatch' landed through admin bypass and was never fixed. gate-blame-needs-evidence today only covers blame aimed at a gate.
Alternative considerations: A new sibling hook was set aside in favor of extending the existing blame hook. Forbidding the word 'flaky' was set aside because a proven flaky test with a filed issue is a legitimate finding.
Implementation details: In engine/hooks/gate-blame-needs-evidence add a second judged question: does the reply dismiss a failure as pre-existing/flaky/environmental/unrelated, and if so, does the transcript show a base-branch run and does the reply name an issue/task id or fix commit. Decide meaning the way the phrase-judge skill in engine/skills/phrase-judge/ prescribes: a phrase dictionary under engine/hooks/llm-judge/phrases/ run through the shared llm-judge, never an ad-hoc regex. A judge result that could not be produced is reported as unchecked, never as clean.
Non-goals: No change to the existing gate-blame detector's behavior or messages. Leave the files of open PRs #482, #500 and #528 alone; if a change would overlap them, stop and report instead.
Layer: domain
Feature state: active
Files:
- engine/hooks/gate-blame-needs-evidence/detect.py
- engine/hooks/gate-blame-needs-evidence/claude_stop_check.py
- engine/hooks/llm-judge/phrases/pre-existing-needs-handle.json
- engine/hooks/gate-blame-needs-evidence/tests/test_hooks.py
Change types:
- engine/hooks/gate-blame-needs-evidence/detect.py: modify
- engine/hooks/gate-blame-needs-evidence/claude_stop_check.py: modify
- engine/hooks/llm-judge/phrases/pre-existing-needs-handle.json: create
- engine/hooks/gate-blame-needs-evidence/tests/test_hooks.py: modify
Acceptance criteria:
- `python3 -m unittest discover -s engine/hooks/gate-blame-needs-evidence/tests -v` exits 0.
- `python3 scripts/check_hook_test_coverage.py` exits 0.
- `python3 scripts/check_no_silent_hook_except.py` exits 0.
- `python3 scripts/check_rules_are_wired.py` exits 0.
- `python3 scripts/check_no_new_comments.py` exits 0.

Exit code: 0
Invoker-Finalize-Id: 22d4b65b-14c0-4bba-865d-76bd05d53f79
…iew claim: `python3 scripts/check_hook_test_coverage.py` passes on the finished branch.

Review lane: proof
Safety invariant: Verification is read-only and alters no repository file.
Effectiveness measurement: The command's exit code is the direct measurement.
Slice rationale: One check per proof task.
Architectural effect: None; verification only.
Goal: Prove the slice.
Motivation: Running the check is the proof.
Alternative considerations: The full suite was not required because the slice touches one component with its own tests.
Implementation details: Run `python3 scripts/check_hook_test_coverage.py`.
Non-goals: No mutations.
Layer: app_regression
Feature state: active
Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry.
Acceptance criteria:
- The command exits 0.

Exit code: 0
…iew claim: `python3 scripts/check_no_new_comments.py` passes on the finished branch.

Review lane: proof
Safety invariant: Verification is read-only and alters no repository file.
Effectiveness measurement: The command's exit code is the direct measurement.
Slice rationale: One check per proof task.
Architectural effect: None; verification only.
Goal: Prove the slice.
Motivation: Running the check is the proof.
Alternative considerations: The full suite was not required because the slice touches one component with its own tests.
Implementation details: Run `python3 scripts/check_no_new_comments.py`.
Non-goals: No mutations.
Layer: app_regression
Feature state: active
Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry.
Acceptance criteria:
- The command exits 0.

Exit code: 0
…iew claim: `python3 scripts/check_rules_are_wired.py` passes on the finished branch.

Review lane: proof
Safety invariant: Verification is read-only and alters no repository file.
Effectiveness measurement: The command's exit code is the direct measurement.
Slice rationale: One check per proof task.
Architectural effect: None; verification only.
Goal: Prove the slice.
Motivation: Running the check is the proof.
Alternative considerations: The full suite was not required because the slice touches one component with its own tests.
Implementation details: Run `python3 scripts/check_rules_are_wired.py`.
Non-goals: No mutations.
Layer: app_regression
Feature state: active
Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry.
Acceptance criteria:
- The command exits 0.

Exit code: 0
…iew claim: `python3 -m unittest discover -s engine/hooks/gate-blame-needs-evidence/tests -v` passes on the finished branch.

Review lane: proof
Safety invariant: Verification is read-only and alters no repository file.
Effectiveness measurement: The command's exit code is the direct measurement.
Slice rationale: One check per proof task.
Architectural effect: None; verification only.
Goal: Prove the slice.
Motivation: Running the check is the proof.
Alternative considerations: The full suite was not required because the slice touches one component with its own tests.
Implementation details: Run `python3 -m unittest discover -s engine/hooks/gate-blame-needs-evidence/tests -v`.
Non-goals: No mutations.
Layer: app_regression
Feature state: active
Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry.
Acceptance criteria:
- The command exits 0.

Exit code: 0
…iew claim: `python3 scripts/check_no_silent_hook_except.py` passes on the finished branch.

Review lane: proof
Safety invariant: Verification is read-only and alters no repository file.
Effectiveness measurement: The command's exit code is the direct measurement.
Slice rationale: One check per proof task.
Architectural effect: None; verification only.
Goal: Prove the slice.
Motivation: Running the check is the proof.
Alternative considerations: The full suite was not required because the slice touches one component with its own tests.
Implementation details: Run `python3 scripts/check_no_silent_hook_except.py`.
Non-goals: No mutations.
Layer: app_regression
Feature state: active
Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry.
Acceptance criteria:
- The command exits 0.

Exit code: 0
Invoker-Finalize-Id: 77cc6e7f-7e87-48f4-a4d2-654c43c759e5
…No ephemeral inter-task handoff files remain in the worktree before the merge gate.

Review lane: cleanup
Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files.
Effectiveness measurement: The script exits non-zero if any handoff artifact remains.
Slice rationale: Required terminal scrub for every implementation workflow.
Architectural effect: None; hygiene only.
Goal: Leave the branch free of handoff artifacts.
Motivation: Handoff files must not reach the PR.
Alternative considerations: Manual cleanup was set aside as non-deterministic.
Implementation details: Run scripts/scrub-handoff-artifacts.sh.
Non-goals: No product edits.
Layer: app_regression
Feature state: active
Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry.
Acceptance criteria:
- The command exits 0.

Exit code: 0
Invoker-Finalize-Id: 49f0d8b1-085b-4687-b1e0-dab31b2c4697
…a997eb26d-7610b992 — Review claim: No ephemeral inter-task handoff files remain in the worktree before the merge gate.

Review lane: cleanup
Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files.
Effectiveness measurement: The script exits non-zero if any handoff artifact remains.
Slice rationale: Required terminal scrub for every implementation workflow.
Architectural effect: None; hygiene only.
Goal: Leave the branch free of handoff artifacts.
Motivation: Handoff files must not reach the PR.
Alternative considerations: Manual cleanup was set aside as non-deterministic.
Implementation details: Run scripts/scrub-handoff-artifacts.sh.
Non-goals: No product edits.
Layer: app_regression
Feature state: active
Layer exception: allowed. Verification and the terminal scrub run after the docs commit so they check the final branch the PR will carry.
Acceptance criteria:
- The command exits 0.
@cursor

cursor Bot commented Sep 13, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_19205f3f-32aa-4219-9cbf-6d59c682080b)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant