[Hooks Cannot See] (2) Require the output to entail the claim, not merely agree - #519
Conversation
…rely agree The session behind slice (1) ran one command against one container image and proved that image had no corepack. That was written up as corepack having been removed from Node 25+, a version-boundary claim no command in the session ever checked. The output was true and the sentence was wider than the output. prove-it already covers a missing check and a hedge. It did not cover a check that ran, came back narrower than the claim, and got filed under the wider heading anyway -- which is the common case, because the cheap check is always the reachable instance. It binds hardest in a correction: restating the original overclaim while pasting a narrower proof relabels the mistake as a fix, which is what happened here before the rule existed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZqCxSQVpmM9Rx37C7irgf Change-Id: Ibc32514be641d37277115faf6068e75ea02e0083
Owner
Author
|
This pull request is part of a Mergify stack:
|
5 tasks
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_bac5280c-bd98-4100-8056-7d56a041abd8) |
EdbertChan
added a commit
that referenced
this pull request
Sep 13, 2026
…ody checker on PR #519's live body after the rewrite. Goal: Prove the body passes as published. Motivation: A PATCH that returned 200 is not proof the live body passes. Safety invariant: Read-only; a failed fetch fails the task instead of passing. Verify: exit code 0 and output PR body validation passed. Effectiveness measurement: The same checker the required GitHub check runs passes on the live body, so the PR is no longer blocked on its text. Exit code: 0
EdbertChan
added a commit
that referenced
this pull request
Sep 13, 2026
…ody checker on PR #519's live body after the rewrite. Goal: Prove the body passes as published. Motivation: A PATCH that returned 200 is not proof the live body passes. Safety invariant: Read-only; a failed fetch fails the task instead of passing. Verify: exit code 0 and output PR body validation passed. Effectiveness measurement: The same checker the required GitHub check runs passes on the live body, so the PR is no longer blocked on its text. Exit code: 0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A written rule tells people how to support statements with command output. It helps readers judge whether evidence truly proves what a sentence says.
One check proved a single container image lacked a package. The report widened that result into a claim about newer runtime versions.
No check tested the version boundary, so the output was true but the sentence reached beyond it.
The new paragraph requires narrowing the sentence to the tested case and naming broader claims as open.
Review Claim
Approve one paragraph that requires pasted output to prove the full statement, not merely point in the same direction.
Review Lane
behavior
Review Unit
corpus-lesson
Safety Invariant
Only the body of pull request #519 changes. No commits, pushes, branch changes, labels, comments, or edits to any other pull request are made. The body is published only after the checker passes, and the live body is checked again after publication.
Slice Rationale
Split from slice (1) because
docs/ecosystem.mdforbids engine runtime and corpus lessons in the same PR, andscripts/check_ecosystem_boundaries.pyrejected the combined branch by name.It sits after (1) because (1) is the mechanical half of the same reflect pass; a reviewer who reads the detectors first has the concrete failure in hand before the general rule.
Non-goals
Does not add a hook. Judging whether pasted output entails a sentence is a meaning question, not a regex one; the mechanical candidate is an
llm-judgeand is not in this slice.Does not touch
engine/. That is slice (1).Does not restate the existing missing-check or hedge rules, which already cover their cases.
Test Plan
Test Plan
Prose-only slice; the gates that apply are the skill and ecosystem checks.
Whole repo suite on the combined branch before the split:
python3 scripts/check_ecosystem_boundaries.pypython3 scripts/check_skill_file_refs.pypython3 scripts/check_skills_three_harnesses.pybash scripts/run_all_tests.sh— 49 directories, 0 failuresRevert Plan
Revert Plan
git revert <sha>