fix: send the block report to stderr so the agent actually receives it - #65
Merged
Conversation
A flagged verdict exited 2 with its report on stdout. Exit 2 is the blocking code, and the PostToolUse contract forwards the hook's stderr to the model and discards stdout, so the report was written to the one stream the host throws away. The block landed with no explanation attached: the model saw only that something blocked it, never which comment or why. Measured against a real host before the change: the payload for a write adding `// set x to 1` above `const x = 1;` produced a correct 611-byte report on stdout with exit 2 and an empty stderr, and the model received nothing from it. Where the hook was reached through a wrapper that emitted anything at all on stderr, that unrelated text became the block reason instead, which is how this presented as "the hook is broken". `print!` becomes `eprint!` on the Block arm only. Exit codes are untouched, and the pass note stays on stdout, where a host sends exit-0 stdout to its debug log. The two existing exit-code tests pass both before and after this change, which is precisely why it shipped: they pinned the code and never the stream. `flagged_payload_reports_on_stderr_only` now pins both halves — report present on stderr, stdout empty — and it fails on the pre-change binary (2 passed, 1 failed) and passes after. Also syncs Cargo.lock, which still recorded 0.1.7 after the crate moved to 0.1.8, so any build regenerated it as an unrelated diff. Verified: cargo fmt --check, cargo clippy --all-targets -D warnings (0 findings), cargo test --all-targets (97 passed, 0 failed).
The stream choice was carried by a comment above `eprint!`. Naming the two emit paths carries it in code: `emit_to_model` is the stream a host forwards on exit 2, `emit_to_debug_log` is the one it keeps to itself. The comment is deleted rather than shortened, since a shortened restatement is still a restatement. Test failure messages lose the same prose for the same reason; the test names already say which contract broke.
Review found the gate certified less than it appeared to. Two mutations
passed it:
- `eprint!("{input}")` — dumping the raw hook JSON. The anchor was
`# TODO: fix this later`, a string the *payload* supplies, so echoing the
input satisfied it. A check keyed on a value its own input provided
certifies nothing.
- `eprint!("# TODO: fix this later\n")` — the anchor and nothing else. The
model would get a bare comment with no file, no line, no reason.
The anchors are now three strings only the report path can produce: the
header, the action footer, and the classifier's reason text. Both mutations
now fail, as does the original stdout regression.
Coverage the gate was missing, each verified by a mutation that used to
pass and now fails:
- the pass note's stream — moving it to stderr was invisible before
- `Edit` and `MultiEdit`, which share the Block arm with `Write` and had no
binary-level test, so a per-tool branch could regress them unnoticed
- `--prompt`, whose substituted report travels the same arm
Docs name the stream they were relying on: the npm page said "prints",
which reads as stdout, and the project README's demo redirected in a way
that would now drop the report. The exit-2 claim both rest on is cited
rather than asserted.
Verified: cargo fmt --check, clippy --all-targets -D warnings (0), cargo
test --all-targets (101 passed, 0 failed); four mutations killed
(input-echo 4 failures, anchor-only 4, stdout regression 4, pass-note 1);
changeset validator OK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
A flagged verdict exited
2with its report on stdout. Exit 2 is the blocking code, and thePostToolUsecontract forwards the hook's stderr to the model and discards stdout (hooks reference). The report was written to the one stream the host throws away, so the block arrived carrying no explanation of what was flagged — the model saw only that something blocked it.Measured against a real host before the change: a
Writepayload adding// set x to 1aboveconst x = 1;produced a correct 611-byte report on stdout with exit 2 and an empty stderr, and the model received nothing from it. Where the hook was reached through a wrapper that emitted anything on stderr, that unrelated text became the block reason instead — which is how this presents as "the hook is broken".The change
print!→eprint!on theBlockarm only, via two named emit paths so the stream choice is greppable rather than commented:emit_to_model(stderr, what a host forwards) andemit_to_debug_log(stdout, what it keeps). Exit codes are untouched; the pass note stays on stdout.Why the existing tests missed it
clean_payload_exits_zeroandflagged_payload_exits_with_the_blocked_contractpass both before and after this change. They pinned the exit code and never the stream — precisely the uncovered half of the contract that let this ship.The new gate was itself reviewed adversarially, and its first version certified less than it looked like. Two mutations passed it:
eprint!("{input}")— dump the raw hook JSON# TODO: fix this later, a string the payload supplies, so echoing the input satisfied iteprint!("# TODO: fix this later\n")— anchor onlycontainswith no structural assertionA check keyed on a value its own input provided certifies nothing. The anchors are now three strings only the report path can produce — the header, the action footer, and the classifier's reason text.
Coverage the gate was also missing, each verified by a mutation that used to pass and now fails:
EditandMultiEdit— they share theBlockarm withWriteand had no binary-level test, so a per-tool branch could regress them unnoticed--prompt— the substituted report travels the same armDocs
The npm page said the hook "prints" the report, which reads as stdout; the project README's demo redirected in a way that would now drop it. Both name the stream, and the exit-2 claim they rest on is cited rather than asserted.
Cargo.lockcarries a one-line version sync — it still recorded0.1.7after the crate moved to0.1.8, so any build regenerated it as an unrelated diff.Verification
cargo fmt --checkcleancargo clippy --all-targets -- -D warnings— 0 findingscargo test --all-targets— 101 passed, 0 failedmain.rsreverted tomaster, the stream tests fail while both pre-existing exit-code tests still passminor, because the entire observable effect of the binary changed for its primary consumerThe classifier is untouched, so no mutation run was required.
Not in scope
check.rs:40-58never reads the file, so anEditpayload is judged as if the patch fragment were the file — fragment-offset line numbers, and a comment-only fragment is acquitted outright.tests/pipeline.rs:63-67and:69-73pin that acquittal as intended behaviour, so making it honest is a design change, not a bug fix, and it is left for a separate decision. This PR is orthogonal: it only changes which stream the verdict travels on.