Hook metrics (3) a per-hook report and a crash notice - #513
Open
EdbertChan wants to merge 8 commits into
Conversation
…port.py prints one row per registered hook, with `no record` for a hook that never ran, and hook-health tells the agent in all three harnesses about failed hook runs since the last prompt. Review lane: behavior Safety invariant: A registered hook with no rows is "no record", never healthy; the notice never blocks; an unreadable log is unchecked, not clean. Effectiveness measurement: Seeded-log tests assert the table, "no record", the notice text, and the unreadable-log behavior. Slice rationale: Reading the metrics is one claim, apart from producing them in steps 1 and 2. Architectural effect: Hook failures become visible to the agent on the next prompt and to the user on demand. Goal: Create report.py and the hook-health hook with its three-harness install. Motivation: Rows nobody reads detect nothing. Alternative considerations: A Stop-hook notice was rejected because it would interrupt the reply that is already finishing; a SessionStart-only notice was rejected because a crash mid-session would wait for the next session. Implementation details: detect.py is pure (rows in, notice text out); entrypoints read the log from a per-session offset and print the notice in each harness's non-blocking output shape. Non-goals: No change to run.py, outcome.py, or wrap_installed.py; no dashboard or remote upload. Layer: contact_surface Feature state: active Files: - engine/hooks/_runner/report.py - engine/hooks/_runner/tests/test_report.py - engine/hooks/hook-health/detect.py - engine/hooks/hook-health/claude_prompt_submit.py - engine/hooks/hook-health/cursor_before_submit.py - engine/hooks/hook-health/codex_prompt_submit.py - engine/hooks/hook-health/claude.hook.json - engine/hooks/hook-health/codex.hook.json - engine/hooks/hook-health/install_claude_hook.py - engine/hooks/hook-health/install_cursor_hook.py - engine/hooks/hook-health/install_codex_hook.py - engine/hooks/hook-health/tests/test_detect.py - engine/hooks/hook-health/tests/test_hooks.py - install.sh Change types: - engine/hooks/_runner/report.py: create - engine/hooks/_runner/tests/test_report.py: create - engine/hooks/hook-health/detect.py: create - engine/hooks/hook-health/claude_prompt_submit.py: create - engine/hooks/hook-health/cursor_before_submit.py: create - engine/hooks/hook-health/codex_prompt_submit.py: create - engine/hooks/hook-health/claude.hook.json: create - engine/hooks/hook-health/codex.hook.json: create - engine/hooks/hook-health/install_claude_hook.py: create - engine/hooks/hook-health/install_cursor_hook.py: create - engine/hooks/hook-health/install_codex_hook.py: create - engine/hooks/hook-health/tests/test_detect.py: create - engine/hooks/hook-health/tests/test_hooks.py: create - install.sh: modify Acceptance criteria: - `python3 -m unittest discover -s engine/hooks/_runner/tests -v` exits 0. - `python3 -m unittest discover -s engine/hooks/hook-health/tests -v` exits 0. - `python3 scripts/check_hook_test_coverage.py` exits 0. - `bash scripts/run_all_tests.sh` exits 0. Exit code: 0 Invoker-Finalize-Id: e5a2c7d4-4239-4d64-aa80-1efc40d7fbe2
…m: engine/hooks/hook-health/README.md and a Report section in engine/hooks/_runner/README.md state what the table and the notice show. Review lane: docs Safety invariant: Only the two Markdown files change; no code, test, or config file is edited. Effectiveness measurement: Every column and message named in the prose appears in report.py or detect.py, checked by reading both. Slice rationale: Prose in its own commit so the code commit stays one claim. Architectural effect: None; prose only. Goal: Create engine/hooks/hook-health/README.md and add a Report section to engine/hooks/_runner/README.md. Motivation: Readers need the table and notice format without reading code. Alternative considerations: Code comments were rejected; the repo forbids new comments. Implementation details: Two Markdown edits. Non-goals: No code, test, or config edits. Layer: docs Feature state: active Files: - engine/hooks/hook-health/README.md - engine/hooks/_runner/README.md Change types: - engine/hooks/hook-health/README.md: create - engine/hooks/_runner/README.md: docs-only Acceptance criteria: - `test -f engine/hooks/hook-health/README.md` exits 0. Exit code: 0 Invoker-Finalize-Id: 5512a433-370f-4af9-98ce-321f2188fc66
… The repo suite, which includes the runner and hook-health tests, and the hook coverage gate pass. Review lane: proof Safety invariant: Verification is read-only and does not alter any repository file. Effectiveness measurement: The two commands are the direct measurement. Slice rationale: One focused proof before review. Architectural effect: None; verification only. Goal: Prove the report and notice behavior. Motivation: Running the tests is the proof. Alternative considerations: The full suite is included because install.sh changes. Implementation details: Run the suite and the coverage gate. Non-goals: No mutations. Layer: app_regression Feature state: active Acceptance criteria: - Exits 0 only when all pass. Exit code: 0 Invoker-Finalize-Id: 260483f4-93d9-470a-8d2a-9e4971a54b18
…ice/g0.t1.a-a6d820dc7-e8e99f8b
…No ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Layer exception: allowed -- the terminal scrub must run after every task in the workflow, including the docs task. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0. Exit code: 0 Invoker-Finalize-Id: 65a5143c-73b6-48fa-b975-b68b58cdfc3b
…aa09fa676-478048c6 — Review claim: No ephemeral inter-task handoff files remain in the worktree before the merge gate. Review lane: cleanup Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files. Effectiveness measurement: The script exits non-zero if any handoff artifact remains. Slice rationale: Required terminal scrub for every implementation workflow. Architectural effect: None; hygiene only. Goal: Leave the branch free of handoff artifacts. Motivation: Handoff files must not reach the PR. Alternative considerations: Manual cleanup was rejected as non-deterministic. Implementation details: Run scripts/scrub-handoff-artifacts.sh. Layer exception: allowed -- the terminal scrub must run after every task in the workflow, including the docs task. Non-goals: No product edits. Layer: app_regression Feature state: active Acceptance criteria: - `bash scripts/scrub-handoff-artifacts.sh` exits 0.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_e44aeb0b-4c7e-4ab6-9ff3-315d6b9e1ff8) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Each saved hook run can now be seen in one table.
The table shows how often each hook ran, what happened, how long it took, and its last error.
A short prompt message points out new hook failures in Claude, Cursor, and Codex, without stopping the prompt.
Review Claim
Review that hook failures become visible on demand and automatically on the next prompt.
Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
A registered hook with no rows is reported as
no record, never healthy; the notice never blocks prompt submission; an unreadable metrics log is reported as unchecked, not clean.Slice Rationale
This slice reads metrics produced by the earlier runner work and exposes them to users and agents, while keeping metric production unchanged.
Non-goals
No changes to
run.py,outcome.py, orwrap_installed.pybehavior, and no dashboard or remote metrics upload.Architecture
The table reader compares registered hook configuration with the metrics log on demand. Prompt entrypoints read new rows from a per-session offset, emit harness-specific context, and advance that offset.
Before
graph TD A["runner metrics log"] --> B["no reader"] C["prompt event"] --> D["no failure context"]After
graph TD A["runner metrics log"] --> B["per-hook table"] A --> C["prompt health reader"] D["registered hook configs"] --> B E["prompt event"] --> C C --> F["Claude/Cursor/Codex context notice"]Test Plan
Test Plan
bash scripts/run_all_tests.sh && python3 scripts/check_hook_test_coverage.pybash scripts/scrub-handoff-artifacts.shRevert Plan
Revert Plan
git revert <sha>Note
Low Risk
Read-only reporting plus non-blocking advisory context on prompt submit; install merges new hook entries without changing runner metric production.
Overview
This PR surfaces hook-runner metrics that were previously only written to
runs.jsonl: a on-demand per-hook report and an advisory prompt notice when wrapped hooks fail.engine/hooks/_runner/report.pyjoins installed catstack hook commands (Claude/Cursor/Codex configs viawrap_installed) with metrics rows in a--sincewindow. It prints outcome counts, p95 duration, and last error; registered hooks with no rows showno record(not “healthy”); stray log rows appear underunregistered:. Missing or bad logs exit2withuncheckedmessages.--jsonmirrors the same structure.hook-healthis a new advisory hook on prompt submit (UserPromptSubmit/beforeSubmitPrompt). Per session it tails the metrics log from a saved byte offset, and on newcrashed/timed_out/caught_errorrows for the current harness it injects a short summary (up to five failures, points atreport.py). It never blocks prompts; unreadable logs get an unchecked notice. Install links the hook for all three harnesses; docs and tests cover report CLI, detect/runtime, and install wiring.Reviewed by Cursor Bugbot for commit 605ba6a. Bugbot is set up for automated code reviews on this repo. Configure here.