Skip to content

Hook metrics (3) a per-hook report and a crash notice - #513

Open
EdbertChan wants to merge 8 commits into
plan/hook-metrics-2-install-wraps-every-hook-with-the-runnerfrom
plan/hook-metrics-3-a-per-hook-report-and-a-crash-notice
Open

Hook metrics (3) a per-hook report and a crash notice#513
EdbertChan wants to merge 8 commits into
plan/hook-metrics-2-install-wraps-every-hook-with-the-runnerfrom
plan/hook-metrics-3-a-per-hook-report-and-a-crash-notice

Conversation

@EdbertChan

@EdbertChan EdbertChan commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Summary

Each saved hook run can now be seen in one table.

The table shows how often each hook ran, what happened, how long it took, and its last error.

A short prompt message points out new hook failures in Claude, Cursor, and Codex, without stopping the prompt.

Review Claim

Review that hook failures become visible on demand and automatically on the next prompt.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

A registered hook with no rows is reported as no record, never healthy; the notice never blocks prompt submission; an unreadable metrics log is reported as unchecked, not clean.

Slice Rationale

This slice reads metrics produced by the earlier runner work and exposes them to users and agents, while keeping metric production unchanged.

Non-goals

No changes to run.py, outcome.py, or wrap_installed.py behavior, and no dashboard or remote metrics upload.

Architecture

The table reader compares registered hook configuration with the metrics log on demand. Prompt entrypoints read new rows from a per-session offset, emit harness-specific context, and advance that offset.

Before

graph TD
    A["runner metrics log"] --> B["no reader"]
    C["prompt event"] --> D["no failure context"]
Loading

After

graph TD
    A["runner metrics log"] --> B["per-hook table"]
    A --> C["prompt health reader"]
    D["registered hook configs"] --> B
    E["prompt event"] --> C
    C --> F["Claude/Cursor/Codex context notice"]
Loading

Test Plan

Test Plan
  • bash scripts/run_all_tests.sh && python3 scripts/check_hook_test_coverage.py
  • bash scripts/scrub-handoff-artifacts.sh

Revert Plan

Revert Plan
  • Safe to revert? Yes
  • Revert command: git revert <sha>
  • Post-revert steps: None
  • Data migration? No

Note

Low Risk
Read-only reporting plus non-blocking advisory context on prompt submit; install merges new hook entries without changing runner metric production.

Overview
This PR surfaces hook-runner metrics that were previously only written to runs.jsonl: a on-demand per-hook report and an advisory prompt notice when wrapped hooks fail.

engine/hooks/_runner/report.py joins installed catstack hook commands (Claude/Cursor/Codex configs via wrap_installed) with metrics rows in a --since window. It prints outcome counts, p95 duration, and last error; registered hooks with no rows show no record (not “healthy”); stray log rows appear under unregistered:. Missing or bad logs exit 2 with unchecked messages. --json mirrors the same structure.

hook-health is a new advisory hook on prompt submit (UserPromptSubmit / beforeSubmitPrompt). Per session it tails the metrics log from a saved byte offset, and on new crashed / timed_out / caught_error rows for the current harness it injects a short summary (up to five failures, points at report.py). It never blocks prompts; unreadable logs get an unchecked notice. Install links the hook for all three harnesses; docs and tests cover report CLI, detect/runtime, and install wiring.

Reviewed by Cursor Bugbot for commit 605ba6a. Bugbot is set up for automated code reviews on this repo. Configure here.

CI Bot and others added 8 commits September 12, 2026 21:27
…port.py prints one row per registered hook, with `no record` for a hook that never ran, and hook-health tells the agent in all three harnesses about failed hook runs since the last prompt.

Review lane: behavior
Safety invariant: A registered hook with no rows is "no record", never healthy; the notice never blocks; an unreadable log is unchecked, not clean.
Effectiveness measurement: Seeded-log tests assert the table, "no record", the notice text, and the unreadable-log behavior.
Slice rationale: Reading the metrics is one claim, apart from producing them in steps 1 and 2.
Architectural effect: Hook failures become visible to the agent on the next prompt and to the user on demand.
Goal: Create report.py and the hook-health hook with its three-harness install.
Motivation: Rows nobody reads detect nothing.
Alternative considerations: A Stop-hook notice was rejected because it would interrupt the reply that is already finishing; a SessionStart-only notice was rejected because a crash mid-session would wait for the next session.
Implementation details: detect.py is pure (rows in, notice text out); entrypoints read the log from a per-session offset and print the notice in each harness's non-blocking output shape.
Non-goals: No change to run.py, outcome.py, or wrap_installed.py; no dashboard or remote upload.
Layer: contact_surface
Feature state: active
Files:
- engine/hooks/_runner/report.py
- engine/hooks/_runner/tests/test_report.py
- engine/hooks/hook-health/detect.py
- engine/hooks/hook-health/claude_prompt_submit.py
- engine/hooks/hook-health/cursor_before_submit.py
- engine/hooks/hook-health/codex_prompt_submit.py
- engine/hooks/hook-health/claude.hook.json
- engine/hooks/hook-health/codex.hook.json
- engine/hooks/hook-health/install_claude_hook.py
- engine/hooks/hook-health/install_cursor_hook.py
- engine/hooks/hook-health/install_codex_hook.py
- engine/hooks/hook-health/tests/test_detect.py
- engine/hooks/hook-health/tests/test_hooks.py
- install.sh
Change types:
- engine/hooks/_runner/report.py: create
- engine/hooks/_runner/tests/test_report.py: create
- engine/hooks/hook-health/detect.py: create
- engine/hooks/hook-health/claude_prompt_submit.py: create
- engine/hooks/hook-health/cursor_before_submit.py: create
- engine/hooks/hook-health/codex_prompt_submit.py: create
- engine/hooks/hook-health/claude.hook.json: create
- engine/hooks/hook-health/codex.hook.json: create
- engine/hooks/hook-health/install_claude_hook.py: create
- engine/hooks/hook-health/install_cursor_hook.py: create
- engine/hooks/hook-health/install_codex_hook.py: create
- engine/hooks/hook-health/tests/test_detect.py: create
- engine/hooks/hook-health/tests/test_hooks.py: create
- install.sh: modify
Acceptance criteria:
- `python3 -m unittest discover -s engine/hooks/_runner/tests -v` exits 0.
- `python3 -m unittest discover -s engine/hooks/hook-health/tests -v` exits 0.
- `python3 scripts/check_hook_test_coverage.py` exits 0.
- `bash scripts/run_all_tests.sh` exits 0.

Exit code: 0
Invoker-Finalize-Id: e5a2c7d4-4239-4d64-aa80-1efc40d7fbe2
…m: engine/hooks/hook-health/README.md and a Report section in engine/hooks/_runner/README.md state what the table and the notice show.

Review lane: docs
Safety invariant: Only the two Markdown files change; no code, test, or config file is edited.
Effectiveness measurement: Every column and message named in the prose appears in report.py or detect.py, checked by reading both.
Slice rationale: Prose in its own commit so the code commit stays one claim.
Architectural effect: None; prose only.
Goal: Create engine/hooks/hook-health/README.md and add a Report section to engine/hooks/_runner/README.md.
Motivation: Readers need the table and notice format without reading code.
Alternative considerations: Code comments were rejected; the repo forbids new comments.
Implementation details: Two Markdown edits.
Non-goals: No code, test, or config edits.
Layer: docs
Feature state: active
Files:
- engine/hooks/hook-health/README.md
- engine/hooks/_runner/README.md
Change types:
- engine/hooks/hook-health/README.md: create
- engine/hooks/_runner/README.md: docs-only
Acceptance criteria:
- `test -f engine/hooks/hook-health/README.md` exits 0.

Exit code: 0
Invoker-Finalize-Id: 5512a433-370f-4af9-98ce-321f2188fc66
… The repo suite, which includes the runner and hook-health tests, and the hook coverage gate pass.

Review lane: proof
Safety invariant: Verification is read-only and does not alter any repository file.
Effectiveness measurement: The two commands are the direct measurement.
Slice rationale: One focused proof before review.
Architectural effect: None; verification only.
Goal: Prove the report and notice behavior.
Motivation: Running the tests is the proof.
Alternative considerations: The full suite is included because install.sh changes.
Implementation details: Run the suite and the coverage gate.
Non-goals: No mutations.
Layer: app_regression
Feature state: active
Acceptance criteria:
- Exits 0 only when all pass.

Exit code: 0
Invoker-Finalize-Id: 260483f4-93d9-470a-8d2a-9e4971a54b18
…No ephemeral inter-task handoff files remain in the worktree before the merge gate.

Review lane: cleanup
Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files.
Effectiveness measurement: The script exits non-zero if any handoff artifact remains.
Slice rationale: Required terminal scrub for every implementation workflow.
Architectural effect: None; hygiene only.
Goal: Leave the branch free of handoff artifacts.
Motivation: Handoff files must not reach the PR.
Alternative considerations: Manual cleanup was rejected as non-deterministic.
Implementation details: Run scripts/scrub-handoff-artifacts.sh.
Layer exception: allowed -- the terminal scrub must run after every task in the workflow, including the docs task.
Non-goals: No product edits.
Layer: app_regression
Feature state: active
Acceptance criteria:
- `bash scripts/scrub-handoff-artifacts.sh` exits 0.

Exit code: 0
Invoker-Finalize-Id: 65a5143c-73b6-48fa-b975-b68b58cdfc3b
…aa09fa676-478048c6 — Review claim: No ephemeral inter-task handoff files remain in the worktree before the merge gate.

Review lane: cleanup
Safety invariant: The scrub script only checks for known handoff artifact names and never touches source, tests, or other repository files.
Effectiveness measurement: The script exits non-zero if any handoff artifact remains.
Slice rationale: Required terminal scrub for every implementation workflow.
Architectural effect: None; hygiene only.
Goal: Leave the branch free of handoff artifacts.
Motivation: Handoff files must not reach the PR.
Alternative considerations: Manual cleanup was rejected as non-deterministic.
Implementation details: Run scripts/scrub-handoff-artifacts.sh.
Layer exception: allowed -- the terminal scrub must run after every task in the workflow, including the docs task.
Non-goals: No product edits.
Layer: app_regression
Feature state: active
Acceptance criteria:
- `bash scripts/scrub-handoff-artifacts.sh` exits 0.
@cursor

cursor Bot commented Sep 12, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_e44aeb0b-4c7e-4ab6-9ff3-315d6b9e1ff8)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant