From e4cd9c4017633ab7d56a2a09ebaaf61308edab7c Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Sun, 13 Sep 2026 02:05:36 -0700 Subject: [PATCH] handback-needs-attempt: flag handing the user a step the agent never tried A background-judged Stop hook. When a reply asks the user to run a command or do a step themselves, it checks the turn for an earlier attempt, and stays silent after a permission denial, a sandbox or classifier refusal, or a human-only step (password, OAuth consent, hardware). A judge result that could not be produced is reported as unchecked, never as clean. Built by Invoker workflow wf-1789281140471-18; its only failing check (check_install_effective.py) reads this machine's install and fails on clean origin/main as well. The cat-mode prose pointer is left for a follow-up PR so this one stays a single engine-runtime review unit. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_016VgzyERsebhUHizFhFqP2t Change-Id: I8fd1c26cf5fa020719f7f9452aa68acdacf2d259 --- docs/ecosystem.md | 1 + engine/hooks/handback-needs-attempt/README.md | 75 +++++++++ .../handback-needs-attempt/claude.hook.json | 16 ++ .../claude_stop_check.py | 17 +++ engine/hooks/handback-needs-attempt/detect.py | 143 ++++++++++++++++++ .../install_claude_hook.py | 38 +++++ .../tests/fixtures/handbacks.json | 6 + .../tests/test_hooks.py | 121 +++++++++++++++ .../phrases/handback-needs-attempt.json | 21 +++ install.sh | 2 + 10 files changed, 440 insertions(+) create mode 100644 engine/hooks/handback-needs-attempt/README.md create mode 100644 engine/hooks/handback-needs-attempt/claude.hook.json create mode 100644 engine/hooks/handback-needs-attempt/claude_stop_check.py create mode 100644 engine/hooks/handback-needs-attempt/detect.py create mode 100644 engine/hooks/handback-needs-attempt/install_claude_hook.py create mode 100644 engine/hooks/handback-needs-attempt/tests/fixtures/handbacks.json create mode 100644 engine/hooks/handback-needs-attempt/tests/test_hooks.py create mode 100644 engine/hooks/llm-judge/phrases/handback-needs-attempt.json diff --git a/docs/ecosystem.md b/docs/ecosystem.md index 32b391fa..5c456db7 100644 --- a/docs/ecosystem.md +++ b/docs/ecosystem.md @@ -73,6 +73,7 @@ again. | `demo-freeze` | hook | | `explicit-failures` | hook (advisory; always on) | | `external-claim-gate` | hook (PreToolUse on `Bash`; blocks a gh issue/comment/release/api write whose body claims a cause or fix with no evidence; blocks as UNCHECKED when the body cannot be read) | +| `handback-needs-attempt` | hook (Stop; background-judges user hand-backs that lacked an agent attempt) | | `playbook-router` | hook (UserPromptSubmit; injects the steps of the one playbook a prompt names) | | `diu-stop` | hook (Stop; blocks a reply over the word limit or with an unproven claim, and asks the background judge whether the reply used wording from its `phrases/` lists, waiting for that answer so a hit blocks the same turn) | | `frustration-watchdog` | hook | diff --git a/engine/hooks/handback-needs-attempt/README.md b/engine/hooks/handback-needs-attempt/README.md new file mode 100644 index 00000000..51258a8e --- /dev/null +++ b/engine/hooks/handback-needs-attempt/README.md @@ -0,0 +1,75 @@ +# handback-needs-attempt + +On Claude `Stop`, flag a reply that hands the user a command or actionable +step the agent could have attempted when the current turn has no attempt before +the hand-back. + +This is the mechanical backstop for cat-mode's rule that a hand-back is an +unverified claim. The hook does not decide that from local wording rules. On +every Stop it hands the latest reply and turn exchange to the background judge +using +[`engine/hooks/llm-judge/phrases/handback-needs-attempt.json`](../llm-judge/phrases/handback-needs-attempt.json). +A hit arrives on a later turn through the shared [`llm-judge`](../llm-judge/README.md) +inbox and carries the dictionary's `on_hit` text. The live reply is never held +up. If the judge result was unchecked, the inbox reports "could not judge" +instead of treating the reply as clean. Fail-open. + +Stay silent after a permission denial, sandbox refusal, classifier refusal, or a +step that only the human can do, such as entering a password, granting OAuth +consent, approving a hardware prompt, or connecting hardware. Quoted or +documented instructions are not hand-backs. + +When prompted, attempt the command or step first and report the result. If it +truly requires the human, name the permission denial, sandbox or classifier +refusal, password, OAuth consent, or hardware requirement that makes it +human-only. `stop_hook_active` is a one-rewrite escape hatch. + +## Model-judged path + +`enqueue_judge` in `detect.py` reads the transcript, takes the current reply, +builds a phrase-dictionary job from the latest exchange, and sends it to +`llm-judge`. The dictionary defines the meaning with `match` and `not_match` +examples and supplies the static `on_hit` follow-up text. + +No job is sent when `stop_hook_active` is set, when this transcript or reply was +already prompted, or when the reply is empty. Inside a judge run +(`CATSTACK_LLM_JUDGE_CHILD=1`) `llm-judge` refuses the job. + +The model call runs in a detached background process, so the reply is never held +up. Runners are tried in `llm-judge` order: `codex` (gpt-5.3-codex-spark), then +`claude` (haiku, hooks off), then `cursor-agent`, first answer wins. + +The verdict reports one turn later. On the next prompt the `llm-judge` inbox +shows a hit as the dictionary's `on_hit` text. If no runner could answer, or the +result could not be checked, the inbox says "could not judge" instead of staying +quiet. A clean verdict shows nothing. + +`llm-judge` is loaded from the sibling folder (`../llm-judge/judge.py`), which +sits next to this one in the repo and in each harness's `hooks/` folder. If the +hook input is malformed JSON, the wrapper returns quietly. If judge enqueue +raises, the hook writes `catstack-hook-error handback-needs-attempt: ` to +stderr and otherwise stays fail-open. + +To grow coverage, add the real text of any miss to the dictionary's `match` +phrases, or the real text of any false alarm to `not_match`. Do not add a +pattern to this hook; the prose meaning belongs in the phrase dictionary. + +## Files + +- `detect.py` - judge enqueue + once-per-transcript state +- `claude_stop_check.py` - Claude `Stop` wrapper +- `claude.hook.json` - Claude hook fragment +- `install_claude_hook.py` - idempotent Claude settings merge +- `tests/` - positive, exception, fail-open, install, and unchecked outcomes +- `../llm-judge/phrases/handback-needs-attempt.json` - phrase dictionary + +## Install + +`./install.sh` from the repo root. Restart the harness. + +## Tests + +```sh +python3 -m unittest discover -s engine/hooks/handback-needs-attempt/tests -v +python3 scripts/check_hook_test_coverage.py engine/hooks/handback-needs-attempt +``` diff --git a/engine/hooks/handback-needs-attempt/claude.hook.json b/engine/hooks/handback-needs-attempt/claude.hook.json new file mode 100644 index 00000000..8daad159 --- /dev/null +++ b/engine/hooks/handback-needs-attempt/claude.hook.json @@ -0,0 +1,16 @@ +{ + "hooks": { + "Stop": [ + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": "python3 $HOME/.claude/hooks/handback-needs-attempt/claude_stop_check.py", + "timeout": 10 + } + ] + } + ] + } +} diff --git a/engine/hooks/handback-needs-attempt/claude_stop_check.py b/engine/hooks/handback-needs-attempt/claude_stop_check.py new file mode 100644 index 00000000..f34cb3b5 --- /dev/null +++ b/engine/hooks/handback-needs-attempt/claude_stop_check.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 +import json +import sys + +from detect import try_enqueue_judge + + +def main() -> None: + try: + payload = json.load(sys.stdin) + except (json.JSONDecodeError, OSError): + return + try_enqueue_judge(payload if isinstance(payload, dict) else {}) + + +if __name__ == "__main__": + main() diff --git a/engine/hooks/handback-needs-attempt/detect.py b/engine/hooks/handback-needs-attempt/detect.py new file mode 100644 index 00000000..019a7e94 --- /dev/null +++ b/engine/hooks/handback-needs-attempt/detect.py @@ -0,0 +1,143 @@ +from __future__ import annotations + +import functools +import hashlib +import importlib.util +import json +import os +import sys +import uuid + +HOOKS_DIR = os.path.dirname(os.path.abspath(__file__)) +LLM_JUDGE_PATH = os.path.join(os.path.dirname(HOOKS_DIR), "llm-judge", "judge.py") +PHRASES_PATH = os.path.join(os.path.dirname(HOOKS_DIR), "llm-judge", "phrases.py") +STATE_DIR = os.environ.get( + "HANDBACK_NEEDS_ATTEMPT_STATE_DIR", + os.path.join(os.path.expanduser("~"), ".cache", "catstack-handback-needs-attempt"), +) + + +def _state_file(key: str) -> str: + digest = hashlib.sha1(key.encode()).hexdigest()[:16] + return os.path.join(STATE_DIR, f"{digest}.prompted") + + +def already_prompted(key: str) -> bool: + return os.path.isfile(_state_file(key)) + + +def mark_prompted(key: str) -> None: + os.makedirs(STATE_DIR, exist_ok=True) + with open(_state_file(key), "w", encoding="utf-8") as handle: + handle.write(key + "\n") + + +def _text(data: dict) -> str: + message = data.get("message") + content = message.get("content") if isinstance(message, dict) else data.get("content") + if isinstance(content, str): + return content + if isinstance(content, list): + parts = [] + for block in content: + if isinstance(block, str): + parts.append(block) + elif isinstance(block, dict): + if block.get("type") == "text": + parts.append(block.get("text") or "") + elif block.get("type") in {"tool_use", "tool_result"}: + parts.append(json.dumps(block, ensure_ascii=False, sort_keys=True)) + return "\n".join(parts) + return "" + + +def _human_user(data: dict) -> bool: + if data.get("type") != "user": + return False + message = data.get("message") + content = message.get("content") if isinstance(message, dict) else data.get("content") + if isinstance(content, str): + return bool(content.strip()) and not content.lstrip().startswith((" str: + for key in ("agent_transcript_path", "transcript_path", "transcriptPath"): + value = payload.get(key) + if isinstance(value, str) and os.path.isfile(value): + return value + return "" + + +def exchange_from_transcript(path: str) -> str: + if not path or not os.path.isfile(path): + return "" + try: + with open(path, encoding="utf-8") as handle: + lines = [json.loads(line) for line in handle if line.strip()] + except (OSError, json.JSONDecodeError): + return "" + start = 0 + for index, data in enumerate(lines): + if isinstance(data, dict) and _human_user(data): + start = index + return "\n".join(_text(data) for data in lines[start:] if isinstance(data, dict)) + + +def last_assistant_text(payload: dict, path: str) -> str: + for key in ("last_assistant_message", "last-assistant-message", "lastAssistantMessage"): + value = payload.get(key) + if isinstance(value, str) and value.strip(): + return value + exchange = exchange_from_transcript(path) + return exchange + + +def decide(_payload: dict) -> None: + return None + + +@functools.cache +def _judge(): + spec = importlib.util.spec_from_file_location("llm_judge", LLM_JUDGE_PATH) + if spec is None or spec.loader is None: + raise ImportError(LLM_JUDGE_PATH) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@functools.cache +def _phrases(): + spec = importlib.util.spec_from_file_location("llm_judge_phrases", PHRASES_PATH) + if spec is None or spec.loader is None: + raise ImportError(PHRASES_PATH) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def enqueue_judge(payload: dict) -> str | None: + if not isinstance(payload, dict) or payload.get("stop_hook_active"): + return None + path = resolve_transcript(payload) + reply = last_assistant_text(payload, path) + exchange = exchange_from_transcript(path) + text = "\n\nLATEST REPLY:\n" + reply + ("\n\nTURN EXCHANGE:\n" + exchange if exchange else "") + key = path or text[:200] + if not reply.strip() or already_prompted(key): + return None + dictionary = _phrases().load("handback-needs-attempt") + job = _phrases().job(dictionary, path, text) + job["id"] = uuid.uuid4().hex + job_id = _judge().enqueue(job) + if job_id is not None: + mark_prompted(key) + return job_id + + +def try_enqueue_judge(payload: dict) -> None: + try: + enqueue_judge(payload) + except Exception as exc: + print(f"catstack-hook-error handback-needs-attempt: {type(exc).__name__}: {exc}", file=sys.stderr) diff --git a/engine/hooks/handback-needs-attempt/install_claude_hook.py b/engine/hooks/handback-needs-attempt/install_claude_hook.py new file mode 100644 index 00000000..8a63bd71 --- /dev/null +++ b/engine/hooks/handback-needs-attempt/install_claude_hook.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python3 +import json +import os + +HERE = os.path.dirname(os.path.abspath(__file__)) +SETTINGS_PATH = os.path.expanduser("~/.claude/settings.json") +FRAGMENT_PATH = os.path.join(HERE, "claude.hook.json") +MARKER = "handback-needs-attempt/claude_stop_check.py" +EVENT = "Stop" + + +def _is_ours(entry: dict) -> bool: + return any(MARKER in h.get("command", "") for h in entry.get("hooks", [])) + + +def merge_hook(settings: dict, fragment: dict) -> bool: + entries = settings.setdefault("hooks", {}).setdefault(EVENT, []) + incoming = fragment.get("hooks", {}).get(EVENT, []) + before = json.dumps(entries, sort_keys=True) + entries[:] = [entry for entry in entries if not _is_ours(entry)] + incoming + return json.dumps(entries, sort_keys=True) != before + + +def main() -> None: + settings = {} + if os.path.exists(SETTINGS_PATH): + with open(SETTINGS_PATH, encoding="utf-8") as handle: + settings = json.load(handle) + with open(FRAGMENT_PATH, encoding="utf-8") as handle: + fragment = json.load(handle) + if not merge_hook(settings, fragment): + print("ok claude Stop handback-needs-attempt already up to date") + return + os.makedirs(os.path.dirname(SETTINGS_PATH), exist_ok=True) + with open(SETTINGS_PATH, "w", encoding="utf-8") as handle: + json.dump(settings, handle, indent=2) + handle.write("\n") + print("link claude Stop handback-needs-attempt merged into settings.json") diff --git a/engine/hooks/handback-needs-attempt/tests/fixtures/handbacks.json b/engine/hooks/handback-needs-attempt/tests/fixtures/handbacks.json new file mode 100644 index 00000000..ba884ebf --- /dev/null +++ b/engine/hooks/handback-needs-attempt/tests/fixtures/handbacks.json @@ -0,0 +1,6 @@ +[ + {"name": "command handback", "reply": "Please run: invoker-cli setup slack ... then tell me when it completes", "match": true}, + {"name": "xcode handback", "reply": "Simulator build still passes. Now on your end in Xcode:", "match": true}, + {"name": "permission denial", "reply": "Permission to use Bash was denied, so please run the command yourself.", "match": false}, + {"name": "oauth consent", "reply": "Please complete the OAuth consent screen on your end.", "match": false} +] diff --git a/engine/hooks/handback-needs-attempt/tests/test_hooks.py b/engine/hooks/handback-needs-attempt/tests/test_hooks.py new file mode 100644 index 00000000..6b89f967 --- /dev/null +++ b/engine/hooks/handback-needs-attempt/tests/test_hooks.py @@ -0,0 +1,121 @@ +from __future__ import annotations + +import io +import json +import os +import sys +import tempfile +import time +import unittest +from contextlib import redirect_stderr +from unittest.mock import patch + +HOOK_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, HOOK_DIR) +import claude_stop_check +import detect +import install_claude_hook + +JUDGE_DIR = os.path.join(os.path.dirname(HOOK_DIR), "llm-judge") +sys.path.insert(0, JUDGE_DIR) +import inbox +import judge + +FIXTURES = os.path.join(HOOK_DIR, "tests", "fixtures", "handbacks.json") +PY = sys.executable +ANSWER_RUNNER = ["fake", [PY, "-c", "import json,sys; p=sys.argv[-1].lower().split('text:\\n')[-1]; print(json.dumps({'match': not any(x in p for x in ['permission to use', 'oauth consent', 'password', 'physical device']), 'closest': ''}))", "{prompt}"]] + + +def transcript(reply: str, denial: str = "") -> str: + lines = [{"type": "user", "message": {"role": "user", "content": "finish setup"}}] + if denial: + lines.extend([ + {"type": "assistant", "message": {"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "Bash", "input": {"command": "invoker-cli setup slack"}}]}}, + {"type": "user", "message": {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1", "is_error": True, "content": denial}]}} + ]) + lines.append({"type": "assistant", "message": {"role": "assistant", "content": reply}}) + handle = tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False, encoding="utf-8") + handle.write("\n".join(json.dumps(line) for line in lines) + "\n") + handle.close() + return handle.name + + +class TestHandbackNeedsAttempt(unittest.TestCase): + def setUp(self): + self.judge_state = tempfile.TemporaryDirectory() + self.hook_state = tempfile.TemporaryDirectory() + self.env = patch.dict(os.environ, {judge.STATE_ENV: self.judge_state.name, judge.RUNNERS_ENV: json.dumps([ANSWER_RUNNER]), "HANDBACK_NEEDS_ATTEMPT_STATE_DIR": self.hook_state.name}) + self.env.start() + detect.STATE_DIR = self.hook_state.name + detect._judge.cache_clear() + detect._phrases.cache_clear() + + def tearDown(self): + deadline = time.monotonic() + 10 + while os.path.isdir(os.path.join(self.judge_state.name, "jobs")) and os.listdir(os.path.join(self.judge_state.name, "jobs")) and time.monotonic() < deadline: + time.sleep(0.05) + self.env.stop() + self.judge_state.cleanup() + self.hook_state.cleanup() + detect._judge.cache_clear() + detect._phrases.cache_clear() + + def run_case(self, reply: str, denial: str = "") -> list[str]: + path = transcript(reply, denial) + try: + detect.enqueue_judge({"transcript_path": path, "last_assistant_message": reply}) + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + messages = inbox.messages(path) + if messages: + return messages + time.sleep(0.05) + return inbox.messages(path) + finally: + os.unlink(path) + + def test_fires_on_command_handback_without_attempt(self): + with open(FIXTURES, encoding="utf-8") as handle: + case = json.load(handle)[0] + messages = self.run_case(case["reply"]) + self.assertEqual(len(messages), 1) + self.assertIn("handback-needs-attempt", messages[0]) + + def test_fires_on_xcode_handback_without_attempt(self): + messages = self.run_case("Simulator build still passes. Now on your end in Xcode:") + self.assertEqual(len(messages), 1) + + def test_stays_silent_after_permission_denial(self): + self.assertEqual(self.run_case("Please run the command yourself.", "Permission to use Bash was denied"), []) + + def test_stays_silent_for_oauth_consent(self): + self.assertEqual(self.run_case("Please complete the OAuth consent screen on your end."), []) + + def test_judge_failure_is_unchecked(self): + os.environ[judge.RUNNERS_ENV] = json.dumps([["missing", ["catstack-no-such-runner", "{prompt}"]]]) + messages = self.run_case("Please run the command yourself.") + self.assertEqual(len(messages), 1) + self.assertIn("could not judge", messages[0]) + + def test_stop_hook_is_fail_open_and_returns_early(self): + err = io.StringIO() + with patch.object(sys, "stdin", io.StringIO(json.dumps({"stop_hook_active": True, "last_assistant_message": "Please run it"}))): + with redirect_stderr(err): + claude_stop_check.main() + self.assertEqual(err.getvalue(), "") + + def test_unreadable_transcript_fails_open(self): + self.assertIsNone(detect.enqueue_judge({"transcript_path": "/no/such/transcript", "last_assistant_message": ""})) + + def test_install_merges_stop_idempotently(self): + with open(install_claude_hook.FRAGMENT_PATH, encoding="utf-8") as handle: + fragment = json.load(handle) + settings = {"hooks": {"Stop": [{"matcher": "*", "hooks": [{"type": "command", "command": "python3 keep.py"}]}]}} + self.assertTrue(install_claude_hook.merge_hook(settings, fragment)) + self.assertFalse(install_claude_hook.merge_hook(settings, fragment)) + commands = [h["command"] for e in settings["hooks"]["Stop"] for h in e["hooks"]] + self.assertIn("handback-needs-attempt/claude_stop_check.py", commands[-1]) + + +if __name__ == "__main__": + unittest.main() diff --git a/engine/hooks/llm-judge/phrases/handback-needs-attempt.json b/engine/hooks/llm-judge/phrases/handback-needs-attempt.json new file mode 100644 index 00000000..91ff5f00 --- /dev/null +++ b/engine/hooks/llm-judge/phrases/handback-needs-attempt.json @@ -0,0 +1,21 @@ +{ + "checker": "handback-needs-attempt", + "reads": "exchange", + "meaning": "The latest assistant reply hands the user a command or actionable step that the agent could have attempted, and the current turn contains no prior attempt of that step and no earlier permission denial, sandbox or classifier refusal, or human-only requirement.", + "match": [ + "Please run: invoker-cli setup slack ... then tell me when it completes", + "Simulator build still passes. Now on your end in Xcode:", + "Run the setup command yourself and send me the output.", + "Open the app and finish this configuration on your machine." + ], + "not_match": [ + "Permission to use Bash was denied, so please run the command yourself.", + "The sandbox refused access to the simulator; please open Xcode and approve it.", + "Please enter your password or complete the OAuth consent screen.", + "Connect the physical device and tap Trust; that requires your hardware.", + "I ran invoker-cli setup slack and it failed; please inspect the error.", + "Please run this in the example below; I am quoting instructions for documentation.", + "I cannot run that command because the classifier refused it." + ], + "on_hit": "handback-needs-attempt: this reply hands back a step the agent could have attempted without trying it. Attempt the command or step first and report the result; if it truly requires the human, name the password, OAuth consent, hardware, or preceding permission/sandbox/classifier refusal that makes it human-only." +} diff --git a/install.sh b/install.sh index 64e6b210..86ddb7d5 100755 --- a/install.sh +++ b/install.sh @@ -230,6 +230,7 @@ link_item "auto-pr" "$REPO_DIR/engine/hooks/auto-pr" "$HOME/.claude/hooks/auto-p link_item "pr-schema-gate" "$REPO_DIR/engine/hooks/pr-schema-gate" "$HOME/.claude/hooks/pr-schema-gate" link_item "external-claim-gate" "$REPO_DIR/engine/hooks/external-claim-gate" "$HOME/.claude/hooks/external-claim-gate" link_item "wrong-check-reflect" "$REPO_DIR/engine/hooks/wrong-check-reflect" "$HOME/.claude/hooks/wrong-check-reflect" +link_item "handback-needs-attempt" "$REPO_DIR/engine/hooks/handback-needs-attempt" "$HOME/.claude/hooks/handback-needs-attempt" link_item "llm-judge" "$REPO_DIR/engine/hooks/llm-judge" "$HOME/.claude/hooks/llm-judge" link_item "hook-health" "$REPO_DIR/engine/hooks/hook-health" "$HOME/.claude/hooks/hook-health" link_item "build-the-lever" "$REPO_DIR/engine/hooks/build-the-lever" "$HOME/.claude/hooks/build-the-lever" @@ -349,6 +350,7 @@ python3 "$REPO_DIR/engine/hooks/auto-pr/install_claude_hook.py" python3 "$REPO_DIR/engine/hooks/pr-schema-gate/install_claude_hook.py" python3 "$REPO_DIR/engine/hooks/external-claim-gate/install_claude_hook.py" python3 "$REPO_DIR/engine/hooks/wrong-check-reflect/install_claude_hook.py" +python3 "$REPO_DIR/engine/hooks/handback-needs-attempt/install_claude_hook.py" python3 "$REPO_DIR/engine/hooks/llm-judge/install_claude_hook.py" python3 "$REPO_DIR/engine/hooks/hook-health/install_claude_hook.py" python3 "$REPO_DIR/engine/hooks/build-the-lever/install_claude_hook.py"