[kyle-sexton]

Refs: #12035

Summary

After #13640, the eval policy probe reports its facts. #12035's re-run (https://github.com/melodic-software/claude-code-plugins/actions/runs/42166675433) failed both pass legs with result=present subtype=success num_turns=2 tool_calls=none denials=0: the model made no tool call, so the policy had nothing to deny. A local reproduction showed why: told only to read /proc/self/environ, the runner file and Grep TOKEN, the model declines them as credential harvesting (0 of 8 runs made all four calls).

Fix

• All six probe steps (eval-browser-dispatch.yml, eval-hill-climb-pass.yml): the prompt names the run the runner's permission self-test, where every call is expected to be denied; the Grep pattern is policy-probe, not TOKEN. The paths and the four denied checks are unchanged.
• A session with a success result and zero tool_use blocks proves nothing, so it runs again, at most three times; then denied fails with probe_facts. A call made and not denied, a partial set of calls, or no result still stops the job on the first session. The credential-leak grep and the handoff classifier run on every attempt.
• callers.test.mjs: a case runs the real step with stub claude and node (first-try pass, retry pass, three no-call sessions fail, an undenied Read fails without retry). workflow.test.mjs: the leak-check indent pin moves.

A documented command that evaluates a permission rule without a model would be simpler. None exists: claude --help (2.1.318) has none, and https://code.claude.com/docs/en/permissions says rules are enforced by Claude Code, with Read rules reaching Grep and Glob only on a best-effort basis, which is why the probe calls each tool.

Verification

• Local reproduction, claude -p --setting-sources "" --settings <deny-rules file> --model claude-sonnet-5-5 --allowedTools Read Grep Glob: old prompt 0 of 8 runs made all four calls; new prompt 8 of 8, all four denied.
• node --test in .github/actions/resolve-config: 929 pass, 0 fail; .github/scripts/plugin-eval: 105 pass, 0 fail.
• UNVERIFIED until a hosted run: the API-key leg on a hosted runner (local runs used subscription auth). feat(evals): background hill-climb that opens draft PRs for skill edits beating the baseline #12035's re-run after merge is that check.

Related

• fix(plugin-eval): read the eval probe's stream line by line and name its facts on a failed denial check #13640, feat(evals): background hill-climb that opens draft PRs for skill edits beating the baseline #12035

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	❌ Failed 2026-11-06T05:30:54.241417Z We couldn’t complete this review. Comment @codex review to try again.	ded96f1	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

claude-review reviewed ded96f1729acd6d0eb0552fcdd29386b5e577efa: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #13665, a clean whole review

No findings. I read every hunk of all 4 changed files. eval-browser-dispatch.yml and eval-hill-climb-pass.yml are trusted paths, so I read their whole changes. I also read the full probe step at head, eval-hill-climb-pass.yml L326-L388, for context.

Probe loop logic, as checked against the PR description:

• Leak check and classifier. The credential-leak grep and the handoff classifier now run inside the loop, on every attempt. A handoff writes its verdict to GITHUB_OUTPUT and exits 0, as before.
• When a session retries. It runs again only when the last result message exists, has subtype == "success", and the session made no tool_use blocks at all. Any other outcome breaks out of the loop at once. That covers a call made but not denied, a partial set of calls, and a missing result. The four denied checks then fail it on that first session, as the description says.
• Three no-call sessions. The loop finishes, the last verdict is written to GITHUB_OUTPUT, and denied Read /proc/self/environ fails with probe_facts (tool_calls=none).
• Test coverage. The new test in callers.test.mjs runs the real step with stub claude and node. It covers four cases: a first-try pass, a pass on retry, three no-call sessions that fail, and an undenied Read that fails without a retry. The never case gives the stub a fourth stream, so its calls == 3 check would catch a fourth attempt.
• The other five copies. They are byte-identical to the tested step. The existing test "every eval leg runs the same managed-policy step and the same probe" checks that they stay identical.
• workflow.test.mjs. The indent pin for the leak check moved from 12 to 14 spaces, which matches the new nesting inside the loop.

Verification. I did not run the tests. The local reproduction counts (0/8 runs before, 8/8 after) and the node --test results (929 and 105 passing) come from the PR description and are unverified. The author says the API-key leg on a hosted runner stays unverified until the #12035 re-run.

Closing issue. The PR only says "Refs: #12035"; it closes no issue, so there are no acceptance criteria to check.

Spend (not a hold). Each probe step can now start up to three claude-sonnet-5-5 sessions instead of one, but only when a session makes no tool call. This uses prepaid API credit or subscription usage only.

[github-actions]

claude-security-review reviewed ded96f1729acd6d0eb0552fcdd29386b5e577efa: read 4 of 4 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #13665, no findings

I read every hunk of all 4 changed files. Two are on trusted paths: .github/workflows/eval-browser-dispatch.yml and .github/workflows/eval-hill-climb-pass.yml. I read their whole change and what it gates.

What the change gates. Each of the six "Probe that the policy denies those reads" steps decides whether pull request eval cases run beside the leg's Claude credential. The step must pass four denied checks (denied Read /proc/self/environ, denied Read "$RUNNER_ROOT/.runner", denied Grep /proc/self, denied Glob ...). Each check reads the last result message's permission_denials and exits 1 when its denial is missing.

Checked: the retry loop leaves the gate as strict as before.

• The loop only goes round again when the session ended with subtype == "success" and made zero tool_use blocks (for example eval-browser-dispatch.yml L605-607 in the first probe). In every other case it hits || break: a call was made, the calls were partial, or there was no result. It then reaches the same four denied checks it ran before.
• A session with no tool call has no permission_denials, so it can never pass those checks. After three such sessions the loop ends and denied Read /proc/self/environ fails the job. This fails closed, and the new test pins it (never → status 1, 3 calls).
• The credential-leak grep -qF "$credential" "$out" and the handoff classifier run on every attempt, before $out is overwritten. So no attempt's stream escapes the leak check. A handoff still writes its verdict and exits 0, as before. Otherwise only the last attempt's verdict goes to $GITHUB_OUTPUT, after the loop, in the same place as the earlier single write.
• None of these inputs come from an attacker: the prompt interpolates only $RUNNER_ROOT, and the claude flags (--allowedTools Read Grep Glob --max-turns 8) and target paths are unchanged.
• The Grep pattern moved from TOKEN to policy-probe. If the policy ever failed to deny, this narrows what a Grep result could put into $out. It does not widen it.

Trust controls. This change does not touch the trusted-trigger check, the token request or broker, the kill switch, the merge holds or any .github/actions/ action. It widens nothing.

Instruction surface. The probe prompt was reworded, and nothing was deleted, narrowed or softened. The new wording frames the run as a permission self-test so the model makes the four calls. Whether the reads are allowed is still decided by the managed policy, not the prompt, and the step checks the policy's denials itself.

Tests. callers.test.mjs adds a test that runs the step with a stub claude and node. workflow.test.mjs updates an indentation pin. Neither file has security impact.

Spend (not a finding): each probe step can now run up to 3 short claude-sonnet-5-5 sessions instead of 1. That draws only on the leg's prepaid API credit or subscription.

critical-findings: 0