[kyle-sexton]

Closes #13854

Summary

The eval policy probe reported a session that made no tool call as a policy
failure ("the managed policy did not record a denied Read of
/proc/self/environ"). It now reports that case as unproven, with its own
error, and still stops the leg, so no case runs on a policy nobody proved.

Fix

• New .github/scripts/plugin-eval/policy-probe.mjs judges the probe's
  stream once per expected call (Read /proc/self/environ, Read
  $RUNNER_ROOT/.runner, Grep /proc/self, Glob /proc/self). Each call is
  sorted as denied, made and not denied, or not made:
  • exit 0: all four calls denied, so the leg proceeds.
  • exit 1: a call was made and not denied. The leg stops at once with the
    old "did not record a denied" message, the only case that now uses it.
  • exit 1: the session errored before making every call. The leg stops
    with its own message.
  • exit 3: the session ended normally without making every call. The step
    runs it again. After 3 attempts it stops with ::error::the policy is unproven: ....
• All six probe steps use it: the three legs in eval-hill-climb-pass.yml
  and the three in eval-browser-dispatch.yml. The issue named four.
• The jq judge, which was duplicated six times, is replaced by one module.
• The claude -p call stays inline in each step on purpose. The credential
  guards in callers.test.mjs (isLauncherStep, the stray-reader check and
  the fallback-token rule) find launchers by a claude line in step.run.
  Moving the call into a shell script would hide the probe from those
  guards.
• The judge runs from .base without the credential (env -u), after the
  leak check, as eval-handoff.mjs does.

Done-when 2 (prove the policy without depending on the model choosing a tool
call) is met only by the issue's "or" arm:

• Run 42221177945
  passed every probe step (pass-a, pass-a2, pass-b, key and subscription
  legs) after fix(ci): frame the eval policy probe as a self-test and rerun a no-call session #13665 added the self-test framing and the retry.
• The failing run
  42166675433
  (05:09Z) ran before fix(ci): frame the eval policy probe as a self-test and rerun a no-call session #13665 merged (05:37Z).
• No model-independent proof is added. claude --help lists no flag that
  forces a tool call.
• A check of managed-settings.json cannot stand in for the live probe. The
  deny step already asserts the /proc rule is in the installed file
  (jq -e '.permissions.deny | index("Read(//proc/**)")'), but Claude Code
  enforces Read deny rules on Grep and Glob only best-effort
  (code.claude.com/docs/en/permissions, "Read & Edit"). A correct file
  therefore does not prove the calls are denied, and letting cases run on a
  file check alone would loosen a security control.

Done-when 3 was already fixed on main:

• Run 42138674762
  (00:41Z) logged eval-handoff: Invalid package config /home/runner/work/claude-code-plugins/claude-code-plugins/package.json.
  lane.cjs, required in place, climbed to the workspace root's
  package.json, which did not parse.
• fix(plugin-eval): load the eval handoff classifier under its own package scope #13636 (929ef62, 01:07Z) copies the classifier under its own
  package.json. The regression test is "the classifier loads under a
  workspace whose root package.json does not parse" in
  eval-handoff.test.mjs.

Verification

Each file was run serially with bash scripts/run-test-file.sh, after
merging origin/main:

• policy-probe.test.mjs: 7/7. It covers the no-call, skipped-call,
  made-not-denied, errored and non-JSON-line cases, and checks that the
  facts carry no stream text.
• callers.test.mjs: 273/273. The step harness now runs the real judge. A
  session that never calls a tool runs 3 times and ends unproven, never
  did not record. A Read that is not denied stops after 1 attempt.
• workflow.test.mjs: 6/6.
• hill-climb-workflow.test.mjs: 11/11.
• eval-handoff.test.mjs: 15/15.
• actionlint: no new findings. The only findings are the existing
  cache-mode key ones, also present on main.

This PR touches no plugin, so eval-browser-dispatch's stage stays off and
the probe does not run against the old .base.

Related

• fix(ci): frame the eval policy probe as a self-test and rerun a no-call session #13665 (self-test framing and retry)
• fix(plugin-eval): load the eval handoff classifier under its own package scope #13636 (classifier package scope)
• Lane sweep: open findings #9922 (lane sweep)

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	❌ Failed 2026-11-07T00:31:59.895186Z We couldn’t complete this review. Comment @codex review to try again.	af46f6c	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

claude-security-review reviewed af46f6c6d05ae335d5f7de4cb0227aca30c637db: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

I read every hunk of all 6 changed files. Both workflows are on the trusted-path list, so I read each of their changes in full, plus the probe steps they sit in and the .base checkout and patch steps that feed those steps.

What changed. In all six eval probe steps (three legs in eval-hill-climb-pass.yml, three in eval-browser-dispatch.yml), the inline jq denied() / probe_facts() judge is replaced by one module, .github/scripts/plugin-eval/policy-probe.mjs.

The policy gate is no weaker.

• The step continues only on judged=0: eval-hill-climb-pass.yml#L365-L369. The module returns 0 only when calls.every((c) => c.denied), meaning all four (tool, prefix) pairs appear in the last result's permission_denials. That is the same four checks, with the same file_path // path startsWith matching, that the old denied calls made.
• Exit 3 (the session ended without making every call) still stops the step with exit 1 after three attempts.
• Every other outcome stops it too: exit 1, a node crash, or a missing module. A missing module is what happens on a run whose .base is older than this PR. judged is reset on every attempt, and an empty reason falls back to a fixed message.
• The credential leak check and the handoff classifier still run before the judge, in the same order. The judge runs with env -u ANTHROPIC_API_KEY -u CLAUDE_CODE_OAUTH_TOKEN, so it never holds the credential.

No stream text reaches the logs.

• Stream values in the printed reason line pass only through fixed-pattern token() checks.
• Tool names are reduced to Read/Grep/Glob/other.
• The call names and prefixes come from expectedCalls. The only outside value in them is RUNNER_ROOT, which comes from the trusted deny step's output.
• No content from a tool result, and no workflow-command text, can reach echo through $reason.

Where the judge now loads from. The judge now loads from .base, not from the workflow file. In eval-browser-dispatch.yml, .base is the base checkout and pull request code is kept in .head. In eval-hill-climb-pass.yml, pass b applies the scanned, hash-checked edit.patch to .base before the probe (L207-L225). eval-handoff.mjs already runs from that same patched tree and can already end the step with exit 0, so this adds no new trust boundary. A forged probe pass also would not remove the managed-settings deny rule, which the deny step still installs. I checked this and it is not a finding.

Test files. callers.test.mjs, workflow.test.mjs and policy-probe.test.mjs change only test assertions and harnesses. No trust control, token scope, kill switch, merge hold or instruction surface is touched, and no plugin manifest is changed.

Spend. No new spend beyond the existing probe: the retry still runs at most 3 claude -p sessions per leg, drawn from prepaid API credit or the subscription.

The PR context contains no embedded instructions aimed at the review. There are no prior findings, so this is a full review rather than a blocking-only one.

critical-findings: 0

[github-actions]

claude-review reviewed af46f6c6d05ae335d5f7de4cb0227aca30c637db: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #13885, no findings

I read all six changed files hunk by hunk. I also read the head probe step in eval-hill-climb-pass.yml, and the .base checkouts and base-sha wiring in both eval workflows and in maintenance-measure-skill-edit.yml.

What the change does

The probe step had a jq judge copied into six places. This PR replaces those copies with one module, .github/scripts/plugin-eval/policy-probe.mjs. The module sorts each of the four expected calls as denied, made and not denied, or not made. It exits 0 when all four are denied. It exits 1 when a call went through without a denial, or when the session errored. It exits 3 when the session ended normally without making every call. All six probe steps now:

• run the session again on exit 3, up to 3 attempts;
• after 3 attempts, stop with a separate the policy is unproven error instead of the policy-failure message.

What I checked

• Order of the checks: leaked, then error subtype, then no call, then a skipped call. A call that was made and not denied always counts as a policy failure, even when the session also errored. missing always exists once the all-denied case has returned.
• Matching the old jq: pathOf's file_path ?? path matches the old jq .tool_input.file_path // .tool_input.path // "" for every input the probe produces. Lines that are not JSON are still skipped, so they cannot hide the denials. The facts still hold only tokens checked against fixed patterns.
• Shell flow: reason=$(…) || judged=$? is safe under bash -e. judged is always set before the case. If node crashes (exit 1, nothing on stdout), the step falls back to the ${reason:-policy-probe.mjs could not judge the probe} message.
• Where the judge runs from: In eval-hill-climb-pass.yml, .base is checked out at inputs.base-sha. The caller sets that to github.sha, so the workflow and .base/…/policy-probe.mjs always come from the same commit. In eval-browser-dispatch.yml, pull-request runs check .base out at the main tip. Before this PR merges, that tip has no judge, so the probe would stop with "could not judge". The PR body says this, and the leg does not run for this PR.
• Tests:
  • The step harness in callers.test.mjs now runs the real judge through the node stub (*policy-probe.mjs case). It covers the no-call retry, stopping as unproven after 3 attempts, and stopping on the first attempt for a Read that is not denied.
  • The leg-parity check still normalises the browser leg's wording. Dropping the echo replacement is correct, because both legs now echo "$reason".
  • The new regexes in workflow.test.mjs match the step text at head.

Closing issue #13854

• Done-when 1 (a probe with no tool call reports a separate error, not "did not record a denied read", in all copies): met in all six steps. It is covered by policy-probe.test.mjs ("a session with no tool call is unproven…", which asserts doesNotMatch(/did not record/)) and by the callers.test.mjs harness assertion doesNotMatch(never.stdout, /did not record a denied/).
• Done-when 2: the PR meets it through the issue's "or" arm: a run got through the probe step (run 42221177945). That rests on the author's word and I have not checked it. It is an operational fact, so no code test could cover it.
• Done-when 3: the PR says fix(plugin-eval): load the eval handoff classifier under its own package scope #13636 already fixed it on main and names the regression test in eval-handoff.test.mjs. The author claims this and I have not checked it; that file is not part of this diff.

I did not run any tests or actionlint. The pass counts in the PR body are the author's claims.

Spend

A session that makes some of the calls but not all four now gets up to 3 claude -p probe sessions, where before it stopped after one. This spends a little more prepaid API credit or subscription usage. It is not a hold.

Not applicable

• No plugin manifest changed, so there are no manifest lines.
• No settled threads or prior findings exist for this PR.
• None of the context files contained embedded instructions.

[github-actions]

Post-merge mutation testing

• Range: bc6a910..e33b467
• Merged pull request: #13885
• Outcome: reported

Killed 5, survived 2, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 71%; mutation score 71%.

File	Killed	Survived	Timeout	No coverage	Invalid	Not run
.github/scripts/plugin-eval/policy-probe.mjs	5	2	0	0	0	0

Surviving mutants

Location	Operator	Disposition	Cause
.github/scripts/plugin-eval/policy-probe.mjs:64	other	productive	input-gap
.github/scripts/plugin-eval/policy-probe.mjs:24	SBR	productive	input-gap

The run's summary has the full report.