[kyle-sexton]

Refs: #13586
Refs: #12035

Summary

In run 42146761553 (education/teach, case te-*), both baseline passes' key legs failed at the safety probe with "the managed policy did not record a denied Read of /proc/self/environ, so no case runs". The subscription and fallback legs were skipped. This PR fixes the one defect the evidence shows in the check, and makes the next failure name its cause. The probe still fails closed.

Fix

What changed in the split. I compared the probe with the pre-#13626 single-leg version (git show 60f5136484:.github/workflows/eval-hill-climb-pass.yml):

• The managed-policy step, the step order (policy, isolation, toolchain, probe), the prompt, the model and the tools are the same on every leg. The new test pins this.
• One change: the probe moved from --output-format json (one object) to stream-json --verbose, so the classifier can read a credit or limit stop.
• With it, the denial check became jq -e -s ... 'last(.[] | select(.type == "result")) ...'. That slurp fails to parse when any stdout line is not JSON. A parse failure reads exactly as "did not record a denied Read".

Evidence:

• jq 1.8.2 run on a stream with one non-JSON line: the slurp check exits 5 with a parse error, and the line-by-line check finds the denial.
• A local claude -p (Claude Code 2.1.318) with a deny rule records the denial in permission_denials under both json and stream-json --verbose, so the stream format itself still carries the field.

The cause on that runner is UNVERIFIED. The probe writes its stream to a file that is never printed, by design, so the log cannot separate:

• a line in the stream that is not JSON;
• a model that made no call;
• an API error on the Console key;
• a policy that let the read through.

I found nothing showing a difference in Claude Code behavior between credential types.

The fix.

• Parse. The check parses the stream line by line (jq -R -s '[split("\n")[] | fromjson? | objects] | ...'), so a line that is not JSON is skipped. It still requires all four denials on the last result line, and still fails closed.

• Diagnostics. A failed check now names the probe's closed facts:

  • whether a result line exists;
  • its subtype, terminal_reason, is_error, api_error_status and num_turns;
  • the tools called (Read, Grep, Glob or other), the denial count, and the count of lines that are not JSON.

  Every value goes through a closed pattern, so no model or tool text reaches the public log. The next run that fails says which of the four cases it is.

• Scope. The same change goes into all six probe steps, in eval-hill-climb-pass.yml and eval-browser-dispatch.yml.

Verification

• New test in callers.test.mjs:

  • every eval leg in both workflows runs the same managed-policy step and the same deny rules;
  • every leg runs the same probe, apart from each workflow's message wording;
  • the policy is written before isolation, and isolation comes before the probe;
  • the probe parses line by line, never with jq -e -s, and its failure message carries probe_facts.

• Simulated probe steps. Every leg's rendered probe step ran in bash with a stub claude:

  • a full set of denials with one non-JSON line passes;
  • a refusal fails, naming tool_calls=none denials=0 is_error=false;
  • an API error fails, naming is_error=true api_error_status=404 terminal_reason=api_error.

• Node suites. After merging main (ci(lanes): run the mutation lane on run-claude-cli, mutants in a job with no secret #13632): 8227 tests, 8227 pass, 0 fail, 0 skipped. callers.test.mjs runs 262 of 262.

• Bash suites:

  Suite	Result
  write-runner-never-rules.test.sh	76/76
  check-artifact-retention.test.sh	39/39
  check-app-key-references.test.sh	9/9
  check-read-caller-keys.test.sh	68/68
  check-lane-coverage.test.sh	41/41
  workflow-self-paths.test.sh	132/132
  credential-sweep.test.sh	75/75
  run-zizmor.test.sh	13/13
  ci-review-lanes.test.sh	245/245
  affected-tests-live.sh	39/39

• Gates. runner-policy passed, and artifact-retention, read-caller-keys, app-key-references, actionlint and zizmor are clean.

Remaining

• Confirm the cause on a hosted run. A re-dispatch of the measurement, after merge, will print the probe's facts if it still fails.

Related

• ci(lanes): run the maintenance eval and propose lanes on the three-leg chain #13626 and fix(plugin-eval): load the eval handoff classifier under its own package scope #13636 (the eval legs and the classifier load fix)
• ci(lanes): run every model lane on the same key, subscription and fallback chain #13586
• feat(evals): background hill-climb that opens draft PRs for skill edits beating the baseline #12035

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-11-06T04:47:06.099511Z We couldn’t complete this review. Comment @codex review to try again.	9fb5565	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

claude-security-review reviewed 80b30ff9d47f8605ae44dc2b884581115542597e: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #13640 (incremental, head 80b30ff)

No security issues found. This review covered only what changed since 9fb55652. It was blocking only (this lane already reviewed an earlier head), so only CRITICAL and IMPORTANT findings would have been reported.

Scope

• One file changed in this push: .github/actions/resolve-config/callers.test.mjs. I read its only hunk, around L9417-9429.
• This test file is under resolve-config, which is a trusted path. The trust-control rules count resolve-config's tests as outside that control.

What changed

• The step-order test used to compare indexOf values directly. A missing step returned -1, which could make the order check pass by mistake.
• It now looks up each step with findIndex and asserts that the index is >= 0 before comparing order.
• This only makes the test stricter. No workflow, action, token request, trusted-actor list, kill switch or merge hold changed, so nothing is widened.

Other checks

• Manifests: none changed.
• Spend: none. This is a test-only change.
• PR context: no embedded instructions to this lane.
• Prior finding: the earlier summary comment reported 0 findings, so there is nothing to restate.

critical-findings: 0

[github-actions]

claude-review reviewed 80b30ff9d47f8605ae44dc2b884581115542597e: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #13640, incremental review since 9fb55652

Scope: incremental and blocking only, since this lane already reviewed an earlier head. One file changed since then, .github/actions/resolve-config/callers.test.mjs. I read its whole hunk and the test around it at head (lines 9400–9429). .github/actions/resolve-config/** is a trusted path, but the change only touches a test assertion and gates nothing at runtime.

What changed

• The step-order check in "every eval leg runs the same managed-policy step and the same probe" no longer uses names.indexOf(...).
• It now looks up each of the three step names with findIndex, asserts the index is >= 0, and only then compares the order: policy, then isolation, then probe.

Prior finding: this fixes the advisory finding from the last review (comment).

• Before, a step name that was not found gave -1, so the order check could pass without checking anything.
• Now a renamed step fails with no step named <n> before any comparison, so the regression the old check missed can no longer pass silently.
• file, id and steps are in scope from the for loop at line 9408, and steps is the job's step array, so the new code reads correctly.

Findings: none.

Closing issues: none. The PR only references #13586 and #12035.

Author's verification: the test counts and the bash and simulation results in the PR description are the author's claims. I ran nothing.

Spend: none new. This change only touches a test.

[kyle-sexton]

claude-review's advisory (callers.test.mjs:9421-9422, an order check that passes when a step name is not found): CORRECT. Fixed in 80b30ff: each step is looked up with findIndex and must be found before the order is compared. node --test callers.test.mjs: 262 pass, 0 fail.