[kyle-sexton]

Refs: #13586

Summary

The maintenance role's lanes move onto the chain every other model lane runs: role API key, then role subscription token, then FALLBACK_CLAUDE_CODE_OAUTH_TOKEN. Each lane also gets the check-key-credit gate and a lane-outcome record per leg. The owner decided in #13586 to add the maintenance key (Q2, "You add a maintenance key (Recommended)") and to make every lane "consistent and uniform". MAINTENANCE_CLAUDE_API_KEY is now an org secret.

Lane	Legs before	Legs after
maintenance-measure-skill-edit.yml propose	subscription, fallback (steps in one job)	key, subscription, fallback steps in one job, new propose-key-credit gate
eval-hill-climb-pass.yml pass (pass-a, pass-a2, pass-b)	subscription only	pass, pass-subscription, pass-fallback jobs behind a new gate job
eval-browser-dispatch.yml eval	subscription only	eval, eval-subscription, eval-fallback jobs behind a new eval-key-credit job

Fix

How the eval passes hand off. claude plugin eval does not launch through run-claude-cli, so the new .github/scripts/plugin-eval/eval-handoff.mjs reads claude plugin eval's output with run-claude-cli's own classifier (option (a) in the issue).

• Inputs. It reads the probe's stream and each errored run's kept trace.
• Classifier. Each leg checks out three files from ci-workflows at the run-claude-cli pin (b8bdd7f): lane.cjs for judgeLeg and linesToMessages, classify.cjs for isCreditStop, and one helper lane.cjs loads.
• Handoff rule. It is restated from lane.cjs handoffReason, which is not exported:
  • key leg: a credit stop, or a 401 or 402;
  • subscription leg, when the fallback token is set: also a limit stop;
  • fallback leg: never.
• Not read. The suite's exit code, because a case below --threshold also exits 1.
• Basis:
  • plugin-evals, What a grader can look at defines a run's trace as "The session as JSON, one message per line".
  • A local --keep-temp run on Claude Code 2.1.318 kept a trace with the SDK's system/assistant/rate_limit_event/result messages, and its result carried is_error and api_error_status.
  • A run with a bogus ANTHROPIC_API_KEY exited 2 with partial: true, partialReason: auth_failed. Its one run had error: "exit 1: Invalid API key" and a trace whose result had api_error_status: 401, and eval-handoff.mjs handed that off as auth on the key leg.
  • plugin-evals, Runs fail with a usage-limit or rate-limit error says such runs score 0, the suite still finishes, and it is not marked partial. So the per-run traces are where a limit or credit stop shows.

Credentials. The eval reads the provider credentials from its environment (plugin-evals, Run evals in CI: "credentials in the environment such as ANTHROPIC_API_KEY").

• Key leg. The key reaches the step as LANE_API_KEY, and the script exports it as ANTHROPIC_API_KEY and unsets LANE_API_KEY before claude runs, so CLAUDE_CODE_SUBPROCESS_ENV_SCRUB strips it from Bash subprocesses. callers.test.mjs reads any ANTHROPIC_* env name as a secret, which is why the step env does not use it directly.
• Subscription and fallback legs. These keep CLAUDE_CODE_OAUTH_TOKEN.

The probe now writes --output-format stream-json --verbose, so a credit or limit stop in it is classified the same way. The stream's result line still carries permission_denials, which the deny checks read with jq -s 'last(...)'. A probe that ends on the credential's own stop hands off instead of failing red. The leak check runs first and now checks for the leg's own credential.

Legs.

• One credential per job. The eval runs --scaffold and, on dispatch, Bash(...) grants, so each eval leg is its own job, held equal to the key leg's steps by a shape test, as the tidy lane is.
• propose stays one job. Its model has Read, Grep and Glob only, no Bash (run-claude-cli's one-job rule), following adjudicate-feedback.
• Empty credentials hand off. An empty subscription token hands to the fallback leg, and the fallback leg starts only when its token is set.
• A leg that hands off writes nothing. It summarizes and uploads nothing, so the pass's outputs pick fallback, else subscription, else key.

Credit gate.

• Marker names. Each key leg uploads key-credit-stop-<workflow>-<key job> after a credit stop. eval-hill-climb-pass.yml is a called workflow whose github context is the caller's (Reusing workflow configurations: "the github context is always associated with the caller workflow"), so its marker is key-credit-stop-maintenance-measure-skill-edit.yml-pass.
• Concurrent uploads. pass-a and pass-a2 can both upload that marker in one run, so the upload is continue-on-error.
• Permissions. The gate jobs hold actions: read only, and the pass callers grant actions: read for theirs.

Telemetry.

• Eval legs. Each eval leg writes a report-lane-outcome record with lane-id, credential, outcome, handoff and fallback reason. durationMs and costUsd come from an execution file of one result message that carries the suite's own durationSeconds and costUsd and nothing a run wrote.
• Skipped keys. A key skipped by the gate writes an outcome: skip, class: credit-stop, handoff: true record.

Catalog and tests.

• Catalog. lane-funding.json lists the new legs, and .github/artifact-retention.json has a row for each of the three new markers.
• Exemptions. The six maintenance LEG_EXEMPTIONS entries are gone, along with the propose entry in RECORDLESS_LEGS.
• Credential tests. callers.test.mjs now accepts a claude plugin eval leg (the claude-cli launcher) reading the role key, and the fallback token reaching such a leg's claude steps or the eval pass workflow.
• Changed test expectations. These follow the new steps:
  • hill-climb-workflow.test.mjs's run-claude-cli count goes from 2 to 3, for the new key step;
  • the probe's leak-check pattern now matches $credential;
  • the scan and upload conditions add steps.judge.outputs.handoff != 'true'.

Verification

• Node suites. node --test ".github/scripts/**/*.test.mjs" ".github/actions/**/*.test.mjs" (this covers .github/scripts/lib): 8210 tests, 8207 pass, 0 fail, 3 skipped. The 3 skipped are the real-classifier cases in eval-handoff.test.mjs, which need EVAL_HANDOFF_CI_WORKFLOWS.

• Real classifier. With EVAL_HANDOFF_CI_WORKFLOWS pointing at the three files at b8bdd7f, eval-handoff.test.mjs ran 7 of 7, using the observed spend-limit and rejected-key shapes from ci-workflows classify.test.cjs.

• callers.test.mjs. 256 pass after merging main. A shape mutation in pass-fallback fails the new leg test.

• Bash suites:

  Suite	Result
  write-runner-never-rules.test.sh	76/76
  check-artifact-retention.test.sh	39/39
  check-app-key-references.test.sh	9/9
  check-read-caller-keys.test.sh	68/68
  check-lane-coverage.test.sh	41/41
  check-scan-carrier-secret-shadow.test.sh	9/9
  workflow-self-paths.test.sh	132/132
  credential-sweep.test.sh	54/54
  run-zizmor.test.sh	13/13
  ci-review-lanes.test.sh	245/245
  plan-test-lanes.test.sh	58/58
  affected-tests-live.sh	39/39

• Gates, all clean:

  • runner-policy.mjs --root . (passed, so no standards contract change is needed);
  • check-read-caller-keys.sh, check-app-key-references.sh and check-artifact-retention.sh --check (120 uploads, 82 families);
  • actionlint and zizmor on the three workflows;
  • check-test-tmp-cleanup.sh, check-silent-skips.sh, check-discriminating-test-skips.sh and check-lane-coverage.sh --check.

• Simulated probe steps. The rendered probe steps of both workflows ran in bash with a stub claude, which also confirmed LANE_API_KEY never reaches claude:

  Leg	Probe stream	Result
  key	credit stop	hands off
  key	limit stop	fails red (no handoff, as run-claude-cli)
  subscription	credit or limit stop	hands off
  either	full denials	passes
  either	missing denials	fails red

Remaining

• UNVERIFIED on a hosted runner:
  • the cross-repository actions/checkout of the public melodic-software/ci-workflows at a pinned SHA with the job token;
  • a real rejected rate_limit_event in an eval trace: the limit-stop rule is tested on fixtures, and a real trace showed only an allowed event.
• Judge calls. A credit or limit stop inside an LLM judge call leaves no run trace, so it does not hand off. The hill-climb passes run deterministic graders only. The dispatch path can run judged cases.
• Real-classifier tests. They skip in CI, which has no ci-workflows checkout.
• post-merge-test-mutation. Its move to run-claude-cli and its LEG_EXEMPTIONS entry follow in ci(lanes): run the mutation lane on run-claude-cli, mutants in a job with no secret #13632, the last pull request for that issue.

Related

• ci(lanes): run every model lane on the same key, subscription and fallback chain #13586
• ci(lanes): run the release, adjudicate, refine and capture lanes on the three-leg chain #13590, ci(lanes): run the code-tidying upkeep sweep on the three-leg chain #13592, ci(lanes): give every three-leg lane the key-credit gate and the empty-subscription handoff #13608 (the same chain on the other lanes)
• test(lanes): fail a model lane that lacks its key, subscription or fallback leg #13594 (the leg guard test)

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-05T23:23:29.841641Z	3dd0429	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

claude-code-plugins/.github/scripts/plugin-eval/eval-handoff.mjs

Line 144 in 3dd0429

`outcome=${verdict.laneClass === "" ? "success" : "failure"}`,

Distinguish missing eval results from successful execution

If an eval crashes or is interrupted before writing its results JSON, readJson() returns null, no streams are judged, and this code reports outcome=success and writes a synthetic successful result message. Both eval-browser-dispatch.yml and eval-hill-climb-pass.yml subsequently publish that verdict through their “Record the leg” steps, even though their summary/read steps fail because no result exists. This makes failed executions appear successful in lane telemetry. Track missing or unreadable expected results separately from credential handoff classification and report an execution failure; this does not require treating a below-threshold suite exit as a credential failure.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

claude-review: 1 inline finding(s) on 3dd0429edd3be9f791932a9dfc4c11557f1f4e37; the summary is in this lane's pull request comment.

claude-lane: claude-review

[github-actions]

claude-review reviewed ae1e5ce91361ae4fd7ec205372934ffb2db92735: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Incremental code review: ec5bbea → ae1e5ce (blocking only)

No blocking findings in this push.

This lane already reviewed an earlier head of this PR, so this round reports blocking findings only and drops advisory ones. This push changed one file, .github/actions/resolve-config/callers.test.mjs, and I read its whole change.

What changed

• Narrower check. In readsFallback, the old check accepted any claude-cli step that held the fallback token. The new evalLegReadsFallback check passes a job only when every step that reads FALLBACK_CLAUDE_CODE_OAUTH_TOKEN (presence checks aside) meets one of two conditions, each with the token as CLAUDE_CODE_OAUTH_TOKEN:
  • it is a claude plugin eval run;
  • it is the probe step (a claude -p) of a job that runs one.
• New test. It covers six cases: an eval leg with and without a probe, a plain claude -p step, a probe in a job with no eval, an eval leg with another step on the token, and the token under another env name.

Checks

• Step matching. isEvalLegStep checks isLauncherStep(step, "claude-cli") first, which needs typeof step.run === "string". So EVAL_RUN.test(step.run) never sees a step with no run:. Neither regex uses the g flag, so .test() keeps no state between calls.
• Real probe steps. They match the probe rule. For example,

  claude-code-plugins/.github/workflows/eval-hill-climb-pass.yml

  Lines 1062 to 1082 in ae1e5ce

  	id: probe
  	if: steps.credential.outputs.run == 'true'
  	env:
  	CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.FALLBACK_CLAUDE_CODE_OAUTH_TOKEN }}
  	CLAUDE_CODE_SUBPROCESS_ENV_SCRUB: "1"
  	RUNNER_ROOT: ${{ steps.deny.outputs.runner-root }}
  	LEG: fallback
  	FALLBACK_READY: ${{ steps.credential.outputs.fallback-ready }}
  	CI_WORKFLOWS: ${{ github.workspace }}/.ci-workflows
  	run: |
  	if [ -n "${LANE_API_KEY:-}" ]; then
  	export ANTHROPIC_API_KEY="$LANE_API_KEY"
  	unset LANE_API_KEY
  	fi
  	credential=${ANTHROPIC_API_KEY:-${CLAUDE_CODE_OAUTH_TOKEN:-}}
  	[ -n "$credential" ] || { echo "::error::the leg's credential is empty"; exit 1; }
  	out=$(mktemp)
  	trap 'rm -f "$out"' EXIT
  	status=0
  	claude -p "Make exactly these four tool calls, in order, and no other: the Read tool on /proc/self/environ; the Read tool on $RUNNER_ROOT/.runner; the Grep tool with pattern TOKEN and path /proc/self/environ; the Glob tool with pattern environ and path /proc/self." \
  	--model claude-sonnet-5-5 --allowedTools Read Grep Glob --max-turns 8 --output-format stream-json --verbose >"$out" 2>/dev/null || status=$?

  has id: probe, the token as CLAUDE_CODE_OAUTH_TOKEN, and a line that starts with claude -p . The credential step's ${{ secrets.FALLBACK_CLAUDE_CODE_OAUTH_TOKEN != '' }} is a presence check and is stripped by withoutPresenceChecks. The credential: FALLBACK_CLAUDE_CODE_OAUTH_TOKEN inputs on the record steps are plain strings, not secret reads.
• Fixture results. I traced each fixture through readsFallback, and every one gives the asserted result. In the "another step" case, the final presence-only clause stays false, because the stray step still reads the token after presence checks are removed.

Prior and settled

• Earlier finding on callers.test.mjs. This lane's earlier finding that the fallback exception was wider than its comment says (thread on original line 2161) is resolved and settled. This push makes it a job-level eval-leg check, as the finding suggested, so I am not raising it again.
• Earlier finding on eval-handoff.test.mjs. That thread is settled, and this push does not touch that file.

Verification

I ran nothing. The "257/257 pass" in the owner's reply is the author's claim and was not checked here.

Other checks

• No plugin manifest changed in this push.
• No embedded instructions were found in the PR context, settled-threads or prior-findings files.
• Spend: none. This push changes only a test file.

[github-actions]

claude-security-review: 1 inline finding(s) on 3dd0429edd3be9f791932a9dfc4c11557f1f4e37; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

[github-actions]

claude-security-review reviewed ae1e5ce91361ae4fd7ec205372934ffb2db92735: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review of PR #13626, changes from ec5bbea to ae1e5ce

What I reviewed. One file changed since the last head this lane reviewed: .github/actions/resolve-config/callers.test.mjs. I read every hunk of it. It matches the trusted-path glob .github/actions/resolve-config/**, so I read the whole change and the helpers it calls (isLauncherStep, claudeReads, withoutPresenceChecks). This lane has reviewed earlier heads, so this review was blocking only: it reports CRITICAL and IMPORTANT findings and drops SUGGESTIONs.

Result: no security issues found.

critical-findings: 0

What changed and what I checked

• The fallback-token check in the test is now stricter. The old rule accepted a job when any claude-cli launcher step put FALLBACK_CLAUDE_CODE_OAUTH_TOKEN into CLAUDE_CODE_OAUTH_TOKEN. The new rule, evalLegReadsFallback, has two conditions:

  • every step that reads the fallback secret, with presence-only checks removed, must be an eval-leg step and pass the token as CLAUDE_CODE_OAUTH_TOKEN;
  • an eval-leg step is either a claude plugin eval run, or a step with id: probe that runs claude -p in a job that also runs claude plugin eval.

  (L2147-L2190)

• The new rule passes nothing the old one rejected. Every job it accepts has a reader step that is a claude-cli launcher with that exact env value, so the old clause accepted that job too. The new rule rejects these cases:

  • a claude -p step in a job with no eval;
  • an extra step that reads the token next to an eval leg;
  • the token passed under a different env name.

  The new unit test covers each case.

• This is a test, not a control. It is part of resolve-config's tests, which grant no token and do not change what the broker mints. No token scope, trusted actor, kill switch or merge hold changes. The change only tightens a guard test.

Prior findings

The earlier SUGGESTION on callers.test.mjs said the fallback-token exception was wider than its comment (discussion_r4487278084). This push narrows the exception as that comment suggested: a token-reading step must be the eval run or the probe of a job that runs one. Nothing in it needs to be raised again.

Spend

None. The change is test-only.

[github-actions]

The pr-verify verifier failed head ae1e5ce91361ae4fd7ec205372934ffb2db92735. What it wrote, as data:

What it wrote (36 lines)

Summary: Issue #13586 is judged as `closes`, not `refs`. The body's `Refs: #13586` line meant it as
  a partial delivery. But its Remaining section ends with "a separate pull request, which closes
  #13586", and GitHub reads that as a closing keyword, so merging this pull request closes the
  issue. Against the whole issue, the first done-when item ("Every model-running workflow runs the
  same three-leg chain ... No lane keeps its own variant") is not met. post-merge-test-mutation.yml
  still runs on claude-code-action with no key leg and no fallback leg, and its LEG_EXEMPTIONS entry
  is kept in callers.test.mjs. No `credited_siblings` or `person_checks` excuse it, nor the
  release-lane, pr-unblock, intake-triage or pr-verify-capture items. The cited #13590, #13592 and
  #13608 have no credited entry, and a sibling the body names without one excuses nothing. As a
  partial delivery the work itself looks correct. eval-browser-dispatch.yml,
  eval-hill-climb-pass.yml and maintenance-measure-skill-edit.yml `propose` each gain a key leg on
  MAINTENANCE_CLAUDE_API_KEY, a subscription leg and a fallback leg. Each lane gets a
  check-key-credit gate holding actions: read only, a lane record per leg, and a skip record for a
  gated key. Each eval leg is its own job holding one credential, and a shape test keeps those jobs
  equal. The key reaches `claude` as ANTHROPIC_API_KEY, with LANE_API_KEY unset first, so env scrub
  applies. The probe's leak check runs before the classifier, and the classifier runs with no
  credential set. lane-funding.json and artifact-retention.json are updated. The exemption entries
  are removed and callers.test.mjs is tightened for the fallback token. test-node, test-bash (0–2)
  and lint-repo all concluded success on the judged head
  (https://github.com/melodic-software/claude-code-plugins/actions/runs/42132825476/job/127613592424).
  The diff has no binary files and changes nothing in the takeover set. No text was found that tries
  to steer the reviewer.
- PR #13626 body, Remaining section: the phrase "which closes #13586" is a GitHub closing keyword,
  so merging this pull request closes #13586 while post-merge-test-mutation.yml's move to
  run-claude-cli and its key and fallback legs are still undelivered. Reword it so the body has no
  closing keyword (for example "which will finish #13586") and keep the `Refs: #13586` line.
  Otherwise, deliver the post-merge-test-mutation work in this pull request and remove its
  LEG_EXEMPTIONS entry.
- .github/actions/resolve-config/callers.test.mjs:9354: the post-merge-test-mutation.yml
  LEG_EXEMPTIONS entry still waives the key and fallback legs. That is fine for a `Refs` partial
  delivery, but under the current closing keyword it leaves done-when item 1 (every model-running
  workflow on the three-leg chain, no lane variant) unmet.
- .github/scripts/plugin-eval/eval-handoff.test.mjs:89: at import time the suite fetches lane.cjs,
  classify.cjs and their helper from raw.githubusercontent.com and asserts on the response, so a
  network or GitHub outage fails every case in the file. Consider skipping the real-classifier
  cases, with a recorded reason, when the fetch fails, instead of failing the whole suite.

This pull request leaves the merge queue, and ci-status holds it out while it carries pr-verify-failed. A pr-verify pass removes the label, at a new commit or at this same head: once a trusted person edits the body, and only trusted people have edited it since the fail, this head is judged again, so a fail on the body or the issues it links needs only the body fixed, no new commit.