[kyle-sexton]

Refs: #13586
Refs: #12035

Summary

In run 42138674762 (maintenance-measure-skill-edit, education/teach), both pass-a and pass-a2 failed at the key leg's probe:

eval-handoff: Invalid package config /home/runner/work/claude-code-plugins/claude-code-plugins/package.json.

The handoff classifier from #13626 never ran, so neither leg judged its credential or handed off, and pass-subscription and pass-fallback were skipped.

Fix

The cause.

• eval-handoff.mjs required lane.cjs in place, from the .ci-workflows sparse checkout. That checkout holds only three files and no package.json.
• So Node's package-scope lookup climbed past it to the workspace root, and the package.json there did not parse.
• I reproduced this locally: the same script, given an unparsable package.json at its working directory's root, exits 2 with the same message, and passes with a valid one.

What wrote that file is UNVERIFIED. The repository checkout is at .base, not the workspace root. The fix no longer depends on it.

The fix.

• loadLauncher copies the three classifier files into a fresh folder with its own package.json ({"type":"commonjs"}), where the lookup ends.
• It then requires them and removes the folder.

A classifier that cannot run fails closed.

• Every probe and judge step in eval-hill-climb-pass.yml and eval-browser-dispatch.yml now stops the leg with ::error::the handoff classifier failed, so this leg cannot tell its credential's stop from a case failure and stops.
• Basis: a handoff on an unknown verdict would spend the next credential and hide a classifier defect behind a green run. A red leg with a named error is what the run's owner acts on.

Check (1), the credit-skip path.

• A key the gate skips already runs nothing: every step after the credential check is gated on steps.credential.outputs.run, or on a step gated on it.
• In this run the gate did not skip the key: the step env shows SKIP_KEY: false and PRESENT: true. The skip notice in the log is that step's script text, not its output.
• A test now holds this path.

Verification

• eval-handoff.test.mjs: 12/12. New cases:

  • the classifier run in a workspace whose root package.json is not json { hands off a credit stop;
  • a classifier that cannot load prints no verdict and exits 2;
  • the files it loads equal the legs' sparse checkout.

• callers.test.mjs:

  • a key leg's steps after the credential check all carry a condition, the first reading run;
  • every classifier call ends in the fail-closed ::error.

• Node suites. node --test ".github/scripts/**/*.test.mjs" ".github/actions/**/*.test.mjs": 8214 tests, 8214 pass, 0 fail, 0 skipped.

• Bash suites:

  Suite	Result
  write-runner-never-rules.test.sh	76/76
  check-artifact-retention.test.sh	39/39
  check-app-key-references.test.sh	9/9
  check-read-caller-keys.test.sh	68/68
  check-lane-coverage.test.sh	41/41
  workflow-self-paths.test.sh	132/132
  credential-sweep.test.sh	75/75
  run-zizmor.test.sh	13/13
  ci-review-lanes.test.sh	245/245
  affected-tests-live.sh	39/39

• Gates, all clean:

  • runner-policy.mjs --root . passed;
  • check-artifact-retention.sh --check, check-read-caller-keys.sh and check-app-key-references.sh;
  • actionlint and zizmor;
  • check-test-tmp-cleanup.sh.

Remaining

• What wrote the invalid workspace package.json on the runner. UNVERIFIED: no step in the leg writes one, and the log shows nothing.
• A real hosted run of the fixed leg. It has not run yet, because the Actions incident is in progress and nothing was re-dispatched.

Related

• ci(lanes): run the maintenance eval and propose lanes on the three-leg chain #13626 (introduced the classifier)
• ci(lanes): run every model lane on the same key, subscription and fallback chain #13586
• feat(evals): background hill-climb that opens draft PRs for skill edits beating the baseline #12035

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-06T01:01:58.216478Z	3000b31	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

claude-security-review reviewed 3000b31225b25475f5241d9a51f365c8d3ba352f: read 5 of 5 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #13636 (head 3000b31)

I found no security issues. I read every hunk of all 5 changed files. Two of them match trusted paths: .github/workflows/eval-browser-dispatch.yml and .github/workflows/eval-hill-climb-pass.yml. I read their whole change and the steps it gates.

What changed

• eval-handoff.mjs gets a new loadLauncher(root). Before, the script required lane.cjs straight from the .ci-workflows sparse checkout. Now it copies the three classifier files into a fresh mkdtempSync folder, writes {"type":"commonjs"} as that folder's package.json, requires the files from there, and removes the folder.
• Every probe and judge call to eval-handoff.mjs in the two eval workflows now ends in || { echo "::error::…"; exit 1; }, so a classifier failure stops the leg. It never turns into a handoff.
• Tests: new cases in eval-handoff.test.mjs and callers.test.mjs.

What I checked

• Where the copied files come from. The three file paths are fixed in the code (CLASSIFIER_FILES). The root folder comes from CI_WORKFLOWS, the same value the base code already used, and it points at the ci-workflows checkout pinned by SHA. Nothing from a pull request, issue or comment decides which files are copied or required. The temp folder's name comes from mkdtempSync, so nobody can predict it or plant files in it first.
• Module lookup. A bare package name required inside the copied files would now be searched for from the temp folder upward, where before the search started from the workspace. Any step that could plant a module along that path could already have edited the .ci-workflows files in the workspace, so the change opens no new trust boundary. The probe call still unsets ANTHROPIC_API_KEY and CLAUDE_CODE_OAUTH_TOKEN before running the classifier (unchanged, e.g. eval-hill-climb-pass.yml L350).
• Credentials and gating. No permissions: block, token request, trigger, if: condition or credential flow changes. The new || exit 1 only makes the steps stricter: a classifier that fails stops the leg red, never as a handoff that would spend the next credential. No trust control is widened. The kill switch, trusted-trigger check, token broker and merge holds are not touched.
• Instruction surfaces and manifests. The diff deletes or softens no standing instruction and touches no plugin manifest.
• PR text. The PR context contains no embedded instructions aimed at this review.

Spend (not a finding). None added. Fail-closed legs can spend less, because a broken classifier no longer hands off to the next credential.

critical-findings: 0

[github-actions]

claude-review reviewed 3000b31225b25475f5241d9a51f365c8d3ba352f: read 5 of 5 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

This was a full review of the head commit 3000b312. This lane had not reviewed this pull request before, so the review was not limited to blocking findings. I read every hunk of all 5 changed files.

What changed

• The fix. loadLauncher in .github/scripts/plugin-eval/eval-handoff.mjs now copies the three classifier files into a fresh temporary folder. That folder gets its own package.json set to {"type":"commonjs"}, so Node stops looking for a package scope there. The code then requires the files and deletes the folder.
• Why it works. Node keeps a required module in its cache, so judgeLeg, linesToMessages and isCreditStop still work after the folder is deleted.
• Shared list. CLASSIFIER_FILES is now exported, and the test imports it.
• Workflows fail closed. Each of the 12 classifier calls in eval-hill-climb-pass.yml and eval-browser-dispatch.yml now ends in || { echo "::error::…"; exit 1; }.
  • With bash -e, a failed verdict=$(…) already ended the step before this change. The real gain is a named error instead of a bare exit 2.
  • A failing classifier still can't produce a handoff, because the probe step's printf to GITHUB_OUTPUT never runs.

What I checked

• The sparse-checkout test. Its regex sparse-checkout: \|\n((?: {12}\S+\n)+) matches the three-line blocks at eval-hill-climb-pass.yml lines 313, 706 and 1054, in the same order as CLASSIFIER_FILES.
• The callers.test.mjs assertion. The fail-closed check expects \n\s+\{ echo … after ) ||. That matches the literal block scalars as the diff writes them.
• Running the classifier after the folder is deleted. The new spawn test runs judgeLeg on a spend-limit probe in a workspace whose package.json doesn't parse, after loadLauncher has deleted its folder. So if lane.cjs required a file lazily, that test would fail.
• What's left unchecked. I couldn't read lane.cjs itself, because the ci-workflows files aren't in this checkout. Its other branches aren't confirmed free of lazy requires, but the subscription and isCreditStop paths run through real = loadLauncher(ROOT) in the existing tests.

Not verified by me
The author reports that eval-handoff.test.mjs passes 12/12, the Node suites pass 8214/8214, and the bash suites and gates are clean. I had no shell, so I ran none of these.

Closing issues and spend

• Closing issues. The PR only refers to ci(lanes): run every model lane on the same key, subscription and fallback chain #13586 and feat(evals): background hill-climb that opens draft PRs for skill edits beating the baseline #12035 ("Refs:"). It closes no issues, so there were no acceptance criteria to check.
• Spend. This change starts no new spend. When the classifier fails, the leg now stops red instead of handing off, which can only save the next credential's usage.

[github-actions]

Post-merge mutation testing

• Range: 297298d..929ef62
• Merged pull request: #13636
• Outcome: reported

Killed 5, survived 2, timeout 0, no coverage 0, invalid 0, not run 0.
Covered-code score 71%; mutation score 71%.

File	Killed	Survived	Timeout	No coverage	Invalid	Not run
.github/scripts/plugin-eval/eval-handoff.mjs	5	2	0	0	0	0

Surviving mutants

Location	Operator	Disposition	Cause
.github/scripts/plugin-eval/eval-handoff.mjs:118	SBR	productive	input-gap
.github/scripts/plugin-eval/eval-handoff.mjs:128	SBR	productive	no-assertion

The run's summary has the full report.