Repository navigation
test(ci): show the plugin eval health line reading skill loaded on a dispatched run #12032
Description
Refs: #6405
Outcome
#6405's first slice, item 4, asks for one health line per case: did the skill load in the with-arm, and how many runs fired. The line is built (.github/scripts/plugin-eval/summarize.mjs:140-142), but no run has shown it answering. The one dispatch, https://github.com/melodic-software/claude-code-plugins/actions/runs/41055400082, was rendered only from its uploaded artifact. That artifact carries no traces, because --keep-temp keeps them on the runner outside the upload (.github/workflows/eval-browser-dispatch.yml:519-547, :679-686). Every case therefore read skill loaded: unknown. The job summary, which runs on the runner where the traces exist (eval-browser-dispatch.yml:613), was never read. This slice shows the health line answering on a real run, and fixes it if it does not.
Done when
- The job summary of a dispatched
eval-browser-dispatch.ymlrun is read, either from run 41055400082 or, if its logs have expired, from a new deterministic-only dispatch on a suite with a skill-fired case. A comment on feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405 links the run and quotes each case's health line. - Each health line reads
skill loaded: yesorskill loaded: no, andfired in N of M runs with a kept trace, notunknown (no kept trace lists skills). - If the summary itself still reads
unknown, a PR makessummarize.mjsfind the runner's kept trace (summarize.mjs:59-77resolvesrun.tracePath), with a test that fails before the fix. A re-dispatch then shows the line answering.
Out of scope
- Uploading traces in the artifact. The upload is limited to dispatched runs on purpose (
eval-browser-dispatch.yml:676-678). - Repairing cases that report
no. That is a separate slice of feat(evals): run plugin eval suites in a CI lane and hill-climb in the background #6405.
Activity
github-actions commentedon Nov 1, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; a maintainer confirms or changes it.
Size: Small
Complexity: Low
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: read the job summary of an eval-browser-dispatch run, either run 41055400082 or a new deterministic-only dispatch, and confirm each case's health line reads
skill loaded: yesornoinstead ofunknown. If it still readsunknown, fixsummarize.mjsto find the runner's kept trace and add a test that fails before the fix. The dispatch and log reading need a real Actions run, so a person or host run may have to do them. Nothing else is missing. No likely duplicates found, apart from the parent #6405.Automated triage. A maintainer reviews it.
kyle-sexton commentedon Nov 1, 2026
Claimed by a Backlog HITL host-run worker (2026-11-01). Branch: host-run/12032-eval-health-line-log.
kyle-sexton commentedon Nov 1, 2026
Closing as completed. Run 41353703294 on main shows the health line
skill loaded: yes, which is now readable from the job log (#12084). Item 3 was not triggered. Full record is on #6405.