Skip to content

ci(coverage): measure the Coverage lane across runner classes - #4658

Closed
thymikee wants to merge 1 commit into
test/host-kit-proc-fixturefrom
ci/coverage-wall-time
Closed

thymikee wants to merge 1 commit into
test/host-kit-proc-fixturefrom
ci/coverage-wall-time

Conversation

@thymikee

@thymikee thymikee commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Temporary. This lane measures the Coverage job's wall time across runner sizes so the change can be chosen from CI numbers instead of guesses. Do not merge: the file is deleted with this PR, and the lane must never reach main.

One job, a matrix over blacksmith-8/16/32vcpu-ubuntu-2404, each running the same unit-ci gate that Coverage runs. Wall time is read from the Actions API's per-step timings, so nothing in the tested tree is changed to observe it: no reporter, no vitest.config.ts edit, no artifact upload.

workflow_dispatch only. A pull_request lane would mark unrelated PRs red on a probe leg, and an unfiltered one also makes every path reachable to every gate, which the gate-manifest audit rejects — it failed a docs-only change still reaches the command-reference gate at head 6a6f0abf06. The cancel-in-progress group is gone for the same reason: a discarded run is a lost sample.

Validation

Head 8a3be5673f, on top of #4662, which fixes the pid-space flake this lane found.

  • node --experimental-strip-types scripts/gate/check.ts — gate manifest ok, 60 checks across 26 lanes.
  • node --experimental-strip-types --test scripts/gate/audit-coverage.test.ts — 7 pass, 0 fail.
  • npx oxfmt --check .github/workflows/coverage-probe.yml — clean.

The numbers themselves are in #4664. This PR is the instrument; it closes rather than merges.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.38 MB 6.38 MB 0 B
Package (unpacked) 6.38 MB 6.38 MB 0 B
Package (download) 1.94 MB 1.94 MB -5 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 57.1 ms 37.0 ms -20.1 ms
CLI --help 109.3 ms 78.7 ms -30.6 ms

@apex-sentinel-staging

Copy link
Copy Markdown

Apex Sentinel · ⛔ Block · review of 029ea20

4 finding(s): 1 blocker, 2 major, 1 minor.

Required fixes

  • Probe workflow breaks the Coverage check on every PR: hidden-path upload guard now fails
  • VITEST_MAX_WORKERS leg measures nothing: variable is never read, and CI ignores it anyway
  • Coverage-leg timing artifact is overwritten by the fuzz-worker leg; coverage wall time never recorded

🔴 Blocker (1)

Where Finding
.github/workflows/coverage-probe.yml:71 Probe workflow breaks the Coverage check on every PR: hidden-path upload guard now fails

🟠 Major (2)

Where Finding
.github/workflows/coverage-probe.yml:51 VITEST_MAX_WORKERS leg measures nothing: variable is never read, and CI ignores it anyway
scripts/vitest-ci-timing-reporter.ts:45 Coverage-leg timing artifact is overwritten by the fuzz-worker leg; coverage wall time never recorded

🟡 Minor (1)

Where Finding
.github/workflows/coverage-probe.yml:7 Probe's cancel-in-progress concurrency group discards the timing samples it exists to collect

Reviewed with this repository's own Sentinel configuration (655ede2796e5).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 4 new findings on 029ea20. The verdict and the full list are in the summary comment.

Comment thread .github/workflows/coverage-probe.yml Resolved
Comment thread .github/workflows/coverage-probe.yml Resolved
Comment thread scripts/vitest-ci-timing-reporter.ts Resolved
Comment thread .github/workflows/coverage-probe.yml Resolved
@thymikee

Copy link
Copy Markdown
Member Author

Reviewed 6a6f0ab. One change is needed before this leaves draft. CI was still running at review time.

coverage-probe.yml:7 uses on: pull_request: with no path, branch or label filter, although the PR body says the lane runs only on workflow_dispatch. Every push to every PR would start four extra jobs: two pid-probe jobs on 8 and 16 vCPU, and two full unit-ci jobs on 16 and 32 vCPU. A red probe leg, such as the x64-16 pid flake this PR investigates, would then show as a failing check on unrelated PRs. While this stays a draft it only affects this PR. Please restore on: workflow_dispatch (or gate pull_request to this branch or a label) so a temporary probe lane never runs on another PR's push, and say in the body that the lane must not reach main.

Not blocking, take or leave: nothing sets AGENT_DEVICE_CI_TIMING_FILE and the workflow has no upload step, so the reporter in vitest-ci-timing-reporter.ts:36 writes nothing in any lane (wire the variable and an upload, or drop the reporter and its vitest.config.ts registration); outputFile() at line 49 only rewrites a trailing .json, so another destination gets the same name in both Vitest runs; the pid fix in host-process-stop-confirm.test.ts:20 is a real flake fix that would be better as its own test(host-kit) PR, ideally by mocking the /proc identity read next to ./exec.ts so the double wins whatever the pid value; and the PR body still describes a different workflow (dispatch trigger, ARM and w16 legs, probe a5fe48c) than this head.

Could the measurement stay entirely in the probe workflow? For example, run unit-ci with --reporter=json --outputFile=... in that job only, upload the file, and read step durations from the Actions API, with no change to vitest.config.ts or scripts/.

On the open threads: the cancel-in-progress one still applies and is minor for a draft. These no longer apply and can be resolved: the upload step is gone from the workflow at this head; the x64-8-w16 leg does not exist at this head; the timing file overwrite is avoided by the per-pid name, and nothing sets the variable anyway.

Before merge, the trigger must be restricted and the Coverage and Probe checks must be green. Please also confirm that the Repo Guards gate-manifest audit accepts whichever trigger you choose.

The Coverage job's wall time is set by how many workers its machine affords, and that is a
configuration question: the same gate on 8, 16 and 32 vCPU answers it. Runs the gate on all three and
reads the wall time from the Actions API's per-step timings, so nothing in the tested tree changes to
observe it. Dispatch-only: a lane that starts on another PR's push would mark that PR red for a probe
leg, and an unfiltered pull_request lane also makes every path reachable to every gate, which the
gate-manifest audit rejects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee force-pushed the ci/coverage-wall-time branch from 6a6f0ab to 8a3be56 Compare November 17, 2026 21:05
@thymikee
thymikee changed the base branch from main to test/host-kit-proc-fixture November 17, 2026 21:06
@thymikee

Copy link
Copy Markdown
Member Author

Apex: all four review threads are addressed at 8a3be5673f, and the PR is retargeted onto #4662 so the pid-space flake fix lands underneath this lane.

The lane is now dispatch-only, has no concurrency group, and carries no measurement code: scripts/vitest-ci-timing-reporter.ts and its vitest.config.ts registration are deleted, so vitest.config.ts is untouched at this head. The probe runs the same unit-ci gate on 8, 16 and 32 vCPU and the numbers come from the Actions API's step timings. The measured change and its before/after table are in #4664. This PR is the instrument and is expected to be closed, not merged, so the workflow file cannot reach main.

@thymikee

Copy link
Copy Markdown
Member Author

The code at 8a3be56 is ready, and the earlier findings from #4658 (comment) are now fixed: the hidden-path upload step, the pull_request trigger, the worker-count leg, the timing reporter and the concurrency group are all gone, and the PR is one dispatch-only workflow file with three runner sizes. Not blocking: GitHub only starts a workflow_dispatch workflow whose file exists on the default branch, so this lane cannot be started from this branch, and the PR body says it must never land on main; the numbers it was meant to produce are already in #4664, so please close this PR rather than label it ready-for-human, and if you want a re-measurement later, use a throwaway branch with a pull_request trigger. I am not labeling it for that reason.

The four inline threads from the earlier head no longer apply, so please resolve them: the probe breaking the Coverage check (#4658 (comment)), the worker-count leg (#4658 (comment)), the overwritten timing artifact (#4658 (comment)) and the cancel-in-progress group (#4658 (comment)).

There are no conflicts. The diff is a dispatch-only workflow that no pull_request job runs, so the pending checks do not exercise it.

@thymikee

Copy link
Copy Markdown
Member Author

Closing: this was the measurement lane for the Coverage runner experiment. Its numbers are recorded in #4664, and a workflow_dispatch probe can't run unless it lands on main, which this lane must never do.

@thymikee thymikee closed this Nov 17, 2026
@thymikee
thymikee deleted the ci/coverage-wall-time branch November 18, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant