Skip to content

ci: measure slim versus standard gate runner latency - #4555

Closed
dgageot wants to merge 1 commit into
mainfrom
worktree-board-d909887108e17690
Closed

dgageot wants to merge 1 commit into
mainfrom
worktree-board-d909887108e17690

Conversation

@dgageot

@dgageot dgageot commented Oct 8, 2026

Copy link
Copy Markdown
Member

TEMPORARY EXPERIMENT - DO NOT MERGE. This is a data-gathering PR, not a
change meant to land. It should stay open only long enough to collect a
handful of matched runs, then get closed.

The existing eight-run baseline on pull_request shows ordinary gate
runs queuing for 2-5 seconds, but two merge_group runs saw 39s and 372s
outliers. This experiment checks whether the runner image itself
(ubuntu-latest vs the slimmer ubuntu-slim) accounts for any of that
scheduling or setup latency.

It adds a gate-slim job to .github/workflows/ci.yml that only runs
on: pull_request, with the exact same needs and timeout-minutes as
the existing required gate job, running the identical dependency check.
The only difference is runs-on: ubuntu-slim instead of ubuntu-latest.
gate-slim is not required, isn't part of the release dependency chain,
and the diff touches no permissions anywhere in the workflow.

For each PR that carries this change, matched gate / gate-slim run
attempts will be compared using REST API timestamps (second precision):
created_at -> started_at scheduling wait, dependency-ready (the
latest gate dependency's completed_at) -> the Check results step's
started_at user-visible wait, setup step duration, and
dependency-ready -> job completed_at end-to-end gate overhead. Several
matched samples are needed before drawing conclusions, and since this
only exercises pull_request, it can't directly explain the
merge_group outliers above - that would need a separate experiment
triggered on merge_group.

task build, task test, task lint, actionlint, and
scripts/workflow-lint.sh all pass, git diff --check is clean, and
this was reviewed and approved.

@aheritier aheritier added area/ci CI/CD workflows and pipeline kind/chore Maintenance, deps, CI, tooling (maps to chore: commit prefix) labels Oct 8, 2026
@dgageot

dgageot commented Oct 8, 2026

Copy link
Copy Markdown
Member Author

First matched runner-latency sample

CI run 37804754451, attempt 1 passed, including both gates. All shared dependencies finished at 2026-10-08 16:04:09 UTC (test-windows was last). Both gate jobs were created at that same second.

Measurement gate / ubuntu-latest gate-slim / ubuntu-slim
Job created → job started (scheduling/start wait) 3 s 3 s
Dependencies finished → check step started 4 s 5 s
Set up job duration <1 s <1 s
Dependencies finished → job completed 6 s 8 s

REST job/step timestamps have second-level precision; a reported zero-second duration means the timestamps fell within the same second, not that the operation was instantaneous. Logs corroborate the result: the first Check results log was at 16:04:13.274 for standard versus 16:04:14.788 for slim. Both completed the actual check well under one second.

Finding: slim is compatible with the unchanged gate script, but this first pair shows no wait-time improvement. A 1–2 second difference in one sample is not evidence of a consistent speed difference. The created→started metric combines scheduling/provisioning and is not a pure provisioning measurement.

For context, eight recent completed CI runs before the experiment had standard-gate scheduling waits of 2–5 seconds except two merge-group runs at 39 seconds and 372 seconds. PR-only samples cannot establish whether slim would eliminate those merge-queue outliers, particularly because both runner types share standard-runner concurrency limits.

This remains a draft, do-not-merge experiment, with no auto-merge. The existing required gate and publishing dependencies are untouched. Collect several further matched run attempts (using “Re-run all jobs” on this PR) before making a latency recommendation. Neither standard runner costs money in this public repository.

Validation passed locally (task build, task test, task lint, actionlint, workflow invariants, diff check) and in CI; all PR checks are green.

@dgageot dgageot closed this Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci CI/CD workflows and pipeline kind/chore Maintenance, deps, CI, tooling (maps to chore: commit prefix)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants