CI perf: floating toolchain versions in ci.yml e2e — upstream uv 0.13.2 red-mained the merge queue for 2.7 h (~4.3k job-min wasted on 10-20; ~0.45k min/day amortized) #1645

Measurement

Window: 2026-10-19 20:16 → 2026-10-20 20:16 UTC. I fetched jobs (filter=all) for all 129 merge_group CI runs, 4 push runs, the nightly, and 10 of 30 PR runs.

• Upstream uv 0.13.2 (PyPI, 10-19 20:59) broke e2e (ubuntu-latest, e2e_vex_build, pipenv:: --ignored, 2026.8.0): uv pip install -r pylock.toml → "The target environment is not supported by the lockfile". In the 18:13 head run that leg ran on base 14eee88 with no PR code involved (job 118803601782).
• The cause was the astral-sh/setup-uv step in the Pipenv / pip legs. It had no version:, so it always installed the latest uv.
• First red: nightly 39376690526 at 05:46. First fix PR: #1639 at 15:47, about 10 h later.
• What actually fixed main: the same pin also rode inside an unrelated PR, Fix yarn copies with invalid manifest names (#1598) #1605, as version: "0.13.1" (commit aea79f5). It merged at 18:38. Fix nightly Pipenv pylock legs broken by uv 0.13.2 #1639 is still open and now duplicates that pin.
• Merge queue stalled 15:54 → 18:38 (2 h 44 min, 0 merges). 17 consecutive queue heads built on red base 14eee88 failed that same leg and were evicted, about one every 6–7 min:
  • 39452719874 15:54
  • 39455777018 16:15
  • 39460266425 16:45
  • 39465635311 17:23
  • 39470447347 17:56
  • 39472900043 18:13
  • … 11 more.
• Wasted compute: 111 of 129 merge_group CI runs were cancelled or failed. They burned 4,320 of 5,154 merge_group job-min (84%), Linux plus Windows. This does not count the fail-fast and janitor watcher runs (128 more ubuntu-latest runs).
• Latency: the 10 PRs merged in the window were first enqueued at 15:50–15:51 and merged at 18:38–19:02, so first enqueue → merge took 167–191 min (p50 ~172). Their fast-host norm is 8–12.

This is the second upstream-drift incident in a week: the 10-14 nightly broke on old-uv legs and needed #1525.

Where the time goes

• Each eviction cycle took about 6–7 min: the pipenv leg fails about 5–6 min in, then fail-fast cancels the run and the queue rebuilds 5 entries.
• 17 cycles × about 5 runs × about 40 job-min each came to roughly 4.3k job-min.
• None of that work could succeed: every entry's base already failed the leg.

Root cause

The e2e legs take some oracle tools at floating versions:

• setup-uv with no version: (now pinned on main).
• python-version: '3.12.x' (ci.yml lines 1428 and 2682).
• node-version: '24.x' (lines 1422, 2241 and 2637).
• go-version: '1.24' (lines 403, 620 and 2702).

pin-check.yml enforces SHA pins for uses: only. Nothing enforces exact tool versions, so any upstream release can turn main red and stall the merge queue until a human lands a pin.

Proposed fix

1. Close Fix nightly Pipenv pylock legs broken by uv 0.13.2 #1639 or rebase it to a no-op. Main already carries version: "0.13.1".

2. Pin the remaining floating toolchains in ci.yml to exact patch versions:

   • python-version: '3.12.x' → '3.12.<n>'
   • node-version: '24.x' → '24.<m>.<p>' (the matrix legs already use 24.21.0)
   • go-version: '1.24' → '1.24.<p>'
   • Add a comment on each pin: "bump deliberately".

3. Extend pin-check.yml (or add scripts/check-tool-pins.py) to fail when:

   • an astral-sh/setup-uv / setup-uv step has no with.version;
   • any *-version: value under with: ends in .x, is a bare major or major.minor (e.g. '1.24'), or is latest.

   Matrix expressions (${{ matrix.* }}) are allowed when the matrix values are themselves exact.

4. Optional: in merge-queue-fail-fast.py, when the failing entry's ref is pr-<n>-<base_sha> and the same job also failed on the latest schedule or push run for base_sha, post a step summary saying "base is red, not this PR's failure". That points a human at the real fix sooner. It is observability only and changes no gating.

Expected saving

• The cost of one incident like 10-20: about 4.3k Linux+Windows job-min, plus about 2.7 h of zero merges (10 PRs delayed about 160 min each).
• Amortized at about 1 incident per 1–2 weeks (10-14 old uv, 10-20 uv 0.13.2): about 0.3–0.6k job-min/day and about 10–20 merge-queue min/day on average. The spikes are far worse.
• Critical path / PR latency on normal days: 0.

Coverage and risk

• Every test still runs in the same place. Only the tool versions change, from floating to pinned.
• New upstream releases then reach CI through deliberate bump PRs, which run the full PR gate, instead of through main.
• ci-ok and clippy names are unchanged.
• Risk: pins go stale. The bump PRs are cheap, and the pins keep CI reproducible.

Effort

S. One workflow edit plus about 40 lines of lint.

ROI

(0.45k job-min/day + ~15 MQ-min/day) × 0.6 confidence ÷ 1 (S) ≈ 9.3. The incident-weighted queue minutes dominate.

Generated by Claude Code