diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bc4bd7f3243..19f0c2f0e80 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -478,31 +478,44 @@ jobs: # margin for a cold Turbo cache; the old 45 left a hung job "running" for # half an hour past any plausible healthy finish. # - # ⚠ RAISED 30 -> 45 (#16173), and the raise is still load-bearing. It was - # made because shard 5/6 was being killed at the 30-minute wall (12+ - # observations at 30:16-30:21 with every other job green; the green band - # topped out at 28:56, a ~1 minute margin), and so the tail could be - # measured uncensored before anyone re-derived - # `scripts/test-shard-timings.json`. Both are done: the dataset was - # refreshed (#20388, then #21826) and the split ran the whole CLI alone on - # shard 1/6 (#21487) until #22075 cut it into 3 slices. HELD at 45 by that - # change: the re-size is owed after it lands, from the measured post-change - # Test Core job walls, never from a prediction (#22075 carries it). + # History: RAISED 30 -> 45 (#16173) while one shard ran the whole CLI and + # was killed at the 30-minute wall; held at 45 until #22075 cut the CLI + # into 3 slices (5919483472). # - # ⛔ REVERT CONDITION, restated against measured wall time, not a card: - # back to `30` only when the slowest shard's JOB wall time stays at or - # under 24 minutes (80% of 30, so the margin the old wall lacked is - # there) on every scheduled run for a week. It does not today. On the 16 - # main runs after the #21826 refresh (37413386379 to 37467882762), shard - # 1/6 — the CLI alone — read 6m14s-35m43s of job wall time, over 30 - # minutes on 9 of them (34m39s run 37415122516, 35m43s run 37453598388, - # 34m08s run 37460325624), so the condition as first written ("once - # #16173 lands") would now kill that shard; and its slowest reading is - # already 79% of this 45. Read the `Test Core (1/6)` job's started and - # completed times before touching this value. It carries no other expiry - # — a raise with no revert condition beside it becomes permanent by + # ⚠ RE-SIZED 45 -> 50 (#22075), derived so that BOTH stall-guard verdicts + # land before this wall on every observed run. Window: every completed, + # non-cancelled CI run whose tested tree carries 5919483472, runs + # 37917885267 to 37956350887 (2026-10-09 10:29Z-16:03Z): 5 schedule, + # 6 push, 30 merge_group and 25 pull_request runs, 396 `Test Core (N/6)` + # jobs; 24 cancelled runs excluded. Slowest job wall per event, in + # minutes: schedule 21.0, push 20.9, merge_group 24.1, pull_request 27.1; + # p90 of all 396 is 22.0. Measured with + # `node scripts/measure-stall-guard-headroom.mjs` over those jobs, the + # worst p + s (job start to the end of "Run this shard's tests") is + # 26m38s, on run 37923333449 `Test Core (3/6)`. Three readings set the + # value, and the largest answer wins: + # * the window verdict, a silent freeze killed 10m after the last + # output: p + s + 10m = 36m38s; + # * the cap-deferred verdict, when the liveness probe sees the suite + # still writing: p + s + 20m = 46m38s, inside 50 by 3m22s. It missed + # 45 on 8 of the 396 jobs and would miss 40 on 60; + # * the 80% wall margin: the slowest job wall, 27m05s on that same job + # (70 packages executed, the CLI as all 3 slices), is within 80% of + # 50 (40.0). + # + # ⛔ REVERT CONDITION, restated against measurement, not a card: a value + # below 50 is allowed only when, on a re-measured window of Test Core + # runs (`node scripts/measure-stall-guard-headroom.mjs` over the jobs + # payloads of every completed pull_request, merge_group, push and + # schedule run in it), BOTH guard verdicts -- the window path + # (p + s + 10m) and the cap-deferred path (p + s + 20m) -- land before the + # new value on every observation, AND the slowest `Test Core (N/6)` job + # wall is at or under 80% of it. Read the pull_request and merge_group + # runs, not the hourly schedule alone: scheduled runs mostly replay the + # cache their commit's push run saved. It carries no other expiry — a + # raise with no revert condition beside it becomes permanent by # forgetting — and the stall guard stays the primary hang detector. - timeout-minutes: 45 + timeout-minutes: 50 permissions: contents: read strategy: