Skip to content

Commit 8ab0fb8

Browse files
committed
ci: wire the suite-duration ceiling check into Test Core and the refresh PR body
Test Core's aggregator runs check-test-suite-ceilings.mjs as its last step, with no continue-on-error, over the shard captures it already downloads; lint.yml runs the check's self-test. The refresh lane's PR body says whether it writes the first ceiling table or holds the existing one, and names the uncapped packages and any package that ran over its held ceiling in the window. Claude-Session: https://claude.ai/code/session_0115N1oNnQS5WqofZ2DzaT3q Co-authored-by: Claude <noreply@anthropic.com>
1 parent fa972f1 commit 8ab0fb8

3 files changed

Lines changed: 106 additions & 2 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1391,7 +1391,10 @@ jobs:
13911391
# redden it. Ordering after the verdict is what makes "cannot delay"
13921392
# true, and the two flags are what make "cannot redden" true — a
13931393
# diagnostics fault here (an artifact-service 403 is the measured one,
1394-
# #14469) must not evict a green run.
1394+
# #14469) must not evict a green run. The suite-duration ceiling check
1395+
# at the end of this job reads the same captures and is the one step
1396+
# here that DOES red the job, on purpose: it is a gate, not a report, and
1397+
# a capture that never arrived reads NOT MEASURED there, never red.
13951398
#
13961399
# ⛔ Steps, deliberately, NOT a new `needs:` entry: this job's
13971400
# `OS_AGGREGATOR_MEMBERS` / `OS_AGGREGATOR_NON_MEMBERS` block is asserted
@@ -1447,6 +1450,38 @@ jobs:
14471450
if-no-files-found: ignore
14481451
retention-days: 14
14491452

1453+
# ── The per-package suite-duration ratchet (#16468) ──────────────────
1454+
#
1455+
# A GATE, unlike the two reporting steps above: NO `continue-on-error`,
1456+
# so a package whose suite ran past its ceiling reds `Test Core`, naming
1457+
# the package and its slowest test files. The ceiling is the package's
1458+
# slowest executed run in the refresh window x 1.25, generated into
1459+
# scripts/test-shard-timings.json (`ceilings`) by the refresh lane and
1460+
# held by every later refresh; it rises only by a ruled entry in the
1461+
# script's RULED_CEILING_RAISES. A `provisional` package prints "no
1462+
# ceiling: provisional" and is not red; a package absent from the dataset
1463+
# is red (the new-package shape). See the script header for the ruling.
1464+
#
1465+
# HERE, in the aggregator, rather than in each shard, because the CLI's
1466+
# file-level slices land on three shards: the three windows are SUMMED
1467+
# and the sum graded against the whole package's ceiling -- the quantity
1468+
# the ceiling was measured in. It reads the shard captures the download
1469+
# step above already fetched (each package's turbo execution window via
1470+
# samplesFromSummary, and every vitest file line); a capture that did not
1471+
# arrive leaves its packages ungraded and says so, never red.
1472+
#
1473+
# NOT MEASURED, exit 0, while the dataset carries no `ceilings`: they are
1474+
# generated only where the run-summary artifacts are downloadable (the
1475+
# refresh lane), so the first refresh after this lands writes them, and
1476+
# until then every run prints a warning annotation saying so.
1477+
#
1478+
# LAST, after the reporting steps, so the timing table is published
1479+
# whatever this decides. `always()`: it reads only what the shards already
1480+
# captured, so a red shard's packages are graded too.
1481+
- name: Check the per-package suite-duration ceilings
1482+
if: always()
1483+
run: node scripts/check-test-suite-ceilings.mjs --captures "$RUNNER_TEMP/test-timings" --label 'Test Core'
1484+
14501485

14511486
# ── Temporal conformance against live, non-UTC servers (ADR-0053 D-A3) ─────
14521487
#

‎.github/workflows/lint.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4369,6 +4369,16 @@ jobs:
43694369
- name: Test-timing parser self-test
43704370
run: node scripts/report-test-timings.mjs --self-test
43714371

4372+
# The per-package suite-duration ratchet's own instrument (#16468). The
4373+
# gate runs in ci.yml's `Test Core` aggregator, where a clean tree with
4374+
# every suite under its ceiling reads OK whether or not the over-ceiling,
4375+
# absent-package and summed-slice rules still work -- so its self-test,
4376+
# which feeds those shapes in, runs here with teeth. Invoked as `node` for
4377+
# the GATE INVOCATION IDIOM reason at the top of this file. In-process
4378+
# fixtures, no IO, milliseconds.
4379+
- name: Suite-duration ceiling check self-test
4380+
run: node scripts/check-test-suite-ceilings.mjs --self-test
4381+
43724382
# Hand-written `.d.mts` mirrors (#10549). `scripts/js-comment-mask.mjs`
43734383
# and `scripts/check-regen-pending.mjs` are untyped `.mjs` that each ship
43744384
# a hand-written declaration beside them, and both files say "keep this in

‎.github/workflows/shard-timings-refresh.yml‎

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,16 @@
3333
# quoted verbatim into the PR body and a human decides. See "THE PINS" below.
3434
# ⛔ It never opens a PR when the regenerated file is byte-identical. A weekly
3535
# no-op PR trains everyone to ignore this PR.
36+
# ⛔ It never moves a suite-duration ceiling (#16468). The dataset also
37+
# carries `ceilings`, which `Test Core` grades every executed package
38+
# against (scripts/check-test-suite-ceilings.mjs). The generator writes
39+
# them in the same pass and HOLDS every ceiling the committed dataset
40+
# already has; a package gets a new one only when it had none (its slowest
41+
# executed run in this window x 1.25). A ceiling rises only by a ruling,
42+
# recorded in that script, so a refresh whose window ran slower REPORTS
43+
# the gap in the PR body instead of absorbing it. The first refresh after
44+
# the check landed writes the first table; until it merges the check reads
45+
# NOT MEASURED.
3646
#
3747
# HOW IT GETS ITS INPUTS, AND WHY IT RUNS HERE RATHER THAN IN AN AGENT WORKTREE
3848
# ----------------------------------------------------------------------------
@@ -634,6 +644,53 @@ jobs:
634644
}
635645
')
636646
647+
# The suite-duration ceilings (#16468), read out of the dataset like the
648+
# two lines above. The prior table is the COMMITTED dataset, read from
649+
# git because the bins step has already copied the refreshed file over
650+
# it. A refresh never moves a held ceiling, so the line counts what is
651+
# new and what is uncapped, and the spread table below marks every
652+
# held ceiling this window would have set differently -- and every one
653+
# a run of this window already went over.
654+
CEILING_LINE=$(node -e '
655+
const fs = require("fs");
656+
const { execFileSync } = require("child_process");
657+
const d = JSON.parse(fs.readFileSync(process.env.RUNNER_TEMP + "/refresh/refreshed.json", "utf8"));
658+
if (!d.ceilings || typeof d.ceilings !== "object" || !d.uncapped || typeof d.uncapped !== "object") {
659+
throw new Error("the refreshed dataset carries no `ceilings` / `uncapped`");
660+
}
661+
const prior = JSON.parse(execFileSync("git", ["show", "HEAD:scripts/test-shard-timings.json"], { encoding: "utf8" }));
662+
const held = prior.ceilings && typeof prior.ceilings === "object" ? prior.ceilings : null;
663+
const names = Object.keys(d.ceilings);
664+
const fresh = held === null ? names : names.filter((n) => !Object.hasOwn(held, n));
665+
const uncapped = Object.entries(d.uncapped).map(([n, why]) => n + " (" + why + ")");
666+
const h = d.provenance.ceilingHeadroom;
667+
const over = (fs.readFileSync(process.env.RUNNER_TEMP + "/refresh/spread.md", "utf8").match(/OVER its held ceiling/g) || []).length;
668+
const lines = [];
669+
if (held === null) {
670+
lines.push(
671+
"⚠️ This refresh writes the FIRST suite-duration ceiling table: " + names.length + " package(s) get a " +
672+
"ceiling, each its slowest executed run in this window x " + h + ". From the merge of this PR the `Test Core` " +
673+
"check (scripts/check-test-suite-ceilings.mjs) grades every executed package against it -- until then it " +
674+
"reads NOT MEASURED. Every later refresh HOLDS these numbers and a ceiling rises only by a ruling, so " +
675+
"review the ceiling column below before merging."
676+
);
677+
} else {
678+
lines.push(
679+
"Suite-duration ceilings: " + (names.length - fresh.length) + " held unchanged (a refresh never moves one; " +
680+
"a raise is a ruling), " + fresh.length + " new" + (fresh.length ? " (" + fresh.join(", ") + ")" : "") +
681+
", each new one its slowest executed run in this window x " + h + "."
682+
);
683+
}
684+
lines.push(uncapped.length ? "No ceiling: " + uncapped.join(", ") + "." : "Every package carries a ceiling.");
685+
if (over > 0) {
686+
lines.push(
687+
"⚠️ " + over + " package(s) ran OVER their held ceiling in this window (marked in the table): the brake " +
688+
"fired on main. This refresh does not raise them; that takes a ruling."
689+
);
690+
}
691+
console.log(lines.join("\n\n"));
692+
')
693+
637694
SHARDS=$(node -e '
638695
const rs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8"));
639696
const r = rs.find((x) => String(x.run_id) === process.env.RUN_ID);
@@ -675,10 +732,12 @@ jobs:
675732
echo "because a weekly lane cannot know which cards a given run ought to retire. If this is"
676733
echo "the first refresh to land, retire those two by hand as part of merging it."
677734
echo
678-
echo "## Sample depth and run-to-run spread"
735+
echo "## Sample depth, run-to-run spread and suite-duration ceilings"
679736
echo
680737
echo "$DEPTH_LINE"
681738
echo
739+
echo "$CEILING_LINE"
740+
echo
682741
cat "$RUNNER_TEMP/refresh/spread.md"
683742
echo
684743
echo "## Measured per-shard suite time on the newest run in the set"

0 commit comments

Comments
 (0)