Skip to content

Commit 2d2e3ce

Browse files
ci(nightly-tiers): the nightly states its own slicing, one CLI slice per shard (#23444)
Fixes #23407 Clause-②: no ## What was wrong (measured) The nightly tiers shard the CLI's e2e tier with Test Core's partitioner, and until now with Test Core's slice map, `FILE_SHARDED_PACKAGES`. That map has changed twice for Test Core's own reasons, and each change took the nightly's second shard away without any step going red: | tree | `FILE_SHARDED_PACKAGES` | `--shard 1/2` | `--shard 2/2` | |---|---|---|---| | `fd96a847` (`9ff74285^`) | cli: 2 | `@objectstack/cli 1/2` | `@objectstack/cli 2/2` | | `9ff74285` (2026-10-02, retired the entry) | empty | `@objectstack/cli` | (nothing) | | `3ca71b6e` (`59194834^`) | empty | `@objectstack/cli` | (nothing) | | `59194834` (2026-10-09, sliced 3 ways) | cli: 3 | `@objectstack/cli` | (nothing): "3 slices cannot spread across this run's shards" | | `e4ec6ec1a5` (run 38517756588) and `4f81f31526` (base) | cli: 3 | `@objectstack/cli` | (nothing): same refusal | Each row comes from `node scripts/nightly-tiers.mjs --packages` and then `node scripts/partition-test-shards.mjs DOC --shard k/2`, run against that tree's own scripts. The nightly runs agree with that history. This is the "Run this shard's nightly tiers" step's duration, read from the Actions jobs API: - 2026-09-30 and 2026-10-01: both shards non-empty, 1183s / 1190s and 1196s / 1168s. - 2026-10-04 onward: shard 2 ran 0s every night. On seven of eight green nights shard 1 took 2361-2615s; 2026-10-07 read 1589s. - 2026-10-14 (run 38517756588): shard 1 was cancelled at the 45-minute limit. Shard 2's log reads "No packages on this shard — nothing to test." So the workflow header's premise (FILE_SHARDED_PACKAGES gives the CLI two slices, one per shard) stopped holding at `9ff74285`, a week before `59194834`. `59194834` did not bring the second slice back, because 3 slices cannot spread over 2 shards. ## The fix The nightly now states its own slicing. Test Core's map no longer reaches it. - `scripts/nightly-tiers.mjs --packages` writes a `sliceAcrossShards` key naming every tier-owning package. Today that is `["@objectstack/cli"]`. - When a document carries that key, `scripts/partition-test-shards.mjs` cuts each named package into as many slices as the run has shards (the N of the run's own `--shard k/N`). `FILE_SHARDED_PACKAGES` does not apply to that run. `planShards()` (NEEDED / SPREADS / WITHIN CAP) and the vitest file floor still apply. - `turbo ls` never writes the key, so every Test Core document reads it as absent and slices exactly as before (proof below). When the key is present it is checked strictly: it must be an array of distinct names, and each name must be an item of the same document. - The planning in `main()` moved into one exported function, `planRun()`, in the same order. `main()` prints its result, and the self-test drives that same function. - No workflow change. The nightly keeps 2 shards. Because the count comes from the run's own `--shard k/N`, changing the nightly matrix later re-slices it with no planner edit. New output on the base tree's inputs: `--shard 1/2` gives `@objectstack/cli 1/2` and `--shard 2/2` gives `@objectstack/cli 2/2`, at 1102.9s predicted each. A third log line reads `slicing map: the document's sliceAcrossShards -- @objectstack/cli cut 2 ways, one slice per shard of this run; FILE_SHARDED_PACKAGES (Test Core's map) does not apply to this run`. At `--shard k/3` the output is `@objectstack/cli 1/3`, `2/3` and `3/3`. Expected wall time per shard: the whole tier ran 2361-2615s on one shard, and the 2-slice runs on 10-01 ran 1196s / 1168s. That points to roughly 1200-1310s of tests per shard (about 20-22 min), plus about 1 min of setup, against the 45-minute limit. ## Pin: a planner change that empties a shard again goes red `partition-test-shards.mjs --self-test` has a new battery, "a document states its own slicing: the nightly tiers", with 16 cases. lint.yml runs this self-test on every pull request. The battery covers: - An absent key slices by FILE_SHARDED_PACKAGES through `planRun()`. - Five malformed shapes are refused by name. - The run map is N ways, and a one-shard run cuts nothing. - One package at 2, 3 and 4 shards lands one distinct slice per shard. - A control case: 3 configured slices over 2 shards leaves one shard empty, so the predicate can go red. - `test-nightly-tiers.yml`'s matrix size equals the N in its `--shard` argument, and the shards are partitioned from `nightly-tiers.mjs --packages`. - Three live pins drive the nightly's own document from the live tree through `planRun()` at the workflow's shard count: the key names every tier package; no shard is empty; the CLI lands as one distinct slice on each shard. Red shown once, then restored. Both mutations were applied with `scripts/ablation-replace.mjs` on the committed fix (`e11d8343`); each restore was proven by blob hash equal to HEAD and an empty `git diff HEAD`: 1. Planner-side. `const sliced = acrossShards === null ? FILE_SHARDED_PACKAGES : sliceMapForRun(acrossShards, shardCount);` was replaced with `const sliced = FILE_SHARDED_PACKAGES;` (anchor 1 to 0, blob `6bc0cdf7948b` to `e08f8b5ec718`). The self-test exited 1 with: `nightly: the live document at the workflow's 2 shards leaves shard(s) 2 with nothing to run, while the split is [@objectstack/cli | ] (@objectstack/cli: whole (... but 3 slices cannot spread across this run's shards))`. That is the production failure. Restored to `6bc0cdf7948b`. 2. Producer-side. The key was dropped from `packageListDocument()` (blob `d75822731830` to `232c14a0299f`). The self-test exited 1 with `nightly document: sliceAcrossShards is undefined for the tier-owning packages ["@objectstack/cli"] ...`. Restored to `d75822731830`. ## Test Core's partition is byte-identical Test Core's partition did not change. Check: same command on both trees, base `4f81f315` (origin/main) and head `448c8745`, and `cmp` on stdout, stderr and exit code. The command is `node scripts/partition-test-shards.mjs DOC --shard k/6 --exclude @objectstack/dogfood`, for k = 1..6, over five documents: - the full `pnpm turbo ls --output=json` list (83 packages); - an affected subset with the CLI; - an affected subset without it; - the CLI alone; - an empty list. Result: 90 of 90 files identical. The concatenated sha256 is `885ce341b5b2d363...` on both trees, and all 30 exits are 0. The full list still puts `@objectstack/cli 1/3`, `2/3` and `3/3` on shards 2, 3 and 4. ## Gates (all on head `448c8745`) - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 32 commands from the 2-path change set. All 32 ran with exit 0, including `pnpm check:pm-dispatch-gates` (fast tier, 1841 cases). `--ran` reconciliation: 32 derived, 32 run, 0 NOT-MEASURED, 0 UNRUN. - The edited scripts' own suites: `nightly-tiers.mjs --self-test` (47 cases) and `partition-test-shards.mjs --self-test`, both exit 0. The partitioner's module graph gained an import, so the self-tests of the scripts that import it also ran: `check-test-completeness`, `check-test-suite-ceilings`, `measure-test-shard-timings` and `check-tier-file-adoption`, all exit 0. - Lint, narrowed to the 2 changed files: `npx eslint --no-inline-config --format json` read 2 files with 0 errors and 0 warnings. Both files are in eslint's population (`isPathIgnored` is false for each; `eslint.config.mjs` covers `**/*.{...,mjs,...}`). Type-aware linting is not enabled (no `project` / `projectService` in `eslint.config.mjs`), so this diff cannot change the verdict on any untouched file. Full `pnpm lint` is left to CI. - `skip-changeset`: this diff publishes nothing. The root `package.json` is private, `scripts/` is not shipped, and 0 public `files[]` entries name these scripts. As a positive control, 72 of 72 public manifests have a `dist` entry. - This PR triggers the nightly workflow itself (its `pull_request` paths filter names `scripts/nightly-tiers.mjs`). That run on real runners is the first two-shard reading. It writes nothing to the board. ## Acceptance notes - NOT MEASURED locally: whether the two vitest slices are disjoint. `vitest list --filesOnly --shard=k/2` ignores the shard: both slices listed all 82 tier files. The evidence that the `--shard=k/n` passthrough partitions the tier is the 2026-10-01 run (1196s / 1168s). This PR's own nightly run is the next reading. - The workflow header's "Sharding — TWO shards" paragraph is stale. So is the `timeout-minutes` comment ("about 13 min a shard"). The header says the shard count is spelled in three places; it is four, because the artifact name `nightly-tiers-reports-...-of-2` is the fourth. Correcting these is a governed (Tier H) workflow edit, so it is left out of this scripts-only PR. Carrier: the next PR that touches `test-nightly-tiers.yml`. - Nothing outside the workflow reads the nightly's shard count. The `report` job downloads with `pattern: nightly-tiers-reports-*`, and `check-test-completeness.mjs --scheduled` reads only its own shard's list. Neither depends on the count. The new self-test reads the matrix, so a mismatch between matrix and `--shard` now goes red. - An empty nightly shard still exits 0 in the workflow ("No packages on this shard"). That green is how this stayed hidden for twelve nights. With this change the planner-side cause is pinned on every pull request. --- _Generated by [Claude Code](https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4369677 commit 2d2e3ce

2 files changed

Lines changed: 309 additions & 14 deletions

File tree

‎scripts/nightly-tiers.mjs‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,34 @@
6767
* request that introduces it, which is the open question the landing card
6868
* records.
6969
*
70+
* ## The nightly's shards: one slice of the tier on EVERY shard, by the nightly's own count
71+
*
72+
* `--packages` is the nightly's whole input to `scripts/partition-test-shards.mjs`,
73+
* and the one tier-owning package (`@objectstack/cli`, 82 tier files at
74+
* 4f81f31526) has to reach every one of the nightly's shards as a slice of its
75+
* own: an item the split does not place on a shard leaves that shard running
76+
* nothing, reading green, while another carries the whole tier.
77+
*
78+
* The nightly used to get its slices by inheriting Test Core's
79+
* `FILE_SHARDED_PACKAGES`, a map derived for a different run -- Test Core's six
80+
* shards and its queue-tier weights -- and that inheritance broke twice with no
81+
* step red. 9ff74285 retired the CLI's entry (2026-10-02): from the next night
82+
* the second shard's test step ran 0s and the first carried the whole tier
83+
* (each slice had run 1196s / 1168s on 2026-10-01; the one shard ran up to
84+
* 2615s after). 59194834 re-sliced the CLI 3 ways (2026-10-09), and three
85+
* slices cannot spread over two shards, so the partitioner kept it whole until
86+
* run 38517756588 killed the first shard at its 45-minute limit.
87+
*
88+
* So the document states the nightly's slicing itself: `sliceAcrossShards`
89+
* names every tier-owning package, and the partitioner cuts each one into as
90+
* many slices as the run's own `--shard k/N` has shards, still subject to its
91+
* per-run slice decision and the vitest file floor. Test Core's map no longer
92+
* reaches the nightly, and a change to the nightly's matrix re-slices it with
93+
* no edit here. `partition-test-shards.mjs --self-test` (run on every pull
94+
* request by lint.yml) pins it on the live tree: this document, at the shard
95+
* count `test-nightly-tiers.yml` declares, yields that many non-empty shards,
96+
* each carrying one distinct slice of the CLI.
97+
*
7098
* ## Where `OS_TEST_TIERS` has to be declared to reach vitest at all
7199
*
72100
* turbo 2.10 runs in STRICT env mode: a variable that `turbo.json` neither
@@ -195,10 +223,15 @@ export function tierPackages(repoRoot = REPO_ROOT) {
195223
* the tier-owning packages with the same partitioner and the same slice
196224
* grammar the Test Core matrix uses. `count` is written because the
197225
* partitioner checks it against `items.length`.
226+
*
227+
* `sliceAcrossShards` names every tier-owning package: the nightly states its
228+
* own slicing rather than inheriting Test Core's `FILE_SHARDED_PACKAGES`, and
229+
* the partitioner cuts each named package into as many slices as the nightly
230+
* has shards -- see "The nightly's shards" in this file's header.
198231
*/
199232
export function packageListDocument(repoRoot = REPO_ROOT) {
200233
const items = tierPackages(repoRoot).map(({ name, path: dir }) => ({ name, path: dir }));
201-
return { packages: { count: items.length, items } };
234+
return { packages: { count: items.length, items }, sliceAcrossShards: items.map(({ name }) => name) };
202235
}
203236

204237
// ---------------------------------------------------------------------------
@@ -462,6 +495,12 @@ export function selfTest() {
462495
const doc = packageListDocument(root);
463496
check(() => assert(doc.packages.count === 2 && doc.packages.items.length === 2, 'the document counts what it lists'));
464497
check(() => assert(eq(doc.packages.items[1], { name: '@x/owner', path: 'packages/owner' }), 'items carry name and repo-relative path'));
498+
check(() =>
499+
assert(
500+
eq(doc.sliceAcrossShards, doc.packages.items.map((i) => i.name)),
501+
`the document states its own slicing, naming every tier-owning package -- got ${JSON.stringify(doc.sliceAcrossShards)}`
502+
)
503+
);
465504

466505
// -- failing files, from a report shaped like vitest's JSON reporter -------
467506
const known = owners.flatMap((o) => o.files.map((f) => `${o.path}/${f}`));
@@ -497,7 +536,7 @@ export function selfTest() {
497536
check(() => assert(judgeCollection('p', tier, ['test/c.test.ts'], ['test/a.e2e.test.ts']).some((p) => /does NOT collect 1 nightly-tier/.test(p)), 'a tier file missing from nightly runs nowhere'));
498537
check(() => assert(judgeCollection('p', tier, [], []).length === 1, 'an empty nightly collection is exactly one problem: the tier files run nowhere'));
499538

500-
const FLOOR = 45;
539+
const FLOOR = 47;
501540
assert(cases >= FLOOR, `${cases} cases ran, below the pinned floor of ${FLOOR} -- cases stopped running`);
502541
console.log(`\u2713 nightly-tiers self-test: ${cases} cases pass.`);
503542
return SELF_TEST_VERDICT;

0 commit comments

Comments
 (0)