Repository navigation

ci(fleet-patrols): prose and comments stop naming the four retired patrol workflow files #64

Workflow file for this run

.github/workflows/test-nightly-tiers.yml at 5fe8c76

	name: Nightly Tiers

	# The `e2e` and `live` test tiers, run nightly on `main` (#16455).
	#
	# Maintainer direction (2026-09-07, verbatim): 「我想的是测试会不会太多，是否都是
	# 必要的，是不是应该砍，每次修改都要完整的测试吗」. The two tiers leave the per-PR
	# and merge-queue runs (ci.yml's Test Core now runs under `OS_TEST_TIERS=queue`)
	# and run here, under `OS_TEST_TIERS=nightly`, selected by their EXISTING
	# filename tiers only — `.e2e.test.` and `.live.test.` — with no test file
	# renamed, deleted or edited to move it. `scripts/nightly-tiers.mjs` is the one
	# reader of the switch; its header carries the contract, the measured
	# population and the reason it is one module rather than a line per config.
	#
	# ## Shape
	#
	# The house nightly shape: `schedule` + `workflow_dispatch`, node 22, pnpm,
	# RESTORE-ONLY caches (a scheduled run reads main's pnpm store and Build Core's
	# turbo seeds; the per-push workflows own saving them, and a nightly that saved
	# would churn the rationed 10 GB pool the way PR-side saves once did — see
	# ci.yml's "Restore Turbo cache"). The turbo cache restore is what keeps the
	# dependency closure's `build` tasks a replay: `OS_TEST_TIERS` is hashed into
	# the `test` tasks ONLY (turbo.json), so the builds hit main's entries while
	# every test task misses by construction and runs for real.
	#
	# ## Sharding — TWO shards, derived rather than picked
	#
	# The package set is not the workspace: `scripts/nightly-tiers.mjs --packages`
	# enumerates exactly the packages that own a tier file, in the `turbo ls`
	# document shape, and `scripts/partition-test-shards.mjs` splits THAT with the
	# same measured weights, the same LPT placement and the same
	# `<pkg> k/n` slice grammar Test Core uses. Measured at 6eba38f5a3: one
	# package, `@objectstack/cli`, owning all 60 tier files; its whole suite was
	# 1231s on a hosted runner (run 34009395649) and the e2e tier is ~75% of it
	# (packages/cli/vitest.config.ts, the priced trade), so ≈ 15 min of tests plus
	# ~5 min of restore/install per runner. Under 60 minutes on ONE shard already —
	# but `FILE_SHARDED_PACKAGES` cuts the CLI into two vitest slices, and the
	# partitioner REFUSES to place two slices of one package in one bin
	# (`assertSlicesSpread`), so one shard is not a legal count and two is the
	# smallest one that is. Each shard then carries one slice, ≈ 8 min of tests.
	# Raise the matrix (three places below: `matrix.shard`, the job `name:`, and
	# `--shard k/N`) only when a second tier-owning package arrives heavy enough
	# for the partitioner's own balance line to say so.
	#
	# The second tier-owning package is `@objectstack/example-showcase`, and it is
	# not that heavy: one `*.live.test.ts` file (#26445), the showcase browser
	# smoke, which used to be its own scheduled workflow (Showcase Smoke) and
	# filed nothing when red. The partitioner keeps it whole ("not the run's
	# floor") and places it on one shard; that shard also builds the Console SPA
	# and installs Chromium for it (≈ 5 min, the "Prepare the showcase browser
	# smoke" step), which no other shard pays.
	#
	# ## What lands where — the house idiom of `fleet-patrols.yml`'s `platform-checklist-watchdog` job
	#
	# On green this workflow files nothing, edits nothing and closes nothing. On
	# red the `report` job — ONE writer, after both shards, never the shards
	# themselves (two matrix legs racing the same lookup would mint two cards) —
	# opens or refreshes ONE card: fixed title `nightly-tiers: red on main`, labels
	# `bug` · `domain:devx` · `priority:p1` on create only (grading is the devx
	# seat's and a refresh must not undo it), body carrying the run link and the
	# failing files, de-duplicated by scanning OPEN issues for that exact title
	# prefix through the repo-scoped REST API with this workflow's own token. A
	# CLOSED card is not reopened: red again after it was answered is a regression,
	# filed fresh.
	#
	# The body marker is plain text, ⛔ never an HTML comment: this platform's body
	# sanitizer is measured to eat short angle-bracket fragments, and a de-dup key
	# that can be swallowed files a duplicate a day. The failing files come from
	# vitest's own JSON reporter (`--reporter=json`, written per package, collected
	# per shard, read by `scripts/nightly-tiers.mjs --failing-files`), never from a
	# regex over the log — the log is presentation, the report is the contract.
	#
	# ⛔ A run on any ref but `main` NEVER writes to the board — a
	# `workflow_dispatch` on a branch, or the `pull_request` run below, proves the
	# transport, the sharding and the rendering on a real runner and publishes the
	# would-be card to the run summary instead. That is how the card that landed
	# this was smoke-tested.
	#
	# ⛔ This job is NOT a required context and must never become one: it runs on
	# no pull request but one editing its own two files, so a ruleset naming it
	# would block every other PR forever on a check that cannot report.

	on:
	schedule:
	# Nightly. The minute is offset off the top of the hour ON PURPOSE (scheduled
	# workflows queue behind everyone else's `:00` cron) and off every sibling
	# patrol's minute, so two of them never contend for the same runner minute.
	#
	# ⚠️ Two readers, one fact. This value is restated as `PATROL_SCHEDULE` in
	# the `env:` block below, where the report step computes the deadline the
	# card's own `Swept` line is judged against (#17720). Change the cron and
	# change that env in the same edit, or the card publishes a deadline nobody
	# schedules.
	- cron: '29 5 * * *'
	workflow_dispatch: {}
	# ⛔ PATHS-FILTERED TO THIS FILE AND THE SWITCH READER IT INVOKES, and that
	# filter is the whole reason this trigger is allowed to exist: a change to
	# the nightly itself is exercised before it merges — the posture every patrol
	# in this repo keeps, and the one `scripts/pm/dispatch-gates.mjs` pins
	# tree-wide (a family reachable only from a scheduled workflow is refused
	# there by name). A trigger that fires only on a pull request editing these
	# two files puts ≈ 15 minutes of e2e on no unrelated PR's critical path.
	#
	# ⛔ Do NOT widen this list toward the tests themselves: a `packages/**`
	# entry here would put the two tiers back on the per-PR path, which is the
	# thing the card this landed removed. A `pull_request` run never writes to
	# the board — its `github.ref` is the merge ref, not `main`, and the board
	# write is gated on `main` by name.
	pull_request:
	paths:
	- '.github/workflows/test-nightly-tiers.yml'
	- 'scripts/nightly-tiers.mjs'

	# Least privilege. The shards read the repo and publish artifacts of their own
	# run; the report job writes issues and nothing else — no label on anybody's
	# PR, no state, no comment stream.
	permissions:
	contents: read
	issues: write

	# One nightly at a time PER REF. A dispatch overlapping the schedule on `main`
	# would have two `report` jobs racing the same lookup, and the loser would mint
	# a duplicate; a `pull_request` run never writes, so it need not queue behind
	# `main`'s run — keying the group by ref keeps exactly the writers serial.
	concurrency:
	group: test-nightly-tiers-${{ github.ref }}
	cancel-in-progress: false

	env:
	# The card's IDENTITY — the title prefix the de-dup scan keys on, and the
	# plain-text body marker that is its second way in. Single-sourced here so
	# the report step and a reader of this file see the same strings.
	CARD_TITLE: 'nightly-tiers: red on main'
	CARD_MARKER: os-nightly-tiers
	# Applied on CREATE only, additively; never rewritten on a refresh.
	CARD_LABELS: 'bug,domain:devx,priority:p1'
	# This workflow's own `cron:`, restated here so the report step can state the
	# cadence that makes the card's `Swept` line decidable (#17720). The sibling
	# patrol anchors carry the same heartbeat reading and one of them fires every
	# six hours, so a reader who carries a cadence over from a neighbour is wrong
	# by 4× — which is why each card states its own rather than leaving the
	# reader to go and read a workflow file.
	#
	# ⛔ A literal, not a parse of this file: the report body is inline
	# `github-script` that no test reaches, so the cheapest honest route is the
	# one with the least inline logic. The cost is the restatement above, and the
	# comment at the `cron:` is what pays it.
	PATROL_SCHEDULE: '29 5 * * *'

	jobs:
	tiers:
	name: Nightly tiers (${{ matrix.shard }}/2)
	runs-on: ubuntu-latest
	# Generous against the ≈ 13 min a shard is expected to take, and still
	# inside the hour the card set as the whole run's bound. A shard killed here
	# produces no vitest report; the report job says so rather than reading it
	# as "no failing files".
	timeout-minutes: 45
	strategy:
	# Both shards run to completion: the card should carry every failing
	# file, not the first shard's.
	fail-fast: false
	matrix:
	shard: [1, 2]

	steps:
	- name: Checkout repository
	uses: actions/checkout@v7

	# Restore-only: scheduled runs read main's store cache; the per-push
	# workflows own saving it.
	- name: Setup workspace
	uses: ./.github/actions/setup-workspace
	with:
	save-store: 'false'

	# Restore-only, from Build Core's namespace: that job builds the whole
	# workspace on every main push, so its entries replay every `build` task
	# this run depends on. The `test` tasks cannot replay from it — their
	# hash carries `OS_TEST_TIERS`, which no push run has ever set to
	# `nightly` — which is the property that makes a nightly a test run and
	# not a cache read.
	- name: Restore Turbo cache
	uses: actions/cache/restore@v6
	with:
	path: .turbo/cache
	key: ${{ runner.os }}-turbo-build-core-${{ github.ref_name }}-${{ github.sha }}
	restore-keys: \|
	${{ runner.os }}-turbo-build-core-${{ github.ref_name }}-
	${{ runner.os }}-turbo-build-core-

	# Two readings before a single test runs, both loud on failure:
	#
	# --check every tier-owning package honours the switch, judged by what
	# vitest COLLECTS under each setting (a package that adopted a
	# tier without reading the switch would otherwise run its whole
	# suite here under the nightly's name, and its tier files in the
	# merge queue — both silently);
	# --packages the tier-owning packages, in the `turbo ls` document shape
	# the partitioner reads, so this run shards exactly them.
	#
	# An EMPTY shard short-circuits the test step: `turbo run test` with zero
	# --filter args runs the entire workspace.
	- name: Compute this shard's package set
	run: \|
	node scripts/nightly-tiers.mjs --self-test
	node scripts/nightly-tiers.mjs --check
	node scripts/nightly-tiers.mjs --packages > "$RUNNER_TEMP/tier-packages.json"
	node scripts/partition-test-shards.mjs "$RUNNER_TEMP/tier-packages.json" \
	--shard ${{ matrix.shard }}/2 \
	> "$RUNNER_TEMP/shard-packages.txt"
	echo 'Items on this shard (a package name, or a package plus a k/n file-level slice):'
	cat "$RUNNER_TEMP/shard-packages.txt"

	# The showcase browser smoke (`examples/app-showcase/test/showcase-smoke.live.test.ts`)
	# drives the Console against a real backend through Playwright, and needs
	# two things `turbo run test` does not produce: the Console SPA the backend
	# serves at /_console (`packages/console/dist`, built by the `build:dist`
	# task — `pnpm console:build`), and a Chromium this `@playwright/test` can
	# launch. Only the shard the partitioner gave that package pays for them.
	# The test's own `^build` closure is the `test` task's, as for every
	# package here.
	#
	# `continue-on-error`, on purpose: the same shard carries a slice of the
	# CLI's e2e tier, and a Console build that fails one night must not take
	# that slice down with it. The night is still red, and still names the
	# file — the smoke then fails every test at once, and its failure text
	# says to read a whole-run red as the environment first; this step's red
	# in the log is the environment.
	- name: Prepare the showcase browser smoke
	continue-on-error: true
	run: \|
	if ! grep -qE '^@objectstack/example-showcase( \|$)' "$RUNNER_TEMP/shard-packages.txt"; then
	echo "@objectstack/example-showcase is not on this shard — nothing to prepare."
	exit 0
	fi
	pnpm console:build
	pnpm --filter @objectstack/example-showcase exec playwright install --with-deps chromium

	# The same leg loop as ci.yml's "Run this shard's tests", under the
	# nightly setting, with vitest's JSON reporter added beside the default
	# one: `--outputFile.json` is RELATIVE so each package writes its own
	# report under its own root (an absolute path would make a second
	# tier-owning package overwrite the first's), and the default reporter
	# stays so the stall guard still sees output flush.
	#
	# `--log-order=stream` is MANDATORY under the guard and `--concurrency=4`
	# matches the hosted runner's cores — both reasons are in ci.yml beside
	# the step this mirrors; NODE_OPTIONS arms the stall guard's SIGUSR2
	# diagnostic reports the same way.
	#
	# ⛔ A failing leg STOPS the remaining ones, as in ci.yml: a second full
	# leg on a job that is already red only risks turning an informative red
	# into a killed job with no report at all.
	#
	# One deliberate difference from ci.yml: the file-level SLICES run first
	# and the whole-package leg last. The whole packages here are the cheap
	# ones (the partitioner keeps a package whole when it is not the run's
	# floor) — today the showcase browser smoke — and a red in one of them
	# must not stop the shard's slice of the CLI's e2e tier from running.
	# The other way round, a red slice skips a few minutes of smoke.
	- name: Run this shard's nightly tiers
	env:
	OS_TEST_TIERS: nightly
	NODE_OPTIONS: --report-on-signal --report-signal=SIGUSR2 --report-directory=${{ runner.temp }}/stall-reports
	run: \|
	if [ ! -s "$RUNNER_TEMP/shard-packages.txt" ]; then
	echo "No packages on this shard — nothing to test."
	exit 0
	fi
	export VITEST_MAX_WORKERS="$(node scripts/vitest-worker-cap.mjs)"
	mkdir -p "$RUNNER_TEMP/stall-reports"

	FILTERS=""
	SLICES=""
	while read -r PKG SLICE; do
	[ -n "$PKG" ] \|\| continue
	if [ -n "$SLICE" ]; then
	SLICES="$SLICES $PKG=$SLICE"
	else
	FILTERS="$FILTERS --filter=$PKG"
	fi
	done < "$RUNNER_TEMP/shard-packages.txt"

	REPORTER="--reporter=default --reporter=json --outputFile.json=.nightly-tiers/vitest-report.json"
	STATUS=0
	LOGS=""
	for LEG in $SLICES __whole__; do
	if [ "$LEG" = __whole__ ]; then
	[ -n "$FILTERS" ] \|\| continue
	LOG="$RUNNER_TEMP/nightly-tiers-packages.log"
	set -- pnpm turbo run test $FILTERS --concurrency=4 --summarize --log-order=stream -- $REPORTER
	else
	PKG="${LEG%%=*}"
	SLICE="${LEG#*=}"
	LOG="$RUNNER_TEMP/nightly-tiers-slice-$(printf '%s' "$PKG" \| tr -c 'A-Za-z0-9' '-').log"
	set -- pnpm turbo run test "--filter=$PKG" --concurrency=4 --summarize --log-order=stream -- "--shard=$SLICE" $REPORTER
	fi
	LOGS="$LOGS $LOG"
	node scripts/run-with-stall-guard.mjs --log "$LOG" --stall-minutes 10 \
	--report-dir "$RUNNER_TEMP/stall-reports" -- "$@" \|\| { STATUS=$?; break; }
	done

	: > "$RUNNER_TEMP/nightly-tiers.log"
	for LOG in $LOGS; do
	if [ -f "$LOG" ]; then
	cat "$LOG" >> "$RUNNER_TEMP/nightly-tiers.log"
	fi
	done
	exit $STATUS

	# Gather what the report job needs, red or green: every package's vitest
	# JSON report (named by the package directory so two packages never
	# collide in the merged download) and the tail of this shard's log.
	# `if: always()` — a red shard is exactly when this earns its keep.
	- name: Collect this shard's reports
	if: always()
	run: \|
	OUT="$RUNNER_TEMP/nightly-reports/shard-${{ matrix.shard }}"
	mkdir -p "$OUT"
	find . -path '/node_modules' -prune -o -path '/.nightly-tiers/vitest-report.json' -print \
	\| while read -r REPORT; do
	PKG_DIR="$(dirname "$(dirname "$REPORT")")"
	cp "$REPORT" "$OUT/$(printf '%s' "$PKG_DIR" \| sed 's#^\./##' \| tr -c 'A-Za-z0-9' '-').json"
	done
	if [ -f "$RUNNER_TEMP/nightly-tiers.log" ]; then
	tail -n 200 "$RUNNER_TEMP/nightly-tiers.log" > "$OUT/log-tail.txt"
	fi
	echo "collected:"
	ls -la "$OUT"

	- name: Publish this shard's reports
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: nightly-tiers-reports-${{ matrix.shard }}-of-2
	path: ${{ runner.temp }}/nightly-reports/
	if-no-files-found: ignore
	retention-days: 14

	# The showcase smoke's screenshots, traces and error context, which
	# Playwright writes under its default `outputDir` beside its config.
	# `always()` and `ignore`: a red smoke leaves them, and so does a green
	# one that passed only on its retry (`retries: 1` under CI) — the flaky
	# first attempt is the evidence a time-dependent red needs. A clean green,
	# and every shard without the showcase, leave nothing and upload nothing.
	- name: Publish the showcase smoke's Playwright diagnostics
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: showcase-smoke-diagnostics-${{ matrix.shard }}
	path: examples/app-showcase/test-results/
	if-no-files-found: ignore
	retention-days: 7

	# The same two questions ci.yml's guard answers about a red suite — was
	# every test vitest COUNTED actually run, and did every scheduled package
	# report at all — asked of this shard's log against this shard's schedule.
	- name: Test completeness guard
	if: always()
	run: \|
	if [ ! -f "$RUNNER_TEMP/nightly-tiers.log" ]; then
	echo "No test log — the test step did not get far enough to produce one."
	exit 0
	fi
	node scripts/check-test-completeness.mjs "$RUNNER_TEMP/nightly-tiers.log" \
	--scheduled "$RUNNER_TEMP/shard-packages.txt" \
	--package-list "$RUNNER_TEMP/tier-packages.json"

	- name: Upload stall diagnostic reports
	if: failure()
	uses: actions/upload-artifact@v7
	with:
	name: stall-reports-nightly-tiers-${{ matrix.shard }}
	path: ${{ runner.temp }}/stall-reports/
	if-no-files-found: ignore
	retention-days: 14

	report:
	name: File or refresh the red card
	needs: tiers
	# `always()` spelled out: this job must run precisely when a shard FAILED or
	# was killed, which is the case the implicit `success()` wrapper would
	# skip. The board write below is further gated on the result and the ref.
	if: always()
	runs-on: ubuntu-latest
	timeout-minutes: 10
	steps:
	- name: Checkout repository
	uses: actions/checkout@v7

	- name: Setup Node.js
	uses: actions/setup-node@v7
	with:
	node-version-file: .nvmrc

	# No `pnpm install`: `scripts/nightly-tiers.mjs --failing-files` imports
	# `node:` builtins and two repo-local helpers only, so installing the
	# workspace here would buy nothing and would give an unattended nightly a
	# lockfile it could fail on.
	- name: Download the shards' reports
	uses: actions/download-artifact@v8
	with:
	pattern: nightly-tiers-reports-*
	path: ${{ runner.temp }}/nightly-reports
	merge-multiple: true

	# The verdict, read from the matrix job's RESULT — a status read, on
	# purpose: `failure` is a red tier, `cancelled` is a shard killed by its
	# timeout or by a human, and neither is a green nightly. The failing
	# files come from vitest's own JSON reports; a red run that produced none
	# (a build failure, a stall kill) says so in the body instead of reading
	# as "nothing failed".
	- name: Read the verdict
	id: verdict
	env:
	TIERS_RESULT: ${{ needs.tiers.result }}
	run: \|
	echo "tiers result: $TIERS_RESULT"
	echo "result=$TIERS_RESULT" >> "$GITHUB_OUTPUT"
	set +e
	node scripts/nightly-tiers.mjs --failing-files "$RUNNER_TEMP/nightly-reports" \
	> "$RUNNER_TEMP/failing-files.txt" 2> "$RUNNER_TEMP/failing-files.err"
	code=$?
	set -e
	# Captured with NO pipe in between: `cmd \| tail` would report the
	# pipe's status, and this exit code decides whether the list below is
	# a reading or a refusal.
	echo "failing_files_exit=$code" >> "$GITHUB_OUTPUT"
	cat "$RUNNER_TEMP/failing-files.err" >&2 \|\| true
	echo "failing nightly-tier files:"
	cat "$RUNNER_TEMP/failing-files.txt"

	- name: File or refresh the card
	# RED only, on MAIN only. On green nothing in this job writes anything;
	# on a branch (the smoke test a card's dev dispatches) the body goes to
	# the run summary and the board is untouched.
	if: steps.verdict.outputs.result != 'success' && github.ref == 'refs/heads/main'
	uses: actions/github-script@v9
	env:
	TIERS_RESULT: ${{ steps.verdict.outputs.result }}
	FAILING_FILES_EXIT: ${{ steps.verdict.outputs.failing_files_exit }}
	with:
	# Delivery is retried, never assumed: this write is the entire product
	# of a red run, and a transient answer from the issues endpoint would
	# otherwise discard a completed sweep.
	retries: 3
	script: \|
	// The body lives in scripts/ci/nightly-tiers-card.mjs (#26086), driven
	// offline by its --self-test; this step hands it the same github/context/core.
	const { fileNightlyCard } = await import(`${process.env.GITHUB_WORKSPACE}/scripts/ci/nightly-tiers-card.mjs`);
	await fileNightlyCard({ github, context, core });

	- name: Publish the verdict to the run summary
	# Always: on green this IS the whole product; on a branch it is the
	# smoke test's evidence; on a red main run it makes the run
	# self-contained for whoever opens it from the card.
	if: always()
	env:
	TIERS_RESULT: ${{ steps.verdict.outputs.result }}
	run: \|
	{
	echo "### Nightly tiers — tiers job result \`${TIERS_RESULT:-unread}\`"
	echo
	if [ "${{ github.ref }}" != "refs/heads/main" ]; then
	echo "_Board write skipped: this run is on \`${{ github.ref }}\`, not \`main\`. A branch run proves the sharding, the switch and the rendering without filing anything._"
	echo
	fi
	if [ "$TIERS_RESULT" = "success" ]; then
	echo "Green: every nightly-tier file passed. Nothing filed, nothing edited, nothing closed."
	else
	echo "Failing nightly-tier files (from the vitest JSON reports; empty means the run died outside vitest — read the log):"
	echo
	echo '```'
	cat "$RUNNER_TEMP/failing-files.txt" 2>/dev/null \|\| echo '(no list produced)'
	echo '```'
	fi
	} >> "$GITHUB_STEP_SUMMARY"

	- name: Fail the run if the verdict was never read
	# LAST, on purpose: the card is filed BEFORE this can turn the job red.
	# A red TIER is not this job's failure — the shard already went red and
	# the card carries it. What must never look green is a report job that
	# finished without reading the matrix result at all: nothing downstream
	# could tell that apart from a green nightly.
	if: always() && steps.verdict.outputs.result == ''
	run: \|
	echo "the verdict step recorded no result, so this run is NOT a reading about the nightly tiers -- it is neither a green one nor a red one. See this run's summary."
	exit 1

Navigation Menu

Repository navigation

ci(fleet-patrols): prose and comments stop naming the four retired patrol workflow files #64

Workflow file

ci(fleet-patrols): prose and comments stop naming the four retired patrol workflow files #64

Uh oh!

Workflow file for this run