ci: select the remaining orphan tests in their owning runners - #4260
Conversation
00454f5 to
0adfec6
Compare
bd9ce21 to
a1b74a0
Compare
0adfec6 to
673f3c4
Compare
a1b74a0 to
bf88e33
Compare
db77982 to
7edec02
Compare
#4153 added supported speed ramps as rate lanes in data-automation. The previously unselected capability tests still required preprocessing. Require the documented rate-lane representation and its precedence over the constant playback rate, matching the runtime resolver.
7edec02 to
38c218b
Compare
jrusso1020
left a comment
There was a problem hiding this comment.
Approving. The claim this PR rests on is that the reachability baseline is genuinely empty rather than merely zeroed, so that is what I spent the review on.
The baseline is earned, not just edited
Ran the tool at this head: --report prints {}, and the gate prints Test reachability verified: 0 baselined test files. with exit 0. Its own suite is 25/25.
That alone doesn't separate "the orphans got wired in" from "the detector went blind", so I checked by injection. Removing one wiring — pulling scripts/registry-primitive-payloads.test.ts back out of test:scripts — makes the tool report no CI runner selects this test and the gate exit 1. Restore it and the gate is back to exit 0. Both directions, so the zero is real. (First pass I read the exit code through a pipe and got the tail status; the numbers above are measured directly.)
Every orphan landed in a runner that can execute it
This is the part that would fail quietly — a test wired into a lane whose runner can't load it just errors at collection. Checked each file's framework import against the lane it was added to:
| File | Lane it was added to | Imports |
|---|---|---|
registry/components/pan-stations/pan-stations.test.mjs |
node --test |
node:test |
scripts/creator-editing-capabilities.test.mjs |
node --test (skills + scripts) |
node:test |
scripts/generate-catalog-previews.test.ts |
node --test via tsx |
node:test |
scripts/registry-primitive-payloads.test.ts |
node --test via tsx |
node:test |
scripts/generate-catalog-payloads.test.ts |
vitest run |
vitest |
packages/producer/tests/…/webm-concat-copy.test.ts |
producer bun integration | bun:test |
packages/producer/tests/…/playback-rate-av-parity.test.ts |
producer bun integration | bun:test |
Every one matches. Ran the three that don't need a browser: capability test 14/14, pan-stations 1/1, registry-primitive-payloads 1/1.
The producer walk doesn't over-widen
discoverProducerTests now walks tests as well as src, and classifyTestSource throws on any file importing neither runner — so a stray .test.ts under tests/ would take the whole producer lane down rather than being skipped. packages/producer/tests holds exactly two *.test.ts files, both bun:test, both in INTEGRATION_TEST_FILES. So the new unit test's deepEqual against those two is exhaustive, not a sample, and the sort by file makes the order it asserts deterministic. The test-classification.mjs sha256 pinned in test-reachability.json matches the file at this head byte for byte.
The capability assertions are a correction, not a relaxation
Worth separating, because swapping "not supported" for "here's how it works" can be either. Ran the old regexes against the repo's current docs: both fail.
keyframes OLD NO MATCH keyframes NEW-1/NEW-2 MATCH
audio OLD NO MATCH audio NEW-1/NEW-2 MATCH
So the orphan wasn't merely unselected — it had been red against the repo's own skill docs and nothing could see it. git log -S puts both doc lines in 0c158be16 (#4153, the rate lane), and the test file's previous touch is 95bea1631 (#3928), which dates the drift exactly. Wiring it in without this edit would have landed a red.
The surrounding CI wiring
prepare-ffmpeg-bin writes a exec ffmpeg "$@" stub when which ffmpeg is empty, and its own description says jobs needing a real binary must apt-install it before calling it. The new install-ffmpeg-linux step sits immediately above it, so that constraint is satisfied. HYPERFRAMES_BROWSER_PATH is a real escape hatch, read at browserManager.ts:200.
main is 0 commits ahead of the merge base — the branch is already sitting on #4267's merge.
What I did not run
generate-catalog-previews.test.ts spawns a real browser render, and generate-catalog-payloads.test.ts pulls the producer through linkedom. My local setup borrows a sibling checkout's node_modules, which resolves workspace packages into that other tree, so both failed on module resolution in my environment. That's my harness, not this branch — I'm not claiming either passes, only that each is pointed at a runner that can load it.
Notes, neither blocking
scripts/creator-editing-capabilities.test.mjsis selected twice: by the skills runner and bytest:scripts. Its neighbourcheck-media-use-copy-parity.test.mjsis already handled the same way, so this reads as the house pattern rather than a slip — flagging it only so the duplication stays a choice.- The Chrome wiring fails open.
browserManagerhonoursHYPERFRAMES_BROWSER_PATHonly when non-empty, so ifsteps.catalog-chrome.outputs.chrome-pathever stops resolving, the env var is empty and the run silently falls back to default resolution instead of failing on the step that was meant to guarantee Chrome. Atest -non the output would make that loud.
— Rames
Selects the seven remaining orphan tests through their owning runners. Producer discovery covers source and fixture directories and assigns the two real-media tests to the integration lane. The scripts runner selects the Node tests and catalog payload Vitest. Registry edits trigger their checks; skills-only edits select the creator-capabilities guard.
The newly selected capability tests exposed stale assertions from before #4153 added speed ramps. Both now require the documented rate lane in data-automation and its precedence over constant playback rate. No tests are deleted or skipped, and no dependency is added. Catalog preview tests receive real FFmpeg and the installed Chrome path.
On head
7edec024d53d5a5d9d754631fe76fd7eaeb54b3d, reachability CI measured seven findings before and zero afterward.Base report:
{ "packages/producer/tests/distributed/_smoke/webm-concat-copy.test.ts": [ "no CI runner selects this test" ], "packages/producer/tests/playback-rate-av-parity/playback-rate-av-parity.test.ts": [ "no CI runner selects this test" ], "registry/components/pan-stations/pan-stations.test.mjs": [ "no CI runner selects this test" ], "scripts/creator-editing-capabilities.test.mjs": [ "no CI runner selects this test" ], "scripts/generate-catalog-payloads.test.ts": [ "no CI runner selects this test" ], "scripts/generate-catalog-previews.test.ts": [ "no CI runner selects this test" ], "scripts/registry-primitive-payloads.test.ts": [ "no CI runner selects this test" ] }Head report:
{}Focused CI output:
The skills job also passed both updated assertions:
The Test and producer integration jobs execute the newly selected scripts and media tests. #4261 removes the empty baseline and makes its presence an error.