Skip to content

ci: select the remaining orphan tests in their owning runners - #4260

Merged
miguel-heygen merged 4 commits into
mainfrom
hfoss39/orphans-test-runners
Sep 22, 2026
Merged

miguel-heygen merged 4 commits into
mainfrom
hfoss39/orphans-test-runners

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Selects the seven remaining orphan tests through their owning runners. Producer discovery covers source and fixture directories and assigns the two real-media tests to the integration lane. The scripts runner selects the Node tests and catalog payload Vitest. Registry edits trigger their checks; skills-only edits select the creator-capabilities guard.

The newly selected capability tests exposed stale assertions from before #4153 added speed ramps. Both now require the documented rate lane in data-automation and its precedence over constant playback rate. No tests are deleted or skipped, and no dependency is added. Catalog preview tests receive real FFmpeg and the installed Chrome path.

On head 7edec024d53d5a5d9d754631fe76fd7eaeb54b3d, reachability CI measured seven findings before and zero afterward.

Base report:

{
  "packages/producer/tests/distributed/_smoke/webm-concat-copy.test.ts": [
    "no CI runner selects this test"
  ],
  "packages/producer/tests/playback-rate-av-parity/playback-rate-av-parity.test.ts": [
    "no CI runner selects this test"
  ],
  "registry/components/pan-stations/pan-stations.test.mjs": [
    "no CI runner selects this test"
  ],
  "scripts/creator-editing-capabilities.test.mjs": [
    "no CI runner selects this test"
  ],
  "scripts/generate-catalog-payloads.test.ts": [
    "no CI runner selects this test"
  ],
  "scripts/generate-catalog-previews.test.ts": [
    "no CI runner selects this test"
  ],
  "scripts/registry-primitive-payloads.test.ts": [
    "no CI runner selects this test"
  ]
}

Head report:

{}

Focused CI output:

# tests 25
# pass 25
# fail 0
Test reachability verified: 0 baselined test files.

The skills job also passed both updated assertions:

ok 293 - keyframes states truthful creator capabilities and ownership boundaries
ok 295 - audio skill owns placed-track fades, automation, ducking, and effects
# fail 0

The Test and producer integration jobs execute the newly selected scripts and media tests. #4261 removes the empty baseline and makes its presence an error.

@miguel-heygen
miguel-heygen force-pushed the hfoss39/orphans-media-use branch from 00454f5 to 0adfec6 Compare September 22, 2026 01:16
@miguel-heygen
miguel-heygen force-pushed the hfoss39/orphans-test-runners branch from bd9ce21 to a1b74a0 Compare September 22, 2026 01:16
@miguel-heygen
miguel-heygen force-pushed the hfoss39/orphans-media-use branch from 0adfec6 to 673f3c4 Compare September 22, 2026 01:49
@miguel-heygen
miguel-heygen force-pushed the hfoss39/orphans-test-runners branch from a1b74a0 to bf88e33 Compare September 22, 2026 01:53
Base automatically changed from hfoss39/orphans-media-use to main September 22, 2026 02:05
@miguel-heygen
miguel-heygen force-pushed the hfoss39/orphans-test-runners branch 2 times, most recently from db77982 to 7edec02 Compare September 22, 2026 02:37
#4153 added supported speed ramps as rate lanes in data-automation. The
previously unselected capability tests still required preprocessing.
Require the documented rate-lane representation and its precedence over
the constant playback rate, matching the runtime resolver.
@miguel-heygen
miguel-heygen force-pushed the hfoss39/orphans-test-runners branch from 7edec02 to 38c218b Compare September 22, 2026 04:16
@miguel-heygen
miguel-heygen marked this pull request as ready for review September 22, 2026 04:42

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The claim this PR rests on is that the reachability baseline is genuinely empty rather than merely zeroed, so that is what I spent the review on.

The baseline is earned, not just edited

Ran the tool at this head: --report prints {}, and the gate prints Test reachability verified: 0 baselined test files. with exit 0. Its own suite is 25/25.

That alone doesn't separate "the orphans got wired in" from "the detector went blind", so I checked by injection. Removing one wiring — pulling scripts/registry-primitive-payloads.test.ts back out of test:scripts — makes the tool report no CI runner selects this test and the gate exit 1. Restore it and the gate is back to exit 0. Both directions, so the zero is real. (First pass I read the exit code through a pipe and got the tail status; the numbers above are measured directly.)

Every orphan landed in a runner that can execute it

This is the part that would fail quietly — a test wired into a lane whose runner can't load it just errors at collection. Checked each file's framework import against the lane it was added to:

File Lane it was added to Imports
registry/components/pan-stations/pan-stations.test.mjs node --test node:test
scripts/creator-editing-capabilities.test.mjs node --test (skills + scripts) node:test
scripts/generate-catalog-previews.test.ts node --test via tsx node:test
scripts/registry-primitive-payloads.test.ts node --test via tsx node:test
scripts/generate-catalog-payloads.test.ts vitest run vitest
packages/producer/tests/…/webm-concat-copy.test.ts producer bun integration bun:test
packages/producer/tests/…/playback-rate-av-parity.test.ts producer bun integration bun:test

Every one matches. Ran the three that don't need a browser: capability test 14/14, pan-stations 1/1, registry-primitive-payloads 1/1.

The producer walk doesn't over-widen

discoverProducerTests now walks tests as well as src, and classifyTestSource throws on any file importing neither runner — so a stray .test.ts under tests/ would take the whole producer lane down rather than being skipped. packages/producer/tests holds exactly two *.test.ts files, both bun:test, both in INTEGRATION_TEST_FILES. So the new unit test's deepEqual against those two is exhaustive, not a sample, and the sort by file makes the order it asserts deterministic. The test-classification.mjs sha256 pinned in test-reachability.json matches the file at this head byte for byte.

The capability assertions are a correction, not a relaxation

Worth separating, because swapping "not supported" for "here's how it works" can be either. Ran the old regexes against the repo's current docs: both fail.

keyframes OLD    NO MATCH     keyframes NEW-1/NEW-2  MATCH
audio     OLD    NO MATCH     audio     NEW-1/NEW-2  MATCH

So the orphan wasn't merely unselected — it had been red against the repo's own skill docs and nothing could see it. git log -S puts both doc lines in 0c158be16 (#4153, the rate lane), and the test file's previous touch is 95bea1631 (#3928), which dates the drift exactly. Wiring it in without this edit would have landed a red.

The surrounding CI wiring

prepare-ffmpeg-bin writes a exec ffmpeg "$@" stub when which ffmpeg is empty, and its own description says jobs needing a real binary must apt-install it before calling it. The new install-ffmpeg-linux step sits immediately above it, so that constraint is satisfied. HYPERFRAMES_BROWSER_PATH is a real escape hatch, read at browserManager.ts:200.

main is 0 commits ahead of the merge base — the branch is already sitting on #4267's merge.

What I did not run

generate-catalog-previews.test.ts spawns a real browser render, and generate-catalog-payloads.test.ts pulls the producer through linkedom. My local setup borrows a sibling checkout's node_modules, which resolves workspace packages into that other tree, so both failed on module resolution in my environment. That's my harness, not this branch — I'm not claiming either passes, only that each is pointed at a runner that can load it.

Notes, neither blocking

  • scripts/creator-editing-capabilities.test.mjs is selected twice: by the skills runner and by test:scripts. Its neighbour check-media-use-copy-parity.test.mjs is already handled the same way, so this reads as the house pattern rather than a slip — flagging it only so the duplication stays a choice.
  • The Chrome wiring fails open. browserManager honours HYPERFRAMES_BROWSER_PATH only when non-empty, so if steps.catalog-chrome.outputs.chrome-path ever stops resolving, the env var is empty and the run silently falls back to default resolution instead of failing on the step that was meant to guarantee Chrome. A test -n on the output would make that loud.

— Rames

@miguel-heygen
miguel-heygen merged commit cff467a into main Sep 22, 2026
65 checks passed
@miguel-heygen
miguel-heygen deleted the hfoss39/orphans-test-runners branch September 22, 2026 04:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants