Skip to content

ci: run media-use tests when media-use changes - #4259

Merged
miguel-heygen merged 1 commit into
mainfrom
hfoss39/orphans-media-use
Sep 22, 2026
Merged

miguel-heygen merged 1 commit into
mainfrom
hfoss39/orphans-media-use

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Runs the skills test job when packaged media-use code or its parity test changes. The runner already selects those tests; the path filter was the missing route.

On head 0adfec6960ccc0219892f2c48c02f5b2ccab8367, CI printed both reachability reports. Counting their entries confirms 83 findings across 45 files before, and 7 findings across 7 files afterward. All 76 media-use filter findings are gone.

Base report excerpt (one of the 38 media-use files):

{
  "packages/cli/src/media-use/lib/adopt.test.mjs": [
    "CI filters exclude packages/cli/src/media-use/lib/adopt.test.mjs (for example packages/cli/src/media-use/lib/adopt.test.mjs)",
    "CI filters exclude packages/cli/src/media-use/lib/** (for example packages/cli/src/media-use/lib/adopt.mjs)"
  ]
}

Complete head report:

{
  "packages/producer/tests/distributed/_smoke/webm-concat-copy.test.ts": [
    "no CI runner selects this test"
  ],
  "packages/producer/tests/playback-rate-av-parity/playback-rate-av-parity.test.ts": [
    "no CI runner selects this test"
  ],
  "registry/components/pan-stations/pan-stations.test.mjs": [
    "no CI runner selects this test"
  ],
  "scripts/creator-editing-capabilities.test.mjs": [
    "no CI runner selects this test"
  ],
  "scripts/generate-catalog-payloads.test.ts": [
    "no CI runner selects this test"
  ],
  "scripts/generate-catalog-previews.test.ts": [
    "no CI runner selects this test"
  ],
  "scripts/registry-primitive-payloads.test.ts": [
    "no CI runner selects this test"
  ]
}

Focused CI output:

# tests 25
# pass 25
# fail 0
Test reachability verified: 7 baselined test files.

Based on main after the reachability gate merged. The seven unselected tests are wired in #4260; #4261 removes the baseline. No test assertions or application behavior change. Draft until full CI and independent review pass.

Base automatically changed from hfoss39/gate-test-reachability to main September 22, 2026 01:14
@miguel-heygen
miguel-heygen force-pushed the hfoss39/orphans-media-use branch from 00454f5 to 0adfec6 Compare September 22, 2026 01:16
@miguel-heygen
miguel-heygen force-pushed the hfoss39/orphans-media-use branch from 0adfec6 to 673f3c4 Compare September 22, 2026 01:49
@miguel-heygen
miguel-heygen marked this pull request as ready for review September 22, 2026 01:51

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at 673f3c40

Small, correct, and every number in the description reproduces. I ran the gate rather than reading it.

The claims, re-derived independently

I checked the base (86e27b00, which is #4257's merge) and the head out side by side and ran scripts/check-test-reachability.mjs --report in each:

BASE  files: 45  findings: 83
HEAD  files:  7  findings:  7
files that CLEARED: 38   files ADDED: 0
  of which media-use: 38   non-media-use cleared: []
  findings removed from media-use files: 76

That is the description's "83 findings across 45 files before, and 7 findings across 7 files afterward" and "all 76 media-use filter findings are gone", to the entry. I also parsed the two JSON blocks out of the PR body and compared them to my own runs: the complete head report is byte-identical to what the checker prints here, and the base excerpt for adopt.test.mjs matches my base report for that file exactly. node --test scripts/check-test-reachability.test.mjs gives # tests 25 / # pass 25 / # fail 0, and the gate itself prints Test reachability verified: 7 baselined test files. — both the same strings the description quotes.

files ADDED: 0 is the one I most wanted to see. Shrinking a baseline by 38 entries is only safe if nothing regressed into it while you were looking; nothing did.

The premise holds: the runner was already there, only the route was missing

test-skills discovers its set with find skills packages/cli/src/media-use -type f -name "*.test.mjs" and then appends scripts/check-media-use-copy-parity.test.mjs explicitly (ci.yml:427, :432). So both paths this PR adds to the skills filter are inputs the runner already selects — the job just never fired for them, because packages/cli/src/media-use/** wasn't in the filter. The two additions match the runner's inputs exactly, with nothing left over: the parity test reads skills/media-use/scripts/lib and packages/cli/src/media-use/lib, and the first of those was already covered by skills/**.

Red-greened in both directions

arm change gate
A honest head green, 7 baselined test files
B revert the two filter lines, keep the 7-entry baseline the route removed red, exit 1 — all 38 media-use files reappear
C keep the filter, restore the 83-entry baseline the baseline left stale red, exit 1 — lower baseline to 0 ×38

Arm B is what makes the baseline shrink load-bearing rather than cosmetic. Arm C is the more interesting one: the baseline is an exact set, not a ceiling, so a future fix that clears a violation is forced to delete its entry in the same commit. That is precisely the property a ratchet like this usually lacks, and it means the gate can't quietly stay slack where a regression would recur.

Three things to know, none of them blocking

  1. "Both review marks recorded" is not true at this head. pulls/4259/reviews returns 0, pulls/4259/comments returns 0, and issues/4259/comments is []. Mine is the first mark on this PR. Flagging it because it's the second time this week a lane message has carried that line ahead of the reviews existing, and a stamp requested on the strength of it is a stamp requested on nothing.
  2. The description still says "Draft until full CI and independent review pass." isDraft is false. Stale line rather than a defect, but it reads as a self-imposed hold that isn't in force.
  3. Neither job involved is a required check. The main ruleset requires Semantic PR title, Test: runtime contract, Typecheck, Build, regression, Test, Render on windows-latest, Tests on windows-latest, Studio and player capturesTest reachability and Test: skills are both absent. So this routes the media-use tests into a job that runs but cannot block a merge, and it tightens a gate that is advisory today. Strictly better than the status quo either way; worth knowing the protection is weaker than "these tests now guard the code" sounds.

Nit, and only that: the new Report base reachability step is a diagnostic living in the same job as the gate, with no continue-on-error. It can't misfire on findings — I confirmed --report exits 0 even with all 83 present — and fetch-depth: 0 is already on that checkout, so $BASE_SHA will resolve. Combined with the job not being required, the exposure is about nil. Mentioning it only so the next person who adds a step there knows the pattern.

No test assertions or application behaviour change, as the description says — I diffed both files and that holds.

— Rames

@miguel-heygen
miguel-heygen merged commit a80d733 into main Sep 22, 2026
53 checks passed
@miguel-heygen
miguel-heygen deleted the hfoss39/orphans-media-use branch September 22, 2026 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants