ci: run media-use tests when media-use changes - #4259
Conversation
00454f5 to
0adfec6
Compare
0adfec6 to
673f3c4
Compare
jrusso1020
left a comment
There was a problem hiding this comment.
APPROVE at 673f3c40
Small, correct, and every number in the description reproduces. I ran the gate rather than reading it.
The claims, re-derived independently
I checked the base (86e27b00, which is #4257's merge) and the head out side by side and ran scripts/check-test-reachability.mjs --report in each:
BASE files: 45 findings: 83
HEAD files: 7 findings: 7
files that CLEARED: 38 files ADDED: 0
of which media-use: 38 non-media-use cleared: []
findings removed from media-use files: 76
That is the description's "83 findings across 45 files before, and 7 findings across 7 files afterward" and "all 76 media-use filter findings are gone", to the entry. I also parsed the two JSON blocks out of the PR body and compared them to my own runs: the complete head report is byte-identical to what the checker prints here, and the base excerpt for adopt.test.mjs matches my base report for that file exactly. node --test scripts/check-test-reachability.test.mjs gives # tests 25 / # pass 25 / # fail 0, and the gate itself prints Test reachability verified: 7 baselined test files. — both the same strings the description quotes.
files ADDED: 0 is the one I most wanted to see. Shrinking a baseline by 38 entries is only safe if nothing regressed into it while you were looking; nothing did.
The premise holds: the runner was already there, only the route was missing
test-skills discovers its set with find skills packages/cli/src/media-use -type f -name "*.test.mjs" and then appends scripts/check-media-use-copy-parity.test.mjs explicitly (ci.yml:427, :432). So both paths this PR adds to the skills filter are inputs the runner already selects — the job just never fired for them, because packages/cli/src/media-use/** wasn't in the filter. The two additions match the runner's inputs exactly, with nothing left over: the parity test reads skills/media-use/scripts/lib and packages/cli/src/media-use/lib, and the first of those was already covered by skills/**.
Red-greened in both directions
| arm | change | gate |
|---|---|---|
| A honest head | — | green, 7 baselined test files |
| B revert the two filter lines, keep the 7-entry baseline | the route removed | red, exit 1 — all 38 media-use files reappear |
| C keep the filter, restore the 83-entry baseline | the baseline left stale | red, exit 1 — lower baseline to 0 ×38 |
Arm B is what makes the baseline shrink load-bearing rather than cosmetic. Arm C is the more interesting one: the baseline is an exact set, not a ceiling, so a future fix that clears a violation is forced to delete its entry in the same commit. That is precisely the property a ratchet like this usually lacks, and it means the gate can't quietly stay slack where a regression would recur.
Three things to know, none of them blocking
- "Both review marks recorded" is not true at this head.
pulls/4259/reviewsreturns0,pulls/4259/commentsreturns0, andissues/4259/commentsis[]. Mine is the first mark on this PR. Flagging it because it's the second time this week a lane message has carried that line ahead of the reviews existing, and a stamp requested on the strength of it is a stamp requested on nothing. - The description still says "Draft until full CI and independent review pass."
isDraftisfalse. Stale line rather than a defect, but it reads as a self-imposed hold that isn't in force. - Neither job involved is a required check. The
mainruleset requires Semantic PR title, Test: runtime contract, Typecheck, Build, regression, Test, Render on windows-latest, Tests on windows-latest, Studio and player captures —Test reachabilityandTest: skillsare both absent. So this routes the media-use tests into a job that runs but cannot block a merge, and it tightens a gate that is advisory today. Strictly better than the status quo either way; worth knowing the protection is weaker than "these tests now guard the code" sounds.
Nit, and only that: the new Report base reachability step is a diagnostic living in the same job as the gate, with no continue-on-error. It can't misfire on findings — I confirmed --report exits 0 even with all 83 present — and fetch-depth: 0 is already on that checkout, so $BASE_SHA will resolve. Combined with the job not being required, the exposure is about nil. Mentioning it only so the next person who adds a step there knows the pattern.
No test assertions or application behaviour change, as the description says — I diffed both files and that holds.
— Rames
Runs the skills test job when packaged media-use code or its parity test changes. The runner already selects those tests; the path filter was the missing route.
On head
0adfec6960ccc0219892f2c48c02f5b2ccab8367, CI printed both reachability reports. Counting their entries confirms 83 findings across 45 files before, and 7 findings across 7 files afterward. All 76 media-use filter findings are gone.Base report excerpt (one of the 38 media-use files):
{ "packages/cli/src/media-use/lib/adopt.test.mjs": [ "CI filters exclude packages/cli/src/media-use/lib/adopt.test.mjs (for example packages/cli/src/media-use/lib/adopt.test.mjs)", "CI filters exclude packages/cli/src/media-use/lib/** (for example packages/cli/src/media-use/lib/adopt.mjs)" ] }Complete head report:
{ "packages/producer/tests/distributed/_smoke/webm-concat-copy.test.ts": [ "no CI runner selects this test" ], "packages/producer/tests/playback-rate-av-parity/playback-rate-av-parity.test.ts": [ "no CI runner selects this test" ], "registry/components/pan-stations/pan-stations.test.mjs": [ "no CI runner selects this test" ], "scripts/creator-editing-capabilities.test.mjs": [ "no CI runner selects this test" ], "scripts/generate-catalog-payloads.test.ts": [ "no CI runner selects this test" ], "scripts/generate-catalog-previews.test.ts": [ "no CI runner selects this test" ], "scripts/registry-primitive-payloads.test.ts": [ "no CI runner selects this test" ] }Focused CI output:
Based on main after the reachability gate merged. The seven unselected tests are wired in #4260; #4261 removes the baseline. No test assertions or application behavior change. Draft until full CI and independent review pass.