Repository navigation
Split the two self-drive sweeps into one case per corpus pair - #26
Merged
Merged
Conversation
xdist distributes whole tests, so a test that loops the corpus pins the whole corpus to one worker. The two self-drive sweeps were the suite's slowest tests for that reason -- ~400 s and ~200 s on a single core while the others idled -- and no `-n` could reach inside them. They are now one case per archive/PDF pair, 21 and 6. The pair runs in 133 s against ~600 s, at 583% CPU, and the suite's slowest test is no longer either of them: the floor drops from ~400 s to ~97 s, which is an unrelated conversion sweep, with the worst self-drive case at 82 s. Full suite green at 3943 passed, 26 skipped. The cost is real and each sweep keeps a companion test for it. A loop can count what it drove and assert a floor -- `checked >= 10`, `available >= 4` -- which is what stopped these passing on a corpus that had quietly stopped being discoverable. One case per pair cannot: an empty parameter set is reported as `1 skipped, got empty parameter set`, and zero cases passing reads exactly like every case passing. Verified, and both companions falsified by truncating their lists: `only 2 self-drivable pairs discovered; assert 2 >= 10` and `only 1 of the named exceptions are present; assert 1 >= 4`. Splitting also made the sweep stricter, by forcing a decision the loop could dodge. The loop moved on silently when a printout stopped pairing with its archive (`if not report.matched: continue`), so a pair could leave the sweep without saying so. A case has to decide, so it asserts the pairing -- and all 21 pass, meaning that branch had never once been taken. Note the two guards now overlap: an emptied parameter set is a *skip* in tests.test_exar_patch, so the CI report check added for the duplicate-run fix reports it as well. A guard phrased as coverage rather than outcome catches failure modes invented after it was written. CLAUDE.md's "one test is the floor" paragraph predicted this fix and is rewritten with the measured numbers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier 2 of the CI speed-up. Tier 1 (#25) removed a duplicate run and took a job from 1817 s to 1066 s, leaving
Testas 97.4% of it. This addresses whatTestis bounded by.The problem
xdist distributes whole tests, so a test that loops the corpus pins the corpus to one worker. The two self-drive sweeps were the suite's slowest tests for exactly that reason — ~400 s and ~200 s on a single core while the others idled — and no
-ncould reach inside a test body.The change
One case per archive/PDF pair: 21 for the self-drive invariant, 6 for the named exceptions.
CMRR_optionscan_P1.exar1)The single-test floor drops ~400 s → ~97 s, and the binding constraint is no longer one test. Full suite green: 3943 passed, 26 skipped.
The cost, and the companion tests that pay it
A loop can assert something a parametrized case cannot. Both sweeps counted what they drove and asserted a floor —
checked >= 10,available >= 4— which is what stopped them passing on a corpus that had quietly stopped being discoverable.One case per pair cannot make that claim. Demonstrated rather than assumed:
Zero cases passing reads exactly like every case passing. So each sweep keeps a companion test asserting the length of the list its cases are generated from. Both falsified by truncating those lists to 2 and 1:
The two guards overlap, which was not designed. An emptied parameter set is a skip in
tests.test_exar_patch, so the CI report check added in #25 reports it too (::error::N .exar1 tests skipped). A guard phrased as coverage rather than outcome catches failure modes invented after it was written.It also got stricter
The loop moved on silently when a printout stopped pairing with its archive (
if not report.matched: continue), so a pair could leave the sweep without saying so. A case has to decide, so it asserts the pairing — and all 21 pass, meaning that branch had never once been taken in the sweep's lifetime.Verification
-n auto)CLAUDE.md's "one test is the floor" paragraph predicted this fix and is rewritten with the measured numbers, the pattern (when the slowest test is a loop, parametrise rather than raise
-n), and the non-vacuity caveat.🤖 Generated with Claude Code