Skip to content

Split the two self-drive sweeps into one case per corpus pair - #26

Merged
bbfrederick merged 1 commit into
mainfrom
split-corpus-sweeps
Sep 30, 2026
Merged

bbfrederick merged 1 commit into
mainfrom
split-corpus-sweeps

Conversation

@bbfrederick

Copy link
Copy Markdown
Owner

Tier 2 of the CI speed-up. Tier 1 (#25) removed a duplicate run and took a job from 1817 s to 1066 s, leaving Test as 97.4% of it. This addresses what Test is bounded by.

The problem

xdist distributes whole tests, so a test that loops the corpus pins the corpus to one worker. The two self-drive sweeps were the suite's slowest tests for exactly that reason — ~400 s and ~200 s on a single core while the others idled — and no -n could reach inside a test body.

The change

One case per archive/PDF pair: 21 for the self-drive invariant, 6 for the named exceptions.

before after
the two sweeps together ~600 s 133 s (583% CPU)
suite's slowest test 400 s (self-drive) 97 s (unrelated conversion sweep)
slowest self-drive case — 82 s (CMRR_optionscan_P1.exar1)

The single-test floor drops ~400 s → ~97 s, and the binding constraint is no longer one test. Full suite green: 3943 passed, 26 skipped.

The cost, and the companion tests that pay it

A loop can assert something a parametrized case cannot. Both sweeps counted what they drove and asserted a floor — checked >= 10, available >= 4 — which is what stopped them passing on a corpus that had quietly stopped being discoverable.

One case per pair cannot make that claim. Demonstrated rather than assumed:

$ pytest -q test_vacuous.py        # a parametrized sweep whose list emptied
s                                                              [100%]
SKIPPED [1] ...: got empty parameter set for (pair)
1 skipped in 0.00s

Zero cases passing reads exactly like every case passing. So each sweep keeps a companion test asserting the length of the list its cases are generated from. Both falsified by truncating those lists to 2 and 1:

AssertionError: only 2 self-drivable pairs discovered; this proves little
assert 2 >= 10
AssertionError: only 1 of the named exceptions are present
assert 1 >= 4

The two guards overlap, which was not designed. An emptied parameter set is a skip in tests.test_exar_patch, so the CI report check added in #25 reports it too (::error::N .exar1 tests skipped). A guard phrased as coverage rather than outcome catches failure modes invented after it was written.

It also got stricter

The loop moved on silently when a printout stopped pairing with its archive (if not report.matched: continue), so a pair could leave the sweep without saying so. A case has to decide, so it asserts the pairing — and all 21 pass, meaning that branch had never once been taken in the sweep's lifetime.

Verification

  • Full suite: 3943 passed, 26 skipped, 0 failed (11:37 with -n auto)
  • The 27 new cases: 29 passed in 2:13
  • Empty-parameter-set hazard demonstrated; both companions falsified; mutation reverted and confirmed gone
  • black, isort, codespell clean

CLAUDE.md's "one test is the floor" paragraph predicted this fix and is rewritten with the measured numbers, the pattern (when the slowest test is a loop, parametrise rather than raise -n), and the non-vacuity caveat.

🤖 Generated with Claude Code

xdist distributes whole tests, so a test that loops the corpus pins the
whole corpus to one worker.  The two self-drive sweeps were the suite's
slowest tests for that reason -- ~400 s and ~200 s on a single core while
the others idled -- and no `-n` could reach inside them.

They are now one case per archive/PDF pair, 21 and 6.  The pair runs in
133 s against ~600 s, at 583% CPU, and the suite's slowest test is no
longer either of them: the floor drops from ~400 s to ~97 s, which is an
unrelated conversion sweep, with the worst self-drive case at 82 s.  Full
suite green at 3943 passed, 26 skipped.

The cost is real and each sweep keeps a companion test for it.  A loop can
count what it drove and assert a floor -- `checked >= 10`, `available >= 4`
-- which is what stopped these passing on a corpus that had quietly stopped
being discoverable.  One case per pair cannot: an empty parameter set is
reported as `1 skipped, got empty parameter set`, and zero cases passing
reads exactly like every case passing.  Verified, and both companions
falsified by truncating their lists: `only 2 self-drivable pairs
discovered; assert 2 >= 10` and `only 1 of the named exceptions are
present; assert 1 >= 4`.

Splitting also made the sweep stricter, by forcing a decision the loop
could dodge.  The loop moved on silently when a printout stopped pairing
with its archive (`if not report.matched: continue`), so a pair could leave
the sweep without saying so.  A case has to decide, so it asserts the
pairing -- and all 21 pass, meaning that branch had never once been taken.

Note the two guards now overlap: an emptied parameter set is a *skip* in
tests.test_exar_patch, so the CI report check added for the duplicate-run
fix reports it as well.  A guard phrased as coverage rather than outcome
catches failure modes invented after it was written.

CLAUDE.md's "one test is the floor" paragraph predicted this fix and is
rewritten with the measured numbers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bbfrederick
bbfrederick merged commit 4798cc2 into main Sep 30, 2026
9 checks passed
@bbfrederick
bbfrederick deleted the split-corpus-sweeps branch September 30, 2026 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant