test: the pytest half of the skip-loop sweep that #998 did not ship (#994)

[jdatcmd]

#998 shipped test/selftest/470 with no pytest half. CONTEXT.md is explicit — "a test that exists in only one harness is not finished" — and neither I nor the reviewer caught it. This is that half.

It is not a port, and that is the reason to write it

	measures	can see
test/selftest/470	the corpus	a loop that drifted from its arms
test_skip_loop_arms.py	the instrument	a classifier that stopped being able to tell

A classifier that filed every site as armless would report zero mismatches, and 470's population premises would still pass on whatever loops remained. So this drives the same tool over planted trees whose right answer is known and asserts the classification itself.

The shell says the corpus is clean; this says the thing that reads the corpus can tell the difference. Where they disagree, one of them is wrong — which is the whole argument for two harnesses.

Six tests, seventeen checks

test	what it pins
..._naming_its_siblings_arms_is_compared_and_agrees	the clean site is compared, not quietly filed as armless or interpolated
..._rename_in_the_sibling_branch_is_caught	the point: a drifted name is a mismatch
..._clean_and_drifted_sites_differ_by_exactly_the_rename	the pair must disagree, or neither arm above is evidence
..._armless_branch_is_counted_as_armless_and_not_compared	a site nobody compared and a site that agreed are not one number
..._interpolated_sibling_is_reported_rather_than_compared_wrongly	it declines where a literal comparison would be false in both directions
..._every_loop_is_classified_into_exactly_one_category	compared + armless + interpolated == loops

Removal proofs

Each mutation was asserted to apply before the run, because a clean pass reads identically whether the code is load-bearing or the edit never landed.

stop reporting mismatches       -> ..._rename_in_the_sibling_branch_is_caught     FAIL
                                   ..._differ_by_exactly_the_rename              FAIL
compare the interpolated site   -> ..._interpolated_sibling_is_reported...        FAIL
                                   ..._classified_into_exactly_one_category       FAIL
tool restored                   -> 0 diff lines

The second is the one worth keeping. Removing the tool's refusal to compare manufactures a false mismatch on a correct site, and a guard that invents a red is the guard people switch off.

Two guards in the tree caught my omissions, which is the argument for both

Neither was told anything; both derived it from the corpus.

test_harness_deps              [1: undeclared:test_skip_loop_arms.py]
test_docs_cover_the_corpus     every test file and every test is named in TESTS.md:
                               got 'test_a_loop_..., test_a_rename_..., test_skip_loop_arms.py, ...'

The first is the sixth time that arm has decided a membership rather than been told one.

Verified

255 passed / 580 checks / 0 fail   the 14 files the CI job runs, derived from NO_CLUSTER
venv pinned to requirements-test.txt (pytest 9.1.1, xdist 3.8.0), psycopg ABSENT
sweep on this branch               loops 8, compared 6, interpolated 1, armless 1 (= main)
CHANGELOG                          0 em or en dashes

No shell file changes, so no ledger movement: check_ledger.tsv covers harness_selftest and native_join_runtime_filter, and this adds no shell check. test/selftest/350 reads only ci.yml now, so no shell part reads the pytest corpus.

What this does not do

• It does not close One skip stands in for 28 named arms across six suites, so a skipped arm and a deleted one are indistinguishable #994. That issue is closed; this repays a debt test: a skipped arm records under the name it would have used (#994) #998 left behind it. No closing keyword.
• It does not add the third sweep counter. Bare suite-level skips are still invisible to the tool, so armless 0, interpolated 0 remains necessary but not sufficient for arming an absence-based gate. That is agreed follow-up work of mine, deliberately not bundled here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EbyGSaU93XYQr8aH4NrUiw

[OffgridwithJD]

Reviewed from the file list first, which is the lesson from the bundles and not a formality. Four text files, nothing derived, nothing binary, nothing outside test/ but the changelog the repo's own rule requires.

Your two questions, answered — and on the first one your own removal proof settles it better than my opinion does.

1. "Planted trees" is a second measurement, not a rationalisation

Compare the interpolated site → the interpolated and partition arms go red. That mutation is invisible to 470 by construction: 470 asks "do the corpus's loops agree with their arms", and a tool that has lost its refusal to compare still answers that, on whatever loops remain, reporting zero mismatches. Only a planted tree with a known answer sees it. Two questions, two subjects, one tool between them — the relationship test_mutation_ledger.py already has to pgc_ledger.py, stated in its docstring rather than after the fact. Not convenient reasoning.

2. Your new control does real work, and I checked the thing that would have made it circular

drift_line is computed from the planted body —

drift_line = next(i + 1 for i, l in enumerate(body.splitlines())
                  if l.lstrip().startswith("for _b in"))

— not from the tool's output. That is the difference between a known answer and an identity. Had it been parsed out of the MISMATCH line, the arm would have asserted that the tool agrees with itself.

And premise: BOTH loops were compared is what makes the pin mean "one of two" rather than "one of one". That premise is the arm.

The per-bucket negatives are what make the partition load-bearing, and you put that in words: a classifier filing everything as armless satisfies compared + armless + interpolated == loops perfectly. Each bucket asserting "and NOT counted among the compared" is what closes it. Both halves or neither, as the file now says.

One finding, non-blocking: the pytest half cannot see an unusable tool

No test asserts the tool's exit code. I mutated main()'s success return to 2 — the tool still prints correct counters — and asserted the behaviour changed before concluding anything (tool rc now = 2):

pytest half   6 passed / 18 checks          BLIND
shell half    1 FAIL
              premise: the sweep found skip loops at all: got [only none] want [yes]

470 catches it because of || _sk_out="TOOL FAILED: $_sk_out", and the seds then find no counters. So the pair is stronger than either half — but the pytest half's claim, the classifier still classifies, would survive the tool becoming unusable, and that is the half a person runs by hand.

The sharpest detail is which arm caught it in the shell half. Not the headline — every comparable skip loop names exactly the arms its sibling would emit still PASSED under the mutation. A premise caught it. That is the argument for premises in one line, and it is why I would add one here rather than treat the rc as a detail: _sweep returns the counters and drops returncode, so one expect.num(r.returncode, 0, ...) inside it covers all six tests at once.

My first attempt at this probe was invalid and the premise is what said so. I appended sys.exit(2) at module level, after the if __name__ == "__main__" guard had already exited — unreachable. The edit applied, the behaviour did not change, and tool rc now: 0 is the line that caught it. An arm asserting only that a mutation applied would have reported the tree as blind on no evidence.

Two things I would not change

test_harness_deps.py's new entry carries its reason rather than just the filename, and says which guard caught the file's arrival. That is the sixth time that arm has decided a membership rather than been told one, and the comment records it.

And the docstring that distinguishes the two halves — corpus versus instrument — belongs in the file rather than in a PR description that nobody reads again.

On the debt itself

470 shipped without a pytest half, I approved it, and you merged it. The standard was visible in the adjacent PR — #993 shipped both halves in one change — and it went past both of us anyway. Worth saying plainly: what caught it was you re-reading CONTEXT.md, not any check. There is no guard that a new test/selftest/ part has a pytest twin, and I am not proposing one tonight, but it is the same shape as the bundles: a rule everyone agrees with and nothing enforces.

Verified here: test_skip_loop_arms.py 6 passed / 18 checks; whole corpus 353 passed / 867 checks, in the container's pinned venv.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[OffgridwithJD]

Approving at ac989b4d on 13 of 13 green. The finding is fixed and I verified the fix the three-step way rather than reading the diff.

unmutated                       6 passed, 24 checks   (18 before: the assertion adds one per test)
1. the edit landed              the last `return 0` -> `return 2`
2. the behaviour moved          tool rc 0 -> 2, counters still printed correctly
3. which checks moved           6 FAILED, including the partition test

All six, not one — which is what putting it in _sweep buys, and routing the partition test through the helper closed the one call site that had its own blindness.

The docstring is the part I would keep. It records that the shell half caught this through || _sk_out="TOOL FAILED" while a premise failed and the headline arm stayed green, and it says why that argues for the helper rather than for one test. Your generalisation of it is sharper than my finding was: an empty finding list and a tool that cannot run are the same observation to any arm that only counts findings. That is true of most headline arms in this harness, and the premise beside each one is the load-bearing half.

It also records that my first probe of this was invalid — sys.exit(2) appended after the __main__ guard, which applied cleanly and changed nothing. Asserting a mutation applied is not asserting the behaviour moved, and a file that says so where the next person will read it is worth more than the arm it explains.

On the substance I reviewed before, unchanged and still right: drift_line computed from the planted body rather than parsed from the tool's own MISMATCH line; premise: BOTH loops were compared making the pin one-of-two; and the per-bucket negatives being what make compared + armless + interpolated == loops load-bearing rather than satisfiable by filing everything as armless.

Verified here: 6 passed / 24 checks, and the whole corpus green in the container's pinned venv.

Merging is not mine; this is the approval only.