Repository navigation
Establish a reproducible coverage baseline (4 conflicting numbers in circulation) #115
Description
Activity
- added a parent issue
on Aug 17, 2026 - added 4 commits that reference this issue
on Aug 18, 2026 Closed — one command, one number, and a floor that fails
$ python -m pytest tests/ -m "not real_world" --ignore=tests/real_world --ignore=tests/integration --cov=clustrix TOTAL 7665 2230 71% Required test coverage of 66.0% reached. Total coverage: 70.91% 1476 passed, 18 skipped, 0 failedCI runs that same command (
tests.yml:81), and no step carriescontinue-on-error.fail_under = 66is inpyproject.tomland was verified to fire rather than decorate — running a single test file under the same command exits 1 withRequired test coverage of 66.0% not reached. Total coverage: 14.40%.The four rival figures are down to one reproducible number.
Two honest caveats, so this does not read better than it is. The floor is a static 66, not a ratchet that rises on its own. And a large part of the movement came from deleting 17,809 lines of untested code, not from testing anything — of the recent ~2.66-point gain, roughly half is attributable to removing
enhanced_notebook_widget.pyalone. 70.91% is a real measurement; it is not 70.91% of the package this issue was filed against.
Part of #108 · Phase 1 · Depends on #110, and on the CI issue
Problem
Four different coverage numbers are in circulation. None of them was the number used for planning.
coverage_detailed_report.txt:77(committed, dated 2025-09-04)TOTAL 14524 13483 4446 33-> 5.69%pytest tests/unit/)TOTAL 14518 13109 4440 75-> 8.21% (15 passed in 3.50s)coverage combine)TOTAL 14518 6089 4440 542-> 56.14%Meanwhile
pyproject.tomlsetsfail_under = 90, a gate the project has never come close to and which is not enforced in CI anyway.An entire epic (#98) with eight child issues was planned against a number that cannot be reproduced. This is the single clearest example of the project's core problem: measurement was assumed rather than performed.
Caveat on the 56.14% figure
It is a slight underestimate, and the reason matters: the full coverage run reproducibly hangs at 98% in a real AWS retry loop (
clustrix/kubernetes/aws_provisioner.py:763,771) — the same landmine as #109. 56.14% comes from combined worker data at that point. A clean 100% figure is not obtainable until #109 lands.Zero-coverage modules worth noting (from the stale report)
cli_credentials.py(511 stmts, 0.00%),notebook_magic_widget.py(839, 0.00%),cloud_providers/aws.py(304, 0.00%),cloud_providers/azure.py(282, 0.00%), and all fourcost_providers/*(0.00%).Note the overlap with the dead-code issue: some of these are 0% because nothing imports them. Deleting orphaned modules will move the coverage number without writing a single test — which is the correct outcome, and another reason to re-baseline after Phase 4 rather than chase the current denominator.
Acceptance criteria
fail_underis set to something slightly below the real measured value, and enforced in CI (notcontinue-on-error)coverage_detailed_report.txt,coverage.json, andhtmlcov/are removed from the tree and gitignored — coverage is a CI artifact, not a committed fileVerification