Repository navigation
Make CI actually run the test suite (currently 15 of 2,280 tests) #113
Description
Activity
- added a parent issue
on Aug 17, 2026 Resolved
CI runs the whole non-billable suite.
.github/workflows/tests.yml:81:pytest tests/ -v --cov=clustrix --cov-report=xml --cov-report=html \ --cov-report=term-missing --cov-report=json \ -m "not real_world" --ignore=tests/real_world --ignore=tests/integration
Not
tests/unit/. That command collects 1258 tests on the current tree, against the 15 this issue reported.The steps can also fail now:
$ grep -n "continue-on-error\|--exit-zero\|if: false" .github/workflows/*.yml (no matches)--exit-zeroon flake8 andcontinue-on-error: trueon mypy are both gone, so the lint and type steps are load-bearing rather than decorative.Recent runs on this branch:
Testssuccess across ubuntu 3.10/3.11/3.12, macos 3.11/3.12 and windows 3.11/3.12;Fast CIsuccess.Two related things fixed since:
- The duplicate
real_world_tests.ymlworkflow — which had failed all 94 of its scheduled runs since 2025-08-24 without ever passing, because it referencedGCP_CREDENTIALSandAZURE_CREDENTIALS, secrets that were never created — was deleted in0e3490e. Onlyreal-world-tests.ymlremains. scripts/check_docs_examples.pynow runs in CI too (Fix all broken README/docs examples and unify the version across 4 locations #124).
One thing I could not verify: whether branch protection on
masteractually requires the test job.gh api repos/ContextLab/clustrix/branches/master/protectionwould settle it. A green workflow that is not a required check can still be merged past.- The duplicate
Part of #108 · Phase 1 · Depends on #109, #110
Problem
CI runs 15 of 2,280 tests (~0.7%).
tests.yml(push/PR)pytest tests/unit/ -m "not real_world" -n 2continue-on-error: truefast_ci.yml(PR)pytest tests/unit/ -m "not real_world and not slow"real_world_tests.ymlcontinue-on-error: true; failed 60/60 recent runsreal-world-tests.ymlif: falsetests/unit/is 5 files, 391 LOC, 15 tests out of 240 files / 89,950 LOC / 2,280 tests.~99.3% of the test suite never executes in CI — which is exactly why all 127 failures documented in the sibling issue are invisible, and why
fast_ci.ymlcould fail for a year unnoticed.There are also two near-duplicate workflow files,
real_world_tests.ymlandreal-world-tests.yml, which is its own source of confusion.Acceptance criteria
continue-on-error: trueremoved — including the mypy step intests.ymlif: false; either it runs, or the file is deletedreal_world_tests.yml/real-world-tests.ymlpair is consolidated to oneNote on sequencing
Turning CI on before fixing the 127 failures will make
masterpermanently red. Land the failure-fix issue first, or gate this behind a temporary allowlist that shrinks to zero — but do not solve it by re-addingcontinue-on-error.Verification
gh run list --workflow=tests.yml --limit 1 --json conclusion # and confirm the run's log reports ~1900 tests collected, not 15