Skip to content

Fix #41: run each mutant's covering tests fastest first - #48

Open
lefticus wants to merge 2 commits into
mainfrom
fix/issue-41-fastest-tests-first
Open

lefticus wants to merge 2 commits into
mainfrom
fix/issue-41-fastest-tests-first

Conversation

@lefticus

@lefticus lefticus commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Fixes #41.

What changes

The per-mutant strategy builds a mutant once, then runs its covering tests until one detects it. Until now, it ran them in the order the tests were selected, so a slow test could run before a fast test that would have killed the mutant on its own. cover_mutants now orders each mutant's tests by baseline duration, fastest first, with the name as a tie-breaker.

  • BaselinedTest and CoveringTest carry the baseline duration rather than the timeout derived from it. mutant_timeout() derives the timeout, and CoveringTest.timeout is now a property, so callers are unchanged.
  • Resume: previous_outcomes looks up every test's recorded outcome instead of stopping at the first test without one. Each run measures its own baselines, so two tests with close durations can swap places between the interrupted run and the resumed one. The old lookup would then run tests that had already run. covered_outcomes runs only the tests with no recorded outcome.
  • The README's --strategy row says what per-mutant does, including the order.

The per-test strategy and the order of the mutants are unchanged.

Tests

  • New end-to-end test (the acceptance in the issue): sample_tests sleeps for a second, and the fast other_tests also kills the mutant of add. Only other_tests runs on that mutant, and every mutant's state matches the existing per-mutant expectations. Without the sort, the test fails with ['sample_tests'] != ['other_tests'].
  • Two existing end-to-end tests expected the selection order. Both fixture tests take a few milliseconds, so with sorting their order depends on timing.
    • The "stops at the first kill" test now has both tests kill add, and checks that exactly one runs. Either test can be the killer, so it checks the combined file score instead of the per-test summary line (fixed in a follow-up commit, after it showed up as flaky).
    • The "every covering test of a survivor" test compares the tests as a set.
  • New unit tests in tests/test_runner.py for resuming: a test recorded out of order isn't run again, and a recorded kill runs nothing. Both fail on the old lookup.
  • python -W error -m pytest: 257 passed, 9 skipped. ruff format --check, ruff check and lizard are clean.

🤖 Generated with Claude Code

The per-mutant strategy stops at the first test that detects a mutant, but
it ran the covering tests in the order they were selected, so a slow test
could run before a fast one that kills the mutant on its own. It now sorts
them by baseline duration, then by name.

BaselinedTest and CoveringTest carry the baseline duration instead of the
timeout derived from it; mutant_timeout() derives the timeout.

Resuming looks up every test's recorded outcome rather than stopping at
the first test without one. Each run measures its own baselines, so two
tests with close durations can swap places between the interrupted run and
the resumed one, and the old lookup would then run tests again.

Fixes #41.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both tests in the stop-at-the-first-kill test kill the mutant of add, and
with the fastest-first order either can run first, so the per-test summary
lines depend on timing. Check the combined file score instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run each mutant's covering tests fastest first

1 participant