Repository navigation
Conversation
The per-mutant strategy stops at the first test that detects a mutant, but it ran the covering tests in the order they were selected, so a slow test could run before a fast one that kills the mutant on its own. It now sorts them by baseline duration, then by name. BaselinedTest and CoveringTest carry the baseline duration instead of the timeout derived from it; mutant_timeout() derives the timeout. Resuming looks up every test's recorded outcome rather than stopping at the first test without one. Each run measures its own baselines, so two tests with close durations can swap places between the interrupted run and the resumed one, and the old lookup would then run tests again. Fixes #41. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both tests in the stop-at-the-first-kill test kill the mutant of add, and with the fastest-first order either can run first, so the per-test summary lines depend on timing. Check the combined file score instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #41.
What changes
The per-mutant strategy builds a mutant once, then runs its covering tests until one detects it. Until now, it ran them in the order the tests were selected, so a slow test could run before a fast test that would have killed the mutant on its own.
cover_mutantsnow orders each mutant's tests by baseline duration, fastest first, with the name as a tie-breaker.BaselinedTestandCoveringTestcarry the baseline duration rather than the timeout derived from it.mutant_timeout()derives the timeout, andCoveringTest.timeoutis now a property, so callers are unchanged.previous_outcomeslooks up every test's recorded outcome instead of stopping at the first test without one. Each run measures its own baselines, so two tests with close durations can swap places between the interrupted run and the resumed one. The old lookup would then run tests that had already run.covered_outcomesruns only the tests with no recorded outcome.--strategyrow says whatper-mutantdoes, including the order.The per-test strategy and the order of the mutants are unchanged.
Tests
sample_testssleeps for a second, and the fastother_testsalso kills the mutant ofadd. Onlyother_testsruns on that mutant, and every mutant's state matches the existing per-mutant expectations. Without the sort, the test fails with['sample_tests'] != ['other_tests'].add, and checks that exactly one runs. Either test can be the killer, so it checks the combined file score instead of the per-test summary line (fixed in a follow-up commit, after it showed up as flaky).tests/test_runner.pyfor resuming: a test recorded out of order isn't run again, and a recorded kill runs nothing. Both fail on the old lookup.python -W error -m pytest: 257 passed, 9 skipped.ruff format --check,ruff checkand lizard are clean.🤖 Generated with Claude Code