Skip to content

Fix #42: run the cheapest mutants first, with an estimate of the time left - #51

Open
lefticus wants to merge 1 commit into
fix/issue-40-memory-limitfrom
fix/issue-42-cheapest-first-eta
Open

lefticus wants to merge 1 commit into
fix/issue-40-memory-limitfrom
fix/issue-42-cheapest-first-eta

Conversation

@lefticus

Copy link
Copy Markdown
Member

Fixes #42. Stacked on #50 (#40), which is stacked on #48 (#41). Merge those first; GitHub then retargets this PR, and the diff shrinks to this commit.

Order

With --strategy per-mutant, mutants now run cheapest first: by the sum of their covering tests' baseline durations, then by file and offset. That's --order cost, the default. --order source keeps the old order by file and offset. With per-test, every mutant of a test runs the same test, so the order doesn't change there.

Time left

strain/estimate.py estimates the time left. A mutant costs a rebuild of its file plus its covering tests' baseline durations. The baseline durations are an upper bound, since a kill stops the tests early. As mutants finish, the estimate learns:

  • each file's rebuild time: everything but the tests, so mutating, journaling and restoring count too. A file not rebuilt yet takes the average.
  • what fraction of the baseline durations the tests really take.

Mutants that finish without a build don't teach it anything: ignored mutants, mutants known to break the build, and fully resumed ones. A mutant that fails to build teaches only the build time.

The estimate shows up in three places:

  • Log: each progress line ends with an ETA:
    [2/8] survived   src/sample.cpp:7:13 [ror] `<` → `>=`  sample_tests: 0.02s, not_reached  id 31b12689  · ETA 2s
    
  • events.jsonl: a progress event after each mutant, {"done", "total", "remaining_seconds"}.
  • TUI: the header counts the runner's estimate down between updates, through a new progress_estimated console hook. Before, it multiplied the mean time per mutant by the mutants left. The mean is still the fallback until the first estimate arrives.

The estimate covers the whole run with per-mutant, and the current test with per-test, since later tests' mutants aren't known until their baselines run. The README has a "Time left" section and an --order row.

In the TUI

From a per-mutant run of the fixture, one frame every half second:

elapsed 0:02  ·  eta 0:07  ·  1.7s/mutant
elapsed 0:03  ·  eta 0:05  ·  1.3s/mutant
elapsed 0:03  ·  eta 0:04  ·  1.3s/mutant
elapsed 0:04  ·  eta 0:03  ·  1.2s/mutant
elapsed 0:04  ·  eta 0:02  ·  1.2s/mutant
elapsed 0:05  ·  eta 0:00  ·  0.9s/mutant

The run finished at 0:06. My first version counted only the build and test time, and showed eta 0:01 at 0:01 for the same run. Counting everything but the tests as "build" fixed that.

Tests

  • The acceptance in the issue (tests/test_estimate.py): a seeded synthetic run of 500 mutants, cheapest first. Rebuild times differ by file, with ±20% noise. Covering tests take between 0.5 s and 85 s, and really use 40–80% of their bound. After 10% of the mutants, the estimate is within 20% of the real remaining time.
  • The other half of the acceptance (end to end): --order cost and --order source test the same mutants with the same results. Mutant add has two covering tests; it runs first in source order and last by cost.
  • Unit tests for the estimate:
    • no estimate until a mutant is built;
    • per-file build times, with unbuilt files taking the average;
    • the test fraction;
    • a mutant that failed to build;
    • format_eta (ETA 45s, ETA 12m30s, ETA 2h10m).
  • LiveState: the runner's estimate counts down, stops at 0 and is dropped with a new queue.
  • CLI: --order and its default.
  • Updated: the JSON UI test expects the progress event, and the per-mutant test expects add last, with an ETA in the output.
  • python -W error -m pytest: 280 passed, 9 skipped. ruff format --check, ruff check and lizard are clean.

🤖 Generated with Claude Code

With --strategy per-mutant, mutants whose covering tests take least time
now run first (--order cost, the default), so most results arrive early
and an interrupted run has covered the most ground. --order source keeps
the order of files and offsets.

The new strain/estimate.py estimates the time left. A mutant costs a
rebuild of its file plus its covering tests' baseline durations. As
mutants finish, the estimate learns each file's rebuild time (everything
but the tests, so mutating and restoring count too) and what fraction of
the baseline durations the tests really take, since a kill stops them
early.

- Each progress line ends with `· ETA 2h10m`.
- events.jsonl gains a `progress` event with done, total and
  remaining_seconds.
- The TUI's header counts the runner's estimate down, through the new
  progress_estimated console hook, instead of multiplying the mean time
  per mutant.

The estimate covers the whole run with per-mutant, and the current test
with per-test.

Fixes #42.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lefticus
lefticus force-pushed the fix/issue-42-cheapest-first-eta branch from d2b3b3c to 59d42b1 Compare November 18, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant