Mutation testing for C++ projects built with CMake and tested with CTest.
strain changes your code in small ways (< becomes <=, && becomes ||) and checks whether your tests notice. Every mutant that survives points at a test that runs the code but doesn't really check it.
strain run --ui live on tests/fixtures/sample_project after the run, with the second survivor selected.
Status: design stage, not yet usable. Design doc:
- Results in minutes: tests are mutated one at a time, so you see the first results without a full-suite baseline.
- Works with your toolchain: GCC or Clang, Ninja or Makefiles. No compiler plugin required.
- Sanitizers count: a mutant caught by ASan or UBSan counts as killed, and is reported separately.
- Action items, not just a score: pseudo-tested functions, tests that kill nothing, and a concrete hint for each survivor.
- Minimal dependencies: the core is the Python standard library only.
- Python 3.11+
- CMake 3.14+ and CTest
- git, with a clean working tree
- GCC 9+ with
gcov(v1), or Clang withllvm-cov(planned) - Optional:
cppcheck: the best mutation enginetree-sitter: the fallback engine (pip)textual: the live TUI,--ui live(pip)
# TODO(claude): replace once packaging exists
pip install strain # core
pip install "strain[ui]" # with the live TUIstrain needs coverage, sanitizers and a compilation database:
cmake -S . -B build-strain -G Ninja \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
-DCMAKE_CXX_FLAGS="--coverage -fsanitize=address,undefined -fno-omit-frame-pointer -D_GLIBCXX_ASSERTIONS -ftrivial-auto-var-init=pattern" \
-DCMAKE_EXE_LINKER_FLAGS="--coverage -fsanitize=address,undefined"
cmake --build build-strainstrain run --build-dir build-strain # all tests
strain run --build-dir build-strain -R parser # only tests matching "parser"
strain run --build-dir build-strain --since main # only lines changed since mainAdd --ui live (with strain[ui] installed) to watch the run in a full-screen TUI, described below.
parser_tests score 87.8% killed 31 sanitizer 4 timeout 1 survived 5
Action items
1. parse_header() is pseudo-tested: nothing checks its result
2. lexer.cpp:88 `==` → `!=` survived (not_reached): add a test where tok != END
3. parser.cpp:97 `+` → `-` survived (not_detected): assert the computed offset
strain run --ui live shows the run in a full-screen TUI built on Textual. Without
Textual installed, it warns and prints plain output instead. Use the keyboard or the mouse.
| View | Shows |
|---|---|
1 Run |
The mutated code as a highlighted diff, the mutant queue with each result and time, findings and the build log |
2 Survivors |
Each survivor with its coverage, the tests it survived, a hint and its source, and the score of each file |
3 Mutants |
Every mutant so far, filtered by state, with the same details |
4 Stats |
The safety nets the configuration let fail, why each matters and how to fix it; scores by file, operator and test; the slowest mutants |
5 Log |
The full console output |
Above the views, the header shows the score with its history, the totals, the elapsed time and an estimate of the time
left. When a safety net fails but its level is warn or ignore, a ! N safety nets off badge appears there; click it
for the details. It is red when one of those nets errors by default, because then a mutant that breaks the code can
survive, and each survivor's details carry the same caution. Under it, a bar has one cell per mutant, colored by its result: hover to name a mutant, click to show it in the
queue.
| Key | Does |
|---|---|
1–5, tab, shift+tab |
Switch views; or click a tab |
↑ ↓, PgUp PgDn |
Move through a list; in Run, look at another mutant while the run goes on |
enter |
In Run, open the selected mutant in Mutants |
f |
In Run, follow the running mutant again; in Mutants, show the next state filter |
p |
Pause before the next mutant starts, with the sources unmutated; press again to resume |
ctrl+c |
Stop the run: strain restores the sources and writes the reports, as with --resume |
q |
Leave once the run has finished, then print the plain summary |
?, ctrl+p |
Show every key; open the command palette, which also changes the theme |
For each selected test:
- Baseline: run the test unmutated and collect its coverage. If it fails, reports UB or times out, it's excluded.
- Generate mutants on the lines this test executed, in a fixed, reproducible order.
- For each mutant:
- Mutate the source, with the original saved to a journal first.
- Reject it early if
-fsyntax-onlyfails, or without touching the source if it is known to break the build. - Rebuild incrementally with
cmake --build. - Run the test and classify the result.
- Restore the file, always, even on Ctrl-C.
Interrupted runs continue with --resume.
A mutant that doesn't compile is a compile_error for every test, so strain builds it only once:
- Later tests in the same run, and later runs, report it straight away, with no mutation and no rebuild.
- The cache lives in
<build-dir>/strain-cache/, one file per commit and compilation database. A new commit or a reconfigured build starts with an empty cache. --no-build-cacheignores the saved cache and doesn't save to it. Within a run, failures are still remembered.
| State | Meaning | Counts toward score |
|---|---|---|
killed |
A test failed | Yes |
killed_by_sanitizer |
ASan or UBSan caught it, the assertions didn't | Yes, flagged |
timeout |
Ran too long | Yes |
survived |
All tests passed | Yes |
skipped_unevaluated |
Operand never evaluated by this test | No |
compile_error |
Didn't compile | No |
ignored |
Suppressed by config or // mutate-off |
No |
Survivors are labeled not_reached (needs a new test input) or not_detected (needs a stronger assertion) when branch coverage is available.
Mutation score = (killed + killed_by_sanitizer + timeout) ÷ (those + survived)
strain checks every translation unit for:
- ASan
- UBSan
- standard library assertions
- pattern auto-init
Each check is error, warn or ignore; the default is error.
strain run --safety-nets=warn| Option | Default | Description |
|---|---|---|
--build-dir |
. |
CMake build folder |
-R / -E |
Include / exclude tests by regex, as in ctest | |
--since <ref> |
Mutate only lines changed since a git ref | |
--strategy |
per-test |
per-test or per-mutant |
--skip-killed |
off | Skip mutants an earlier test already killed |
--no-build-cache |
off | Don't reuse or save the build cache |
--engine |
auto |
cppcheck, tree-sitter, lexer |
--coverage |
auto |
gcov, llvm-cov, lcov, gcovr |
--ui |
auto |
live, log, json |
--safety-nets |
error |
error, warn, ignore |
--threshold |
Exit non-zero below this score | |
--strict |
off | Abort if any test is excluded |
--max-per-line |
Cap mutants per source line | |
--resume |
Continue an interrupted run | |
--all |
off | Show every survivor, not just the top actions |
# strain.toml
exclude = ["tests/**", "third_party/**"]
arid = ["LOG_*", "spdlog::*"]
[safety_nets]
asan = "error"
ubsan = "error"
stdlib_assertions = "error"
auto_var_init = "warn"
[ignore]
mutants = ["a1b2c3d4"] # equivalent mutants, by IDTo suppress mutants in source, add // mutate-off to the line.
Written to strain-report/ :
report.json: full results, in the mutation-testing-elements schemaevents.jsonl: every event of the run- compiler-style warnings on the console (
file:line:col: warning: survived [ror]), clickable in editors
| Part | Options |
|---|---|
| Mutation engine | cppcheck, tree-sitter, lexer |
| Coverage | gcov, llvm-cov, lcov, gcovr |
| Strategy | per-test, per-mutant |
| UI | live, log, json |
- Tests must be deterministic
- One build per mutant, so large header changes are slow
- MSVC is not supported
- No parallel workers yet
- v1: GCC + gcov, cppcheck/lexer engines, per-test strategy, log/json/live UI
- Clang + llvm-cov, tree-sitter engine
- Condition/region coverage to skip unevaluated operands
- per-mutant strategy and parallel workers
- Equivalent-mutant detection by comparing object files
- Mutant schemata (no rebuild per mutant)
