Skip to content

Fix #43: report constexpr code no test runs, and mutate it with --compile-time - #53

Open
lefticus wants to merge 2 commits into
fix/issue-42-cheapest-first-etafrom
fix/issue-43-compile-time-code
Open

lefticus wants to merge 2 commits into
fix/issue-42-cheapest-first-etafrom
fix/issue-43-compile-time-code

Conversation

@lefticus

@lefticus lefticus commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Fixes #43. Stacked on #51 (#42), which is stacked on #50 and #48. Merge those first; GitHub retargets each PR as its base merges.

The gap

gcov counts what runs. I checked on a small project: a header whose constexpr and consteval functions are used only in static_asserts doesn't appear in the coverage data at all. strain mutates only executed lines, so it never looked at that code, and nothing said so.

What changes

strain/compile_time.py finds constexpr and consteval function bodies with strain's lexer. It skips constexpr variables, brace-initialised objects, declarations without a body, lambdas assigned to variables, if constexpr, and comments; it handles operator= and nested lambdas. strain walks the project's C++ sources and headers, skipping the build folder, dot-folders, test code and exclude patterns, and reads only files that mention constexpr or consteval. The lines no test ran come from the union of every baseline's coverage.

  • By default, a new section lists the lines of those functions that hold mutants but that no test runs:

    ▸ Not mutated: evaluated at compile time only
      · include/sample.hpp:8-9 clamp_index(): 5 mutants on 2 lines
      · include/sample.hpp:15 first_of(): 1 mutant on 1 line
      no test runs these constexpr lines; --compile-time mutates them, and a failed static_assert kills them
    

    It appears on the console, in summary.md and index.html, and as compileTimeOnly in report.json. If a run has no mutants at all, the error now points at these lines and at --compile-time.

  • With --compile-time, strain mutates those lines after the tests, builds everything with each mutant, and restores the source. A new phase, "compile-time checks", appears in the log and the TUI.

    • killed_at_compile_time (new state, counted as detected): a static_assert or a constant evaluation failed. The detail names the failed check.
    • survived: the mutant builds, so no compile-time check caught it, and no test runs the line. The hint says "add a static_assert where < and <= give different results".
    • compile_error: invalid code, as before.
    ▸ Compile time
      · 6 mutants on constexpr lines no test runs
      [1/6] survived                 include/sample.hpp:8:13 [ror] `<` → `<=`  builds, and no test runs this line
      [2/6] killed_at_compile_time   include/sample.hpp:8:13 [ror] `<` → `>=`  killed at compile time: tests/sample_tests.cpp:3:34: error: static assertion failed
      ...
      [6/6] compile_error            include/sample.hpp:15:15 [aor] `*` → `/`  build failed
    
  • Any mutant, not only with --compile-time: when -fsyntax-only or the build fails, its diagnostics decide the state. A failed compile-time check means killed_at_compile_time; anything else stays compile_error. A failed static_assert is a real detection wherever the mutated line is, and -fsyntax-only evaluates static_asserts too. This changes results: mutants that static_asserts catch used to be excluded from the score as compile errors.

    • The patterns cover GCC's static assertion failed, non-constant condition for static assertion and … is not a constant expression, which I checked against g++ here.
    • They also cover Clang's static assertion failed due to requirement, static_assert expression is not an integral constant expression and must be initialized by a constant expression. The macOS CI runners (Apple Clang) produced static assertion failed due to requirement 'clamp_index(-1, 4) == 0' and classified it correctly. The other two Clang messages come from its documentation, not from a run.
  • Build cache: it stores the description (killed at compile time: …), so a cached failure keeps its state, as the issue suggested. Diagnostic paths are made relative to the project.

  • TUI: the new state's short label is constexpr. The header totals and the Mutants filters now use the short labels (sanitizer, memory, constexpr, no build). With full names, the totals ran into the timing:

    before: ✔ killed 1   ✔ killed_at_compile_time 3   ✗ survived 2   ⊘ compile_…elapsed 0:04  ·  0.6s/mutant
    after:  ✔ killed 1   ✔ constexpr 3   ✗ survived 2   ⊘ no build 1            elapsed 0:05  ·  0.6s/mutant
    
  • README: a "Compile-time code" section, a row in the result states table, the score formula and the CLI table.

This also covers the "lines not covered at runtime" group from #44 (#52): they get their own section instead of mixing with survivors.

Tests

  • End to end (the acceptance in the issue): a header with clamp_index, checked only by static_asserts, and first_of, where a *→/ mutant makes const Value *start invalid code.
    • By default, both functions' lines are reported as not mutated, with the right line ranges and counts, in the console, summary.md and report.json. No header mutant is run.
    • With --compile-time, I worked out each state by hand: <→>=, >=→< and -→+ are killed_at_compile_time; <→<= and >=→> survive, since no static_assert sits at 0 or at the size; *→/ stays compile_error. The failed check is named, and the survivors get the static_assert hint.
  • tests/test_compile_time.py:
    • the function finder on templates, noexcept, if constexpr, variables, brace initialisation, declarations, lambdas, operator=, nested lambdas and comments;
    • GCC and Clang diagnostics, and invalid code that isn't a failed check;
    • the state of a fresh and a cached failure;
    • grouping into NotMutated with line ranges.
  • CLI: --compile-time.
  • python -W error -m pytest: 288 passed, 9 skipped. ruff format --check, ruff check and lizard are clean.

🤖 Generated with Claude Code

lefticus and others added 2 commits November 18, 2026 04:56
gcov counts what runs, so a constexpr or consteval function checked only
by static_assert has no executed lines and was never mutated. A header
used only that way isn't even in the coverage data.

The new strain/compile_time.py finds constexpr and consteval function
bodies with the lexer, skipping variables, declarations, lambdas and
`if constexpr`.

- By default, the lines of those functions that hold mutants but that no
  test runs are listed under "Not mutated: evaluated at compile time
  only": on the console, in summary.md and index.html, and as
  compileTimeOnly in report.json. The error for a run with no mutants
  points at them too.
- With --compile-time, strain mutates those lines after the tests and
  builds everything with each mutant. A failed static_assert or constant
  evaluation kills it as the new killed_at_compile_time state, which
  counts as detected. A mutant that builds survives, with a hint to add a
  static_assert. Invalid code stays compile_error.
- Any build or -fsyntax-only check that fails on a compile-time check now
  counts as killed_at_compile_time, using GCC's and Clang's diagnostics.
  The build cache keeps the description, so a cached failure keeps its
  state. Diagnostic paths are made relative to the project.
- The TUI header and the Mutants filters use the short state labels, so
  the totals fit beside the timing.

Fixes #43.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On macOS the compiler reports /var/folders/... while strain resolves the
source folder to /private/var/folders/..., so the plain prefix replacement
left the paths long. Any absolute path in a diagnostic that resolves into
the project is now made relative to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lefticus
lefticus force-pushed the fix/issue-42-cheapest-first-eta branch from d2b3b3c to 59d42b1 Compare November 18, 2026 03:56
@lefticus
lefticus force-pushed the fix/issue-43-compile-time-code branch from 05c4940 to 283f437 Compare November 18, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant