Skip to content

Fix #45: document a CI job and say why a run tested nothing - #49

Open
lefticus wants to merge 1 commit into
mainfrom
fix/issue-45-ci-recipe
Open

lefticus wants to merge 1 commit into
mainfrom
fix/issue-45-ci-recipe

Conversation

@lefticus

Copy link
Copy Markdown
Member

Fixes #45.

CI section in the README

  • What the exit status means:

    • non-zero when no mutant was generated, without --since;
    • non-zero below --threshold;
    • zero with --since and nothing to mutate.
  • Why shell: bash matters: without a shell: key, GitHub Actions runs bash -e with no pipefail, so strain run … | tee strain.log passes whatever strain found. With shell: bash, GitHub adds -o pipefail.

  • A GitHub Actions job that:

    • configures the strain build with the quick start's cmake line;
    • runs strain run --ui log --since origin/<base> --threshold 80 | tee strain.log with shell: bash;
    • appends strain-report/summary.md to $GITHUB_STEP_SUMMARY;
    • uploads strain-report/ and the log. The last two steps run with if: always(), so the report survives a failing run.
  • Tests in tests/test_ci.py that read that job from the README and check:

    • the step that pipes strain's output has shell: bash;
    • the summary and upload steps run even when strain fails;
    • --since comes with fetch-depth: 0.

    I also parsed the job with PyYAML to check its structure.

  • The Reports section now lists index.html and summary.md.

"Nothing was mutation tested", with the reason

Without --since, a run with no mutants already fails (require_mutants). With --since, it passes, which is right: a change that touches no line the tests run has nothing to test. But the summary said only "No mutants were generated.", which can read like success. Now summary.md, index.html and the console summary say Nothing was mutation tested, followed by one of these reasons:

  • "No source line changed since origin/main, so there was nothing to mutate."
  • "No mutant was generated on the 3 lines changed since origin/main: the selected tests don't execute them, or they hold nothing to mutate."

What I left out

  • The optional composite action (cpp-best-practices/mutant_strain/action@…). It would need a release and tag scheme for the action, and strain isn't published to PyPI yet. The documented job is the same few steps, and the test keeps it correct.
  • The exit status with --since and nothing to mutate stays 0. Making it non-zero would fail every pull request that changes only docs or untested code, even with --threshold. The issue's acceptance asks for the summary to say so explicitly, which it now does.

Tests

  • End to end:
    • --since HEAD with no changes exits 0. Its summary.md is exactly the heading plus "Nothing was mutation tested. No source line changed since HEAD, so there was nothing to mutate."
    • --since HEAD~1 after a commit that only adds a comment says no mutant was generated on the 1 changed line.
  • Unit: the markdown summary and the HTML report carry the reason, and the HTML escapes it.
  • python -W error -m pytest: 259 passed, 9 skipped. ruff format --check, ruff check and lizard are clean.

🤖 Generated with Claude Code

A CI step that pipes strain's output, such as `strain run … | tee log`,
passes whatever strain returns unless the step says `shell: bash`, because
GitHub Actions only adds -o pipefail then. The README gains a CI section:

- what the exit status means;
- why `shell: bash` matters;
- a GitHub Actions job that tests a pull request's changes with --since,
  appends summary.md to the job summary and uploads the report, both even
  when strain fails.

tests/test_ci.py checks the documented job keeps those properties.

With --since, a run without mutants passes, but its summary used to say
only "No mutants were generated", which reads like a clean run. Now
summary.md and index.html start with "Nothing was mutation tested" and say
why: no source line changed since the ref, or none of the changed lines is
executed and has something to mutate. The console summary says the same.

The Reports section also lists index.html and summary.md.

Fixes #45.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document (or ship) a CI recipe that fails when nothing was mutation tested

1 participant