Skip to content

ci(stryker): align mutation score thresholds with measured baseline - #231

Merged
samtrion merged 1 commit into
mainfrom
ci/stryker-thresholds-unreachable
Oct 29, 2026
Merged

samtrion merged 1 commit into
mainfrom
ci/stryker-thresholds-unreachable

Conversation

@samtrion

@samtrion samtrion commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Problem

stryker-config.json sets thresholds high 90 / low 80 / break 0. The latest scheduled mutation run (run 40488214820) ends with The final mutation score is 59.95 % and is green. break: 0 means the score could drop to 1% and the run would still pass, so there is no regression gate. The score is also permanently below low, so the dashboard badge and log colouring are always red.

Root cause

The thresholds are aspirational values that were never ratcheted to the real baseline.

Change

Only stryker-config.json (the file used by the scheduled run and the workflow_dispatch default): high 90 -> 75, low 80 -> 60, break 0 -> 55. break is deliberately a few points below the measured 59.95% because the score varies between runs. The three stryker-config.roslyn*.json files are untouched: they only run via manual dispatch with an explicit configurationFile and have no measured baseline.

Before / After

metric before after
thresholds (high/low/break) 90 / 80 / 0 75 / 60 / 55
mutation score 59.95% (2110 mutants created) 59.95% (2110 mutants created, identical; config-only change)
run result green, no gate (break 0) green with break 55: https://github.com/dailydevops/analyzer/actions/runs/40815416914
gate proof (throwaway branch, break 65) n/a run failed with Final mutation score is below threshold break. Crashing...: https://github.com/dailydevops/analyzer/actions/runs/40818066383

Note: Stryker rejects configs where low < break; the first gate-proof attempt (low 60, break 70) failed at config validation, so it was repeated with low 65 / break 65.

Risk / what this does not change

  • A score below 55% now fails the scheduled run (intended).
  • Scores were measured before the other mutation findings land; break should be raised as the score improves.
  • Does not improve the score itself.

The thresholds were high 90 / low 80 / break 0, while the scheduled run
(stryker-config.json, Roslyn5_6 project) scores about 60 percent. With
break 0 the run can never fail, even on a collapse of the score, and the
dashboard badge is permanently red.

Set high 75 / low 60 / break 55: break sits a few points below the
measured score to absorb run-to-run variance (compile errors, flaky
coverage), so a real regression fails the run. Only the config used by
the scheduled run is changed; the roslyn4_4, roslyn4_14 and roslyn5_6
variants are manual-dispatch only and have no measured baseline.
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.72%. Comparing base (8c86c2b) to head (64dc318).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #231   +/-   ##
=======================================
  Coverage   94.72%   94.72%           
=======================================
  Files          48       48           
  Lines        2557     2557           
  Branches      449      449           
=======================================
  Hits         2422     2422           
  Misses         58       58           
  Partials       77       77           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@samtrion
samtrion merged commit 14f121e into main Oct 29, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant