Repository navigation
ci(stryker): align mutation score thresholds with measured baseline - #231
Merged
Merged
Conversation
The thresholds were high 90 / low 80 / break 0, while the scheduled run (stryker-config.json, Roslyn5_6 project) scores about 60 percent. With break 0 the run can never fail, even on a collapse of the score, and the dashboard badge is permanently red. Set high 75 / low 60 / break 55: break sits a few points below the measured score to absorb run-to-run variance (compile errors, flaky coverage), so a real regression fails the run. Only the config used by the scheduled run is changed; the roslyn4_4, roslyn4_14 and roslyn5_6 variants are manual-dispatch only and have no measured baseline.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #231 +/- ##
=======================================
Coverage 94.72% 94.72%
=======================================
Files 48 48
Lines 2557 2557
Branches 449 449
=======================================
Hits 2422 2422
Misses 58 58
Partials 77 77 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
stryker-config.jsonsets thresholdshigh 90 / low 80 / break 0. The latest scheduled mutation run (run 40488214820) ends withThe final mutation score is 59.95 %and is green.break: 0means the score could drop to 1% and the run would still pass, so there is no regression gate. The score is also permanently belowlow, so the dashboard badge and log colouring are always red.Root cause
The thresholds are aspirational values that were never ratcheted to the real baseline.
Change
Only
stryker-config.json(the file used by the scheduled run and theworkflow_dispatchdefault):high 90 -> 75,low 80 -> 60,break 0 -> 55.breakis deliberately a few points below the measured 59.95% because the score varies between runs. The threestryker-config.roslyn*.jsonfiles are untouched: they only run via manual dispatch with an explicitconfigurationFileand have no measured baseline.Before / After
Final mutation score is below threshold break. Crashing...: https://github.com/dailydevops/analyzer/actions/runs/40818066383Note: Stryker rejects configs where
low < break; the first gate-proof attempt (low 60, break 70) failed at config validation, so it was repeated with low 65 / break 65.Risk / what this does not change
breakshould be raised as the score improves.