Skip to content

fix(analyzer): remove ExcludeFromCodeCoverage so Stryker mutates analyzers - #680

Merged
samtrion merged 1 commit into
mainfrom
fix/stryker-analyzers-excluded-from-coverage
Oct 29, 2026
Merged

samtrion merged 1 commit into
mainfrom
fix/stryker-analyzers-excluded-from-coverage

Conversation

@samtrion

@samtrion samtrion commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The mutation run reports a score of 33.33%, but it only measures Helper.cs.

  • 246 mutants got status Ignored. Reason: Removed by exclude from code coverage filter (plus 6 "block already covered")
  • 10 total mutants will be tested of 294 created (3 mutated variant projects: Roslyn4_7, Roslyn4_14, Roslyn5_6)
  • final mutation score is 33.33 %
  • Per file: IsPackableProjectDiagnosticAnalyzer.cs Ignored 59 + CompileError 4, OldProjectSupportDiagnosticAnalyzer.cs Ignored 23, Helper.cs Killed 10 + Ignored 2.

Root cause

Both analyzer classes carry [ExcludeFromCodeCoverage]. Stryker honours that attribute and removes all mutants in those types. The only real production logic (NED0001-0009, OLD0001) is therefore never mutated, although 25 tests exercise it. The stryker-config.json mutate globs do not exclude these files, so the attribute is the only cause.

Change

Remove [ExcludeFromCodeCoverage] and the now unused using System.Diagnostics.CodeAnalysis; from both analyzers. No other coverage exclusion config exists in the repo (the CodeCoverage package is referenced only by test projects).

Before / After

Run: https://github.com/dailydevops/defaults/actions/runs/40804488424

metric before after
mutants created 294 294
ignored by exclude-from-coverage filter 246 0
ignored by block-already-covered filter 6 48
compile errors 12 12
no coverage 20 136
tested 10 98
killed 10 75
survived 0 3
mutation score 33.33% 35.05%

The attribute-driven ignores dropped from 246 to 0 and tested mutants rose from 10 to 98 (75 killed, 3 survived). The score barely moves because the Roslyn4_14 variant still finds 0 tests ("Number of tests found: 0"), so its mutants (plus others) became NoCoverage (136). That is a separate problem and needs its own fix. The 3 survivors point to concrete test gaps and are not addressed here.

Risk / what this does not change

  • Analyzers now appear in code-coverage reports; acceptable since tests cover them.
  • The score may go down: new survivors will point to real test gaps, and the Roslyn4_14 variant reports 0 tests found, so its mutants may become NoCoverage. That is a separate problem and is not addressed here.
  • Compile-error mutants (4 per project) stay compile errors. The "84 per project" figure is an upper bound.

…yzers

Stryker drops every mutant in types carrying [ExcludeFromCodeCoverage]
("Removed by exclude from code coverage filter"), so all NED/OLD analyzer
logic was invisible to mutation testing. 25 tests already exercise both
classes.
@samtrion
samtrion requested a review from a team as a code owner October 29, 2026 02:07
@samtrion
samtrion requested review from benwirren and removed request for a team October 29, 2026 02:07
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (2f30f7b) to head (b29b5c9).

Additional details and impacted files
@@            Coverage Diff             @@
##              main      #680    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files            1         3     +2     
  Lines           16       134   +118     
  Branches         1        15    +14     
==========================================
+ Hits            16       134   +118     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@samtrion
samtrion merged commit 9472783 into main Oct 29, 2026
13 checks passed
@samtrion
samtrion deleted the fix/stryker-analyzers-excluded-from-coverage branch October 29, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant