Skip to content

Commit 0b59984

Browse files
danmardanmar
authored
Fix #15047 (baseline: add script and documentation) (#8863)
1 parent cc3fbfc commit 0b59984

5 files changed

Lines changed: 296 additions & 1 deletion

File tree

‎man/manual-premium.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -776,6 +776,120 @@ You can write comments about a suppression as follows:
776776
// cppcheck-suppress warningid ; some comment
777777
// cppcheck-suppress warningid // some comment
778778

779+
# Generating and using a baseline
780+
781+
When you first run Cppcheck on an existing codebase it's common to get a
782+
large number of warnings. A "baseline" lets you suppress all of today's
783+
warnings and see only *new* warnings introduced from now on.
784+
785+
This works by asking Cppcheck to include a content-based `hash` for every warning
786+
(computed from the surrounding code, not the line number), then converting today's
787+
warnings into an XML suppressions file keyed on `id` + `fileName` + `hash`. Because
788+
the hash is based on content rather than line number, warnings stay suppressed even
789+
after unrelated lines above them are added or removed. A warning only reappears if
790+
the code it actually points at changes, or a new warning shows up elsewhere.
791+
792+
## 1. Generate the baseline
793+
794+
Run Cppcheck with `--xml` and capture stderr (where Cppcheck writes its XML) to a
795+
file:
796+
797+
```sh
798+
cppcheck --enable=style --xml src 2> baseline-results.xml
799+
```
800+
801+
Use whatever combination of `--enable`/defines/include paths you normally
802+
analyze the project with — the suppressions you get out only cover the
803+
checks you ran.
804+
805+
## 2. Convert the results into a suppressions file
806+
807+
A [script](https://github.com/cppcheck-opensource/cppcheck/blob/main/tools/generate-baseline-suppressions.py) can be used to generate the baseline:
808+
```sh
809+
python3 generate-baseline-suppressions.py baseline-results.xml suppressions.xml
810+
```
811+
812+
This produces a `suppressions.xml` like:
813+
814+
```xml
815+
<?xml version="1.0"?>
816+
<suppressions>
817+
<suppress>
818+
<id>uninitvar</id>
819+
<fileName>src/file1.c</fileName>
820+
<hash>12345678</hash>
821+
</suppress>
822+
</suppressions>
823+
```
824+
825+
`suppressions.xml` needs to be shared by everyone who runs Cppcheck on this
826+
codebase, including CI.
827+
828+
Having a script instead of using some special cppcheck flags has the advantages:
829+
* it's flexible. You can tweak it if needed.
830+
* there is fewer flags for us to maintain and document, and for you to learn.
831+
832+
## 3. Use the baseline on future runs
833+
834+
```sh
835+
cppcheck --enable=style --xml --suppress-xml=suppressions.xml src
836+
```
837+
838+
Only warnings that aren't in the baseline are reported: new warnings in changed
839+
code, and warnings for checks/files that weren't covered when the baseline was
840+
generated.
841+
842+
## 4. Refreshing the baseline
843+
844+
Re-run steps 1-2 whenever you want to accept the current state as the new
845+
baseline (e.g. after cleaning up a batch of warnings, or deliberately accepting a
846+
new one). Regenerating overwrites `suppressions.xml` with an entry for every
847+
warning present at that time.
848+
849+
## The baseline doesn't need to be kept in sync
850+
851+
Once code that a baselined warning pointed at is fixed, refactored away, or
852+
deleted, its `suppress` entry in `suppressions.xml` becomes dead: nothing will
853+
ever match it again. You don't need to go find and remove it.
854+
855+
Cppcheck's `unmatchedSuppression` check (part of `--enable=all`) normally warns
856+
about suppressions that never matched anything, on the theory that a
857+
suppression nobody needs is probably a mistake. But it does *not* fire for
858+
`suppress` entries that carry a `<hash>` — which is every entry the baseline
859+
script generates. So a baseline file with plenty of dead entries produces no
860+
noise, and developers never need to prune it by hand — it only needs to be
861+
regenerated (step 4) when you deliberately want to reset what's accepted.
862+
863+
## Caveats
864+
865+
- A warning can only be suppressed this way if it carries a `hash` attribute in
866+
the XML output. Almost all checks compute one. Critical errors, i.e. syntax
867+
errors do not get hash and must be fixed. Certain information messages do not
868+
get hash neither.
869+
- Changed Cppcheck options might produce new warnings that are not suppressed
870+
by the baseline.
871+
- Cppcheck upgrades don't affect the hash directly (it isn't version-tagged),
872+
but if a new release for instance changes a check's message wording, the hash
873+
changes with it — so upgrading Cppcheck can resurrect baselined warnings for
874+
checks whose messages were reworded, even though nothing in the analyzed code
875+
changed.
876+
877+
## Strategies for gradually shrinking the baseline
878+
879+
A baseline makes it possible to adopt Cppcheck in CI immediately without
880+
fixing everything first, but nothing about it enforces that the accepted set
881+
of warnings actually shrinks over time.
882+
883+
Generic advice:
884+
885+
- **Prioritize by severity** Checks like `uninitvar` or
886+
`nullPointer` are more valuable to clear than `style` warnings; a baseline
887+
makes it possible to drive the highest-severity checks to zero first while
888+
deliberately leaving lower-risk ones suppressed longer.
889+
- **Be careful** Every fix is a code change, and every code change carries some
890+
risk of introducing a new bug. It can make sense to leave some things suppressed
891+
to minimize the risk that bugs are introduced in working code.
892+
779893
# XML output
780894

781895
Cppcheck can generate output in XML format. Use `--xml` to enable this format.

‎man/manual.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -777,6 +777,120 @@ You can write comments about a suppression as follows:
777777
// cppcheck-suppress warningid ; some comment
778778
// cppcheck-suppress warningid // some comment
779779

780+
# Generating and using a baseline
781+
782+
When you first run Cppcheck on an existing codebase it's common to get a
783+
large number of warnings. A "baseline" lets you suppress all of today's
784+
warnings and see only *new* warnings introduced from now on.
785+
786+
This works by asking Cppcheck to include a content-based `hash` for every warning
787+
(computed from the surrounding code, not the line number), then converting today's
788+
warnings into an XML suppressions file keyed on `id` + `fileName` + `hash`. Because
789+
the hash is based on content rather than line number, warnings stay suppressed even
790+
after unrelated lines above them are added or removed. A warning only reappears if
791+
the code it actually points at changes, or a new warning shows up elsewhere.
792+
793+
## 1. Generate the baseline
794+
795+
Run Cppcheck with `--xml` and capture stderr (where Cppcheck writes its XML) to a
796+
file:
797+
798+
```sh
799+
cppcheck --enable=style --xml src 2> baseline-results.xml
800+
```
801+
802+
Use whatever combination of `--enable`/defines/include paths you normally
803+
analyze the project with — the suppressions you get out only cover the
804+
checks you ran.
805+
806+
## 2. Convert the results into a suppressions file
807+
808+
A [script](https://github.com/cppcheck-opensource/cppcheck/blob/main/tools/generate-baseline-suppressions.py) can be used to generate the baseline:
809+
```sh
810+
python3 generate-baseline-suppressions.py baseline-results.xml suppressions.xml
811+
```
812+
813+
This produces a `suppressions.xml` like:
814+
815+
```xml
816+
<?xml version="1.0"?>
817+
<suppressions>
818+
<suppress>
819+
<id>uninitvar</id>
820+
<fileName>src/file1.c</fileName>
821+
<hash>12345678</hash>
822+
</suppress>
823+
</suppressions>
824+
```
825+
826+
`suppressions.xml` needs to be shared by everyone who runs Cppcheck on this
827+
codebase, including CI.
828+
829+
Having a script instead of using some special cppcheck flags has the advantages:
830+
* it's flexible. You can tweak it if needed.
831+
* there is fewer flags for us to maintain and document, and for you to learn.
832+
833+
## 3. Use the baseline on future runs
834+
835+
```sh
836+
cppcheck --enable=style --xml --suppress-xml=suppressions.xml src
837+
```
838+
839+
Only warnings that aren't in the baseline are reported: new warnings in changed
840+
code, and warnings for checks/files that weren't covered when the baseline was
841+
generated.
842+
843+
## 4. Refreshing the baseline
844+
845+
Re-run steps 1-2 whenever you want to accept the current state as the new
846+
baseline (e.g. after cleaning up a batch of warnings, or deliberately accepting a
847+
new one). Regenerating overwrites `suppressions.xml` with an entry for every
848+
warning present at that time.
849+
850+
## The baseline doesn't need to be kept in sync
851+
852+
Once code that a baselined warning pointed at is fixed, refactored away, or
853+
deleted, its `suppress` entry in `suppressions.xml` becomes dead: nothing will
854+
ever match it again. You don't need to go find and remove it.
855+
856+
Cppcheck's `unmatchedSuppression` check (part of `--enable=all`) normally warns
857+
about suppressions that never matched anything, on the theory that a
858+
suppression nobody needs is probably a mistake. But it does *not* fire for
859+
`suppress` entries that carry a `<hash>` — which is every entry the baseline
860+
script generates. So a baseline file with plenty of dead entries produces no
861+
noise, and developers never need to prune it by hand — it only needs to be
862+
regenerated (step 4) when you deliberately want to reset what's accepted.
863+
864+
## Caveats
865+
866+
- A warning can only be suppressed this way if it carries a `hash` attribute in
867+
the XML output. Almost all checks compute one. Critical errors, i.e. syntax
868+
errors do not get hash and must be fixed. Certain information messages do not
869+
get hash neither.
870+
- Changed Cppcheck options might produce new warnings that are not suppressed
871+
by the baseline.
872+
- Cppcheck upgrades don't affect the hash directly (it isn't version-tagged),
873+
but if a new release for instance changes a check's message wording, the hash
874+
changes with it — so upgrading Cppcheck can resurrect baselined warnings for
875+
checks whose messages were reworded, even though nothing in the analyzed code
876+
changed.
877+
878+
## Strategies for gradually shrinking the baseline
879+
880+
A baseline makes it possible to adopt Cppcheck in CI immediately without
881+
fixing everything first, but nothing about it enforces that the accepted set
882+
of warnings actually shrinks over time.
883+
884+
Generic advice:
885+
886+
- **Prioritize by severity** Checks like `uninitvar` or
887+
`nullPointer` are more valuable to clear than `style` warnings; a baseline
888+
makes it possible to drive the highest-severity checks to zero first while
889+
deliberately leaving lower-risk ones suppressed longer.
890+
- **Be careful** Every fix is a code change, and every code change carries some
891+
risk of introducing a new bug. It can make sense to leave some things suppressed
892+
to minimize the risk that bugs are introduced in working code.
893+
780894
# XML output
781895

782896
Cppcheck can generate output in XML format. Use `--xml` to enable this format.

‎releasenotes.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@ Changed interface:
2020
-
2121

2222
Infrastructure & dependencies:
23-
- compile_commands.json - flexible handling of -isystem, --sysroot and -I flags through the script tweak-compile_commands.py.
23+
- flexible handling of -isystem, --sysroot and -I flags in compile_commands.json through the script tweak-compile_commands.py.
24+
- baseline functionality.
2425

2526
Other:
2627
- Added configuration file for Microsoft.GSL (Guideline Support Library).

‎tools/generate-baseline-suppressions.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/usr/bin/env python3
2+
"""Convert a cppcheck XML results file into a cppcheck XML suppressions file.
3+
4+
Usage: generate-baseline-suppressions.py <results.xml> <suppressions.xml>
5+
6+
Each <error> in the results file that has a non-zero "hash" attribute is
7+
converted into a <suppress> entry using the error's id, the file of its
8+
first <location> (the primary location used by cppcheck's suppression
9+
matching) and the hash.
10+
"""
11+
import sys
12+
import xml.etree.ElementTree as ET
13+
14+
15+
def convert(results_path, suppressions_path):
16+
tree = ET.parse(results_path)
17+
root = tree.getroot()
18+
errors_el = root.find('errors')
19+
20+
seen = set()
21+
suppressions = []
22+
for error in errors_el.findall('error'):
23+
error_id = error.get('id')
24+
error_hash = error.get('hash')
25+
location = error.find('location')
26+
if error_id is None or location is None:
27+
continue
28+
if not error_hash or error_hash == '0':
29+
continue
30+
file_name = location.get('file')
31+
key = (error_id, file_name, error_hash)
32+
if key in seen:
33+
continue
34+
seen.add(key)
35+
suppressions.append(key)
36+
37+
out_root = ET.Element('suppressions')
38+
for error_id, file_name, error_hash in suppressions:
39+
suppress = ET.SubElement(out_root, 'suppress')
40+
ET.SubElement(suppress, 'id').text = error_id
41+
ET.SubElement(suppress, 'fileName').text = file_name
42+
ET.SubElement(suppress, 'hash').text = error_hash
43+
44+
ET.indent(out_root, space=' ')
45+
out_tree = ET.ElementTree(out_root)
46+
out_tree.write(suppressions_path, encoding='UTF-8', xml_declaration=True)
47+
print(f'Wrote {len(suppressions)} suppression(s) to {suppressions_path}')
48+
49+
50+
if __name__ == '__main__':
51+
if len(sys.argv) != 3:
52+
print(f'Usage: {sys.argv[0]} <results.xml> <suppressions.xml>', file=sys.stderr)
53+
sys.exit(1)
54+
convert(sys.argv[1], sys.argv[2])

‎tools/readme.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,18 @@ message.
8484
Script to compare the error IDs in the expected `testrunner` output (without executing it) with the `--errorlist` output.
8585
It will report missing test coverage for an ID and missing IDs in the `--errorlist` output.
8686

87+
### * tools/generate-baseline-suppressions.py
88+
89+
Script that converts a Cppcheck XML results file (`--xml`) into a Cppcheck XML suppressions file, so that
90+
existing warnings can be suppressed as a baseline and only new warnings are reported afterwards. For each
91+
`<error>` in the results file that has a non-zero `hash` attribute, it writes a `<suppress>` entry with the
92+
error's id, the file of its first `<location>` and the hash. Usage:
93+
```shell
94+
$ cppcheck --xml --xml-version=2 ./src 2> results.xml
95+
$ python tools/generate-baseline-suppressions.py results.xml suppressions.xml
96+
```
97+
The generated `suppressions.xml` can then be passed to Cppcheck with `--suppress-xml=suppressions.xml`.
98+
8799
### * tools/tweak-compile-commands.py
88100

89101
Script to tweak `-isystem`/`--sysroot`/`-I` options in a `compile_commands.json` file, for example to make

0 commit comments

Comments
 (0)