Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -159,16 +159,34 @@ jobs:
# The richer corpus is `examples/`, which holds one of every kind and is
# checked by the `example` job above. What gives *this* step its teeth
# is the mutation below, not the size of the number.
- name: and it was measured over something
#
# logbook#69 is the other half of that admission, and it is now the
# tool's own job rather than this comment's: almost every real
# assertion in this file is a Python `assert` or a `diff`, deadassert
# reads neither, and the `unread` count is it saying so. The assertions
# below cannot be vacuous -- the lines they describe are in this file.
- name: and it was measured over something, and it says what it missed
run: |
set -euo pipefail
deadassert . --json > out.json
python - <<'PY'
import json
d = json.load(open("out.json"))
print(f"{d['assertions']} assertions in {d['scripts']} scripts")
print(f"{d['assertions']} assertions in {d['scripts']} scripts, "
f"{len(d['unread'])} not read in {d['unread_scripts']} of them")
assert d["assertions"] >= 2, f"only {d['assertions']} assertions seen"
assert d["findings"] == [], d["findings"]

kinds = {x["kind"] for x in d["unread"]}
assert "assert" in kinds, kinds
assert "diff" in kinds, kinds
assert d["unread_scripts"] >= 4, d["unread_scripts"]

# Anchored, not a substring: this is a repository about assertions
# that match less than they look like.
commands = [x["command"] for x in d["unread"]]
wanted = "diff -u examples/expected-quiet.txt"
assert any(c.startswith(wanted) for c in commands), commands
PY

# The assertion above is only worth anything if it can go red. Break
Expand Down
129 changes: 114 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ stopped saying that, this tells you. It reads the `run:` scripts in
`.github/workflows`, works out which way round each assertion fails, and
reports the ones that cannot fail at all.

It also counts the assertions it did *not* read — a `diff`, a `jq -e`, a
Python `assert` — so that a clean run says what it covered rather than
implying it covered everything.

## Install

```sh
Expand Down Expand Up @@ -90,9 +94,17 @@ ci.yml:67 decorative 'skipped'
if grep -qF 'skipped' seen.txt; then

4 assertions here do not check what they read as checking (1 unwitnessed, 1 prefix, 1 swallowed, 1 decorative).
7 assertions in 6 scripts across 1 workflow file, 1 of them negated.
7 assertions in 7 scripts across 1 workflow file, 1 of them negated. Not counted above: 2 assertions in 1 of those scripts (1 assert, 1 diff), listed below.

not read:
ci.yml:96 assert assert d['dead'] == 2, d['dead']
ci.yml:99 diff diff -u examples/expected-porcelain.txt got.txt
```

The last two lines are not findings — they are the two assertions in that
file `deadassert` never read, and the reason the four above them are a
verdict about the greps rather than about the file. See `not read:` below.

## The kinds

| kind | what it means |
Expand All @@ -103,6 +115,11 @@ ci.yml:67 decorative 'skipped'
| `masked` | its exit status is thrown away by its position in a pipeline |
| `decorative` | it picks a branch, and neither branch can fail the step |

`not read` is not in that table on purpose. The five kinds are verdicts on
an assertion; `not read` is the admission that there was an assertion there
and no verdict was reached. It is never a finding and never changes the exit
code.

## `--against`, and the one guess in here

`prefix` is the only kind that rests on what somebody *meant*. `grep -qF 'x'`
Expand Down Expand Up @@ -144,13 +161,80 @@ stops being reported.
is a real assertion, but the characters in the file are not the characters
grep gets, so it goes in `not decided:` and is left out of the counts.
- **It only knows about matchers.** `grep`, `egrep`, `fgrep`, `rg`. A
verdict reached by `diff`, `test` or a language's own assert is invisible
to it — which is a reason to prefer those, not a gap to fill. See below.
verdict reached by `diff`, `test` or a language's own assert is not read —
which is a reason to prefer those, not a gap to fill. See below. It is no
longer *invisible*, though: those get counted, and the count is printed.
- **`unwitnessed` is a smell, not a proof.** It says nothing in *these
workflows* establishes the string can appear. The proof may well live in a
unit test, where it is arguably better placed. What the finding is really
telling you is that the CI step, read on its own, does not stand up.

## `not read:` — the assertions it did not read

A clean run used to read as *the assertions in this file can fail*. It only
ever meant *the greps in this file can fail*, and the difference is not
academic. On the night `deadassert` was swept across this organisation
looking for dead assertions, it walked past one — in the repository it was
auditing, two steps below a finding it did report:

```sh
python3 -c "
loose = json.load(open('loose.json'))['shapes'][0]['candidates']
assert loose > tight * 5, (loose, tight)
"
```

An assertion whose truth depends on the runner image rather than on the
thing it names. Same defect, different punctuation, and the run said `2
findings` over `32 assertions in 20 scripts` without a word to say that one
of those 20 scripts asserted something it had not read.

So it counts them now:

```console
$ deadassert .
Every assertion here fails when the thing it names stops being true.
2 assertions in 15 scripts across 1 workflow file, 0 of them negated. Not counted above: 11 assertions in 5 of those scripts (9 assert, 2 diff), listed below.

not read:
ci.yml:37 assert assert total >= 110, f"only {total} tests collected"
ci.yml:85 diff diff -u examples/expected-quiet.txt got.txt
...
```

Three shapes, picked by counting what the workflow files in this
organisation actually use: a language's own `assert` (nearly always Python,
via `-c` or a heredoc), `diff` and `cmp`, and `jq -e`.

**This decides nothing.** It does not judge whether a `diff` can fail; it
reports that one is there and was not read. The same move the tool already
makes with `not decided:` and with the non-POSIX shells it skips — and for
the same reason, which is that a tool about assertions that quietly check
nothing has no business quietly covering nothing. It is coverage, not a
complaint: `diff` is what the section below tells you to prefer, and a
repository full of them gets a count, not a finding. Nothing here affects
the exit code, and `-q` still prints findings only.

The sharpest case was the verdict over no greps at all. `deadassert
uncollected` used to print:

```
No assertions here. Read 12 run: scripts in 1 workflow file and found no
command whose exit status decides a step. Nothing to check.
```

over a CI file holding nine Python asserts. Two other repositories in this
organisation got the same answer.

**`test` is deliberately not counted.** `test "$s" -eq 0` and `[ -f out ]`
are the most common assertions in these files and are usually the soundest
thing in them — the status *is* the assertion, with no pattern to rot. A
count that included them would be non-zero nearly everywhere on day one and
would stop being read, which is the one failure mode that would make this
worse than silence. The cost is real and this is where it is written down: a
`test` on a status captured from the wrong command is not covered here
either.

## `grep` is a poor way to assert a whole output

Having written this, the conclusion it kept pushing me towards is that most
Expand All @@ -170,22 +254,37 @@ is small on purpose, and it is printed rather than hidden.

## Measured on something real

Run across the 30 repositories of the organisation this was written in — 38
workflow files, 337 `run:` scripts:
Run across the 31 repositories of the organisation this was written in — 37
workflow files, 360 `run:` scripts:

| | count |
| --- | --- |
| assertions found | 157 |
| negated (`needs-no-match`) | 22 |
| `unwitnessed` | 12 |
| `prefix` | 21 |
| assertions read | 170 |
| negated (`needs-no-match`) | 24 |
| `unwitnessed` | 6 |
| `prefix` | 3 |
| `swallowed` / `masked` / `decorative` | 0 |

Twelve of the thirty repositories have at least one finding. The three
can't-fail kinds scoring zero is the result I did not expect: every `|| true`
in that organisation is on a *tool invocation*, deliberately allowing a
non-zero findings exit before the output is asserted, which is correct usage.
An early version of this tool reported all of them and was wrong to.
| not read | 74, in 24 scripts |

Seven of the thirty-one repositories have at least one finding. The first
run of this tool found 33 — 12 `unwitnessed` and 21 `prefix`, across 12
repositories — and the 24 that have gone are a repo-at-a-time sweep, not a
`sed`; the remaining 9 are the tail of it.

The three can't-fail kinds scoring zero is the result I did not expect:
every `|| true` in that organisation is on a *tool invocation*, deliberately
allowing a non-zero findings exit before the output is asserted, which is
correct usage. An early version of this tool reported all of them and was
wrong to.

The last row is the one that changes how you should read the others. **74
assertions in these files are not read at all** — 67 Python asserts, 5
`diff`, 2 `jq -e` — against 170 that are, and they are concentrated: 15
repositories have at least one, and 3 have nothing *but* unread assertions,
so the verdict on them used to be `Nothing to check.` The 9 findings above
are a verdict about the greps in this organisation, which is a smaller thing
than it looks, and the point of printing the number is that you no longer
have to take my word for how much smaller.

`shellcheck` 0.9.0 reports none of these at default settings. With
`--enable=all` it finds the two pipeline cases as `SC2312` (info) and
Expand Down
20 changes: 20 additions & 0 deletions examples/.github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,3 +77,23 @@ jobs:
countdead examples/ > /dev/null; test $? -eq 1
countdead /nonexistent > /dev/null 2>&1; test $? -eq 2
countdead examples/ -q | grep -qxF 'examples/a.txt:3: never-read'

# The two assertions deadassert does not read, and the reason the run
# ends with a `not read:` block instead of implying the four findings
# above are the whole story (logbook#69).
#
# The `assert` decides the step and can rot exactly as a grep can, and
# a tool that reads matchers sees nothing here at all. The `diff` is
# the shape this project's own README recommends -- counting it is not
# a complaint, it is the tool saying how much of this file its four
# findings were actually measured over.
- name: the json mode and the porcelain mode agree
run: |
countdead examples/ --json > out.json || test $? -le 1
python3 -c "
import json
d = json.load(open('out.json'))
assert d['dead'] == 2, d['dead']
"
countdead examples/ --porcelain > got.txt || test $? -le 1
diff -u examples/expected-porcelain.txt got.txt
6 changes: 5 additions & 1 deletion examples/expected-against.txt
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,8 @@ ci.yml:67 decorative 'skipped'
if grep -qF 'skipped' seen.txt; then

4 assertions here do not check what they read as checking (1 unwitnessed, 1 prefix, 1 swallowed, 1 decorative).
7 assertions in 6 scripts across 1 workflow file, 1 of them negated.
7 assertions in 7 scripts across 1 workflow file, 1 of them negated. Not counted above: 2 assertions in 1 of those scripts (1 assert, 1 diff), listed below.

not read:
ci.yml:96 assert assert d['dead'] == 2, d['dead']
ci.yml:99 diff diff -u examples/expected-porcelain.txt got.txt
18 changes: 17 additions & 1 deletion src/deadassert/analyse.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
import re
from dataclasses import dataclass, field

from . import shell, workflows
from . import shell, unread, workflows
from .shell import Assertion
from .workflows import Script, WorkflowError

Expand Down Expand Up @@ -90,6 +90,11 @@ class Result:
against: str | None = None
#: Patterns `--against` showed matching nothing in the output.
absent: list[tuple[str, str]] = field(default_factory=list)
#: Assertion-shaped constructs not read at all, as `(where, kind, text)`.
#: Never findings, and never part of the exit code: see `unread.py`.
unread: list[tuple[str, str, str]] = field(default_factory=list)
#: How many `run:` scripts hold at least one of those.
unread_scripts: int = 0

@property
def live(self) -> list[tuple[Script, Assertion]]:
Expand Down Expand Up @@ -282,6 +287,17 @@ def analyse(
result.foreign.append((f"{os.path.basename(rel)} {label}", shell_name))
for script in workflow.scripts:
result.scripts += 1
others = unread.scan(script.body)
if others:
result.unread_scripts += 1
for item in others:
result.unread.append(
(
f"{os.path.basename(rel)}:{script.line_of(item.line)}",
item.kind,
item.text,
)
)
for assertion in shell.scan(script.body):
where = f"{os.path.basename(rel)}:{script.line_of(assertion.line)}"
if assertion.polarity == "undecided":
Expand Down
Loading
Loading