Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,59 @@ jobs:
assert any(c.startswith(wanted) for c in commands), commands
PY

# logbook#73 and logbook#66, at the CLI rather than in pytest: both
# were about what the tool does to a whole file, and both were false
# on shapes this file does not itself contain.
#
# `printf` rather than a heredoc, for the reason the `exit codes` step
# uses it: a heredoc's lines are physical lines of *this* script, so a
# fixture written that way puts its greps into this repository's own
# assertion count, and a fixture about dead assertions would start
# arriving as findings against the file that built it.
- name: the shapes logbook#73 and logbook#66 were filed for
run: |
set -euo pipefail
mkdir -p shapes/.github/workflows
printf '%s\n' \
"'on': push" \
"jobs:" \
" j:" \
" steps:" \
" - run: |" \
" grep -qxF 'ok' out.txt" \
" echo hi | grep -qxF 'ok' out.txt || {" \
" echo 'absent'" \
" exit 1" \
" }" \
" if grep -qxF 'ok' out.txt; then exit 1; fi" \
" - run: |" \
" guard() {" \
" grep -qF 'ok' out.txt" \
" }" \
" guard" \
> shapes/.github/workflows/ci.yml
deadassert shapes | tee shapes.txt
deadassert shapes --json > shapes.json
python - <<'PY'
import json
d = json.load(open("shapes.json"))

# Three assertions read, not two: the multi-line `|| { ... }` is one
# of them, and it used to be reported `decorative` because the
# `exit 1` two lines down was never seen.
assert d["findings"] == [], d["findings"]
assert d["assertions"] == 3, d["assertions"]

# The one-line `if ... then exit 1; fi`. Read as `decorative` until
# `then` counted as a command separator, so the negated count is
# what pins it.
assert d["negated"] == 1, d["negated"]

# The guard in the function body: declined, and said so.
unread = [(x["kind"], x["command"]) for x in d["unread"]]
assert unread == [("in function", "grep -qF 'ok' out.txt")], unread
PY

# The assertion above is only worth anything if it can go red. Break
# one assertion in a copy of this file and check that it does.
- name: a broken assertion in this file is caught
Expand Down
54 changes: 33 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,12 @@ stops being reported.
verdict reached by `diff`, `test` or a language's own assert is not read —
which is a reason to prefer those, not a gap to fill. See below. It is no
longer *invisible*, though: those get counted, and the count is printed.
- **A matcher inside a shell function body is not read.** Whether its
verdict reaches the step depends on where the function is *called* — bare
under `set -e` it does, in an `if` condition it does not, and an uncalled
function asserts nothing at all. That is not worked out here, so the body
is declined rather than half-read, and every matcher in one is printed
under `not read:`.
- **`unwitnessed` is a smell, not a proof.** It says nothing in *these
workflows* establishes the string can appear. The proof may well live in a
unit test, where it is arguably better placed. What the finding is really
Expand Down Expand Up @@ -194,17 +200,22 @@ So it counts them now:
```console
$ deadassert .
Every assertion here fails when the thing it names stops being true.
2 assertions in 15 scripts across 1 workflow file, 0 of them negated. Not counted above: 11 assertions in 5 of those scripts (9 assert, 2 diff), listed below.
2 assertions in 16 scripts across 1 workflow file, 0 of them negated. Not counted above: 15 assertions in 6 of those scripts (13 assert, 2 diff), listed below.

not read:
ci.yml:37 assert assert total >= 110, f"only {total} tests collected"
ci.yml:85 diff diff -u examples/expected-quiet.txt got.txt
...
```

Three shapes, picked by counting what the workflow files in this
Four shapes. Three were picked by counting what the workflow files in this
organisation actually use: a language's own `assert` (nearly always Python,
via `-c` or a heredoc), `diff` and `cmp`, and `jq -e`.
via `-c` or a heredoc), `diff` and `cmp`, and `jq -e`. The fourth is
`in function`, and it is the only one here that is a `grep` — a matcher in a
function body, which this tool could read and declines to, for the reason in
the caveat above. It was silent about those until it wasn't: two identical
guards in one file, one at the top level and one in a function, and only the
first was ever reported.

**This decides nothing.** It does not judge whether a `diff` can fail; it
reports that one is there and was not read. The same move the tool already
Expand Down Expand Up @@ -255,36 +266,37 @@ is small on purpose, and it is printed rather than hidden.
## Measured on something real

Run across the 31 repositories of the organisation this was written in — 37
workflow files, 360 `run:` scripts:
workflow files, 364 `run:` scripts:

| | count |
| --- | --- |
| assertions read | 170 |
| negated (`needs-no-match`) | 24 |
| `unwitnessed` | 6 |
| `prefix` | 3 |
| assertions read | 183 |
| negated (`needs-no-match`) | 22 |
| `unwitnessed` | 0 |
| `prefix` | 0 |
| `swallowed` / `masked` / `decorative` | 0 |
| not read | 74, in 24 scripts |
| not read | 79, in 24 scripts |

Seven of the thirty-one repositories have at least one finding. The first
run of this tool found 33 — 12 `unwitnessed` and 21 `prefix`, across 12
repositories — and the 24 that have gone are a repo-at-a-time sweep, not a
`sed`; the remaining 9 are the tail of it.
No findings left anywhere, which is not the tool scoring zero — it is the
end of a sweep. The first run found 33, 12 `unwitnessed` and 21 `prefix`
across 12 repositories, and they were closed a repository at a time over
three weeks rather than with a `sed`. The number to watch now is the last
row, not the empty ones.

The three can't-fail kinds scoring zero is the result I did not expect:
every `|| true` in that organisation is on a *tool invocation*, deliberately
allowing a non-zero findings exit before the output is asserted, which is
correct usage. An early version of this tool reported all of them and was
wrong to.

The last row is the one that changes how you should read the others. **74
assertions in these files are not read at all** — 67 Python asserts, 5
`diff`, 2 `jq -e` — against 170 that are, and they are concentrated: 15
repositories have at least one, and 3 have nothing *but* unread assertions,
so the verdict on them used to be `Nothing to check.` The 9 findings above
are a verdict about the greps in this organisation, which is a smaller thing
than it looks, and the point of printing the number is that you no longer
have to take my word for how much smaller.
The last row is the one that changes how you should read the others. **79
assertions in these files are not read at all** — 71 Python asserts, 5
`diff`, 2 `jq -e` and 1 in a function body — against 183 that are, and they
are concentrated: 15 repositories have at least one, and 3 have nothing
*but* unread assertions, so the verdict on them used to be `Nothing to
check.` An all-clear here is a verdict about the greps in this organisation,
which is a smaller thing than it looks, and the point of printing the number
is that you no longer have to take my word for how much smaller.

`shellcheck` 0.9.0 reports none of these at default settings. With
`--enable=all` it finds the two pipeline cases as `SC2312` (info) and
Expand Down
Loading
Loading