Repository navigation
feat: count the assertions it does not read, and say so - #2
Conversation
`deadassert` reads matchers. A `run:` block can assert with a Python `assert`, a `diff` or a `jq -e`, and logbook#69 is the case where that mattered: stockbox's dogfood job asserted `loose > tight * 5` in Python -- the same runner-image-dependent defect deadassert reported in a grep two steps up in the same file -- and the run said `2 findings` and `32 assertions in 20 scripts` with nothing to say one of those 20 scripts asserted something it had not read. `unread.py` finds those constructs and counts them. It evaluates nothing: the point is that a clean run say what it covered rather than imply it covered everything, the same move already made by `not decided:` and by the skipped non-POSIX shells. `test "$s" -eq N` is deliberately not counted, per the caveat on the issue -- it is the commonest assertion in the org and almost always sound, so counting it would put a non-zero number on nearly every file and the number would stop being read. Reuses `shell.command_head` for the fiddly parts (`!`, `VAR=` prefixes, a path to the program) rather than re-deriving them, which is why that function is now parameterised by program name.
Three places the verdict could overclaim, in rising order of how wrong it was: * the denominator, which now carries `Not counted above: N assertions in M of those scripts`; * `--json`, which gains `unread` and `unread_scripts`; * `No assertions here ... Nothing to check.`, which was the broadest false statement the tool made. `deadassert uncollected` said it over a CI file holding nine Python asserts, and `whichhook` and `unapplied` got the same answer. That path now says what it does not read instead. Not findings and not in the exit code: a repository asserting entirely in `diff` is doing the thing this README recommends, and the count is coverage, not a complaint. `-q` is unchanged, since it is a list of findings for another program to read. The worked example gains the two shapes so the shipped output demonstrates the block, and this repo's dogfood job now asserts the count -- it cannot be vacuous, because the lines it describes are in the file it reads.
The "What it will not tell you" list already admitted that a verdict reached by `diff`, `test` or a language's own assert was invisible to this tool, and recommended preferring those -- advice that quietly shrank what its own clean runs meant. logbook#69 is that admission turning into a number. Re-measured over the org: 170 assertions read against 74 not read, 9 findings down from 33 after the sweep. The not-read row is the one that changes how the others should be read, so it says so.
There was a problem hiding this comment.
Ran it clean. pytest → 150/150, ruff check/ruff format --check clean, deadassert examples/ matches the README byte for byte, deadassert . on this repo gives the "2 greps read, 11 not read" line the PR claims, and the uncollected before/after (Nothing to check. → 9 listed asserts) reproduces exactly via the worktree trick in the PR body. All 7 CI checks green. Threw bad input at it too — missing path, empty dir, malformed YAML with an undefined alias — all exit 2, nothing on stdout, matching the documented contract.
One real gap, in the area you flagged uncertainty about yourself ("where the status reaches the step is not analysed"). Line comments on it below. It doesn't block merge — it's not live anywhere in the org today (I grepped all 38 workflow files for the shape and found none), and it inherits a precedent already shipped in shell.scan rather than inventing a new one — but the PR's own framing of the risk direction is backwards, and I'd like it tracked rather than left implicit.
Also: Closes \logbook#69`is backticked, so GitHub won't parse it as a closing keyword even if the barelogbook#69` form worked cross-repo (it doesn't — cross-repo closing references need the owner prefix). I'll close #69 by hand after merge rather than block on it.
What I let slide: the -q / nothing_to_check inconsistency and the shell: python-steps-already-disclosed-as-foreign overlap — both flagged by you, both genuinely arguable, neither worth a fight. The test "$s" -eq N exclusion is the right call for the reasons given.
| Two exclusions, and only two, because this is a disclosure and not a | ||
| verdict: | ||
|
|
||
| * inside `$(...)` or backticks -- a capture, feeding a variable, exactly |
There was a problem hiding this comment.
Reproduces as a real gap, not just a theoretical one: under GitHub Actions' default bash -e shell, x=$(diff -u a b) and v=$(jq -e ".ok" f.json) do fail the step when diff finds a difference or jq -e is false/null — I checked with bash -c 'set -e; x=$(diff -u a b); echo reached' and it never reaches echo. So "a capture, feeding a variable, not deciding the step" isn't true here the way it is for the || true-guarded example in shell.py's own docstring (point 3) — that one is swallowed by the trailing || true on the whole assignment, not by being inside $(...).
Net effect: these two shapes go completely unmentioned — not even in not read: — when wrapped in a bare var=$(...) with no trailing || true. That's worse than "not read," which at least shows up in the count; this is silent in exactly the way logbook#69 is about. Not live anywhere in the org today (grepped all 38 workflow files for =\$\((diff|cmp|jq)\b, zero hits), and it's the same exclusion shell.scan already ships for grep, so I'm not blocking on it — but worth its own logbook issue alongside #66, and the PR body's "the number is an upper bound" framing (below, in "Where the status reaches the step is not analysed") undersells it: this exclusion can undercount to zero, not just over-count.
What this is
deadassertreadsgrepassertions out ofrun:blocks. A CI step canassert in other ways, and until now a clean run over a file full of them said
nothing about it — so a clean run read as the assertions here can fail when
it only ever meant the greps here can fail.
This counts the assertions it does not read, and prints the count. Three
shapes: a language's own
assert(nearly always Python, via-cor aheredoc),
diff/cmp, andjq -e. It evaluates none of them. It iscoverage, not a verdict: nothing here is a finding, nothing changes the exit
code, and
-qis untouched.Closes
logbook#69, which was filed from thelogbook#64sweep whenstockbox's dogfood job turned out to assertloose > tight * 5in Python —the same runner-image-dependent defect
deadassertdid report in a grep twosteps up in the same file, invisible because it was not a grep.
The sharpest case was the verdict over no greps at all. On
uncollected,before and after:
Nine Python asserts in that file.
whichhookandunappliedgot the sameNothing to check.Three repositories where the tool's answer was flatlywrong rather than merely narrow.
Measured over the org, with the tool from this branch and each repo at
main:170 assertions read, 74 not read (67
assert, 5diff, 2jq -e) in 24scripts across 15 of 31 repositories. 17 repositories report nothing, which is
the number I was most interested in — a count that fired everywhere would stop
being read.
How to check it works
The case from
logbook#69, against the repo it was found in:git clone --depth 1 https://github.com/committed-nightly/stockbox /tmp/stockbox deadassert /tmp/stockbox # ci.yml:234 assert assert loose > tight, (loose, tight)And the before/after, which needs two checkouts:
CI covers it two ways: the
examplejobdiffs the whole--againstoutputagainst a committed fixture, and the
dogfoodjob now asserts the count,the kinds, and that one specific
diffline in.github/workflows/ci.ymlappears in it — so the step cannot be vacuous, since the lines it describes
are in the file it reads.
What I'm unsure about
testis excluded, and that is the big judgement.test "$s" -eq 0and[ -f out ]are the commonest assertions in these files. The issue's owncaveat argued for leaving them out — they are usually the soundest thing in
a file, since the status is the assertion and there is no pattern to rot — and
including them would put a non-zero count on nearly every repo on day one.
I followed that. But it does mean a
teston a status captured from the wrongcommand is not covered either, and if you think the count should be complete
rather than readable, that is the line to redraw. It is one tuple.
diffis counted even though this project recommends it. I went back andforth. Counting it means this repo's own run reports 11 unread, which reads
like a complaint about having taken its own advice. I kept it because the
alternative is a count that is silent about the shape doing most of the real
work, and I tried to make the wording carry that ("coverage, not a
complaint") — but the wording is doing a lot of load-bearing work there and
you may just find it reads wrong.
Where the status reaches the step is not analysed. For the three shapes, I
exclude only two things: inside
$(...), and|| true. Adiffpartway downa pipeline is counted even though its status may be discarded. Deciding that
is exactly the analysis this module declines to do, and guessing it away would
put back the silence — but it does mean the number is an upper bound.
One inconsistency I noticed and left.
-qsuppresses thenot read:block when there are findings, but
nothing_to_checkignores-qentirely(it did before this PR too), so
deadassert -qon a repo with no greps printsthe prose and the block. Defensible either way; I did not want to change
-q's existing behaviour in a PR about something else.shell: pythonsteps are still not counted here. Their asserts are realand unread, but those steps are already disclosed on stderr as
foreign, soadding them to this count would report the same thing twice in two different
vocabularies. Arguable.
Deliberately not done
diffor ajq -ecan failis a real question and a bigger one than this.
logbook#69explicitly askedfor the count first and I did not go past it.
--no-unreador--unread-only. Nobody has wanted oneyet and the block is three lines in the common case.
logbook#64's remaining 9 findings are untouched. This is the backlogitem I did not take tonight: strict density order said
whichrc(2) orunmade(2) next. I took#69ahead of them because it is the tool those7 remaining entries will be audited with, and entry 5 nearly shipped a red
check precisely because the tool was silent about a Python assert in the job
it was reporting on. Doing the tail first means auditing 7 repos with a tool
that can walk past the finding again. Your call if you disagree — the sweep
is unchanged and still nine findings from done.
logbook#66(matchers inside shell function bodies, silently skipped) isthe same family — "say what you didn't read" — and is still open. It wants
its own shift; it is a change to
shell.scan, not a new counter.Session
https://github.com/committed-nightly/ops/actions/runs/42545890756/attempts/1