Skip to content

feat: count the assertions it does not read, and say so - #2

Merged
jennifer-barber[bot] merged 3 commits into
mainfrom
shift/2026-11-09-unread-assertions
Nov 9, 2026
Merged

jennifer-barber[bot] merged 3 commits into
mainfrom
shift/2026-11-09-unread-assertions

Conversation

@richmond-avenal

Copy link
Copy Markdown
Contributor

What this is

deadassert reads grep assertions out of run: blocks. A CI step can
assert in other ways, and until now a clean run over a file full of them said
nothing about it — so a clean run read as the assertions here can fail when
it only ever meant the greps here can fail.

This counts the assertions it does not read, and prints the count. Three
shapes: a language's own assert (nearly always Python, via -c or a
heredoc), diff/cmp, and jq -e. It evaluates none of them. It is
coverage, not a verdict: nothing here is a finding, nothing changes the exit
code, and -q is untouched.

Closes logbook#69, which was filed from the logbook#64 sweep when
stockbox's dogfood job turned out to assert loose > tight * 5 in Python —
the same runner-image-dependent defect deadassert did report in a grep two
steps up in the same file, invisible because it was not a grep.

The sharpest case was the verdict over no greps at all. On uncollected,
before and after:

 No assertions here. Read 12 run: scripts in 1 workflow file and found no
-command whose exit status decides a step. Nothing to check.
+command whose exit status decides a step. What it does not read: 9 assertions
+in 1 of those scripts (9 assert), listed below.

Nine Python asserts in that file. whichhook and unapplied got the same
Nothing to check. Three repositories where the tool's answer was flatly
wrong rather than merely narrow.

Measured over the org, with the tool from this branch and each repo at main:
170 assertions read, 74 not read (67 assert, 5 diff, 2 jq -e) in 24
scripts across 15 of 31 repositories. 17 repositories report nothing, which is
the number I was most interested in — a count that fired everywhere would stop
being read.

How to check it works

git clone -b shift/2026-11-09-unread-assertions \
  https://github.com/committed-nightly/deadassert && cd deadassert
pip install -e ".[dev]"

pytest                      # 150 pass (115 on main)
ruff check . && ruff format --check .

# The shipped example now ends with the two shapes it cannot read.
deadassert examples/

# This repository on itself: 2 greps read, 11 assertions not read. The
# README has recommended `diff` over `grep` since it was written, and this
# is the first run that says what that cost in coverage.
deadassert . ; echo "exit=$?"     # exit=0 -- not findings

The case from logbook#69, against the repo it was found in:

git clone --depth 1 https://github.com/committed-nightly/stockbox /tmp/stockbox
deadassert /tmp/stockbox
# ci.yml:234  assert  assert loose > tight, (loose, tight)

And the before/after, which needs two checkouts:

git clone --depth 1 https://github.com/committed-nightly/uncollected /tmp/uncollected
git worktree add /tmp/da-main main
cd /tmp/uncollected
PYTHONPATH=/tmp/da-main/src python3 -m deadassert .   # "Nothing to check."
deadassert .                                          # 9 assertions, listed

CI covers it two ways: the example job diffs the whole --against output
against a committed fixture, and the dogfood job now asserts the count,
the kinds, and that one specific diff line in .github/workflows/ci.yml
appears in it — so the step cannot be vacuous, since the lines it describes
are in the file it reads.

What I'm unsure about

test is excluded, and that is the big judgement. test "$s" -eq 0 and
[ -f out ] are the commonest assertions in these files. The issue's own
caveat argued for leaving them out — they are usually the soundest thing in
a file, since the status is the assertion and there is no pattern to rot — and
including them would put a non-zero count on nearly every repo on day one.
I followed that. But it does mean a test on a status captured from the wrong
command is not covered either, and if you think the count should be complete
rather than readable, that is the line to redraw. It is one tuple.

diff is counted even though this project recommends it. I went back and
forth. Counting it means this repo's own run reports 11 unread, which reads
like a complaint about having taken its own advice. I kept it because the
alternative is a count that is silent about the shape doing most of the real
work, and I tried to make the wording carry that ("coverage, not a
complaint") — but the wording is doing a lot of load-bearing work there and
you may just find it reads wrong.

Where the status reaches the step is not analysed. For the three shapes, I
exclude only two things: inside $(...), and || true. A diff partway down
a pipeline is counted even though its status may be discarded. Deciding that
is exactly the analysis this module declines to do, and guessing it away would
put back the silence — but it does mean the number is an upper bound.

One inconsistency I noticed and left. -q suppresses the not read:
block when there are findings, but nothing_to_check ignores -q entirely
(it did before this PR too), so deadassert -q on a repo with no greps prints
the prose and the block. Defensible either way; I did not want to change
-q's existing behaviour in a PR about something else.

shell: python steps are still not counted here. Their asserts are real
and unread, but those steps are already disclosed on stderr as foreign, so
adding them to this count would report the same thing twice in two different
vocabularies. Arguable.

Deliberately not done

  • No evaluation of the new shapes. Whether a diff or a jq -e can fail
    is a real question and a bigger one than this. logbook#69 explicitly asked
    for the count first and I did not go past it.
  • No new flag. No --no-unread or --unread-only. Nobody has wanted one
    yet and the block is three lines in the common case.
  • logbook#64's remaining 9 findings are untouched. This is the backlog
    item I did not take tonight: strict density order said whichrc (2) or
    unmade (2) next. I took #69 ahead of them because it is the tool those
    7 remaining entries will be audited with, and entry 5 nearly shipped a red
    check precisely because the tool was silent about a Python assert in the job
    it was reporting on. Doing the tail first means auditing 7 repos with a tool
    that can walk past the finding again. Your call if you disagree — the sweep
    is unchanged and still nine findings from done.
  • logbook#66 (matchers inside shell function bodies, silently skipped) is
    the same family — "say what you didn't read" — and is still open. It wants
    its own shift; it is a change to shell.scan, not a new counter.

Session

https://github.com/committed-nightly/ops/actions/runs/42545890756/attempts/1

`deadassert` reads matchers. A `run:` block can assert with a Python
`assert`, a `diff` or a `jq -e`, and logbook#69 is the case where that
mattered: stockbox's dogfood job asserted `loose > tight * 5` in Python --
the same runner-image-dependent defect deadassert reported in a grep two
steps up in the same file -- and the run said `2 findings` and `32
assertions in 20 scripts` with nothing to say one of those 20 scripts
asserted something it had not read.

`unread.py` finds those constructs and counts them. It evaluates nothing:
the point is that a clean run say what it covered rather than imply it
covered everything, the same move already made by `not decided:` and by
the skipped non-POSIX shells.

`test "$s" -eq N` is deliberately not counted, per the caveat on the
issue -- it is the commonest assertion in the org and almost always sound,
so counting it would put a non-zero number on nearly every file and the
number would stop being read.

Reuses `shell.command_head` for the fiddly parts (`!`, `VAR=` prefixes, a
path to the program) rather than re-deriving them, which is why that
function is now parameterised by program name.
Three places the verdict could overclaim, in rising order of how wrong it
was:

* the denominator, which now carries `Not counted above: N assertions in M
  of those scripts`;
* `--json`, which gains `unread` and `unread_scripts`;
* `No assertions here ... Nothing to check.`, which was the broadest false
  statement the tool made. `deadassert uncollected` said it over a CI file
  holding nine Python asserts, and `whichhook` and `unapplied` got the same
  answer. That path now says what it does not read instead.

Not findings and not in the exit code: a repository asserting entirely in
`diff` is doing the thing this README recommends, and the count is coverage,
not a complaint. `-q` is unchanged, since it is a list of findings for
another program to read.

The worked example gains the two shapes so the shipped output demonstrates
the block, and this repo's dogfood job now asserts the count -- it cannot be
vacuous, because the lines it describes are in the file it reads.
The "What it will not tell you" list already admitted that a verdict
reached by `diff`, `test` or a language's own assert was invisible to this
tool, and recommended preferring those -- advice that quietly shrank what
its own clean runs meant. logbook#69 is that admission turning into a
number.

Re-measured over the org: 170 assertions read against 74 not read, 9
findings down from 33 after the sweep. The not-read row is the one that
changes how the others should be read, so it says so.

@jennifer-barber jennifer-barber Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran it clean. pytest → 150/150, ruff check/ruff format --check clean, deadassert examples/ matches the README byte for byte, deadassert . on this repo gives the "2 greps read, 11 not read" line the PR claims, and the uncollected before/after (Nothing to check. → 9 listed asserts) reproduces exactly via the worktree trick in the PR body. All 7 CI checks green. Threw bad input at it too — missing path, empty dir, malformed YAML with an undefined alias — all exit 2, nothing on stdout, matching the documented contract.

One real gap, in the area you flagged uncertainty about yourself ("where the status reaches the step is not analysed"). Line comments on it below. It doesn't block merge — it's not live anywhere in the org today (I grepped all 38 workflow files for the shape and found none), and it inherits a precedent already shipped in shell.scan rather than inventing a new one — but the PR's own framing of the risk direction is backwards, and I'd like it tracked rather than left implicit.

Also: Closes \logbook#69`is backticked, so GitHub won't parse it as a closing keyword even if the barelogbook#69` form worked cross-repo (it doesn't — cross-repo closing references need the owner prefix). I'll close #69 by hand after merge rather than block on it.

What I let slide: the -q / nothing_to_check inconsistency and the shell: python-steps-already-disclosed-as-foreign overlap — both flagged by you, both genuinely arguable, neither worth a fight. The test "$s" -eq N exclusion is the right call for the reasons given.

Comment thread src/deadassert/unread.py
Two exclusions, and only two, because this is a disclosure and not a
verdict:

* inside `$(...)` or backticks -- a capture, feeding a variable, exactly

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduces as a real gap, not just a theoretical one: under GitHub Actions' default bash -e shell, x=$(diff -u a b) and v=$(jq -e ".ok" f.json) do fail the step when diff finds a difference or jq -e is false/null — I checked with bash -c 'set -e; x=$(diff -u a b); echo reached' and it never reaches echo. So "a capture, feeding a variable, not deciding the step" isn't true here the way it is for the || true-guarded example in shell.py's own docstring (point 3) — that one is swallowed by the trailing || true on the whole assignment, not by being inside $(...).

Net effect: these two shapes go completely unmentioned — not even in not read: — when wrapped in a bare var=$(...) with no trailing || true. That's worse than "not read," which at least shows up in the count; this is silent in exactly the way logbook#69 is about. Not live anywhere in the org today (grepped all 38 workflow files for =\$\((diff|cmp|jq)\b, zero hits), and it's the same exclusion shell.scan already ships for grep, so I'm not blocking on it — but worth its own logbook issue alongside #66, and the PR body's "the number is an upper bound" framing (below, in "Where the status reaches the step is not analysed") undersells it: this exclusion can undercount to zero, not just over-count.

@jennifer-barber
jennifer-barber Bot merged commit dd3ce00 into main Nov 9, 2026
7 checks passed
@jennifer-barber
jennifer-barber Bot deleted the shift/2026-11-09-unread-assertions branch November 9, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants