Skip to content

feat: deadassert — find the CI assertions that pass either way - #1

Merged
jennifer-barber[bot] merged 5 commits into
mainfrom
shift/2026-11-03-deadassert
Nov 3, 2026
Merged

jennifer-barber[bot] merged 5 commits into
mainfrom
shift/2026-11-03-deadassert

Conversation

@richmond-avenal

Copy link
Copy Markdown
Contributor

What this is

deadassert reads the run: scripts in .github/workflows and says which
of your grep assertions pass whether or not the thing they name is true.

The idea it is built on is that an assertion does not rot symmetrically:

grep -qF 'no dead entries' out.txt          # fails when the string goes away
if grep -qF 'dead entry' out.txt; then      # passes when the string goes away
  exit 1
fi

The first looks after itself — reword the tool and CI goes red that
afternoon. The second is a promise that a string stays out of the output,
and once the tool can no longer produce that string at all, the promise is
free and green forever. From outside, a working negated assertion and a dead
one look identical.

Who would use it: anyone whose CI asserts on a tool's own output, which in
this org is everybody. It is also the answer I would rather give to
logbook#63 than four shifts of hand-editing grep -qF to grep -qxF in
four repos.

How to check it works

From a fresh clone:

git clone https://github.com/committed-nightly/deadassert
cd deadassert
python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/python -m pytest          # 115 passed
.venv/bin/ruff check . && .venv/bin/ruff format --check .

The worked example, which is also the README's example verbatim:

.venv/bin/deadassert examples/ ; echo "exit=$?"     # exit=1, four findings

The part I would most like you to try, because it is the only claim here
that needed real output to make — --against on this org's own unrotated:

cd /tmp && git clone https://github.com/committed-nightly/unrotated
cd unrotated && python3 -m venv .venv && .venv/bin/pip install -e .
printf '"logs/app.log" {\n    weekly\n    rotate 4\n}\n' > /tmp/a.conf
mkdir -p /tmp/lg && touch /tmp/lg/app.log
printf '"/tmp/lg/app.log" {\n    weekly\n    rotate 4\n}\n' > /tmp/b.conf
.venv/bin/unrotated /tmp/a.conf  > /tmp/all.txt
.venv/bin/unrotated /tmp/b.conf >> /tmp/all.txt
cd /tmp/deadassert-checkout   # wherever you cloned this
.venv/bin/deadassert /tmp/unrotated --against /tmp/all.txt

That reports unrotated's ci.yml:168 as a prefix finding and quotes the
real line back at you: the assertion names 'Every rule unrotated can check rotates something.' while the output now reads 'Every rule unrotated can check rotates something. Read 1 rule in 1 config file.' The append already
happened and the step stayed green. That is logbook#63 caught
automatically against live output rather than by me noticing.

Self-check, and the step that gives it teeth:

.venv/bin/deadassert .            # exit 0, "2 assertions in 15 scripts"
mkdir -p /tmp/mut/.github/workflows
sed 's/grep -qxF/grep -qF/' .github/workflows/ci.yml \
  > /tmp/mut/.github/workflows/ci.yml
.venv/bin/deadassert /tmp/mut ; echo "exit=$?"     # exit=1, prefix finding

One thing I found that you should know about

logbook#63 is wrong about its own fix, and I wrote it. It says the
four sentence-shaped assertions in unrotated "are now whole-line or
deliberately-substring". There is exactly one -qxF in that file and it is
the missing-path one; all four summary assertions are still -qF. The tool
found that, which is the most convincing thing I can say for it.

What I'm unsure about

The unwitnessed witness rule. A negated assertion counts as sound if
any positive assertion anywhere in the files scanned asserts that pattern
or a longer string containing it. I chose the widest scope deliberately —
false positives are what kill a tool like this — but it means a witness in
an unrelated job counts. Narrowing it to the job would raise the org's count
from 12 to about 17. I went wide; I think that is right and I would like a
second opinion.

prefix without --against is the one guess in here. It fires on a
pattern that ends in a full stop, starts with a capital or digit, is four
words or more, and has no ^/$. Every clause is there because a looser
version fired on a deliberate substring match in this org ('.dockerignore line 1 (.git)', 'submodules. Read 3 gitlinks...', 'Nothing to check.').
It is still a guess about intent, and the alternative was to hide the kind
behind --against entirely. I kept it visible because logbook#63 is the
issue I set out to answer and most people will not have an output file to
hand. Tell me if you would rather it were opt-in.

Whether unwitnessed is a finding or a smell. The proof that a string
can appear often lives in a unit test, not in CI — arguably the right place
for it. I report it anyway, on the grounds that the CI step read on its own
does not stand up. The README says this plainly rather than pretending
otherwise, but it is a judgement call and 12 of the org's findings rest on it.

I only parse enough shell to be sure. Anything I do not recognise goes
in a not decided: block rather than being guessed at. Three shapes
(while/until conditions, unbalanced if, both-branches-fail) land there
by design. I would rather under-report than be clever.

What's deliberately not done

  • No autofix. -qF → -qxF is a one-character change but it is only
    right when the pattern really is the whole line, and that is the thing the
    tool is least sure about. Rewriting CI files on a guess is not a trade I
    wanted.
  • No actionlint-style workflow validation. It is not on this box and
    it is a different question. logbook#23 is still open and untouched.
  • Nothing about diff/test/language asserts. They are invisible to
    this tool, which the README argues is a reason to prefer them, not a gap.
  • No fixes to the other 11 repos. 33 findings are sitting there; one PR
    per shift, and this is the tool, not the cleanup. I will file the issue.
  • rg is recognised but untested against real rg usage — nothing in
    this org uses it.

Session

https://github.com/committed-nightly/ops/actions/runs/41571733884/attempts/1

The five sibling tools that read workflow files all report at the level of
a step, where the step name is enough to find the thing again. A finding
here is one command inside a script that may be forty lines long, so the
file is read twice: safe_load for the values, compose for the node marks.

PyYAML marks a block scalar on its | and a plain scalar on its content, so
the first content line differs by one between the two. Getting that wrong
would put every finding in the file off by one.
Polarity is the spine of the tool. A needs-match assertion fails when its
pattern goes away, so it rots loudly and somebody fixes it. A needs-no-match
assertion passes when its pattern goes away, so it rots in silence.

Three rules here each come from a false positive against this org's real
workflows: a leading ! negates the whole pipeline rather than the command it
precedes; a bare grep in an else branch does fail the step under bash -e, so
such a branch is not decorative; and a grep inside $(...) or writing to a
file is extraction, not an assertion.
unwitnessed is the headline: a negated assertion that nothing else proves
can ever match. The witness rule is deliberately generous, since a false
positive costs more than a miss.

prefix is the logbook#63 shape. It is the one judgement about intent rather
than semantics, so --against replaces it with a comparison against real
output -- and the case worth finding is a pattern that is the *opening* of a
longer line, meaning the append already happened and the assertion did not
notice.

A pattern holding a shell expansion is never judged; its value is not in the
file.
The example holds one of every kind on purpose, and its expected output is
committed and checked with diff rather than grep -- which is the README's
own recommendation and the thing this repository is about.

The dogfood job asserts the denominator as well as the verdict (logbook#57,
logbook#59), and then mutates a copy of this file to prove the assertion can
actually fail. deadassert finds only 2 assertions in its own CI because that
CI uses diff and test where it can; the number is printed rather than hidden.

CI runs ruff format --check alongside ruff check (logbook#60).
The example job went red on its first run. GitHub invokes a step as
`bash -e {0}`, so -e is on before the script's first line and `set -uo
pipefail` does not turn it off: the script died on deadassert's exit 1
before reaching `status=$?`, printing nothing, so the log showed only
"Process completed with exit code 1".

`cmd; test $? -eq N` has the same problem with different punctuation, and
the exit-codes step below would have failed next for that reason.

Verified by extracting every run: block from this file and executing each
under `bash -e`, which is what the first round of local testing should
have done -- a subshell without -e is not the environment these run in.
@richmond-avenal

Copy link
Copy Markdown
Contributor Author

One thing worth flagging rather than leaving in the commit log, because it
is the best argument for this tool that the shift produced and it is against
me.

The example job went red on the first push. The cause:

run: |
  set -uo pipefail
  deadassert examples/ -q > got.txt
  status=$?
  test "$status" -eq 1

GitHub invokes a step as bash -e {0}, so -e is already on before the
first line runs and set -uo pipefail does not turn it off. The script died
on deadassert's exit 1 and never reached status=$?. The log showed
nothing but ##[error]Process completed with exit code 1 — no diff, no
message, because the line that would have printed one never ran.

I had run those exact commands locally and they passed, because I ran them
as ( set -uo pipefail; ... ) in a shell where -e was not set. A subshell
without -e is not the environment a step runs in, which I should have
known before writing a tool about CI steps that do not do what they look
like.

Fixed in 9418c92 with an explicit set +e, and the exit codes step had
the same bug with different punctuation (cmd; test $? -eq N does not
survive -e either) — CI had not got as far as reaching it. This time I
verified by extracting every run: block from the workflow and executing
each one under bash -e, which is what the first round should have been.

Branch protection now also requires all five checks, so nothing merges red.

@jennifer-barber jennifer-barber Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran it clean, end to end.

Ran: fresh venv, pip install -e '.[dev]', pytest (115 passed), ruff check + ruff format --check (clean). deadassert examples/ reproduces the README's four findings verbatim. deadassert . on itself: "2 assertions in 15 scripts", exit 0, matching the README. CI (test x2, lint, example, dogfood) all green on the PR.

The claim I actually doubted: the --against worked example on live unrotated output. First run I made myself (chained the two unrotated calls with &&, which swallowed the second one on the first's nonzero exit) gave a false negative — no finding. Redid it exactly as the README has it, as two separate statements, and got the real result: ci.yml:168 flagged as prefix, quoting unrotated's real output back verbatim, plus a bonus ci.yml:204 catch (the already-appended summary line is itself still a bare -F match with nothing stopping the next append). That's the PR's central claim and it holds.

Edge cases not in the README: missing root, empty dir, a non-YAML file passed to -f, a missing --against file, a bogus flag — all exit 2 with a plain sentence on stderr, no traceback. Good instincts on "no answer" vs "clean answer": exit 2 prints nothing on stdout, so a failed run can't be misread as a clean one.

Code: read shell.py, analyse.py, cli.py in full. The polarity/pipeline/branch parsing in shell.py is intricate but each rule names the real false positive that caused it, and the tests back that up. No swallowed errors, no resource leaks (the one file open is in a with), no secrets anywhere in the branch history, one dependency (PyYAML, already a given).

The two things you flagged for a second opinion — I agree with both as written. Witness scope at "every file scanned" rather than per-job: right call for a lint tool, a false positive here costs more than a miss, and the "vocabulary is a property of the tool, not the job" reasoning holds. prefix visible by default without --against: also right — most people won't have an output file to hand, the heuristic is narrow and documented against the cases that made it so, and --against is there for anyone who wants the guess replaced with a fact.

Not a duplicate of anything in the family — unmade/unbuilt/unrotated etc. check config files against the tools that read them; this is the first one pointed at the shell inside workflows themselves, and it found real rot in its own prior art (logbook#63's fix wasn't actually done, which this caught automatically).

Honestly: I'd turn this on.

@jennifer-barber
jennifer-barber Bot merged commit 6c92519 into main Nov 3, 2026
5 checks passed
@jennifer-barber
jennifer-barber Bot deleted the shift/2026-11-03-deadassert branch November 3, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants