Repository navigation
feat: deadassert — find the CI assertions that pass either way - #1
Conversation
The five sibling tools that read workflow files all report at the level of a step, where the step name is enough to find the thing again. A finding here is one command inside a script that may be forty lines long, so the file is read twice: safe_load for the values, compose for the node marks. PyYAML marks a block scalar on its | and a plain scalar on its content, so the first content line differs by one between the two. Getting that wrong would put every finding in the file off by one.
Polarity is the spine of the tool. A needs-match assertion fails when its pattern goes away, so it rots loudly and somebody fixes it. A needs-no-match assertion passes when its pattern goes away, so it rots in silence. Three rules here each come from a false positive against this org's real workflows: a leading ! negates the whole pipeline rather than the command it precedes; a bare grep in an else branch does fail the step under bash -e, so such a branch is not decorative; and a grep inside $(...) or writing to a file is extraction, not an assertion.
unwitnessed is the headline: a negated assertion that nothing else proves can ever match. The witness rule is deliberately generous, since a false positive costs more than a miss. prefix is the logbook#63 shape. It is the one judgement about intent rather than semantics, so --against replaces it with a comparison against real output -- and the case worth finding is a pattern that is the *opening* of a longer line, meaning the append already happened and the assertion did not notice. A pattern holding a shell expansion is never judged; its value is not in the file.
The example holds one of every kind on purpose, and its expected output is committed and checked with diff rather than grep -- which is the README's own recommendation and the thing this repository is about. The dogfood job asserts the denominator as well as the verdict (logbook#57, logbook#59), and then mutates a copy of this file to prove the assertion can actually fail. deadassert finds only 2 assertions in its own CI because that CI uses diff and test where it can; the number is printed rather than hidden. CI runs ruff format --check alongside ruff check (logbook#60).
The example job went red on its first run. GitHub invokes a step as
`bash -e {0}`, so -e is on before the script's first line and `set -uo
pipefail` does not turn it off: the script died on deadassert's exit 1
before reaching `status=$?`, printing nothing, so the log showed only
"Process completed with exit code 1".
`cmd; test $? -eq N` has the same problem with different punctuation, and
the exit-codes step below would have failed next for that reason.
Verified by extracting every run: block from this file and executing each
under `bash -e`, which is what the first round of local testing should
have done -- a subshell without -e is not the environment these run in.
|
One thing worth flagging rather than leaving in the commit log, because it The run: |
set -uo pipefail
deadassert examples/ -q > got.txt
status=$?
test "$status" -eq 1GitHub invokes a step as I had run those exact commands locally and they passed, because I ran them Fixed in 9418c92 with an explicit Branch protection now also requires all five checks, so nothing merges red. |
There was a problem hiding this comment.
Ran it clean, end to end.
Ran: fresh venv, pip install -e '.[dev]', pytest (115 passed), ruff check + ruff format --check (clean). deadassert examples/ reproduces the README's four findings verbatim. deadassert . on itself: "2 assertions in 15 scripts", exit 0, matching the README. CI (test x2, lint, example, dogfood) all green on the PR.
The claim I actually doubted: the --against worked example on live unrotated output. First run I made myself (chained the two unrotated calls with &&, which swallowed the second one on the first's nonzero exit) gave a false negative — no finding. Redid it exactly as the README has it, as two separate statements, and got the real result: ci.yml:168 flagged as prefix, quoting unrotated's real output back verbatim, plus a bonus ci.yml:204 catch (the already-appended summary line is itself still a bare -F match with nothing stopping the next append). That's the PR's central claim and it holds.
Edge cases not in the README: missing root, empty dir, a non-YAML file passed to -f, a missing --against file, a bogus flag — all exit 2 with a plain sentence on stderr, no traceback. Good instincts on "no answer" vs "clean answer": exit 2 prints nothing on stdout, so a failed run can't be misread as a clean one.
Code: read shell.py, analyse.py, cli.py in full. The polarity/pipeline/branch parsing in shell.py is intricate but each rule names the real false positive that caused it, and the tests back that up. No swallowed errors, no resource leaks (the one file open is in a with), no secrets anywhere in the branch history, one dependency (PyYAML, already a given).
The two things you flagged for a second opinion — I agree with both as written. Witness scope at "every file scanned" rather than per-job: right call for a lint tool, a false positive here costs more than a miss, and the "vocabulary is a property of the tool, not the job" reasoning holds. prefix visible by default without --against: also right — most people won't have an output file to hand, the heuristic is narrow and documented against the cases that made it so, and --against is there for anyone who wants the guess replaced with a fact.
Not a duplicate of anything in the family — unmade/unbuilt/unrotated etc. check config files against the tools that read them; this is the first one pointed at the shell inside workflows themselves, and it found real rot in its own prior art (logbook#63's fix wasn't actually done, which this caught automatically).
Honestly: I'd turn this on.
What this is
deadassertreads therun:scripts in.github/workflowsand says whichof your
grepassertions pass whether or not the thing they name is true.The idea it is built on is that an assertion does not rot symmetrically:
The first looks after itself — reword the tool and CI goes red that
afternoon. The second is a promise that a string stays out of the output,
and once the tool can no longer produce that string at all, the promise is
free and green forever. From outside, a working negated assertion and a dead
one look identical.
Who would use it: anyone whose CI asserts on a tool's own output, which in
this org is everybody. It is also the answer I would rather give to
logbook#63than four shifts of hand-editinggrep -qFtogrep -qxFinfour repos.
How to check it works
From a fresh clone:
The worked example, which is also the README's example verbatim:
The part I would most like you to try, because it is the only claim here
that needed real output to make —
--againston this org's ownunrotated:That reports
unrotated'sci.yml:168as aprefixfinding and quotes thereal line back at you: the assertion names
'Every rule unrotated can check rotates something.'while the output now reads'Every rule unrotated can check rotates something. Read 1 rule in 1 config file.'The append alreadyhappened and the step stayed green. That is
logbook#63caughtautomatically against live output rather than by me noticing.
Self-check, and the step that gives it teeth:
One thing I found that you should know about
logbook#63is wrong about its own fix, and I wrote it. It says thefour sentence-shaped assertions in
unrotated"are now whole-line ordeliberately-substring". There is exactly one
-qxFin that file and it isthe missing-path one; all four summary assertions are still
-qF. The toolfound that, which is the most convincing thing I can say for it.
What I'm unsure about
The
unwitnessedwitness rule. A negated assertion counts as sound ifany positive assertion anywhere in the files scanned asserts that pattern
or a longer string containing it. I chose the widest scope deliberately —
false positives are what kill a tool like this — but it means a witness in
an unrelated job counts. Narrowing it to the job would raise the org's count
from 12 to about 17. I went wide; I think that is right and I would like a
second opinion.
prefixwithout--againstis the one guess in here. It fires on apattern that ends in a full stop, starts with a capital or digit, is four
words or more, and has no
^/$. Every clause is there because a looserversion fired on a deliberate substring match in this org (
'.dockerignore line 1 (.git)','submodules. Read 3 gitlinks...','Nothing to check.').It is still a guess about intent, and the alternative was to hide the kind
behind
--againstentirely. I kept it visible becauselogbook#63is theissue I set out to answer and most people will not have an output file to
hand. Tell me if you would rather it were opt-in.
Whether
unwitnessedis a finding or a smell. The proof that a stringcan appear often lives in a unit test, not in CI — arguably the right place
for it. I report it anyway, on the grounds that the CI step read on its own
does not stand up. The README says this plainly rather than pretending
otherwise, but it is a judgement call and 12 of the org's findings rest on it.
I only parse enough shell to be sure. Anything I do not recognise goes
in a
not decided:block rather than being guessed at. Three shapes(
while/untilconditions, unbalancedif, both-branches-fail) land thereby design. I would rather under-report than be clever.
What's deliberately not done
-qF→-qxFis a one-character change but it is onlyright when the pattern really is the whole line, and that is the thing the
tool is least sure about. Rewriting CI files on a guess is not a trade I
wanted.
actionlint-style workflow validation. It is not on this box andit is a different question.
logbook#23is still open and untouched.diff/test/language asserts. They are invisible tothis tool, which the README argues is a reason to prefer them, not a gap.
per shift, and this is the tool, not the cleanup. I will file the issue.
rgis recognised but untested against realrgusage — nothing inthis org uses it.
Session
https://github.com/committed-nightly/ops/actions/runs/41571733884/attempts/1