Repository navigation
test(ci): witness the declined-check guard, and give it a denominator - #2
Conversation
`deadassert` reported the `^not checked:` guard in the dogfood job as `unwitnessed` (logbook#64): it fails only when its pattern appears, and nothing in these workflows established that the pattern can appear. The day `is_shallow` or `default_branch` stopped answering, the guard - and the step above it that exists to show the `fetch-depth: 0` advice is needed - would have gone on passing. The witness cannot come from this repository, since the guard requires the string's absence here. So the job now builds one corpus per cause and greps each for the line it should produce: a fresh `git init` with no origin for the default-branch skip, a `--depth 1` clone of it for the branch-and-tag skip. Both corpora are constructed rather than inherited from the runner, because plain `actions/checkout` leaves `refs/remotes/origin/HEAD` unset and would witness the second cause for free until the day it doesn't. Also pins the guard's denominator whole-line. The absence check says nothing about how much was read, and a second workflow file here would read as `2 workflows checked, clean` without the guard noticing its subject had changed.
`deadassert`'s witness rule is generous by design - one positive assertion containing the pattern clears the finding. Confirmed by measurement: deleting the anchor from the shallow witness still gives a clean `deadassert` run, because the other witness is enough for it. So the controls added in the previous commit would stay green while covering one of two causes, and nothing would say which. These two tests check the relationship the controls rest on: every `report.skip()` in `core.check` names its check with a literal, and every one of those names is grepped for in ci.yml. Verified by mutation: a third literal skip fails the first test, a skip whose name comes from a variable fails it too (3 calls, 2 literals), and an unanchored witness fails the second. The behaviour of the skips is already tested in test_cli.py (`TestSkippedChecks`, `TestHealthy`) and is better placed there, so this does not repeat it.
There was a problem hiding this comment.
Checked out the branch, built the venv per README, ran the full suite (179 passed, matches the PR's claim). Extracted the dogfood job's actual run: blocks with the same yaml-slicing trick the PR describes and ran them under bash -e — guard step exits 0, controls step exits 0 and prints exactly the two not checked: lines quoted in the PR body.
Did the mutation checks myself rather than trusting the description:
- Patched
is_shallowingitcmd.pyto always returnFalseand re-ran the controls script: exit 1, as claimed. Reverted cleanly. - Dropped the
^anchor from thebranch and tag filtersgrep inci.ymland rantests/test_dogfood.py:test_every_declinable_check_has_a_witness_in_the_dogfood_jobfails exactly as described, naming the uncovered cause. Reverted.
Read tests/test_dogfood.py — the source-text coupling (grepping report.skip( literals against core.py and ci.yml) is a real cost but the PR is upfront about it and the counting trick (calls vs. named) closes the obvious gap of a skip named from a variable. No objection to taking that trade.
No new dependencies, no secrets in the diff, CI green on all 5 checks. This is logbook#64 sweep entry 8, scoped exactly to the dogfood job's own CI reading of itself — no src/ changes, which matches the PR's stated scope.
Sweep entry 8 of
committed-nightly/logbook#64, first of tonight's pair. Jenasked on
whichrc#3fordeadtriggerandgroupclashto be taken in oneshift as two PRs, since they are tied at one finding each and guard the
identical string. This is the
deadtriggerhalf; the other iscommitted-nightly/groupclash#2.What this is
The dogfood job's last step reads the tool's report of this repository and
fails if any check declined to run:
deadassertreported it asunwitnessed. It fails only when its patternappears, and nothing in these workflows established that the pattern can
appear at all — so if
is_shallowordefault_branchstopped answering, thestep would have gone on passing, and so would the one above it that exists to
show the
fetch-depth: 0advice is needed.Two changes, plus two tests.
A witness per cause. It cannot come from this repository: the guard
requires the string's absence here. So the job builds two corpora and greps
each for the line it should produce — a fresh
git initwith no origin forthe default-branch skip, a
--depth 1clone of it for the branch-and-tagskip. Both are constructed rather than inherited, because plain
actions/checkoutleavesrefs/remotes/origin/HEADunset and would witnessthe second cause for free until the day it stops.
A denominator. The absence check says nothing about how much was read.
The empty case is already covered —
deadtriggerexits 2 on no workflowfiles and
bash -ekills the step at the redirect — but a second workflowfile here would read as
2 workflows checked, cleanand the guard would notnotice its subject had changed.
grep -qxF '1 workflow checked, clean', beforethe guard.
tests/test_dogfood.py, for the one thing a control cannot tell you.deadassert's witness rule is generous by design: one positive assertioncontaining the pattern clears the finding. I measured that — deleting the
anchor from the shallow witness still gives a clean
deadassertrun, becausethe other witness satisfies it. So a third
report.skip()incore.checkwould leave the guard part-witnessed with nothing to say so. The two tests
check that every skip call names its check with a literal, and that every one
of those names is grepped for in
ci.yml.It deliberately does not retest the skips themselves. That behaviour is
already covered in
test_cli.py(TestSkippedChecks, andTestHealthy.test_nothing_is_skipped_in_a_full_clone_with_a_default_branch),and
deadassert's own README says a unit test is the better home for it.How to check it works
The two controls, run the way CI runs them — extract each
run:block andexecute it under
bash -e, since that is the shell a step gets and-eisalready on:
Each control fails when its own cause stops working — the check that the old
guard could not make:
And the two tests, by mutation — add a third skip to
core.checkand watchtest_every_declinable_check_names_itself_with_a_literalfail; delete the^from one witness inci.ymland watch the other test fail whiledeadassert .still reports clean.What I am unsure about
The tests read source text, and that is a cost. They match
report\.skip\(againstcore.pyand look for a string insideci.yml,because the thing being checked is a relationship between a Python file and a
YAML file and nothing at runtime holds both. The blind spot I closed on
purpose: a skip whose name comes from a variable escapes the name regex, so
the test also counts the calls and fails when the two numbers disagree
(verified: 3 calls, 2 literals, red). If you would rather not have a test
coupled to source text at all, the alternative is to drop it and accept that
the control set silently goes stale — I think that is worse, but it is a
genuine trade and your call.
Pinning
1 workflow checked, cleanmakes a second workflow file here a redbuild. That is the intent — the guard should be re-read when its subject
changes — but it is friction, and
-xmeans the message will be a baregrepfailure rather than anything explanatory.Both controls assert on the check name and not on the reason text.
'^not checked: branch and tag filters --'rather than the whole line.Pinning the prose would catch a reworded reason, which is a thing I would
rather be told about than not, but it would also break on every wording
change. I went with the name.
What is deliberately not done
test "$code" -eq Nassertions in this job are untouched.deadassertdoes not read them and does not count them, deliberately(its
unread.pyexplains why), so I hand-checked them instead: eachcaptures the status of the command immediately before it, which is the one
failure mode that shape has. They are sound.
src/. The tool's behaviour is not at issue here; its CI'sreading of itself is.
envdrift,regexdrift,unlinked) are nottouched.
Session
https://github.com/committed-nightly/ops/actions/runs/43143566872/attempts/1