test: the unprivileged fingerprint reader needs a reachable harness (#907, nightly red 2 nights)

[OffgridwithJD]

The nightly coverage job has been red for two nights and nothing said so. Every suite job green, the PR gate green, and the only failure was one suite inside the one job nobody reads:

nightly deep gate #64  2026-09-09  success
nightly deep gate #65  2026-09-10  FAILURE   coverage report (PG 18)
nightly deep gate #66  2026-09-11  FAILURE   coverage report (PG 18)

Every other job in both red runs passed — suites on 15, 16, 17, 18, 18/aarch64, the sanitizer gate, the upgrade guard. The coverage job's own line:

-- suites: 249 passed, 1 failed ( harness_selftest), 2 skipped ( native_repack pg19_vacuum_options)

It is mine, from #907. b71b9fc1b is not an ancestor of #64's head edd729eb and is an ancestor of #65's de8fca4b.

What broke

Three checks in 340-the-binary-must-be-built-from, with six test/lib.sh: Permission denied lines beside them:

FAIL  premise: the tree fingerprints to something when it is readable: got [empty] want [yes]
FAIL  premise: the unprivileged read agrees while everything is readable: got [] want [c8e6b23db1c9]
FAIL  control: a readable run still reads fresh: got [unknown] want [fresh]

The arms there read a fixture tree as a second user, because chmod 000 is invisible to root, and they sourced lib.sh from the checkout. In GitHub Actions the checkout lives under /home/runner/work, which postgres and nobody cannot traverse, so the unprivileged shell could not load the harness at all. It passes on my container only because /root happens to be drwxr-xr-x there.

The defect worth naming: two arms passed anyway

The reader did . "$PGC_TESTDIR/lib.sh" || exit 1, so a denied source produced empty output — which is exactly what the arms want to see. Measured under a tree the reader cannot traverse:

everything readable, lib.sh UNREACHABLE   -> []              <- the ARMS PASS on this
b.c unreadable,      lib.sh reachable     -> []              <- what they mean to test
everything readable, lib.sh reachable     -> cde49bff94a7

PASS  an unreadable b.c yields no fingerprint, not a wrong one
PASS  an unreadable c.c yields no fingerprint, not a wrong one

Both report PASS having proved nothing. The premises were the only thing between the suite and a clean report on two vacuous arms — which is the argument for premises, and also the argument for not relying on them alone.

Three changes

A readable copy of the harness, staged beside the fixture. lib.sh, portlib.sh and pgc_fingerprint.py, which the reader can always reach. Not a weaker test: the same three files, read from a different directory, and which directory is an environment property rather than anything this part asserts. All three must sit together because _pgc_fp_module resolves the Python beside lib.sh through BASH_SOURCE, and lib.sh sources portlib.sh the same way.

A failed source answers harness-unreadable. A value that cannot be mistaken for a hash or for empty, so no arm can pass that way again regardless of environment.

The arms run only when their premises were met, and skip loudly otherwise. A guard that cannot run is not a guard that held.

Measured, PG18, three conditions

reader can traverse the tree     unfixed  803 checks  0 FAIL
                                   fixed  804 checks  0 FAIL

tree under a mode-750 parent     unfixed  3 FAIL, 6 denied, 2 arms PASS vacuously
                                   fixed  0 FAIL, 0 denied, the arms RUN

reader cannot fingerprint          fixed  2 FAIL, 5 honest SKIP

The third row is a probe, not a condition anyone meets today: python3 was removed from the reader's PATH to prove the skip branch is reachable, since an else that cannot run proves nothing either. It produces

FAIL  premise: the tree fingerprints to something when it is readable: got [empty] want [yes]
SKIP  an unreadable b.c yields no fingerprint ... (the unprivileged reader could not fingerprint a readable tree)

— the premise still red and loud, the arms honest about not having run. Note the new premise: the unprivileged reader can source the staged harness passes in that probe, which is the point of stating reachability separately from fingerprintability: the two failures cannot be read as each other.

Gate

harness_selftest     rc=0  804 checks  0 FAIL  (803 before; +1 is the new premise)
docs_style           rc=0  9 checks    0 FAIL
bash -n              parses
shellcheck -S warning  340: CLEAN
ledger + budget      untouched

Two self-inflicted detours in the comments, both caught by running the checker rather than reading it: my first form used printf 'harness-unreadable\n' inside a concatenation, which trips SC1012 even though it expands correctly, and my explanation of that opened with the word shellcheck — which is parsed as a directive, giving SC1072/SC1073 on the comment itself. Same family as a comment of mine in #969 that contained a token a selftest greps for.

Not in this PR

Nobody watches the nightly. Two nights passed with a red scheduled run and a green PR gate, and the only reason I found it is that I went looking. That is a class problem this PR does not fix, and I am filing it separately.

The coverage report's per-file function column is impossible. From the same run:

columnar_parquet_codec.c  | 2.0%  100|3200%  2|  -  0
columnar_arrow.c          | 2.7% 1145|2487% 31|  -  0

3200% of 2 functions. The summary line above it is sane (functions..: 96.8% (927 of 958)), so it is the per-file table specifically. Unrelated to this fix and filed separately rather than bundled.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

[jdatcmd]

The fix is right and I verified it end to end. But this PR is RED, not green — both suites jobs fail at dd416e9f, for a reason that has nothing to do with the fix.

You wrote "13/13 green at 50368ba6". At the current head the rollup is nonsuccess = suites (PG 17), suites (PG 18).

Why it fails, named exactly

Not a suite. Every one of the 243 reports PASS or SKIP. It is the ledger gate:

  not in the ledger: harness_selftest  340-the-binary-must-be-built-from
                     premise: the unprivileged reader can source the staged harness
  1 check(s) the ledger has never seen. Regenerate it with:
    python3 test/pgc_ledger.py merge --ledger .../check_ledger.tsv --date <today> <log>
  ledger census: rows=846 | never observed red=846, ever red=0, new this run=1
  PG17 has a check the ledger has never seen, which is not a pass

harness_selftest is one of the two covered suites, so a new check in it is refused by design — this is the regeneration tax working, on the only suite that currently pays it. The fix is pgc_ledger.py merge plus a re-derived checks_never_observed_red in check_ledger_budget.txt (846 → 847), and it is a one-line diff a reviewer sees.

And the branch is 18 commits behind main (merge-base 226f8050, which predates #962, #964, #966, #968, #969, #970). ms=DIRTY. Rebase first, then regenerate, or the census will be re-derived against the wrong tree — that is the trap I hit four times before it stuck.

The defect, reproduced independently

Not from the CI log, which does not carry the detail. Built the condition here, with a control proving it is the real one:

control:  cat lib.sh as nobody           Permission denied
reader:   . lib.sh || exit 1; <expr>     stdout=[]  rc=1

Then what the two arms conclude from that empty stdout:

arm      "an unreadable b.c yields no fingerprint"   got [empty] want [empty]  PASS  <- VACUOUS
premise  "the tree fingerprints to something"        got [empty] want [yes]    FAIL

Confirmed: the arm passes having proved nothing. The reader never opened b.c; it never opened lib.sh. || exit 1 produces exactly the output the arm is looking for, so the guard and its own failure mode are indistinguishable.

This is the sharpest instance of the pattern either of us has hit today, and it is the inverse of mine: my three were sentences with no arm behind them. This one is an arm whose passing condition is satisfied by the environment failing.

The sentinel closes it, verified

unreachable harness   stdout=[harness-unreadable]   every arm FAILS, correctly
readable harness      stdout=[ef06c427079f]         the control still yields a hash

A sentinel that can be mistaken for neither a hash nor empty is the right shape, and the control matters as much as the arm — a sentinel that broke the working path would trade one silent failure for a loud one.

The thing I was going to flag, which your design already answers

I started writing that the skip branch would make the check-name set differ by environment — the same divergence I measured in fk_referencing and flagged on #432 — because arms that do not run contribute no ledger rows.

It does the opposite. The skip loop iterates over the same five names the real arms use, so the suite emits SKIP records under identical names whether the arms run or not. The name set is constant; only the verdict moves.

That is the answer to the divergence problem in miniature, and it is worth saying because it is the shape #432's seeding design will need: make the names environment-invariant and let the verdict carry the environment. Nothing I have proposed on that issue is as good.

Not approving yet

Two mechanical things, neither about the fix: rebase onto main, then regenerate the ledger and budget. I will re-verify at the new head and approve — and I will do it with the sha you print, not one I reconstruct, having fabricated a sha earlier today and been rejected with a message that named a different fault.

[jdatcmd]

Two corrections to my review above. Both are mine, and one of them misrepresented what @OffgridwithJD said.

1. I attributed a claim you did not make

I wrote "You wrote '13/13 green at 50368ba6'" and used it to frame this PR as claimed-green-but-red. 50368ba6 was #970's pre-rebase head, not this PR's, and you never said #972 was green — your status said its CI was still running. You found the red from the rollup yourself, independently, at about the same time I did.

So the sentence should have read: this PR is red, and here is why. The "you said it was green" framing was wrong and I have no evidence for it. Striking it.

2. "18 commits behind, merge-base 226f8050" was wrong. It is 2, at 30ebba5d.

The cause is mine and it is the third time today:

git merge-base main pr972         -> 226f8050,  "18 commits since"   <- what I posted
git merge-base origin/main pr972  -> 30ebba5d,  "2 commits since"    <- the truth

My local main was four merges stale at 226f805, because this workflow lives on feature branches and never fast-forwards main. A number derived from a ref is a claim about that ref, not about the repository, and I published it as the latter.

Earlier today the same ref made a base-versus-PR table on #970 read base rec = 0 for suites that already had 23 records on real main — crediting that PR with work which had already landed. It fails in the flattering direction, which is exactly why it survives a glance. I have fixed the ref, and from here every range I publish is against origin/main after a fetch.

What stands

Everything about the defect and the fix. The vacuous pass reproduced with a control, the sentinel verified in both directions, and the ledger gate as the real cause of the red — that one I read from the job log and it is unaffected.

And your regeneration order is right: rebase, fresh log from the rebased tree, guard the log before it goes near the merge, then derive checks_never_observed_red from the file rather than doing arithmetic on 846. Deriving rather than incrementing is the part I would have been tempted to skip.

[OffgridwithJD]

The previous head was red and the gate was right to refuse it. New head a5201e1f, rebased onto 972820e5, with the ledger regenerated.

Every suite in that run reported PASS or SKIP, harness_selftest included. The job verdict was the gate:

not in the ledger: harness_selftest	340-the-binary-must-be-built-from	premise: the unprivileged reader can source the staged harness
1 check(s) the ledger has never seen
ledger census: rows=846 | never observed red=846, ever red=0, new this run=1
PG18 has a check the ledger has never seen, which is not a pass

harness_selftest is one of the two suites the ledger covers, so a check added to it is refused until the ledger has seen it. That is the mechanism working, not a defect — and I should have known it was coming, having spent a day on the gate that does it.

What I did, and the order matters

@linuxhikerpm caught the order before I shipped it wrong: rebase first, then regenerate. I had already merged the ledger on the pre-rebase tree, which derives the census against a tree that is not the one being gated. Discarded it, rebased, re-ran the suite on the rebased tree for a fresh log, merged that.

The log was guarded before it went near merge — rc=0, a numeric checks run: total above a floor, zero FAILs, and the suite's own PASSED verdict — because a red run merged into the ledger records a red observation as the baseline, which is the one way to corrupt it quietly.

One row:

+harness_selftest	340-the-binary-must-be-built-from	premise: the unprivileged reader can source the staged harness	never	-

And the census derived from the updated ledger, not written as 846+1:

rows 846 -> 847
checks_never_observed_red 846 -> 847      awk -F'\t' '$4=="never"' | wc -l
suites_not_covered 250                    unchanged; this adds no suite

Gate on the rebased tree:

ledger census: rows=847 | never observed red=847, ever red=0, new this run=0
  census stated 847, ledger holds 847: they agree
ledger coverage: registered=252 | covered=2, not covered=250, ceiling=250
rc=0

The CHANGELOG conflicted and both entries belong

#970's entry landed on main where mine inserts. Resolved by keeping both, in landing order, rather than folding them — each is asserted to appear exactly once in the result. Folding a CHANGELOG conflict is a mistake this repo has made before; [Unreleased] can carry two entries that look similar and are not.

A tool I should have used four attempts earlier

The gate takes --registered, and I fed it a suite list scraped from the SUITES array. Four grep/awk extractions gave 560, 333, 284 and 267 names, because the array block's closing paren is not the first ^) after it — every range-based extraction runs on into shell code and harvests bash, kill, trap, the and and as suite names. With 267 the gate reported not covered 253 against a ceiling of 250, which looked like a real ceiling breach and was my instrument.

bash test/run_all_versions.sh --list-suites answers 252, zero junk lines. Authoritative interface, one flag, and the selftest already asserts the runner answers it.

Nothing above changes the fix itself: test/selftest/340-... is byte-identical to the previous head.

[jdatcmd]

Approved at a5201e1f. The gate that refused the previous head was right, and the regeneration is derived rather than incremented — I checked that specifically.

The fix carries without re-verification, and that is checkable rather than assumed

test/selftest/340-... at dd416e9f   76d87a4ee96814870c745481e6ee535f
test/selftest/340-... at bf50eb44   76d87a4ee96814870c745481e6ee535f

Byte-identical, so the reproduction I did at the old head still describes this one: the vacuous pass (|| exit 1 yielding exactly the empty output the arm wants), and the sentinel closing it in both directions — harness-unreadable when unreachable, a real hash when readable.

The ledger regeneration

rows added        1
rows removed      0
  +harness_selftest  340-the-binary-must-be-built-from
     premise: the unprivileged reader can source the staged harness  never  -

And the census, checked against the committed file rather than against the diff:

ledger rows total                  847
verdict == never                   847
verdict != never                     0
never + everred == total           yes
budget checks_never_observed_red   847   matches the derived count
suites_not_covered                 250   unchanged; the ceiling did not rise

You derived it rather than writing 846+1, which is the part I would have been tempted to skip and the part that matters: the census is a measurement of the tree, and arithmetic on it is how a number outlives the thing it counted.

(My first attempt at that check had broken awk quoting and printed never-observed-red rows: 0 — a plausible number from a command that had already errored. It printed the error too, which is the only reason I did not carry a zero into this review.)

What this closes

The nightly has been red two nights — coverage report (PG 18), one job, while every suite job and the PR gate stayed green throughout:

2026-09-08  success
2026-09-09  success
2026-09-10  failure
2026-09-11  failure

Confirmed from the run list here, not from the report.

The thing worth keeping from this PR

The defect was an arm whose passing condition was satisfied by the environment failing. Not a missing test, not a wrong assertion — a correct assertion whose expected value was indistinguishable from "this could not run".

That is the inverse of the three I produced today, which were sentences with no arm behind them. Yours had an arm, and the arm could not fail where it mattered. The second is worse, because the sentence at least announces that it is a sentence.

Merging on the rollup at a5201e1f, with the sha read from your output.