Skip to content

test: a transfer artifact must not be tracked either — nine git bundles out of the tree - #1001

Merged
jdatcmd merged 2 commits into
commandprompt:mainfrom
OffgridwithJD:fix/a-transfer-artifact-must-not-be-tracked
Sep 12, 2026
Merged

test: a transfer artifact must not be tracked either — nine git bundles out of the tree#1001
jdatcmd merged 2 commits into
commandprompt:mainfrom
OffgridwithJD:fix/a-transfer-artifact-must-not-be-tracked

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

Nine git bundles, 76,194 bytes, are tracked in main. I put them there. They went in with ee2f8f18 — a merged commit whose subject is a check name in sorted_pathkeys.sh.

b983.bundle      12239        b993r2.bundle    12344        b998.bundle       6766
b983r.bundle     12342        b995.bundle       5522        b998b.bundle     15543
b990.bundle       2646        b997r.bundle      5563
b991.bundle       3229                                      total           76194

I create bundles in my clone to move a branch into the audit container, and I staged with git add -A. That is precisely the mechanism part 310's own comment already describes for a stray .pyc:

a tracked build artifact joins whichever commit is next.

Nothing caught it. Five suites and the whole selftest ran green either side of that commit, because no suite has an opinion about files it does not read. What found it was a later rebase printing the filenames in a list I happened to read — which is not a mechanism, and is why this PR is not only a deletion.

What this does

removes the nine files from the tree they are reachable in history either way; see the limits below
*.bundle in .gitignore ignoring is what stops the deletion being undone by the next git add -A
four arms in 310-a-compiled-artifact-must-not-be.sh so the next one fails a check rather than riding into a commit

Part 310 deferred this question on purpose, and this answers it for one class only

SCOPE. Python only. […] Whether every derived file in the tree deserves one rule is a larger judgement and is deliberately not decided here.

A bundle is a transfer artifact, which is why it belongs beside the .pyc and not beside the Iceberg and Parquet fixtures the same note protects. It is derived from commits already in the history; it is named after whichever branch was in flight; nothing in the tree opens one; and the next person to make one will choose a different name — so it can never become a fixture anything depends on. The scope note still declines the general question, and so do I.

Same two-part rule as the Python half, the same no-repo discipline (every arm answers no-repo where there is no repository, rather than the silence ls-files gives by default), and a control.

Red to green, measured on both trees

RED   main's tree, 9 bundles tracked, with this part and .gitignore dropped in
        FAIL  no git bundle is tracked: got [9 tracked] want [0 tracked]
        FAIL  and the tracked BUNDLE list names none of them: got [[b983.bundle b983r.bundle ...]]

GREEN this branch
        PASS  no git bundle is tracked
        PASS  and the tracked BUNDLE list names none of them
        PASS  and the tree ignores a git bundle, so the next add cannot re-add it
        PASS  control: a source file named like a bundle is not ignored
        rc=0  checks run 867  records 867  distinct keys 867  colliding 0  FAIL 0

The control is load-bearing: the rule keys on the suffix, and without it a check-ignore that answered "ignored" to everything would make the ignore arm vacuous. test/bundle_notes.sh is not ignored; some-branch.bundle is.

Three mistakes of mine inside this fix, all caught by the tree rather than by me

1. I appended the arms after the part's unset. The helpers were gone, so every arm returned empty.

2. I compared against "". check_text refused all three on the ground that "nothing was compared" and "the two agreed" are the same observation:

FAIL  no git bundle is tracked: a side is empty, so nothing was compared: got [] want []

Two independent reasons, three refused arms, and the harness named both. The arms now use the counted idiom the Python half uses (0 tracked) rather than an empty string.

3. I created a new #982 collision. My second arm was named and the tracked list names none of them — which the Python half four arms above already owns. One ledger row would have covered two checks, and the ledger's duplicate-name detector caught it on the first merge:

duplicate check name in one run, so one ledger row covers 2:
  harness_selftest  310-a-compiled-artifact-must-not-be  and the tracked list names none of them
distinct checks this merge=866 against checks run=867

In the session that finished removing the last of #982's 24 instances. Renamed to and the tracked BUNDLE list names none of them, and the colliding-key count is now asserted in the guard rather than only the record count: 867 records, 867 distinct keys.

Ledger

merge   rows 909 -> 913
census  DERIVED from the file: rows=913 never=913, partition closes
gate    census 913/913 agree | ceiling 250 -> 250, which does not rise | rc=0

Limits of what this fixes

  • The blobs stay reachable in the repository. Removing a file from the tree does not unwrite it, and rewriting main is not something a stray artifact justifies — that is a judgement for whoever owns the history, not a consequence of this PR. What this stops is the tree carrying them, and git add -A re-adding them.
  • One class, not every derived file. Part 310's general question is still open and still deliberately undecided.
  • .gitignore does not stop git add -f or an explicit path. The arms do, on the next run; nothing stops a deliberate -f and nothing should.
  • pg18a only, and these arms touch no cluster — they ask git three questions.

docs_style 9/9; shellcheck -S error -s bash over the harness: 0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

OffgridwithJD and others added 2 commits September 11, 2026 21:37
…t of the tree)

I put them there. 76,194 bytes across nine git bundles went into c697c8c -- a
merged commit whose subject is a check name in sorted_pathkeys.sh. I make bundles in
my clone to move a branch into the audit container, and I staged with `git add -A`.

That is exactly the mechanism part 310's own comment calls out for a stray .pyc: a
tracked build artifact joins whichever commit is next.

NOTHING CAUGHT IT. Five suites and the whole selftest ran green either side of that
commit, because no suite has an opinion about files it does not read. What found it
was a later rebase printing the filenames in a list I happened to read, which is not
a mechanism.

Part 310 deferred this on purpose:

    Whether every derived file in the tree deserves one rule is a larger judgement
    and is deliberately not decided here.

This decides it for one more class and only that class. A BUNDLE IS A TRANSFER
ARTIFACT: derived from commits already in the history, named after whatever branch
was in flight, never opened by anything in the tree, and the next one will have a
different name -- so it can never be a fixture anything depends on. That is why it
belongs beside the .pyc rather than beside the Parquet fixtures, which are inputs.

Three arms, the same two-part rule and the same `no-repo` discipline as the Python
half:

    no git bundle is tracked
    and a git bundle is ignored, so the next add cannot re-add it
    control: a source file named like a bundle is NOT ignored

The control matters because the rule keys on the suffix: without it, a check-ignore
that answered "ignored" to everything would make the second arm vacuous.

The blobs stay reachable in history. Removing a file from the tree does not unwrite
it, and rewriting main is not something a stray artifact justifies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
From a guarded harness_selftest run on this tree: rc=0, checks run 867, RESULT
records 867, 867 distinct (suite, part, name) keys, zero colliding, zero FAILs, the
suite's own PASSED verdict.

    merge   rows 909 -> 913
    census  DERIVED from the file: rows=913 never=913, partition closes
    gate    census 913/913 agree | ceiling 250 -> 250, which does not rise | rc=0

THE COLLIDING-KEY COUNT IS IN THAT LIST FOR A REASON. The first run of these arms
reported `distinct checks this merge=866` against `checks run=867`, because I had
named one of them `and the tracked list names none of them` -- which the Python half
four arms above already owns. One ledger row would have covered two checks: commandprompt#982's
defect, created in the session that finished removing the last of its 24 instances,
and caught by the ledger's own duplicate-name detector on the first merge rather than
by me.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
OffgridwithJD pushed a commit to OffgridwithJD/pgcolumnar that referenced this pull request Sep 12, 2026
From a guarded harness_selftest run on this tree: rc=0, checks run 903, RESULT
records 903, 903 distinct (suite, part, name) keys, ZERO COLLIDING, zero FAILs, the
suite's own PASSED verdict.

    merge   rows 909 -> 949
    prune   2 orphans removed -- the two part-330 rows this PR exists to reach
    census  DERIVED from the file: rows=947 never=947, partition closes
    gate    census 947/947 agree | ceiling 250 -> 250, which does not rise | rc=0

The colliding-key count is in the guard now, not just the record count. It earned its
place on the sibling PR commandprompt#1001, where a new arm of mine shared a name with one four
arms above it and the merge reported `distinct checks this merge=866` against
`checks run=867` -- commandprompt#982's defect, created in the session that finished removing the
last of its 24 instances.

TWO DERIVES WHERE A REBASE MOVED THE LEDGER. Rebasing across commandprompt#998 left the committed
pair disagreeing: the ledger took both sides' rows while the budget kept one side's
number. Four arms failed and all four traced to that one cause -- one asserts the pair
agrees, three run the real gate, which correctly refuses a contradiction. The order
for a ledger PR across a rebase is derive-to-reconcile, run, merge, prune, derive.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I reviewed this from the file list first, which is the thing I failed to do on #992 and the reason those bundles are in main at all. Half of that failure is mine: I verified your one-line fix behaviourally — composed the tree, ran the sweep, clean with it and MISMATCH sorted_pathkeys.sh:283 without — and a behavioural check stays true whatever else rides along. I had run gh pr view --json files on #993 and #997 an hour earlier and skipped it here because I already knew which file the fix was in.

Driven, not read

On a worktree at your head:

tracked bundles after this PR []
some-branch.bundle ignored
test/bundle_notes.sh (the control) not-ignored
a nested bundle, test/sub/deep/x.bundle ignored
does ls-files -- '*.bundle' reach nested? yes — planted test/sub/nested.bundle, git add -f, and the guard listed it

That fifth row is the one I most wanted to check and it is the one your arms depend on. A pathspec that only matched the root would have left the guard passing on a bundle one directory down, which is exactly where the next one will land if somebody works inside test/.

Ledger: 913 rows, census 913, never 913, 0 duplicate (suite, part, name) keys, ceiling 250 unchanged. The four new rows are the four new arms.

The classification is right and the argument is the load-bearing part

A bundle beside the .pyc rather than beside the Parquet fixtures, because it is derived from commits already in the history, named after whatever branch was in flight, never opened by anything in the tree, and the next one will be named differently — so it can never become a fixture something depends on. That last clause is what actually separates it from the .avro/.puffin inputs, and it is the sentence I would keep if the comment ever needs shortening.

The two-part rule is right too. Deleting without ignoring lasts until the next git add -A in a clone where somebody moved a branch — which is precisely how these arrived.

The frame this does not cover, stated because every fix has one

The rule keys on the suffix. A bundle written without one slips both halves:

git check-ignore b999   ->  NOT ignored
git ls-files '*.bundle' ->  does not list it

git bundle create takes whatever filename it is given, and yours happened to be bNNN.bundle. I am not asking for this in the PR — a content sniff (# v2 git bundle in the first bytes) would catch any name, but it is a different shape of guard and the suffix rule is proportionate to the failure that actually happened. It belongs in the scope note's list of things deliberately not decided, if anywhere.

On your three self-caught faults

The third is the one worth putting in the body rather than only in a comment. A new arm named and the tracked list names none of them, colliding with a name the Python half four arms above already owns — a fresh #982 collision in the session that finished removing the last of its 24, caught by the ledger's own duplicate-name detector at distinct checks this merge=866 against checks run=867.

That is the strongest argument the detector will ever get: it caught its own author, the same night, on the same subject. Worth saying out loud, because the next person deciding whether that guard earns its keep will read this PR.

And check_text refusing your three arms for comparing against "" is the harness making the same call twice in one night — it refused mine on #1000 for the same reason. "Nothing was compared" and "the two agreed" being one observation is a rule that keeps paying.

Verdict

Approved. The blobs stay reachable in history and that is correct — removing a file from the tree does not unwrite it, and a stray artifact does not justify rewriting main. What this stops is the tree carrying them and the next git add -A putting them back.

@jdatcmd
jdatcmd merged commit 32658e3 into commandprompt:main Sep 12, 2026
13 checks passed
OffgridwithJD pushed a commit to OffgridwithJD/pgcolumnar that referenced this pull request Sep 12, 2026
From a guarded harness_selftest run on this tree: rc=0, checks run 907, RESULT
records 907, 907 distinct (suite, part, name) keys, ZERO COLLIDING, zero FAILs, the
suite's own PASSED verdict.

    reconcile  rows 951 against a 947 budget -> 951, BEFORE running
    merge      no new rows: the run's checks were already all ledgered
    prune      0 orphans, 0 unprunable
    census     DERIVED: rows=951 never=951, partition closes
    gate       census 951/951 agree | ceiling 250 -> 250, which does not rise | rc=0

THE RECONCILE STEP IS WHY THIS COMMIT IS SELF-CONSISTENT. Rebasing across commandprompt#1001 left
rows=951 against a budget still stating 947: git merges both sides' rows into the tsv
and keeps one side's number in the budget. An earlier version of this commit carried
that contradiction, which would have left a commit in the history whose own selftest
fails -- four arms, all tracing to that one cause. The order is now written into
pgc_ledger.py's docstring so the next person does not rediscover it.

The colliding-key count is asserted beside the record count because it earned its
place on commandprompt#1001: a new arm of mine shared a name with one four arms above it, and the
merge reported `distinct checks this merge=866` against `checks run=867` -- commandprompt#982's
defect, in the session that removed the last of its 24.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
OffgridwithJD pushed a commit to OffgridwithJD/pgcolumnar that referenced this pull request Sep 12, 2026
…mandprompt#983)

Suggested by @jdatcmd after watching this PR pay the tax four times, the fourth
because commandprompt#1001 merged while it was in flight.

A rebase moves the LEDGER without moving the BUDGET -- git merges both sides' rows
into the tsv and keeps one side's number in the budget -- so the committed pair
contradicts itself before anything runs. Four arms then fail and all four trace to
that one cause: one asserts the pair agrees, three run the real gate, which correctly
refuses a contradiction. An hour to diagnose from those four failures; one paragraph
to avoid.

    1. rebase onto the new base FIRST
    2. DERIVE to reconcile, before running
    3. run on the REBASED tree, and guard the log
    4. merge the guarded run
    5. prune orphans, if any
    6. DERIVE again

Two derives: step 2 makes the tree self-consistent enough for the suite to pass at
all, step 6 records the result of 4 and 5. Both read back from the file, because
`old + n` is right once and wrong every time after.

NO CHECK CHANGES, and that is verified rather than assumed. Nothing in either harness
reads this file's TEXT: the five greps in 410 and the one in test_mutation_ledger.py
all read run_all_versions.sh for `pgc_ledger.py" gate`. So the check set cannot move,
and the previous commit's run stands for this tree. "A docstring cannot matter" is
exactly how a guard that reads a docstring gets broken, so it was checked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants