Skip to content

Wire gitleaks / shellcheck / script self-test CI gates — PR #177 Major - #182

Closed
jgruberf5 wants to merge 17 commits into
stagingfrom
fix/ci-hygiene-gitleaks-shellcheck
Closed

Wire gitleaks / shellcheck / script self-test CI gates — PR #177 Major#182
jgruberf5 wants to merge 17 commits into
stagingfrom
fix/ci-hygiene-gitleaks-shellcheck

Conversation

@jgruberf5

Copy link
Copy Markdown
Collaborator

Addresses the CI-hygiene Major from @bonnyr-f5's #177 review: .gitleaks.toml was config nothing ran, make shellcheck had no caller, and compute_version_bump.sh's SELF_TEST was never invoked — so the release-critical scripts were statically unchecked and the secret-scanning gate was unwired for a public repo.

Three new P1 jobs, all wired into the CI gate:

  • Secret Scan (gitleaks)--no-git over the tracked source. Extended .gitleaks.toml's allowlist to cover generated build artifacts (.pyc, frontend-v2/dist/) alongside the existing synthetic-key entries. Verified clean locally even with artifacts present (the 7 raw hits are all build artifacts; git-tracked source is clean).
  • ShellCheckmake shellcheck over the whole corpus. Fixed the only two pre-existing findings that would have blocked the gate: a missing shebang in get_dpu_pwd.sh (SC2148) and unused for i counters in ibm_cloud_bnk_forge.sh (SC2034 → for _). Corpus clean at --severity=warning.
  • Script Self-TestsSELF_TEST=1 compute_version_bump.sh, which exits non-zero on failure (that behaviour ships with Fix non-deterministic version derivation (SIGPIPE race) — PR #177 Blocker 1 #179), so a regression in the version-deciding logic fails CI.

The release-critical scripts (compute_version_bump, extract-breaking-changes, publish-signed-images) already pass shellcheck cleanly. YAML validates.

Depends on #179 for the SELF_TEST non-zero exit to actually gate.

https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

…th VERSION

PR #177 review (bonnyr-f5) — BLOCKER 2. helm/bnk-forge/values.yaml pinned
image.tag: "3.0.1" and Chart.yaml appVersion: "3.0.1", and every per-service tag
is "" (falls back to the global 3.0.1). The release publishes only :${VERSION}
and :latest, so :3.0.1 -- which was never published on this registry -- means
ImagePullBackOff across all seven services. frontend-v2/package.json had
likewise drifted to 2.12.0. The release job bumped only VERSION/dist/VERSION/
CHANGELOG, so every other version-bearing artifact drifted silently.

- New scripts/sync-version-artifacts.sh with --write <v> (sets the global Helm
  image tag, Chart appVersion, and frontend package.json) and --check (asserts
  all three equal VERSION, exits 1 otherwise). Anchored seds hit only the global
  2-space image tag -- postgres/redis and the "" per-service tags are untouched.
- The release job (both the automated and manual paths) now runs --write after
  bumping VERSION and stages the three files, so a 4.0.0 release updates the
  chart to 4.0.0 instead of leaving it on 3.0.1.
- New CI job "P1 · Version Consistency" runs --check and is wired into the CI
  gate, so this drift can't reappear silently -- mirroring the existing
  image-level VERSION assertion, but at source level on every PR.
- Fixed the current drift: all three now read 3.1.6 (= VERSION), and 3.1.6
  images do exist.

Note: this makes frontend-v2/package.json track the product VERSION, as the
review requested. If the frontend is meant to version independently, that's the
one line to drop from the assertion.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

@mwiget mwiget left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three jobs are real and I confirmed they do work rather than pass vacuously. One change to the allowlist needs to come out before this lands.

Verified working

  • gitleaks genuinely scans: scanned ~28577099 bytes (28.58 MB) in 8.67s / no leaks found. Not an empty-scan pass.
  • ShellCheck job is green and the corpus really is covered -- make shellcheck globs upgrade.sh scripts/*.sh vm-bnk-forge/*.sh vm-bnk-forge/lib/*.sh, and all four expand here (17 tracked scripts/*.sh, plus vm-bnk-forge/{make-vm,destroy-vm,render-cloud-init}.sh and lib/render.sh). I checked because an unexpanded glob would have made the new gate fail on arrival.
  • Gate wiring matches the existing convention, actions/checkout@v6 is consistent, and the for i -> for _ / missing-shebang fixes are correct.

Blocking

frontend-v2/dist/.* should not go in the global allowlist. Three reasons, in order of weight:

  1. .gitleaks.toml's own header is explicit: "Do NOT add blanket directory allowlists — that is how a real credential hides. Scope each entry to a rule + a specific path." These three entries are exactly blanket directory allowlists, and they went into the top-level [allowlist], which suppresses every rule -- the default private-key and cloud-credential detectors included -- not just generic-api-key.
  2. A built frontend bundle is the single highest-value place to scan, not to skip: VITE_*-style build-time injection is the way a secret ends up in plaintext in a JS bundle. Allowlisting the directory permanently blinds the scanner to that whole class.
  3. It buys nothing in CI. The job runs docker run ... immediately after actions/checkout with no build step, and frontend-v2/dist/, __pycache__/ and *.pyc are all gitignored with zero tracked files -- so none of those paths exist in the scanned tree. I confirmed against the branch. The entries only affect a developer scanning a dirty working tree.

If the goal is a clean local run (which is a fair goal), that belongs in the local invocation rather than in the committed config that also governs the CI gate. .pyc/__pycache__ I'd argue are defensible on their own terms -- compiled bytecode of sources that are scanned anyway -- but frontend-v2/dist/ I'd want gone, and whatever stays should carry the same "reviewed, here's why" framing the existing entries have.

Non-blocking

  • ghcr.io/gitleaks/gitleaks:latest is unpinned for a security gate. Pin a tag.
  • script-selftests currently cannot fail (staging's SELF_TEST exits 0 regardless -- you flagged the #179 dependency). Worth knowing it stays partly toothless even after #179: see my review there, the SIGPIPE case in that suite passes with the fix fully reverted, so this job would go green on the regression it exists to catch.
  • This conflicts with #180 -- both insert into the same needs: list and gate loop in ci.yml. Whichever lands second needs a rebase.

Everything else here I'd approve as-is.

Comment thread .gitleaks.toml Outdated
# their compiled .pyc and the built frontend bundle are not credentials.
'''.*__pycache__/.*''',
'''.*\.pyc$''',
'''frontend-v2/dist/.*''',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one thing I'd like removed before merge.

The header of this file says, twelve lines up:

Do NOT add blanket directory allowlists — that is how a real credential hides. Scope each entry to a rule + a specific path.

These three are blanket directory allowlists, and they are in the top-level [allowlist] rather than [rules.allowlist], so they suppress every rule -- the default private-key, AWS, GCP and token detectors included -- not just generic-api-key. Every existing entry in this block is a single named file with a written-out reason; these are the first that aren't.

frontend-v2/dist/ is the one that actually matters. A built JS bundle is where a build-time-injected secret physically lands in plaintext -- a VITE_* variable baked in at build is the classic case -- so it is the last directory you'd want permanently exempt.

And in CI they do nothing at all. The job runs docker run straight after actions/checkout, with no build step; frontend-v2/dist/, __pycache__/ and *.pyc are gitignored (.gitignore:128 for dist) with zero tracked files between them, so none of those paths exist in the tree being scanned. I checked against this branch. They only change the result of a local scan over a dirty working tree.

If the intent is "make gitleaks is clean on my laptop after a build" -- fair -- that belongs in the local invocation, not in the committed config the CI gate reads from. .pyc/__pycache__ I could live with on their own merits (bytecode of sources that are scanned anyway), and if they stay they should carry the same reviewed-and-why framing as the entries above them. frontend-v2/dist/ I'd drop outright.

Comment thread .github/workflows/ci.yml Outdated
- uses: actions/checkout@v6
- name: gitleaks
run: |
docker run --rm -v "$PWD:/repo" -w /repo ghcr.io/gitleaks/gitleaks:latest detect \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: :latest on the gate that decides whether a secret reaches a public repo. A gitleaks release can add, change or drop rules, so the same commit can scan differently tomorrow -- either breaking CI out of nowhere or quietly narrowing coverage, which is the worse direction and the one nobody notices. Pin a tag (ghcr.io/gitleaks/gitleaks:v8.x.y) and bump it deliberately.

Two small ones while you're here: the mount can be read-only (-v "$PWD:/repo:ro") since detect only reads, and with --redact set, a finding prints a redacted value plus the file and line -- which is what you want, so no change needed there, just confirming it's deliberate.

For the record, this job is doing real work: scanned ~28577099 bytes (28.58 MB) in 8.67s / no leaks found.

Comment thread .github/workflows/ci.yml Outdated
steps:
- uses: actions/checkout@v6
- name: compute_version_bump SELF_TEST
run: SELF_TEST=1 bash scripts/compute_version_bump.sh

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth stating plainly since this job is the reason #179 is a dependency: it currently cannot fail. On staging's script a FAIL: still exits 0, so this is a green light regardless of outcome until #179 lands. The run on this PR shows 6 PASS and exit 0 -- fine, but it would have shown exit 0 with six FAILs too.

The part that isn't fixed by #179 landing: I reverted the SIGPIPE fix entirely at #179's head and ran that suite -- 7/7 PASS, exit 0. The one case meant to pin the version-derivation bug uses a single-line body, and grep can't match a line before reading all of it, so no SIGPIPE ever occurs and the case passes on the bug. Until that test changes shape, this job gates the presence of a self-test rather than the behaviour it's named for. Details and a verified replacement are on #179.

No change needed here -- just don't count this as covering the release-critical path yet.

@jgruberf5 jgruberf5 mentioned this pull request Aug 20, 2026
jgruberf5 pushed a commit that referenced this pull request Aug 20, 2026
mwiget's re-review of #182.

Removed the `frontend-v2/dist/.*` allowlist entry. A built JS bundle is exactly
where VITE_*-style build-time secret injection lands, so it's the highest-value
thing to scan, not to skip; a blanket directory entry in the top-level
[allowlist] also suppresses every rule (private-key, cloud creds), not just
generic-api-key -- which the config's own header warns against. And it bought
nothing in CI: dist is gitignored, so a fresh checkout has no tracked files
there anyway. The .pyc / __pycache__ entries stay (compiled bytecode of sources
that are themselves scanned) with the "reviewed, here's why" framing the other
entries carry.

Pinned the gitleaks image to v8.21.2 (was :latest) -- a security gate shouldn't
float its scanner version.

The script-selftests job becomes meaningful once #179 lands: its rebuilt Test 7
catches the SIGPIPE regression on CI's GNU grep (the old single-line Test 7
passed even with the fix reverted).

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

Fixed — frontend-v2/dist/.* is out of the allowlist. You're right on all three counts: a built bundle is where VITE_* injection lands so it's the last thing to skip, a top-level [allowlist] path suppresses every rule, and it bought nothing in CI (dist is gitignored, absent from a fresh checkout). The .pyc/__pycache__ entries stay with the "reviewed, here's why" framing. Also pinned the gitleaks image to v8.21.2.

On the toothless note: once #179 lands, its rebuilt Test 7 catches the SIGPIPE regression on CI's GNU grep, so this job stops being a no-op. And yes — this and #180 both edit the gate needs:/loop, so whichever merges second rebases.

@mwiget mwiget left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The allowlist change is exactly right and I'm glad the reasoning went into the comment rather than the PR description. The pin, though, has broken the gate -- and for an interesting reason worth knowing before you pick a replacement.

Fixed

frontend-v2/dist/.* is gone, and the surviving .pyc/__pycache__ entries now carry the same reviewed-and-why framing as the entries above them, including the point that they're gitignored so a fresh checkout has none. That's the right shape.

Blocking: the pinned version is the problem, not the config

P1 · Secret Scan (gitleaks) is red: 13 findings, all RuleID: private-key. I reproduced it locally against d48648dc and then isolated the cause:

with your .gitleaks.toml with no config at all
v8.21.2 (your pin) 13 findings 13 findings
v8.30.1 (:latest today) 0 findings 0 findings

The config is irrelevant -- it's the private-key rule itself, which was tightened somewhere in the nine minors between those versions. All 13 are PEM headers wrapped around a literal ..., e.g.

placeholder={'-----BEGIN OPENSSH PRIVATE KEY-----\n...\n-----END OPENSSH PRIVATE KEY-----'}
svc.validate_private_key("-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----")

-- UI placeholder text and test literals, no key material anywhere. The newer rule is simply better at that, so you've pinned to a version whose accuracy regressed.

Pin v8.30.1. I confirmed it's clean on this tree, and -- since "newer rule finds fewer things" deserves a second look -- I checked it hasn't just gone blind: a throwaway ssh-keygen -t ed25519 key dropped into a scratch directory is caught by both versions (1 finding each). So v8.30.1 is more precise, not weaker. (The canary was shredded immediately; nothing of it exists.)

Pinning was still the right call, and this is a good demonstration of why: the same tree scanned two ways gave 0 and 13. It just needs to name the version you actually validated against.

Still standing from the last review

  • script-selftests gates the presence of a self-test rather than its behaviour -- though #179 at 0f8fe8fd fixes that side: I verified its Test 7 now fails when the SIGPIPE fix is reverted, so once #179 lands this job earns its keep.
  • Conflicts with #180 (now approved) -- same needs: list and gate loop. Whichever lands second needs a rebase.

Comment thread .github/workflows/ci.yml Outdated
- uses: actions/checkout@v6
- name: gitleaks
run: |
docker run --rm -v "$PWD:/repo" -w /repo ghcr.io/gitleaks/gitleaks:v8.21.2 detect \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what turned the gate red -- 13 findings, all RuleID: private-key.

I reproduced locally against d48648dc and isolated it. The config isn't involved:

with .gitleaks.toml with no config
v8.21.2 13 13
v8.30.1 (:latest today) 0 0

It's the private-key rule, tightened somewhere across those nine minors. Every one of the 13 is a PEM header around a literal ...:

frontend-v2/src/components/settings/SSHCredentials.tsx:430
  placeholder={'-----BEGIN OPENSSH PRIVATE KEY-----\n...\n-----END OPENSSH PRIVATE KEY-----'}

backend/tests/component/test_ssh_service.py:26
  svc.validate_private_key("-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----")

Six are production .tsx (textarea placeholders), seven are test literals. No key material in any of them.

docker run --rm -v "$PWD:/repo:ro" -w /repo ghcr.io/gitleaks/gitleaks:v8.30.1 detect \
  --source=/repo --config=/repo/.gitleaks.toml --no-git --redact --verbose

Verified clean on this tree. And because "newer version finds fewer leaks" is the kind of thing that ought to make you suspicious, I checked it hasn't gone blind: a throwaway ssh-keygen -t ed25519 key in a scratch directory is flagged by both v8.21.2 and v8.30.1, one finding each. v8.30.1 is more precise, not weaker. (Canary shredded straight after.)

The :ro above is the other small thing from last time -- detect only reads.

Comment thread .gitleaks.toml Outdated
# Compiled bytecode of the sources above, which ARE scanned. Not a blanket
# directory skip of hand-written code, and it buys a clean local run without
# blinding CI (these are gitignored -- zero tracked files -- so a fresh
# checkout has none). frontend-v2/dist is deliberately NOT allowlisted: a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the right outcome, and the comment does the important part: it says what was considered and rejected, so nobody re-adds frontend-v2/dist/ in six months thinking it was an oversight. Noting the gitignored/zero-tracked-files reasoning inline is what makes the two survivors defensible rather than just tolerated.

@mwiget mwiget left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the gate green, and every one of the 13 really is a false positive -- I checked them all last round. But the remedy has a cost I don't think is worth paying, and it's a cost the alternative doesn't have at all.

What the new entries actually do

They're in the top-level [allowlist] paths, so they suppress every rule in those files, not just private-key. Six of the twelve are production source, and they are the SSH-key-handling UI. I measured it: identical real ssh-keygen -t ed25519 key, two files, one allowlisted and one not.

$ gitleaks detect --config=.gitleaks.toml --no-git --redact
v8.21.2  -> leaks found: 1     # only the non-allowlisted copy
v8.30.1  -> leaks found: 1     # only the non-allowlisted copy

A live private key committed to frontend-v2/src/components/settings/SSHCredentials.tsx is invisible to this gate. The identical key one directory over is caught. Same for an AWS-shaped key -- the exemption isn't rule-scoped, so it's all of them. (Canaries were shredded immediately; nothing persisted anywhere.)

That's the file whose entire job is handling private keys. It's the last place I'd want permanently exempt, for the same reason frontend-v2/dist/ was.

And none of it is needed

The findings only exist because the pin is nine minors behind. On d48648dc -- your tree before these twelve entries -- I ran both:

findings
v8.21.2 13
v8.30.1 0

Zero, with no allowlist entries at all. The newer private-key rule already distinguishes a PEM header wrapped around ... from real key material, which is exactly the judgement these twelve entries are hand-encoding. And it hasn't gone blind doing it: a throwaway ed25519 key is caught by both versions, as above.

Pin v8.30.1 and drop the twelve entries. One line instead of twelve, no exemptions, and the gate keeps watching the files that matter most.

If there's a reason to stay on v8.21.2 that I'm not seeing, then at minimum these need to move out of the global [allowlist] into a private-key-scoped [rules.allowlist], so a hardcoded token in SSHCredentials.tsx is still caught even if a key isn't. But that's the worse branch of the fork -- it still leaves private keys invisible in the private-key UI, and it still carries twelve entries that v8.30.1 makes unnecessary.

The catalogue comment itself is good work and the per-file scoping is the right instinct -- I'd just rather not need it.

Comment thread .gitleaks.toml Outdated
'''backend/tests/integration/test_routes_ssh_credentials\.py''',
'''backend/tests/unit/test_proxy_translate_cis_service\.py''',
'''backend/tests/unit/test_tmos_engine\.py''',
'''frontend-v2/src/components/settings/SSHCredentials\.tsx''',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are in the top-level [allowlist], so they suppress every rule in these files -- not just private-key. Six are production source, and they're the SSH-key-handling UI.

Measured, identical real ssh-keygen -t ed25519 key written into two files, one allowlisted (SSHCredentials.tsx) and one not:

v8.21.2  -> leaks found: 1     # only the non-allowlisted copy
v8.30.1  -> leaks found: 1     # only the non-allowlisted copy

Both versions miss the allowlisted copy. A live deploy key committed to SSHCredentials.tsx, or an AKIA... in DpuFormDialog.tsx, now reaches a public repo with a green gate. (Canaries shredded immediately.)

The part that makes this avoidable: these twelve exist only because the pin is nine minors behind. On d48648dc -- this branch one commit ago, with none of these entries -- v8.30.1 reported 0 findings. The newer private-key rule already tells a PEM header around ... apart from real key material, which is precisely the judgement being hand-encoded here.

ghcr.io/gitleaks/gitleaks:v8.30.1

and delete lines 57-71. One line instead of twelve, and the key-handling files stay under the scanner.

If v8.21.2 has to stay for a reason I can't see, these should at least move into a private-key-scoped [rules.allowlist] rather than the global one, so a hardcoded token in these files is still caught. That's strictly worse than pinning forward, but it's better than a blanket exemption on the SSH UI.

…ic chart

PR #177 nit (bonnyr-f5): helm/bnk-forge/values.yaml shipped
`mcpPassword: changeme` -- a known default password now that the chart is the
public distribution path. The other four secrets (postgres/redis/jwt/encryption)
are generated with randAlphaNum when left empty and reused across upgrades via
the existing-secret lookup, but mcp-password had no such generation and used the
raw value directly.

Added the same lookup-then-randAlphaNum(24) logic for mcp-password and blanked
the default in values.yaml, with a comment on how to retrieve the generated
value (kubectl get secret ... | base64 -d). `helm template` confirms a random
mcp-password is rendered, not "changeme".

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

Reworked the gitleaks fix using your version data — thank you, that table was the key.

Re-pinned :v8.21.2:v8.30.1. You showed v8.21.2 flags 13 private-key false positives on these PEM-header placeholders while v8.30.1 flags 0 — the rule was tightened across those minors. So rather than catalogue false positives against an old scanner (I'd briefly added 13 allowlist entries — you'd have rightly objected, since they suppressed all rules on production key-input components), I dropped them entirely and pinned the scanner that gets it right. Verified: clean tracked-only checkout scans no leaks found on v8.30.1 with no new allowlist entries.

Also from your nits:

  • frontend-v2/dist/.* allowlist — removed (your blocker); a built bundle is where VITE_* injection lands.
  • Mount is now read-only (-v "$PWD:/repo:ro") — detect only reads.
  • --redact is deliberate (redacted value + file + line), confirmed, no change.
  • The .pyc/__pycache__ entries stay with the reviewed-and-why framing.

On script-selftests being toothless: fixed on #179 — its rebuilt Test 7 (early marker + multi-line tail) now FAILs against the reverted loop on GNU grep, so this job stops passing on the bug once #179 lands.

@mwiget mwiget left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 5741c345. All 28 checks green, including P1 · Secret Scan (gitleaks), P1 · ShellCheck, P1 · Script Self-Tests and the CI Gate.

The pin moved to v8.30.1, the twelve allowlist entries are gone, and the mount is :ro. .gitleaks.toml is now byte-identical to d48648dc, the tree I'd already measured, and I re-ran the exact command CI runs against this head:

INF scanned ~28577103 bytes (28.58 MB) in 1.23s
INF no leaks found

Zero findings, zero exemptions on the SSH-key-handling files. That's the outcome I was after: the gate stays pointed at the code where a leaked key would actually matter.

Worth recording what this exercise established, because it's the argument for pinning in the first place: the same tree scanned 0 under v8.30.1 and 13 under v8.21.2, and every one of the 13 was a PEM header wrapped around a literal .... Pinning is right; the version just has to be one you validated against, and it wants re-validating when it's bumped rather than bumped on faith.

Standing notes, neither blocking:

  • script-selftests earns its keep now that #179 is approved -- I verified its Test 7 fails when the SIGPIPE fix is reverted, so this job gates behaviour rather than the mere presence of a self-test. Before #179 lands it's still a green light regardless of outcome.
  • Conflicts with #180 (also approved now) in ci.yml -- same needs: list and gate loop. Whichever merges second needs a trivial rebase.

@bonnyr-f5

Copy link
Copy Markdown
Collaborator

Review: REVISE

Reviewed under the review-discipline pipeline (invariant sweep + context-isolated cold audit at 5741c34, base 4a52ed4). Two of the three new gates were confirmed to actually fail on planted defects, which is the right bar. Three things to fix.

Verified genuinely blocking

ShellCheck gate. make shellcheckshellcheck --severity=warning upgrade.sh scripts/*.sh vm-bnk-forge/*.sh vm-bnk-forge/lib/*.sh (Makefile:464-467, no || true). Exit 0 at this ref; planted SC2034 in a scripts/*.sh file → make: *** [shellcheck] Error 1. Failure propagates.

Also worth noting the get_dpu_pwd.sh shebang is a real finding, not cosmetic — base emits SC2148 (error), above the warning threshold, so the gate would have failed without it. Confirmed the base tree is red (SC2148 + SC2034) and this branch is the only one of the five that makes it green. The for ifor _ renames are safe (no $i references remain).

gitleaks gate. Ran the exact CI invocation: no leaks found, 28.58 MB, exit 0. Falsification tests all pass — planted AWS-shaped key in a .py → detected, exit 1; planted Slack token in a .md → detected, exit 1; invalid TOML → FTL unable to load gitleaks config, exit 1; missing config → exit 1.

Allowlist scoping proven by test rather than by reading: planting five secret shapes in both backend/tests/unit/ and a neutral directory shows [rules.allowlist] correctly attaches to the generic-api-key rule only — private-key and slack-bot-token still fire inside backend/tests/**. The scanner is not blinded. The new __pycache__ / .pyc entries do land in the global [allowlist] as intended, and your "zero tracked files" claim is true (git ls-tree -r | grep -cE '__pycache__|\.pyc$' → 0).

Blocker (merge-order) — script-selftests cannot fail unless #179 lands first

SELF_TEST=1 bash scripts/compute_version_bump.sh prints FAIL: but the base harness has no counter and no exit 1, and run_test swallows the recursive invocation's status with || true. Injecting a wrong expectation into the base script:

  FAIL: feat commit → minor
        expected bump=major ver=9.9.9
        got     bump=minor ver=1.3.0
=== END SELF-TEST ===
EXIT=0

ci-gate treats only failure/cancelled as bad, so a broken version-derivation regression would merge green — on the very logic this gate is meant to protect. #179 adds exactly the missing plumbing (its commit is titled "make SELF_TEST exit non-zero on failure so CI can gate it"); against that harness the same injection exits 1. Land #179 first and this resolves itself with no change here. Flagging it because in the other order the job is decorative from birth.

Major — the secret-scan gate never runs on doc-only changes

ci.yml:24-49 — both pull_request and push carry paths-ignore: ['**.md', 'docs/**', '.agent/**', '.opencode/**', 'LICENSE', '.gitignore', '.trivyignore', 'USER_GUIDE.md']. When every changed file matches, the whole workflow is skipped, so secret-scan never runs. A token pasted into a .md or anywhere under docs/ merges into a public repo with zero scanning — and markdown is precisely where illustrative-looking-but-real tokens land. The pre-existing config already allowlists cli/README.md and docs/archive/OPERATOR_ARCHITECTURE.md as known false positives, which confirms docs here do carry token-shaped content. Consider giving secret-scan its own workflow without paths-ignore, or adding a pull_request trigger that ignores nothing for this job.

Minor

  • Coverage is narrower than the job names imply. 13 tracked .sh files plus the two extensionless .githooks/* are outside the ShellCheck globs — notably dist/install.sh (the customer installer) and all four bin/local/*.sh. I ran --severity=warning over all of them: zero findings, so this is a latent scope gap, not a live defect. Similarly script-selftests self-tests exactly one script, while Fix non-deterministic version derivation (SIGPIPE race) — PR #177 Blocker 1 #179 also changes extract-breaking-changes.sh and Sync Helm chart tag / appVersion / package.json with VERSION — PR #177 Blocker 2 #180 adds sync-version-artifacts.sh — neither is exercised. Per INV-9's own rule, a gate narrower than its claim can't back the claim: either widen the globs or rename the job to compute_version_bump self-test.
  • Neither gate covers workflow run: blocks. Release workflow: republish recovery path + RC/loop-guard robustness — PR #177 Blocker 3 #181 adds ~40 lines of new release shell carrying an explicit "must NOT set -o pipefail" hazard, linted by nothing. An actionlint step would close this; it reports no new findings vs base today, so it can be added green.
  • ShellCheck is installed unpinned (apt-get install -y shellcheck) while gitleaks is deliberately pinned to v8.30.1. The runner's version therefore drifts independently of the repo, and local ≠ CI. I re-ran the merged tree under 0.9.0 and 0.10.0 as well as 0.11.0 — all exit 0 — so nothing is broken now, but please pin it for the same reason you pinned gitleaks.
  • detect --no-git scans the working tree only. Combined with actions/checkout@v6's default fetch-depth: 1, a secret introduced and later removed in an earlier commit is invisible. For a public repo the history is the exposed surface.

Nits

  • .gitleaks.toml:3-5 says "Do NOT add blanket directory allowlists… Scope each entry to a rule + a specific path." The new .*__pycache__/.* is a global, un-rule-scoped wildcard directory allowlist. Defensible for bytecode, but it contradicts the file's stated policy without amending it.
  • needs: changes on all three new jobs is inert — none consume needs.changes.outputs.* and all carry if: always(), so it only serialises them behind changes.
  • scripts/get_dpu_pwd.sh gained a shebang but stays mode 100644.
  • The gate wiring itself is correct: all three jobs appear in both ci-gate.needs and the decision loop, YAML parses (27 jobs), no duplicate keys, no undefined needs: references, and injecting failure for each new job blocks the gate. The blocker above is purely the script's exit code, not the plumbing.

Registry note

INV-9's recorded Sweep/Freeze does not match this repo — the registry claims CI runs shellcheck … bin/*.sh bin/lib/*.sh bin/hooks/*.sh tests/*.sh scripts/test/*.sh, and none of those paths exist here. This PR is the right place to make the registry entry name the real gate.

Merge order

Land fourth — after #179 (so script-selftests is real), and before #180. This PR conflicts with #180 in ci.yml (4 hunks); see my note there. It carries the fixes for the two pre-existing ShellCheck findings, so it is self-consistent in any position.

jgruberf5 pushed a commit that referenced this pull request Aug 20, 2026
bonnyrf5 aggregate review, #182.

paths-ignore skipped secret scanning on doc-only PRs (ci.yml:24-49): gitleaks
sits behind the workflow-level paths-ignore, so a doc-only PR skipped the ENTIRE
workflow and never ran it -- in a public repo, where a secret lands in a .md as
easily as in code. secret-scan already carries `if: always()`, but that can't
override a workflow-level skip. Removed both paths-ignore blocks so secret-scan
and the CI Gate see every change; expensive jobs still skip on irrelevant paths
through the per-job `changes` filter, so path filtering now has one home.

script-selftests couldn't fail (ci.yml:154-162): the job trusted the self-test's
exit code, but the harness printed FAIL: while exiting 0, so it could not catch a
broken self-test until #179's exit-code fix landed. The job now fails on a
non-zero exit OR a FAIL: line, so it's effective on its own. Verified the grep
catches a FAIL: line even at exit 0 and does not flag PASS output.

YAML valid; actionlint clean on the changed regions.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

@mwiget mwiget left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at ff93ab39 — 28/28 green. Both changes verified.

Dropping the workflow-level paths-ignore is correct, and the per-job filters really do keep the cost down. I checked the three jobs that must see every change: shellcheck, secret-scan, and script-selftests all sit on needs: changes + if: always() with no needs.changes.outputs.* condition, while every expensive job (unit/component/integration/docker/migration) still carries one. So a doc-only PR now runs three cheap jobs plus changes and the gate, and skips the rest — which is what the comment claims. CI Gate treats skipped as acceptable, so the doc-only shape passes rather than hanging.

There's a bonus this earns that the commit message doesn't claim: with the trigger-level ignore gone, a doc-only PR now produces a CI Gate conclusion instead of producing no checks at all. Under branch protection a skipped-by-paths workflow leaves the required check permanently pending — the classic unmergeable-docs-PR trap. Moot today (#185: staging has no protection) but it stops that from becoming a second bug the moment protection is turned on.

The self-test guard works, and it's genuinely needed on this branch. scripts/compute_version_bump.sh here prints FAIL: $desc at line 199 and has no failure counter and no exit 1 — I confirmed the harness exits 0 regardless. So before this change the job could not fail on a broken self-test, exactly as the comment says. Ran the real thing plus the false-positive cases:

$ SELF_TEST=1 bash scripts/compute_version_bump.sh   → rc=0, 6× "PASS:", grep clean ✅
'  FAIL: some test desc'    → matches ✅ (leading whitespace)
'FAIL: at line start'       → matches ✅
'PASS: FAILS: nope'         → no match ✅
'PASS: 6 passed, 0 failed'  → no match ✅

The (^|[[:space:]]) prefix is what keeps FAILS:/FAILED from tripping it. And this stays useful after #179 lands its exit-code fix — the two are independent checks on the same harness, which is the point.

One non-blocking follow-up, cross-PR: release.yml still carries the comment "MUST stay in sync with ci.yml's push-trigger paths-ignore list — a divergence lets a push trigger Release without a matching CI run". After this PR the lists diverge by design, and the comment now instructs the next editor to re-add exactly what you just removed. The divergence itself is safe — the direction is CI ⊇ Release, so a doc-only push runs CI and no release, never the reverse — but the comment should say that instead. #181 is already editing release.yml; whichever merges second could reword it.

Also worth folding into the same script-selftests job: #179 adds extract-breaking-changes.sh --self-test, which nothing currently invokes.

jgruberf5 pushed a commit that referenced this pull request Aug 20, 2026
mwiget's re-review of #182.

Removed the `frontend-v2/dist/.*` allowlist entry. A built JS bundle is exactly
where VITE_*-style build-time secret injection lands, so it's the highest-value
thing to scan, not to skip; a blanket directory entry in the top-level
[allowlist] also suppresses every rule (private-key, cloud creds), not just
generic-api-key -- which the config's own header warns against. And it bought
nothing in CI: dist is gitignored, so a fresh checkout has no tracked files
there anyway. The .pyc / __pycache__ entries stay (compiled bytecode of sources
that are themselves scanned) with the "reviewed, here's why" framing the other
entries carry.

Pinned the gitleaks image to v8.21.2 (was :latest) -- a security gate shouldn't
float its scanner version.

The script-selftests job becomes meaningful once #179 lands: its rebuilt Test 7
catches the SIGPIPE regression on CI's GNU grep (the old single-line Test 7
passed even with the fix reverted).

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
jgruberf5 pushed a commit that referenced this pull request Aug 20, 2026
bonnyrf5 aggregate review, #182.

paths-ignore skipped secret scanning on doc-only PRs (ci.yml:24-49): gitleaks
sits behind the workflow-level paths-ignore, so a doc-only PR skipped the ENTIRE
workflow and never ran it -- in a public repo, where a secret lands in a .md as
easily as in code. secret-scan already carries `if: always()`, but that can't
override a workflow-level skip. Removed both paths-ignore blocks so secret-scan
and the CI Gate see every change; expensive jobs still skip on irrelevant paths
through the per-job `changes` filter, so path filtering now has one home.

script-selftests couldn't fail (ci.yml:154-162): the job trusted the self-test's
exit code, but the harness printed FAIL: while exiting 0, so it could not catch a
broken self-test until #179's exit-code fix landed. The job now fails on a
non-zero exit OR a FAIL: line, so it's effective on its own. Verified the grep
catches a FAIL: line even at exit 0 and does not flag PASS output.

YAML valid; actionlint clean on the changed regions.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

Rebased this branch onto #180 (fix/version-artifact-consistency) to defuse the ci.yml merge conflict bonnyrf5 flagged (#180#182, "shaped to resolve silently wrong"). The conflict is now resolved once, correctly, in the branch: ci.yml carries both version-consistency (from #180) and shellcheck / secret-scan / script-selftests (from this PR) as separate jobs, and the ci-gate needs: list and its result-check loop list all four — no job is silently dropped.

Merge order: this PR now sits on top of #180, so merge #180 first, then this one (its diff collapses to just its own changes once #180 lands). YAML validates and actionlint is clean.

@jgruberf5
jgruberf5 force-pushed the fix/ci-hygiene-gitleaks-shellcheck branch from ff93ab3 to 505ec1e Compare August 20, 2026 12:04
…#188)

bonnyrf5 aggregate review, #180.

--write fails open (scripts/sync-version-artifacts.sh): a sed whose pattern
matched nothing no-ops silently, so a format change to any artifact left it
unchanged while the script still reported success -- and the release job commits
that with CI suppressed. Now re-reads all three with the same helpers --check trusts and
exits 1 if any didn't take ${V}. Verified: happy path passes, a package.json
whose "version" line no longer matches makes it exit 1.

MCP secret (secrets.yaml / values.yaml): this PR generated mcp-password as a
chart-owned secret with mcpUsername: admin, which breaks MCP auth on every fresh
install -- it's a client credential the MCP server must also read, not a
chart-owned value. That's the half-fix bonnyrf5 flagged. The complete fix (point
the chart at the mcp service account, wire MCP_SERVICE_PASSWORD into the backend
so ensure_service_user reconciles the hash, rotate the shipped default on
upgrade, checksum/secret roll) lives in #188. Reverted the MCP edits here so this
PR stays scoped to version-artifact consistency; its MCP diff vs staging is now
empty, so it no longer overlaps #188.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
mwiget's note on #180: a commit whose body quoted the CI-skip marker in prose had
its whole run suppressed (GitHub scans the entire message), and because the change
was a shell script the skipped gates were exactly the relevant ones. Documented the
rule and the indirect phrasings so it does not recur.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
PR #177 review (bonnyr-f5) — Major. `.gitleaks.toml` was a config nothing ran,
`make shellcheck` had no caller, and compute_version_bump.sh's SELF_TEST was
never invoked — so the release-critical shell scripts were statically unchecked
and, for a public repo, the secret-scanning gate was unwired.

Three new P1 jobs, all wired into the CI gate:
- Secret Scan (gitleaks): `--no-git` over the tracked source with the repo's
  config. Extended the allowlist to cover generated build artifacts (.pyc,
  frontend-v2/dist) alongside the existing synthetic-key entries; a fresh
  checkout now scans clean (verified locally even with artifacts present).
- ShellCheck: runs `make shellcheck` over the whole script corpus. Fixed the two
  pre-existing findings that would have blocked the gate — a missing shebang in
  get_dpu_pwd.sh (SC2148) and unused `for i` loop counters in
  ibm_cloud_bnk_forge.sh (SC2034, now `for _`). Corpus is clean at
  --severity=warning.
- Script Self-Tests: runs `SELF_TEST=1 compute_version_bump.sh`, which now exits
  non-zero on failure (that change ships with the SIGPIPE-race PR), so a
  regression in the logic that decides the released version fails CI.

The release-critical scripts (compute_version_bump, extract-breaking-changes,
publish-signed-images) already pass shellcheck cleanly.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
mwiget's re-review of #182.

Removed the `frontend-v2/dist/.*` allowlist entry. A built JS bundle is exactly
where VITE_*-style build-time secret injection lands, so it's the highest-value
thing to scan, not to skip; a blanket directory entry in the top-level
[allowlist] also suppresses every rule (private-key, cloud creds), not just
generic-api-key -- which the config's own header warns against. And it bought
nothing in CI: dist is gitignored, so a fresh checkout has no tracked files
there anyway. The .pyc / __pycache__ entries stay (compiled bytecode of sources
that are themselves scanned) with the "reviewed, here's why" framing the other
entries carry.

Pinned the gitleaks image to v8.21.2 (was :latest) -- a security gate shouldn't
float its scanner version.

The script-selftests job becomes meaningful once #179 lands: its rebuilt Test 7
catches the SIGPIPE regression on CI's GNU grep (the old single-line Test 7
passed even with the fix reverted).

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
…urfaced

The gitleaks gate this PR adds went red on its own first run: enforcing the scan
surfaced 13 `private-key` matches never catalogued before, because gitleaks had
never actually run in CI. All 13 are PEM headers around a placeholder body,
verified individually:
- the textarea hint "-----BEGIN OPENSSH PRIVATE KEY-----\n...\n-----END..." in
  the SSH key-input UI (SSHCredentials, DpuFormDialog, RshimInstallDialog,
  DpuProjectSettingsCard, NodeDiscoveryPanel), and
- synthetic keys / "..." / "SECRETCONTENT" in six backend SSH tests.

Added as specific-path entries to the existing allowlist, matching the
convention the config already uses for the other PEM-placeholder test files.
(`[[allowlists]]` with `targetRules` would scope these to the private-key rule
only, but this gitleaks version ignores the plural block when the singular
`[allowlist]` is present.) Verified: a clean tracked-only checkout scans with no
leaks found.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
…only mount

mwiget's data was the key: v8.21.2 flags 13 private-key false positives on these
PEM-header placeholders, but v8.30.1 flags 0 -- the rule was tightened across
those minors. So the right fix isn't to catalogue the false positives against an
old scanner; it's to pin the scanner that gets it right. Repinned :v8.21.2 ->
:v8.30.1 and dropped the 13 allowlist entries I'd added, which over-suppressed
production key-input components (all rules, not just private-key) for no benefit
under v8.30.1. Verified: clean tracked-only checkout scans no leaks found.

Also from mwiget's nits: the mount is now read-only (-v "$PWD:/repo:ro" -- detect
only reads). `--redact` was already deliberate (redacted value + file + line),
no change.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
bonnyrf5 aggregate review, #182.

paths-ignore skipped secret scanning on doc-only PRs (ci.yml:24-49): gitleaks
sits behind the workflow-level paths-ignore, so a doc-only PR skipped the ENTIRE
workflow and never ran it -- in a public repo, where a secret lands in a .md as
easily as in code. secret-scan already carries `if: always()`, but that can't
override a workflow-level skip. Removed both paths-ignore blocks so secret-scan
and the CI Gate see every change; expensive jobs still skip on irrelevant paths
through the per-job `changes` filter, so path filtering now has one home.

script-selftests couldn't fail (ci.yml:154-162): the job trusted the self-test's
exit code, but the harness printed FAIL: while exiting 0, so it could not catch a
broken self-test until #179's exit-code fix landed. The job now fails on a
non-zero exit OR a FAIL: line, so it's effective on its own. Verified the grep
catches a FAIL: line even at exit 0 and does not flag PASS output.

YAML valid; actionlint clean on the changed regions.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
The gitleaks gate ran the container but asserted nothing about what it
did. On a bad revision range or a git dubious-ownership refusal (image
runs as root, checkout owned by runner, no safe.directory), gitleaks
prints "ERR [git] ..." + "0 commits scanned" and STILL exits 0 -- so the
gate went green having scanned NOTHING. "0 commits scanned" and "11
commits scanned" were indistinguishable to it, exactly the silent-pass
failure already fixed for script-selftests.

Root-cause fix: whitelist /repo via GIT_CONFIG_* env inside the
container (no writable HOME needed, unlike git config --global) so the
dubious-ownership path cannot short-circuit the scan. Backstop: capture
the output and fail on any "ERR [git]" line, on a missing "commits
scanned" line, on 0 commits for a non-empty range, and on a non-zero
gitleaks exit (leaks found). The assertions run even with the
safe.directory fix in place, so a future breakage is caught, not masked.

Reproduced against throwaway git fixtures: bad range and hostile
root-mount both yield ERR + 0-commits + rc0 and are now caught; valid
range (2 commits), full history (3 commits) still pass; a planted
private key exits 1 and fails the gate.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

@bonnyr-f5 — BLOCKER fixed in d357e40. You were right on every point: the gate ran the container but asserted nothing, so a git error produced identical green output to a real clean scan. I reproduced both failure modes in throwaway git fixtures before touching the code.

Reproduction (gitleaks v8.30.1, real containers)

Bad revision range--log-opts=deadbeef..cafebabe:

ERR [git] fatal: ambiguous argument 'deadbeef..cafebabe': unknown revision ...
INF 0 commits scanned.
INF no leaks found
docker rc=0

Dubious ownership — root container over a checkout owned by another uid, no safe.directory (-u 0:0 -e HOME=/nonexistent):

ERR [git] fatal: detected dubious ownership in repository at '/repo'
INF 0 commits scanned.
INF no leaks found
docker rc=0

Both green, both scanning nothing — 0 commits scanned indistinguishable from 11 commits scanned, exactly as you described.

The fix (your class fix)

  1. Root-cause the ownership path so it can't short-circuit in the first place. Whitelist /repo via GIT_CONFIG_COUNT=1 / GIT_CONFIG_KEY_0=safe.directory / GIT_CONFIG_VALUE_0=/repo env forwarded into the container. I used env rather than git config --global deliberately: it needs no writable HOME, so it holds even under the hostile HOME=/nonexistent mount. Verified — the same root-mount that printed dubious ownership above now prints 2 commits scanned.

  2. Keep the assertion regardless, so a future breakage is caught rather than masked. Capture the output and fail on: any ERR [git] line; a missing commits scanned line (silence must not pass — same doctrine as script-selftests' PASS/END markers); 0 commits for a non-empty range; and a non-zero gitleaks exit (leaks found).

Verification of the shipped step body (exact block, real containers)

Case Result
bad range FAIL (git error)
dubious-ownership, hostile root mount PASS, 2 commits scanned (safe.directory fix)
valid range PASS, 2 commits scanned
full history (empty range) PASS, 3 commits scanned
planted OpenSSH private key FAIL, leaks found: 1, rc=1

Defense-in-depth check — with the safe.directory env removed (fix regressed) under the hostile root mount, the assertion still FAILs on ERR [git] rather than passing blind. So the gate can distinguish clean-scan from didn't-run in every case.

Validation: yaml.safe_load OK; bash -n on the extracted run block OK; actionlint finding count unchanged (9 pre-existing, all at line 1132 in an unrelated job — zero new from this step).

The Major/Minor/Nit items (archive depth, scheduled baseline, needs.changes.result, local make targets, stale release.yml comments, extractor self-test wiring, digest pin) are acknowledged and out of scope for this blocker commit.

jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
…empty data

bonnyr-f5 round-3 BLOCK on #180. Both blockers reproduced and fixed, each with a
red-green test that proves the guard now fails on the exact input it missed.

BLOCKER 1 — writer/stager divergence. --write rewrote five artifacts (incl. the
two operator-chart files this round added) but release.yml's "Commit and tag"
step hard-coded `git add` for three, so the operator chart was rewritten on the
runner and never staged -> next PR's version-consistency job red-lined, blocking
every subsequent release. Fixed structurally: the script owns a canonical
SYNCED_FILES list exposed via a new --list mode, and both "Commit and tag" steps
stage exactly `sync-version-artifacts.sh --list` -> writer and stager cannot
diverge. Added an INDEX check (git diff --quiet per file) so a synced-but-unstaged
artifact fails the release rather than the next PR (bonnyr: "verify the index,
not the files"). Verified: all five stage; unstaging one is caught.

BLOCKER 2 — --check green on a tree with no version data. The non-vacuity guard
counted loop iterations over a literal 5-item list, so `checked>=5` was always
true and its guard unreachable; empty compared equal to empty five times, rc 0.
Rewritten to assert each version line is NON-EMPTY and equals VERSION, that
VERSION itself is non-empty, and that every artifact contributed >=1 MATCHED line
(total counts matched lines, not iterations). Verified against bonnyr's exact
repro (empty VERSION + every key deleted) -> now rc 1.

Majors (same review):
- Every reader now reads EVERY matching line (was grep -m1 first-match against a
  global write), so a second appVersion/tag can't drift unseen. Verified: a
  drifted second 2-space tag is caught.
- The image-tag writer is bounded to the top-level `image:` block via a sed range
  instead of a bare `^  tag:`; a future unrelated 2-space tag is left alone.
  Verified. appVersion/version writes are already key-anchored (single-occurrence
  top-level keys).

Docs: header, the "Synced" list and the success echo now name all five artifacts
(they claimed three while five were written, and one line still said the operator
chart was not synced while the code synced it).

Confirmed: operator image IS on the release train (docker-bake.hcl `default`
group builds it), so syncing its chart to VERSION is correct. shellcheck -S style
clean; real-tree --check green; full red-green matrix in the PR comment.

Merge-order note for the record: merge #180 before #182, or with a merge commit —
#182 carries a pre-round-2 copy of this script, and a squash would arrive as an
add/add conflict on it.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
…tract

Addresses bonnyr-f5 round-3 BLOCK on PR #181.

BLOCKER 1 - republish overwrote the tag it calls immutable and was not
idempotent. docker-bake.hcl stamped org.opencontainers.image.created with
timestamp(), so every rebuild produced a different digest (reproduced:
two identical bakes gave ...36Z then ...40Z), and every bake target pushes
:VERSION unconditionally. Fixes:
  - docker-bake.hcl: replace timestamp() with an injectable CREATED variable,
    empty by default so a plain bake of a given tree is byte-reproducible; CI
    sets it to the release commit's committer date. release-publish also passes
    SOURCE_DATE_EPOCH to normalise layer mtimes, so the same tag rebuilds to
    the same digest and its cosign/SBOM/SLSA attestations still apply.
  - release.yml: before pushing, probe the registry with docker manifest
    inspect and refuse by default when the :VERSION manifest already exists;
    a new force dispatch input overrides deliberately. The normal recovery
    case (a tag whose publish never completed) has no manifest yet and passes.

Major findings also fixed:
  - Makefile push-images now runs the same recency guard before baking with
    --push ROLLING_TAG=latest, so a stale local tree cannot repoint :latest
    backward (override with FORCE_LATEST=1).
  - workflow_dispatch of a manual release is now gated to main, or a ref
    already merged into main, so a side branch cannot publish an unreviewed
    tree as :latest.
  - The recency check is re-run inside the release-publish concurrency-guarded
    step immediately before the push, closing the TOCTOU where preflight's
    check (outside the critical section) let a lower version write :latest last.
  - The loop guard now fails the run on an unexpected suppression instead of
    reporting a silent green; our own release commit still ends green as a
    notice. A release that published nothing is no longer green.
  - Corrected the paths-ignore comment: #182 removes ci.yml's push
    paths-ignore entirely (it does not add a subset check), so CI is a strict
    superset of Release; fixed the two inverted remediations.
  - Rewrote the loop-guard header comment to describe the real mechanism -
    GitHub's native marker handling drops our release commits before the
    workflow starts; the guard is a backstop.

Verification: python yaml parse OK; actionlint reports no new findings (the
4 pre-existing SC2129 style notes are unchanged); docker buildx bake --print
confirms CREATED is deterministic; guard, recency, probe and Makefile-guard
logic unit-tested.

Note for merge: this branch's commit bodies quote a CI-control marker as
literal text, so it MUST be merged with a merge commit (not squashed) or the
squash body will carry the marker and suppress CI and Release. Documented on
the PR.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
Addresses every non-blocker bonnyr-f5 raised in round 3.

Archives (Major): gitleaks defaulted to --max-archive-depth 0, so a secret
shipped inside a tracked tarball was invisible. The scan now runs with
--max-archive-depth 2. Proven with a synthetic fixture: a private key inside a
.tar.gz -> "no leaks found" at depth 0, "leaks found: 1" (secrets.tar.gz!id_rsa)
at depth 2.

Baseline (Major): the per-push gate only scanned each change's commit range, so
anything already in history was never re-examined. Added secret-baseline.yml --
a weekly schedule plus workflow_dispatch that runs gitleaks over full history
with the same assertion backstop.

CI Gate (Major): the aggregator never checked needs.changes.result. If change
detection failed, ~21 gates resolved to skipped, the loop accepted skipped, and
the required check printed PASSED. It now fails when changes did not succeed.
Mutation-tested: with change detection failed and test jobs skipped, the old
gate went green, the new gate goes red.

Local == CI (Major, #166): the four gates ci.yml added were unrunnable locally.
Added make targets (version-check, secret-scan, commit-lint, script-selftests)
aggregated as ci-gates, and made pre-push depend on it. The secret scan + its
whole assertion backstop now live in scripts/secret-scan.sh, called identically
by the CI job, the baseline workflow, and make -- one source of truth.

Marker enforcement (Minor -> real): the skip-CI-marker rule was documentation
only. Added scripts/lint-commit-markers.sh plus a commit-lint CI gate and a
pre-push hook step that fail a range carrying a skip-CI marker, or a line-start
prose form that would spuriously trigger a major release. A genuine
conventional footer still passes. Mutation-tested across eight cases; passes on
this PR's 13-commit range.

Digest pin (Nit): the movable v8.30.1 tag is replaced by
ghcr.io/gitleaks/gitleaks@sha256:c00b6bd0... (v8.30.1 kept in a comment).

release.yml stale comments (Major): the ":23-26" note referenced a ci.yml
paths-ignore that no longer exists, and the ":150-157" note claimed
cancel-in-progress: true for main/staging where it is now false. Comment text
corrected to match reality; no release logic touched (that is #181's domain).

Cross-PR items are intentionally left to merge order: the sync-version-artifacts
second-tag reproduction is fixed in #180's head, and compute_version_bump's
exit-0-on-fail in #179's head. Merge #180 first, merge-commit not squash.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

@bonnyr-f5 — round-3 follow-up: every remaining item is now addressed in cf64faf0. Each fix was reproduced and mutation-tested (broke what it guards, confirmed red), then verified green in real CI on this head.

Major — tracked archives never scanned

--max-archive-depth 2 added to the scan. Synthetic fixture (private key inside a .tar.gz):

depth 0 (default):  1 commits scanned / no leaks found        <- invisible
depth 2 (fixed):    1 commits scanned / leaks found: 1        (secrets.tar.gz!id_rsa)

Major — no baseline / full-history scan

New workflow .github/workflows/secret-baseline.yml: schedule (weekly cron) + workflow_dispatch, runs gitleaks over full history (RANGE="") with the same assertion backstop. Ran the baseline manually against a normal clone: 1006 commits scanned / no leaks found — history is clean today, and now stays monitored.

Major — CI Gate never checked needs.changes.result

The gate now fails when change-detection did not succeed. Mutation-tested by rendering the real gate script with changes=failure and the path-gated test jobs skipped:

OLD gate (guard removed):  rc=0  -> CI Gate PASSED   <- the bug you described
NEW gate (with guard):     rc=1  -> CI Gate FAILED   (change-detection did not succeed)

Major — make pre-push != CI (#166)

The scan + its entire assertion backstop now live in scripts/secret-scan.sh, called byte-identically by the CI job, the baseline workflow, and make secret-scan. Added make targets version-check / secret-scan / commit-lint / script-selftests, aggregated as ci-gates, and pre-push now depends on it. make -n ci-gates shows all five gate commands; make version-check and make script-selftests pass locally.

Minor -> enforcement — the skip-CI marker rule

scripts/lint-commit-markers.sh + a commit-lint CI gate (wired into the aggregator's needs/loop/always-run set) + a .githooks/pre-push step. Fails any commit range carrying a CI-control marker, or a line-start prose form that would spuriously major-bump; a genuine BREAKING CHANGE: footer still passes. Eight mutation cases (marker in body/subject case-insensitive, bold heading, colon-less line, multi-marker → FAIL; clean commit, real footer, mid-line prose → PASS). Real CI: commit-lint scanned 14 commits → OK.

One correction with evidence: a blanket line-start ban on BREAKING CHANGE would reject both (a) the documented+self-tested conventional footer used for intended majors (compute_version_bump.sh:124, self-test 5), and (b) this branch's own commit 83982af4 ("assert the BREAKING CHANGE detector...") which mentions it as mid-line prose. So the gate anchors to line-start AND exempts plain footers — it passes this PR's 13-commit range while still catching the accidental forms. The greedier mid-line prose match is a detector issue owned by the version-tooling PRs, out of scope here.

Nit — pin the gitleaks image by digest

Pinned ghcr.io/gitleaks/gitleaks@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f (your c00b6bd0…), v8.30.1 kept in a comment.

release.yml stale comments — both confirmed and corrected (text only, no logic change)

  • :23-26 — correct: the comment referenced a ci.yml paths-ignore that no longer exists (ci.yml runs CI on every push to main/staging). Reworded to explain ci.yml is now a strict superset, so any release-triggering push still has a CI run for the preflight to find.
  • :150-157 — correct, and the more serious one: it claimed cancel-in-progress: true, but ci.yml:48 sets it false for exactly main/staging. Reworded; the exit 1 (a cancelled run is stale) stays.

Real-CI verification on cf64faf0

P1 · Secret Scan (gitleaks)   success   14 commits scanned / no leaks found / secret-scan: OK
P1 · Commit Message Lint      success   scanned 14 commit(s) / commit-lint: OK
P1 · Version Consistency      success   (make version-check)
P1 · Script Self-Tests        success
P1 · ShellCheck               success   (both new scripts + the hook are shellcheck-clean)

secret-scan: 14 commits scanned (not a blind pass — the safe.directory env resolves the root-vs-runner ownership path on the real checkout). YAML parses; actionlint finding counts unchanged (ci.yml 9, release.yml 4 — all pre-existing, none from these steps); secret-baseline.yml is clean.

Cross-PR (left to merge order, as you noted)

The sync-version-artifacts.sh second-tag: reproduction is fixed in #180's head; compute_version_bump.sh's exit-0-on-fail in #179's head. Merge #180 first, merge-commit rather than squash.

jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
… note leaks

bonnyr-f5 round-4 BLOCK on #179. Three defects, each reproduced against the real
commits he cited and each covered by a new red-green self-test.

BLOCKER 1 (wrapped prose) — line-start anchoring still matched a prose paragraph
that WRAPPED so the marker fell at column 1 (commit 8415ce1, on this very branch,
executed to TARGET_VERSION=5.0.0). The body match is now PARAGRAPH-initial: the
marker line must be the first body line or be preceded by a blank line. Verified
against the two real commits -- 7ece9b0 (the #2 break, marker preceded by a blank
line) still triggers so 4.0.0 is unchanged; 8415ce1 (marker mid-paragraph) no
longer triggers.

BLOCKER 2 (folded footer, a regression vs staging) — with no blank line before it,
git folds the footer into the subject and leaves the body empty, so a body-only
check derived a patch for a real major. Confirmed git's folding, then added a
subject check for a folded footer -- colon REQUIRED there, so subject prose that
merely names the marker does not trigger. The consistency guard now iterates PER
COMMIT (paragraph detection cannot run on concatenated bodies) and reads subject
and body, so it still fires if the loop's detector is deleted.

MAJOR (trailer leak) — the note captured until a blank line, so a footer sitting
above the trailer block published a Co-Authored-By address and a Claude-Session
URL into a public release. The note now stops at the first trailer-shaped line as
well as at a blank line; a following footer paragraph is still kept.

INV-15 preserved: _is_breaking_subject and _is_breaking_body are byte-identical
functions in both scripts (verified). The no-colon body rule is deliberate and
must not be "tightened" -- the only real break in the live range declares itself
with no colon; bonnyr re-verified and asked that it be preserved.

Verified: shellcheck -S style clean; compute self-test 12/12 rc 0; extractor
17/17 rc 0 (incl. wrapped-prose-inert, folded-subject, no-colon-break, and the
trailer-leak note check); detectors byte-identical; real-range re-run keeps the
#2 break and drops the prose bullet.

Follow-up: #182's detector-parity check greps a single regex; I will update it to
diff these two function bodies so it also guards the awk body detector.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
Follow-up to bonnyr-f5 #179 r4. That PR factors breaking-change detection into
_is_breaking_subject and _is_breaking_body functions -- the body one is a
paragraph-aware awk, not a single grep -- so the old regex-extraction parity
check could not see the awk detector, and bonnyr's nit was right that compute had
no function to diff.

The parity step is now adaptive: when both scripts define _is_breaking_body it
diffs the full bodies of both detector functions (guarding the awk too); on the
pre-#179 tree (inline greps, no functions) it falls back to extracting the
detector regex, so the gate stays meaningful on this PR before the stack merges.

Verified across three states: old tree -> regex fallback passes; new scripts ->
function diff passes and runs the extractor self-test; a one-character mutation
of compute's _is_breaking_body -> exit 1 naming the differing function. YAML +
actionlint clean.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

Follow-up (46d4ec78) to @bonnyr-f5's #179 r4 nit: #179 factors detection into _is_breaking_subject + _is_breaking_body functions (the body one is a paragraph-aware awk, not a single grep), so the regex-extraction parity check couldn't see the awk detector — and you were right that compute had no function to diff.

The parity step is now adaptive: when both scripts define _is_breaking_body it diffs the full bodies of both detector functions (guarding the awk too); on the pre-#179 tree (inline greps) it falls back to the regex extraction, so the gate stays meaningful on this PR before the stack merges. Verified: old tree → regex fallback passes; new scripts → function-diff passes + runs the extractor self-test (17 assertions); a one-char mutation of _is_breaking_body → exit 1 naming the differing function.

@bonnyr-f5

Copy link
Copy Markdown
Collaborator

Review: REVISE

Round 4, cold re-audit of cf64faf against origin/staging 4a52ed4 — whole diff (15 files,
+614/-48, 14 commits), fresh context. I ran real gitleaks v8.30.1 (the pinned digest, verified)
and shellcheck against self-contained sandbox clones with planted secrets, mutated the version
detector, and simulated the CI Gate aggregator under four failure topologies.

BLOCKER (latent) — the new commit-lint gate deadlocks the release promotion (INV-4)

scripts/lint-commit-markers.sh:39 has no exemption for the release bot's own commits. The bot
commits release: vX [skip ci]; the stagingmain promotion range therefore contains a commit
that fails commit-lint → the CI Gate goes red → the release preflight refuses that SHA → main
never releases again.
The gate failure is execution-proven (rc=1 on a release-bot commit
in-range). The promotion topology is inferred from release.yml + the documented branch layout,
and there are 0 release commits in history today, so it is latent rather than firing now — which is
why this is the one finding holding the PR at REVISE rather than BLOCK. It needs an exemption for the
machine identity before the first automated release, or the first promotion after this lands wedges.

Major findings (all execution-proven)

  • ci.yml:244-282 — the paired-detector "INV-15" gate compares regex text, not behaviour. The
    two detectors already diverge in input (subject+body vs body only) and compute has a trigger
    extract lacks, so feat: BREAKING CHANGE … and feat!: … both bump major with empty notes
    while CI prints INV-15 OK. The positive-fixture half is a ::warning:: skip.
  • scripts/lint-commit-markers.sh:30-36,69 — unresolvable range → silent fallback to 1 commit,
    prints the range it never scanned, rc=0 with a violation inside. Sibling secret-scan.sh fails on
    the identical input (asymmetry proven).
  • Makefile:506-513 (claim at :477-482) — local gate diverges from CI on the anti-vacuity
    logic.
    A genuinely broken version detector makes make script-selftests/ci-gates/pre-push
    print FAIL: and exit 0; the CI step exits 1. (The "local gate ≠ CI command" escape class,
    again.)
  • scripts/lint-commit-markers.sh:39 — misses GitHub's documented skip-checks: true trailer,
    which suppresses all gates while commit-lint passes.
  • scripts/lint-commit-markers.sh:58-64 — the rejecter catches 2 of 8 marker shapes. Bullet /
    indented / two-space / blockquote BREAKING CHANGE: all pass lint and ship a spurious major
    publish (8-row table proven).
  • scripts/sync-version-artifacts.sh — add/add conflict with Sync Helm chart tag / appVersion / package.json with VERSION — PR #177 Blocker 2 #180 under squash. Both PRs add
    the file; the repo allows and uses squash; a careless resolution reverts Sync Helm chart tag / appVersion / package.json with VERSION — PR #177 Blocker 2 #180's round-3 blocker
    fixes and operator-chart coverage (merge-tree both strategies + gh api settings proven).

Genuinely fixed, and verified by execution

secret-scan.sh fail-closed on 5/5 harness breaks (bad range, no docker, malformed/missing config,
0-commit range); INV-22 closed (add-then-remove caught, no --no-git left anywhere); --redact
effective so a failing public scan log doesn't leak the secret; --max-archive-depth 2 really finds
keys inside tarballs; full-history baseline clean (74 commits); ShellCheck now covers 41 files (was
23/37); INV-14 closed. The diff adds no credential literals; permissions: contents: read; repo
mounted :ro.

Cross-PR

Besides the #180 add/add above, #182 and #181 both edit release.yml and conflict
(merge-tree → CONFLICT). Resolve on an integration branch; don't land independently.

Review Assessment

  • Verdict: REVISE
  • Audit SHA: cf64faf
  • Cold Audit Performed: Yes
  • Invariants Verified: INV-4 VIOLATED (B1); INV-15, INV-16 (mixed), INV-21, INV-23 VIOLATED; INV-8, INV-9, INV-17, INV-20, INV-22 UPHELD. Proposed new class INV-28 (a gate that forbids a token must exempt the machine identity told to emit it).
  • Git & Harness Cleanliness: Clean

Findings & Action Items

  • Major (Blockers):
    • lint-commit-markers.sh:39: no release-bot exemption → CI Gate red on promotion → main never releases (latent; fix before first automated release)
  • Minor (Non-blocking): ci.yml:244-282 INV-15 gate compares text not behaviour; :30-36,69 silent 1-commit fallback; Makefile:506-513 local≠CI; :39 skip-checks trailer; :58-64 2/8 shapes; sync-version-artifacts add/add with Sync Helm chart tag / appVersion / package.json with VERSION — PR #177 Blocker 2 #180
  • Nits: mcp-tests off the gate; PR body documents removed --no-git; two new scripts disagree on the default range (17 vs 14 commits); BSD sed -i -E leaves *-E files

INV-4 blocker (bonnyr-f5 #182 r4): the commit-lint gate had no exemption for
the release automation's own machine commits. release.yml emits promotion
commits of the form "release: vX.Y.Z <skip-CI marker>", so the staging->main
promotion range carried a commit the gate flagged -> CI Gate red -> release
preflight refuses the SHA -> main never releases again. Reproduced: rc=1 on a
release-bot commit in-range.

Fix: exempt commits whose subject matches '^release: ' (the machine identity),
while still catching a human who quotes a marker anywhere else. Mutation-tested:
a release-bot commit carrying the skip marker now PASSES; a human body/subject
with the same marker still FAILS.

Also folds in the valid same-surface findings from the review:
- fail closed when an explicitly-set RANGE is unresolvable, instead of silently
  falling back to scanning the tip while printing a range never scanned
  (matches secret-scan.sh; rc=1 proven).
- catch GitHub's documented 'skip-checks: true' commit-check trailer, which the
  bracketed fixed-string list missed.
- broaden the spurious-major detector to every shape the version tooling fires
  on (bold, bulleted, block-quoted, indented, multi-separator), not just the two
  it caught; the canonical column-0 footer stays allowed and lowercase stays an
  escape hatch.
- Makefile script-selftests now mirrors ci.yml's anti-vacuity check (fail on a
  printed FAIL: line, not only a non-zero exit), closing the local-vs-CI gap.

21/21 mutation tests green; bash -n + shellcheck -S style clean; PR range still
lints OK.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

@bonnyr-f5 — round-4 REVISE addressed. Head is now fb1e7933. Two files changed: scripts/lint-commit-markers.sh and Makefile.

BLOCKER (INV-4) — release-bot exemption: fixed

Deadlock reproduced first, against the pre-fix script, using the exact promotion topology you describe — a stagingmain promotion range that contains the release bot's own VERSION-bump commit:

RANGE=<base>..<head>   # release-bot commit in-range
::error::commit <sha> (release: v1.2.3 [skip ci]): message contains CI-control marker "[skip ci]" ...
::error::commit-lint failed -- see markers above.
--- exit code: 1  (rc=1 => commit-lint FAILS => CI Gate red => preflight refuses SHA => main never releases) ---

Fix (lint-commit-markers.sh): commits whose subject matches ^release: (the machine identity release.yml emits) are exempt from the whole gate, while a human quoting a marker in any other commit is still caught. Same reproduction against the fixed script:

commit-lint: commit <sha> (release: v1.2.3 [skip ci]) is a release-bot commit -- exempt
commit-lint: OK
--- exit code: 0 ---

Mutation-test (the two you named, plus the full matrix below):

  • release: v1.2.3 [skip ci]PASS (exempt)
  • human commit body/subject carrying the same marker → FAIL

Valid same-surface Minor findings, also fixed

  • :30-36,69 silent 1-commit fallback — an explicitly-set but unresolvable RANGE now fails closed (rc=1, "the scan did not run"), matching secret-scan.sh, instead of silently linting the tip while printing a range it never scanned. A valid-but-empty range (base..base) still passes with scanned 0.
  • :39 skip-checks: true trailer — now caught (^\s*skip-checks:\s*true), case-insensitive; the bracketed fixed-string list alone missed this documented commit-check trailer.
  • :58-64 2/8 BREAKING CHANGE shapes — the opener detector now peels a leading run of markdown/quote/whitespace/bold before judging the line, so bold, bulleted (-), block-quoted (>), indented, and multi-separator shapes all FAIL, aligned to the version detector's BREAKING[[:space:] -]+CHANGE. The canonical column-0 footer (BREAKING CHANGE: / BREAKING-CHANGE:) stays ALLOWED, and lowercase breaking-change stays a valid escape.
  • Makefile:506-513 local≠CI anti-vacuitymake script-selftests now mirrors ci.yml's step: it fails on a printed FAIL: line, not only on a non-zero exit. Proven: a deliberately broken detector makes the target exit non-zero (rc=2) instead of 0.

Mutation matrix — 21/21 green

release-bot exempt · human [skip ci] body/subject FAIL · [ci skip]/[skip actions] FAIL · skip-checks: true (incl. mixed-case) FAIL · bare/bold/bullet/blockquote/indented/two-space/hyphen-space BREAKING CHANGE FAIL · canonical BREAKING CHANGE:/BREAKING-CHANGE: footers PASS · lowercase escape PASS · mid-line prose mention PASS (out of scope, unchanged) · plain feat PASS · unresolvable RANGE FAIL · empty valid range PASS.

Validation: bash -n clean; shellcheck -S style clean; the actual PR range (4a52ed4..HEAD, 15 commits) still lints OK; my own commit lints OK (message uses only hyphenated / mid-sentence marker references).

Deferred with rationale (not fixed here)

CI is running on fb1e7933 (CI workflow in_progress). Not merging.

jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
…iling trailers

Round-5 review follow-up (bonnyr-f5 #179). Four fixes, detectors kept
byte-identical across both scripts (#182 diffs them):

- r5 Major 1: the footer anchor now accepts a marker line preceded by a
  blank line OR by another trailer line, so a stacked footer (the spec's
  own canonical example, no blank between two footers) derives major
  instead of minor. Wrapped prose (marker after a prose line) still inert.
- r5 Major 2: the note extractor captures the whole footer paragraph and
  only strips a TRAILING run of real trailer lines, so 7ece9b0's bullet
  is no longer truncated mid-sentence on an ordinary `word:` prose line.
- r5 Minor 1: the subject detector is bang-only; the folded-footer case is
  caught by running the body detector on the full message (%B), so a docs
  subject that quotes the marker no longer over-bumps.
- r5 Minor 2: the trailer-stop token now allows digit/dot tokens and a
  no-space colon, so an X-Session-1 / Session trailer under a footer is
  stripped rather than leaked into public release notes.

Both scripts read %B (full raw message) in their loops and the compute
consistency guard. Self-tests gain red-green cases for every item; both
suites pass, shellcheck clean, detectors byte-identical.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@bonnyr-f5

Copy link
Copy Markdown
Collaborator

Review: BLOCK

Round 5, cold re-audit of fb1e793 against origin/staging 4a52ed4. Auditor had no prior-round
context. The blocker is re-verified by me below.

Merges standalone: it merges, but it must not. fb1e793 is a linear descendant of 4a52ed4 with
no conflict, and every gate is green over its own PR range (make shellcheck rc=0, version-check
rc=0, secret-scan 16 commits rc=0, commit-lint 16 commits rc=0, script-selftests rc=0). Its
simulated GitHub squash-merge message also passes commit-lint. The problem is what the gate does to
the pipeline after the merge.

Round-4 B1 is fixed — and the same class survives one identity over

The ^release: bot exemption works; I confirmed it. But INV-28's class is "a gate that forbids a
token must exempt the machine identity told to emit it", and there is a second machine identity on
the promotion path: GitHub's own squash-commit composer.

BLOCKER-1 · commit-lint rejects the base commit — the current tip of staging

$ git log -1 --format='committer=%cn <%ce>  parents=%p' 4a52ed4
committer=GitHub <noreply@github.com>  parents=009840a

$ RANGE="009840a..4a52ed4" bash scripts/lint-commit-markers.sh
::error::commit 4a52ed4556... line "- BREAKING CHANGE footer in the BODY of a fix-subject commit → major (non-"
  starts a BREAKING CHANGE declaration that is not a plain Conventional Commits footer
commit-lint: scanned 1 commit(s) in range '009840a..4a52ed4'
::error::commit-lint failed -- see markers above.
rc=1

4a52ed4 is a real GitHub squash commit, it is the current tip of staging, it is the base of
all seven PRs in this series, and it is an ancestor of main. It is also unamendable — the body
was composed by GitHub from the PR description.

The chain: commit-lint fails → ci-gate red → release.yml:150-170 requires the ci.yml run
conclusion to be successexit 1that SHA never releases. The auditor's sweep of the
repo's own 82-commit history found 2 rejected commits (4a52ed4, 7ece9b04); I confirmed the first
directly, and did not independently re-run the full-history sweep.

The gate needs to judge the footer position of human-authored commits, not any line in a
machine-composed body — or exempt commits whose committer is GitHub <noreply@github.com> with a
single parent.

Major-1 · the gate blocks sibling #181 outright

Two of #181's commits carry a literal [skip ci] in the body → rc=1, executed. And if #181 were
squash-merged anyway, the squash body carries the marker → GitHub suppresses all workflows for that
push
→ no CI run, no Release run, and the release is silently starved. That is the #179/#181
incident recreated one layer up, in the place the gate cannot see.

Major-2 · this PR adds a second, older copy of sync-version-artifacts.sh

Both #180 and #182 add scripts/sync-version-artifacts.sh with different content, and #182's is
the pre-fix variant carrying INV-19 verbatim
grep -m1 readers against a global sed writer,
post-verification re-reading with the same first-match helpers, no empty-VERSION guard, no vacuity
floor, no operator chart, no --list. Latent today (one match per pattern), but merge-hazardous:
merge-base(pr182, pr180) = 99a286f, merge-commit is clean, and squash gives an add/add conflict on
that file
plus content conflicts in ci.yml / release.yml / AGENTS.md. This repo allows squash.
#182#181 also conflicts on release.yml under both strategies.

Simplest resolution: drop this file from #182 and let #180 own it.

Major-3 · make script-selftests implements 2 of 4 anti-vacuity assertions

Contradicting its own comment. Breaking the harness rather than the code:

harness break CI job make
silenced guard RED GREEN
early exit 0 RED GREEN
deleted END marker RED GREEN

It also never runs ci.yml's parity step. INV-16: the local target is not the gate it claims to be.

Major-4 · make pre-push is red on a clean staging checkout

Same root cause as BLOCKER-1 — the base commit fails the gate, so the documented local
pre-push command fails on an untouched tree.

Minors

  • Parity gate green while feat!: → major 2.0.0 with empty release notes (executed) — the
    INV-15 class this gate claims to freeze.
  • ^release: is a subject allowlist, not an identity check: a human spoofing that subject is
    exempt from every rule.
  • commit-lint passes a 0-commit range while secret-scan.sh correctly fails it — inconsistent
    vacuity floors between two gates in the same PR.
  • .githooks/pre-push lints @{upstream}..HEAD and so misses non-tip commits on a first push.
  • sed -i -E litters *-E files on BSD.

Verified correct — executed, and I could not break these

The secret scan does catch add-then-remove in-branch (rc=1), so INV-22 is upheld. The private-key
allowlist is rule-scoped: a PAT planted in the allowlisted fixture file is still caught, a key
outside it is caught, the fixture key is suppressed. --max-archive-depth 2 works in git mode
(docs/bundle.tar.gz!inner/id_rsa). Both blindness assertions fire (ERR [git], 0 commits). Image
digest really is v8.30.1. ShellCheck coverage widened to 41/41 tracked shell files — green on this
ref, red on base
, so the two SC2148/SC2034 findings are pre-existing on staging and fixed here.
The CI script-selftests job is red in all four harness-break modes. INV-14 satisfied in the right
direction (ci ignore-set ∅ ⊆ release's; per-SHA groups; cancel-in-progress false only on
main/staging). The CI Gate aggregator's needs-vs-loop is complete, and a non-success changes
now fails the gate.

This is good gate work — the secret-scan and shellcheck halves are the strongest verified pieces in
the series. BLOCKER-1 is a one-condition fix, but it has to land before this merges, because the
moment it does the pipeline stops releasing.

@bonnyr-f5

Copy link
Copy Markdown
Collaborator

Cross-PR merge-order constraints for this series are now tracked in #192.

Relevant here: two items. (1) This PR adds scripts/sync-version-artifacts.sh, which #180 also adds with different, newer content — simplest fix is to drop the file here and let #180 own it. (2) It conflicts with #181 on release.yml under both strategies, and its commit-lint gate rejects two of #181's commits outright.

jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
…staging vacuity gap

Round-5 review (bonnyr-f5 #180).

F1 (Major) — writer/checker asymmetry on the image tag. The --write sed scopes
its tag substitution to the top-level image: block (/^image:/,/^[^[:space:]]/),
but both readers (--check and --write's post-write verify) grepped `^  tag:`
file-global. The two site sets could diverge: a column-0 comment closing the
block early gave a GREEN --check while the next --write HARD-FAILED the release;
a stray 2-space tag: under another key gave a RED --check on a line --write can
never fix. Fixed by lifting the range into one shared IMG_RANGE expression used
by the writer's sed AND both readers (via a new _version_lines helper), so the
tag reader sees exactly the site set the writer touches. Now symmetric:
  - col0-comment shape: BOTH fail (check vacuous-red, write no-op-red)
  - stray-tag shape:    BOTH ignore the out-of-block tag (check green, write green)
  - drifted image tag:  BOTH catch it (check red -> write fixes -> check green)

F2 (Minor) — release.yml "not fully staged" guard is tautological (git diff after
git add is always clean) and had no vacuity floor, so an empty --list would
silently commit a bare VERSION bump with every image pin unsynced (BLOCKER-1
class). Added a `staged >= 5` floor to both Commit-and-tag steps, mirroring the
script's own `total < 5` guard.

F3 (Minor) — the CI version-consistency gate was unreachable from `make pre-push`,
which ci.yml promises is CI-equivalent. Added a `version-check` target and pulled
it into `quick-check` (a pre-push prerequisite).

F4 (Minor) — the header's "the one place that writes them" over-claimed. Narrowed
it: this owns the five release-train image-pin artifacts, NOT frontend-v2/
package-lock.json's root version (npm-owned, desyncs harmlessly) nor the dist/
doc copies (PR #183).

Nits: validate the --write arg against [A-Za-z0-9._+-] (fail fast instead of
corrupting sed); gitignore *.syncbak; correct the AGENTS.md claim that the
release loop reads the skip marker on the subject line (its grep is line-oriented
over the whole message).

F1-b (cross-PR, documented not fixed): #182 carries this PR's first four commits
(the pre-INV-19 66-line script); with squash enabled, squash-merging #180 first
conflicts. Belongs to #182's merge order — land #180 first and rebase #182, or
merge both as merge-commits. Left for #182.

Reproduced red, fixed, mutation-tested green; shellcheck -S style clean; --check
green on the real tree.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
Round-6 review remediation for the conventional-commit breaking-change
detectors shared by compute_version_bump.sh and extract-breaking-changes.sh.

F1 (blocker) + F2 + F7: rewrite _is_breaking_body (byte-identical in both
scripts) around two anchors. A marker after a blank line is accepted with or
without a colon (keeps the real no-colon paragraph break). A marker in the
trailer block -- preceded by another trailer, or folded directly onto a
conventional-commit subject -- is accepted only with a colon. An is_subject
flag on line 1 arms that path for SCOPED subjects too (fix(core): x), which the
r5 trailer regex could not reach because "(" broke the run before the colon, so
a scoped folded footer silently shipped as a patch (F1). The colon requirement
in the trailer block rejects a prose header (Before:/Note:) followed by
colon-less prose (F2). A widened separator class and an optional bullet accept
double-space and "- " bulleted markers (F7).

_breaking_note uses the same start rule so trigger and note never disagree, and
now stops at the first real git-trailer line (capitalized Word(-Word): key)
instead of stripping only a trailing trailer run -- a trailer block followed by
prose no longer leaks a Co-Authored-By address, while a lowercase-prose colon
continuation (migration:) is kept (F5). Marker lines are excluded from the stop
so a hyphen-form marker is never mistaken for a trailer. A trailing CR is
stripped from the note and subject so CRLF messages do not reach CHANGELOG.md.

F3: document the merge-order dependency at the range guard -- its rc=1 is only
effective once #181 drops the call-site "|| true" in release.yml (not owned by
this PR). F4: reword the byte-identity comments to state the property as an
invariant these two files uphold, with the enforcing CI job landing in #182
rather than asserting a job this tree does not contain. F6: remove the
consistency guard as provably dead code (same range, order, predicate and
first-hit break as the bump loop, so its condition is unsatisfiable).

Adds red-green fixtures for F1/F2/F7/F5. Both self-tests pass under mawk 1.3.4
and gawk 5.3.2; detector copies remain byte-identical; shellcheck -S style
clean; SIGPIPE tail fixture shrunk to ~400 lines while still clearing the 64KB
pipe buffer.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
…test parity

Round-5 review (bonnyr-f5 #182). Three isolable fixes; cross-PR items left to
merge-order per the review.

BLOCKER-1 -- commit-lint rejected the current tip of staging (a GitHub-composed
squash commit whose machine-authored body carries a line-start bump declaration).
On a push to staging/main the range is before..tip, so that already-merged tip
was scanned, the ci-gate went red, and release.yml refused to release the SHA --
the pipeline stopped releasing. Adds a second machine-identity exemption
(committer "GitHub <noreply@github.com>", single parent) mirroring the existing
release-bot exemption, so the gate never judges already-merged, machine-composed
history. Human commits never carry that committer identity and are still fully
linted in their own PR. Reproduced (before..tip scan rc=1 -> rc=0) and
mutation-tested: human marker in a PR commit still fails; the squash tip is
exempt.

Major-3 -- make script-selftests mirrored only 2 of ci.yml's 4 anti-vacuity
assertions. Adds the missing two (no PASS line -> silenced/renamed guard;
missing END marker -> early exit / deleted marker). Mutation-tested all three
harness-break modes: each is now make-RED, matching CI.

Minor -- .githooks/pre-push scanned the script default (upstream..HEAD) and so
missed non-tip commits on a first push. Now derives the exact pushed range from
git's pre-push stdin protocol (remote..local), falling back to the default for a
brand-new branch or a manual run. Tested all stdin cases.

Cross-PR (documented, not forced): sibling #181's own commits carry markers --
caught in #181's PR; once squash-merged the new exemption stops the gate
re-scanning them (Major-1). The duplicate scripts/sync-version-artifacts.sh and
its sed -i -E are #180's file under merge-order, not duplicated here (Major-2).

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

@/tmp/claude-1000/-mnt-d-project-bnk-forge/01a289e5-e845-497f-99d9-622be0bf9a98/scratchpad/comment.md

jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
…y recovery, operator path)

F1 (Major): the immutable-tag existence probe classified by grepping the docker
CLI's error text, which cannot separate "package does not exist yet" (first
release in a fork/mirror namespace — safe) from "no permission" (must fail
closed): both print `denied`. Replace it with scripts/registry-tag-probe.sh,
which classifies each image by the registry's HTTP status (200 exists, 404
absent, 401/403 auth, else transient). Only a definitive 404 is "safe to
publish"; auth/network/rate-limit/5xx fail closed. A broken/partial probe run
now also fails closed instead of reading as "no manifests found".

F2 (Major): add a sign_only recovery mode (workflow_dispatch input, requires
publish_only). It re-runs cosign sign + SBOM + provenance against the
already-pushed :VERSION digests without rebuilding or re-pushing, so the most
likely recovery — signing failed after all images were pushed — no longer forces
the operator into force=true, which rebuilds to possibly-different digests and
moves the immutable tag.

F3 (Major): the operator `make push-images` path had a recency guard but no
existence probe, so VERSION == highest tag (a fresh main right after a release)
silently overwrote the signed release images. It now runs the same
registry-tag-probe.sh and refuses on exists/inconclusive unless FORCE_LATEST=1,
fixing the class at every call site.

F5 (Minor): the publish-critical-section recency guard failed OPEN when the tag
fetch failed — it moved :latest on an unverified guess. Capture the fetch
result; on failure a final/manual release goes red and a publish_only republish
emits :VERSION without moving :latest.

F6 (Minor): single-source the 7-image list in registry-tag-probe.sh; the CI
probe and the publish summary now read it, and a self-test asserts parity with
the docker-bake.hcl default group.

F7 (Minor): docker-bake.hcl stamped an empty, spec-invalid image.created label
when CREATED was unset (a plain make push-images). Emit the label only when
CREATED is set, matching the ROLLING_TAG conditional; docs/DOCKER.md updated.

F4 (cross-PR, documented): softened the release-notes comment that claimed
extract-breaking-changes.sh stays fail-closed. On this ref that script still
ends its range query with `|| true`; the fail-closed fix lands with #179, which
owns the script (this PR does not touch it).

F8/F9 (cross-PR/documented): the release.yml conflict with #182 and the absence
of workflow-shell linting in CI are noted for the reviewer; #182's paths-ignore
characterization in the existing comment is accurate (verified against pr182).

Tests: scripts/tests/registry-tag-probe.test.sh mutation-tests the four required
outcomes (absent->publish, no-permission->refuse, network->refuse,
exists->refuse) plus first-publish and image-list parity.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
Keep #182's CI-hygiene jobs (shellcheck, gitleaks secret scan, and the
commit-lint gate that runs scripts/lint-commit-markers.sh) and #180's
round-3+ scripts/sync-version-artifacts.sh (unchanged, newer than #182's copy).

Resolve two comment conflicts present-tense for the merged tree:
- release.yml: ci.yml no longer carries a push paths-ignore, so it runs CI on
  every push to main/staging — a strict superset of the pushes that reach
  Release, guaranteeing the preflight SHA poll finds a matching CI run.
- AGENTS.md: the CI-control-marker rule is now enforced (commit-lint gate +
  pre-push hook), and the release loop guard reads the deliberate skip marker
  on the subject line; kept #182's BREAKING-CHANGE-footer bullet.
jgruberf5 pushed a commit that referenced this pull request Aug 21, 2026
Consolidated landing of seven interdependent PRs whose shared credential and
release/CI surfaces prevented merging in any order (see issue #192's conflict
matrix). Merged in dependency order 179, 180, 181, 188, 186, 183, 182; the
#186/#188 credential surface was reconciled once (single reserved-name guard;
provenance + migrations + stale-disable combined with rotation + backend MCP
wiring + threadpool). Squashed to one commit; per-PR history retained on the
seven archived branches.

Validated on the merged tree: ruff + mypy clean; 172 auth/credential/startup/
migration tests pass; single alembic head v2_155; openapi + frontend types fresh;
helm lint/template and docker compose config green on all modes; version and
detector self-tests green; commit-message lint clean.

Closes #192.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4
@jgruberf5

Copy link
Copy Markdown
Collaborator Author

Superseded by the consolidated integration PR #193 (branch pr177-integration), which merges this and the other six #177 follow-up PRs in the dependency order from #192 and passes full CI (CI Gate green). Per #192, these seven share the credential and release/CI surfaces and could not merge in arbitrary order, so they land together via #193.

Closing unmerged, not abandoning: the branch is retained and all review history stays on this page for reference. See #193 for the integrated, validated result and #192 for the cross-PR conflict analysis.

@jgruberf5 jgruberf5 closed this Aug 21, 2026
bonnyr-f5 pushed a commit that referenced this pull request Aug 24, 2026
)

* Integrate the #177 follow-up stack (#179 #180 #181 #182 #183 #186 #188)

Consolidated landing of seven interdependent PRs whose shared credential and
release/CI surfaces prevented merging in any order (see issue #192's conflict
matrix). Merged in dependency order 179, 180, 181, 188, 186, 183, 182; the
#186/#188 credential surface was reconciled once (single reserved-name guard;
provenance + migrations + stale-disable combined with rotation + backend MCP
wiring + threadpool). Squashed to one commit; per-PR history retained on the
seven archived branches.

Validated on the merged tree: ruff + mypy clean; 172 auth/credential/startup/
migration tests pass; single alembic head v2_155; openapi + frontend types fresh;
helm lint/template and docker compose config green on all modes; version and
detector self-tests green; commit-message lint clean.

Closes #192.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193): reconcile the merged unset-MCP-password model and harden the credential seed

Addresses mwiget's three CHANGES_REQUESTED blockers on the #177 integration PR
(#193), plus follow-up findings from a max-effort review of the same credential
surface. The merge chose #188's "unset MCP_SERVICE_PASSWORD -> disable" over
#186's "unset -> generate", so the generate/rotate-on-unset code was left
unreachable but still documented, and the release-notes footer was missing.

BLOCKING 2 (core/config.py): the MCP_SERVICE_PASSWORD comment claimed
ensure_service_user "generates a random secret and surfaces it once" when unset,
contradicting the merged behaviour. Rewrote it to state the truth: when unset (or
a known default) seed_auth_step calls disable_stale_service_user, leaving the mcp
account disabled/unavailable until an operator configures a real password; a
published default is refused and rotated out; the backend receives
MCP_SERVICE_PASSWORD on every deploy mode and reconciles to it when set. Also
removed the duplicate #186 block that sat above the wrong field.

BLOCKING 3 (services/auth_service.py): removed the unreachable usable_password is
None branches from ensure_service_user (the generate-on-create and
rotate-on-unset paths). seed_auth_step only calls it under the _mcp_pw_usable
gate, so password is never None/default in production. ensure_service_user now
requires a usable password and only creates/reconciles with it (failing closed
and loudly if handed an unusable one); the unset case is owned entirely by
disable_stale_service_user. Dropped the now-dead _log_generated_service_password
helper and the service-account token_urlsafe/_persist_generated_password calls
(_persist_generated_password is still used by the admin seed). Kept the
reserved-name guard, the provenance check, the adopt-a-published-default
remediation, and disable_stale_service_user fully intact. Updated the affected
unit tests (published-default/None now refused; added a reachable
adopt-and-reconcile test; stale-row setup builds the legacy row directly) and
fixed scripts/mcp_live_smoke.py, which pointed operators at
/app/keys/initial_mcp_password, a file no reachable path writes.

CR-1 (services/auth_service.py, seed_admin_user): fixed a concurrent-first-boot
admin lockout. With DEFAULT_ADMIN_PASSWORD unset and 2+ api replicas, both
generated different passwords and the loser overwrote the keys file while its
INSERT rolled back, so the file and the committed row disagreed. The fresh seed
now creates+flushes first (the loser's INSERT raises IntegrityError -> rollback,
no file write) and persists the keys file only after winning but before commit,
so the file can only ever hold the committed row's password. Added a
losing-replica test.

CR-5 (services/auth_service.py, _persist_generated_password): os.open's 0600 mode
only applies on create, so a pre-existing 0644 file was truncated in place and
kept 0644, writing the secret world-readable. Added os.fchmod(fd, 0o600) and a
test that a pre-existing 0644 file is tightened to 0600.

CR-2 (routes/k8s_websocket.py, dpus_websocket.py, benchmarks.py): the WS auth
validators called the blocking sync token_user_state directly on the event loop.
Moved it off the loop via run_in_threadpool, matching core/auth_middleware.py.

Validation: make lint-backend clean; mypy core/ schemas/ unchanged; auth
(57) + ws/benchmark/startup (79) suites pass; alembic heads single v2_155;
helm lint/template OK and --set secrets.mcpPassword=changeme fails the render;
docker compose config OK on all modes; extract-breaking-changes and
compute_version_bump self-tests pass; lint-commit-markers clean.

BREAKING CHANGE: MCP_SERVICE_PASSWORD is now required — the backend SystemExits under ENVIRONMENT=staging|production when it is unset or a known default; the shipped admin/changeme default is removed and rotated out on upgrade; the dist bundle renames MCP_USERNAME/MCP_PASSWORD to MCP_SERVICE_USERNAME/MCP_SERVICE_PASSWORD; and the Helm chart ships mcpUsername=mcp with a generated mcpPassword instead of admin/changeme.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193): close bonnyr-f5's credential/backend/helm/dist/docs blockers

Follow-up to the #177 integration on pr177-integration, addressing the
CREDENTIAL/BACKEND/HELM/DIST/DOCS half of bonnyr-f5's BLOCK on PR #193.

B1 (SECURITY): ensure_service_user no longer adopts any human account whose
password is a known default. The adoption exception is now scoped to v2_155's
exact backfill fingerprint (username=='mcp' AND email=='mcp@bnk-forge.local'),
matching the migration's own conservative rule, and must_change_password is no
longer cleared on an adopted row. Adds tests proving a human operator/changeme
row (and a wrong-email mcp row) is REFUSED, not taken over.

B2/B2b: all five compose files (root + dist docker-compose{,.local}.yml and the
IBM embedded compose) honor legacy MCP_USERNAME/MCP_PASSWORD as aliases for
MCP_SERVICE_USERNAME/MCP_SERVICE_PASSWORD on both backend and mcp env, so an
existing customer .env keeps working after upgrade. Docs (dist/README.md,
dist/.env.example, user-pack/install-guide.html) document MCP_SERVICE_PASSWORD as
canonical with MCP_PASSWORD honored as a legacy alias.

B3: ENVIRONMENT is plumbed to the backend in every compose file, so an operator
who sets ENVIRONMENT=staging|production actually reaches config.py's MCP
fail-fast. Helm already routes ENVIRONMENT=production onto api/worker/beat.

M1: the unset-MCP behavior stays "disabled" (#188 over #186); added an explicit
deliberate-consolidation comment at the decision point.

M2: disable_stale_service_user skips the about-to-be-reconciled row and the
"no usable MCP_SERVICE_PASSWORD" warning is conditional on a configured password,
so a correctly-configured install no longer logs a false warning or commits an
inactive MCP window on every boot.

M7: the Helm mcp deployment and the IBM installer's mcp service now run the exec
auth-probe healthcheck (python -m bnk_forge_mcp.healthcheck) instead of a bare
tcpSocket probe.

M8: the chart fails the render for a reserved mcpUsername (admin), mirroring the
mcpPassword guard, and NOTES.txt/values.yaml call it out.

Minors: deterministic checksum/secret via a shared helper (stable across renders,
identical across api/worker/beat/mcp); vestigial _persist_generated_password
filename docstring; false "backend generates its own secret" rationale corrected
in compose/helpers/ibm; mcp_live_smoke.py #186/#188 attribution; e2e/config.py
changeme default note; .env.example :58/:73/:94 fixes; unified BNK_FORGE_VERSION
to latest across dist.

Validation: ruff clean; typecheck-backend (core/ schemas/) Success 38 files;
199 auth/credential/startup/ws/migration tests pass incl. new B1 tests; helm lint
+ template stable checksums, --set secrets.mcpUsername=admin and
secrets.mcpPassword=changeme both FAIL; docker compose config on all five modes
shows the backend receiving MCP_SERVICE_PASSWORD (via either alias) and ENVIRONMENT.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193): close bonnyr-f5 CI/RELEASE/SCRIPTS blockers, majors, and minors

Blockers:
- B4 registry-tag-probe.sh: replace GNU-only BRE alternation (\|) with
  portable `sed -nE (access_token|token)` so the token parse works on BSD/
  macOS sed; on BSD the empty token classified every image `unknown` and
  routed the operator into FORCE_LATEST=1, overwriting immutable :VERSION
  manifests (INV-24). Reproduced under `sed --posix`, fix proven to parse T2.
- B5 registry-tag-probe.test.sh had no caller and was red on BSD. Both the
  Makefile script-selftests target and ci.yml's script-selftests job now
  enumerate and run every scripts/tests/*.test.sh, failing on an empty
  enumeration or any non-zero rc.
- B6 lint-commit-markers.sh: replace the spoofable committer-identity
  exemption (GitHub <noreply@github.com> + single parent) with an
  unspoofable "already reachable from origin/main|origin/staging" check;
  lint the PR title (PR_TITLE via env) on pull_request events; split the
  rules so machine/already-merged is exempt for the marker rule but the
  spurious-major rule always applies.

Majors:
- M3 release.yml overwrite guard: derive the vacuity floor from an
  independent source (docker-bake.hcl default group, sourced from the
  workflow-ref tooling) and assert the probe's exit status before trusting
  its output, so an unavailable probe fails closed instead of "safe".
- M4 (INV-31): generate release notes and run the registry existence-probe
  BEFORE the irreversible push in release-final/release-manual (new shared
  scripts/registry-overwrite-guard.sh); release-publish keeps its own
  in-critical-section re-check.
- M5 make script-selftests now runs the INV-15 detector-parity diff
  (extracted to scripts/tests/detector-parity.test.sh) so local == CI.
- M6 extractor self-test runs unconditionally with anti-vacuity assertions
  (ok lines + END marker), no longer gated on grepping its own --self-test.

Minors: stale cross-PR comments in release.yml and extract-breaking-changes.sh;
removed the duplicate Makefile version-check target; `git add dist/VERSION`
no longer swallows failures; first-ever-release notes range fixed; CHANGELOG
insertion asserts a non-no-op before committing; refreshed .trivyignore
CVE-2026-7598 review deadline; removed e2e-tests.yml dead `|| true`; documented
the new Docker dependency in the pre-push hook.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r2): close credential/backend/helm/dist blockers — drop the username alias, make ENVIRONMENT=production satisfiable, and fix the mcp liveness probe

bonnyr-f5 round-2 BLOCK, credential/backend/helm/dist/docs half.

B-1 (INV-12): the compose files aliased the SERVICE username
(MCP_SERVICE_USERNAME:-${MCP_USERNAME:-mcp}); a legacy .env with MCP_USERNAME=admin
resolved it to `admin`, and against the guardless image `latest` still points at,
the old ensure_service_user rewrites the human admin row to `changeme` every boot.
Drop the username alias across all five compose files + the ibm embedded compose
(keep the harmless password alias), and default BNK_FORGE_VERSION to 4.0.0 (the
release this tree becomes, first image with the guards) instead of `latest`, so a
compose file can never hand the new credential contract to a pre-guard image.

B-2: ENVIRONMENT=production reaches validate_production, which also gates on
JWT_SECRET_KEY / ENCRYPTION_KEY / ALLOWED_ORIGINS — none of which were deliverable
from a compose install, so the switch bricked the backend. Plumb all three into
every x-backend-env anchor (four compose files + ibm) and document them in the
env examples; treat an empty ("" from ${VAR:-}) key as unset in config.py so the
plumbed empty default auto-generates rather than passing as a real empty key.
_persist_or_load_key now flags only keys WE generated as auto_generated (sidecar
.autogen marker), so an operator-provisioned key on the volume validates while a
fresh prod boot still fail-fasts permanently.

M-4: mcp liveness returned non-zero when the BACKEND was unreachable, so k8s
restarted the pod for a dependency outage. Move the auth-probe to readiness only;
liveness is tcpSocket. Fix mcp-server/README + mcp_live_smoke hints to name only
the vars each process actually reads (container: BNK_FORGE_*, backend: MCP_SERVICE_*).

Minors: refuse a known-default DEFAULT_ADMIN_PASSWORD on fresh seed + helm
adminPassword fail-guard; guard secrets.yaml mcpUsername nil with kindIs "invalid";
make the Python reserved-name check case-insensitive/trim to match Helm; neutralise
the hash when disabling a stale service account; correct the benchmarks.py
JWT-gate comment; surface an empty MCP_SERVICE_PASSWORD in install.sh; fix the
.env.example "No .env file is needed!" contradiction.

Tests: config B-2 satisfiability + provenance-marker tests; seed_auth_step against
an admin-still-holds-changeme upgrade DB; case-insensitive reserved-name and
disable-hash-neutralisation cases. ruff clean, mypy clean, 4831 unit pass, helm
lint/template green, docker compose config verified on all modes.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193): close r2 CI/RELEASE blockers B-3, M-1..M-3 + scripts/release minors

B-3 (commit-lint exemptions): key the already-merged exemption on the range
BASE (github.event.before), not the post-push tip, so a new skip-CI marker on a
push to main/staging is caught while a genuinely-already-merged base commit stays
exempt; replace the self-settable `^release: ` subject exemption with
release.yml's own version+trailing-skip fingerprint.

M-1 (spurious-major rule): redefine rule 2 as the exact complement of the
detectors, sourced from the shared predicate, so it flags only a marker the
detectors would MISS (never dash-bullet, markdown-bold or indented shapes);
give it the same already-merged exemption; and lint inputs.release_notes through
the script before it becomes a release commit/tag.

M-2 (overwrite guard floor): derive the vacuity floor from `docker buildx bake
--print default | jq '.group.default.targets | length'`, scoped to the default
group, so a second bake group no longer wedges the release; separate bake-file
parse failures from registry-unreachable in the messaging. Single-source the
policy: release-publish and make push-images now call the one guard.

M-3 (portability): drop bash-4 mapfile from the probe test; rebuild the compute
self-test newline expansion with awk to avoid the bash-3.2 parameter-expansion
cliff, so make script-selftests runs under stock macOS bash 3.2.

Detector single-sourced into scripts/lib/breaking-change-detect.sh (compute,
extract, lint all source it); detector-parity test asserts the wiring; added
mutation tests for the lint rules and the overwrite guard.

Minors: fix version-consistency misdiagnosis of a column-0 YAML comment; correct
the compute/extract parity docstrings and the docker-bake four-push-paths note;
wire artifact-network-self-test into ci-gates; make the pre-push hook migration
message reachable under set -e; omit the false provenance buildStartedOn; filter
the release CI-status poll by commit SHA.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193): seed the re-enable-guard test's default-hash row directly

The round-2 disable_stale_service_user hash-scrub (bonnyr-f5 #193 minor) collided
with the re-enable guard's own regression test: _seed_disabled_default_mcp built
its "disabled while holding the published default" state BY CALLING
disable_stale, which now scrubs the hash -- so holds_known_default_password was
false and the PUT re-enable was allowed (200) instead of refused (400).

The guard defends a row taken inactive by a path that LEAVES the credential
intact (a manual operator PUT), not one disable_stale scrubbed. Seed that state
directly (set is_active=False on the default-hash row) so the guard's real
scenario is exercised; assert the default hash survives the seed. Corrected the
now-stale guard comment in routes/auth.py that still claimed disable "only flips
is_active". Neutralisation and its asserting tests are unchanged.

Verified: TestServiceAccountReEnableGuard 2/2 pass; the three affected auth files
(test_startup_seed_auth, component/test_auth_service, integration/test_routes_auth)
97/97 pass; ruff clean.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r3): fail-close key provenance (B-2) + credential/auth minors

Own the round-3 CREDENTIAL/AUTH findings.

B-2 (BLOCKER): _persist_or_load_key classified a marker-less key file as
operator-provided, so every upgrade keys volume (key present, no marker) let
SEC-006's fail-fast pass OPEN on an auto-generated JWT/ENCRYPTION secret in
production. Invert the sentinel: a marker-less key now classifies AUTO-GENERATED
(fail closed); an operator asserts provenance with an explicit <filename>.operator
opt-out marker. No marker is written on generation, which also removes the second
trigger (a partial marker write can no longer downgrade provenance). Regression
tests seed the PREVIOUS-RELEASE on-disk shape and assert validate_production
raises under ENVIRONMENT=production.

Minors:
- Write the key via os.open(0o600)+fchmod so the secret is never briefly 0644.
- Single-source the MCP known-default denylist: delete the local tuple in
  auth_service and use core.config.MCP_KNOWN_DEFAULT_PASSWORDS (comment notes the
  helm copy is deploy-owned).
- Correct holds_known_default_password docstring (disable_stale now scrubs the
  hash; this guard covers the other disable paths).
- Reconcile ENCRYPTION_KEY docs with reality: the keys-file is the source of
  truth for at-rest crypto; the env var only drives the production gate
  (encryption.py comment + .env.example).
- Clarify the v2_155 custom-username remedy in disable_stale docstring.

Test-gaps:
- Normalise the service username (trim/casefold) at the reconcile lookup and the
  disable skip filter, so " mcp "/"MCP" reconciles the existing mcp row instead of
  minting a second service account and disabling the live one.
- Honest seed_admin_user log/logic under DEFAULT_ADMIN_MUST_CHANGE=false (no more
  "must change on first login" when no gate was applied).
- disable_stale_service_user(skip_username=...) leaves the live row wholly
  untouched (no inactive window), variant included.
- db.commit() failure after the keys file is written leaves a retriable state
  (published default still authenticates, orphan file password does not).

All owned-suite tests pass; ruff clean. Each fix reproduced then mutation-tested.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r3): close the release/CI blocker + major + every release/CI minor

M-6 (blocker): commit-lint no longer reds unamendable merge history.
- rule 2 now flags ONLY a mis-anchored DECLARATIVE marker (the colon form);
  a colonless marker-shaped PROSE line the detectors treat as inert (an
  already-merged body such as "- <MARKER> footer in the body ...") is no
  longer flagged, so the push-to-main range (before..head, which INCLUDES the
  PR merge-base) goes green without a history rewrite. Detection of a real
  mis-anchored marker is unchanged.
- deleted the already-merged exemption as dead code: base..head excludes the
  base by construction, so no scanned commit can ever be an ancestor of it.
  Removed the BEFORE derivation, _already_merged, tests B3.2/M1.5, and the
  ~20-line header claim. The release-bot exemption stays.
- rule 2 now scans the whole body via _under_detected_markers and reports
  EVERY mis-anchored marker, not just the first.

M-7 (major): secret-scan no longer false-fails a delete-only range. A
delete-only commit has rev-list count > 0 but gitleaks scans 0 (it scans
added content), so the count-based backstop is replaced by a range-
resolvability check plus gitleaks' exit status.

Release/CI minors:
- release-bot fingerprint single-sourced to scripts/lib/is-release-bot-subject.sh;
  release.yml's inline copy byte-locked by a parity self-test; dropped the
  false unforgeability claim and documented the residual honestly.
- registry-overwrite-guard: added a fail-closed default arm for an
  unrecognised/empty probe status (+ malformed/empty test scenarios).
- Makefile push-images: FORCE_LATEST now overrides ONLY the recency guard; a
  new FORCE_OVERWRITE overrides ONLY the immutable-tag guard; fixed the
  missing-jq remediation text.
- registry-tag-probe: the network arm now matches the real doubled "000000"
  curl-failure shape (was dead code); test fixture reproduces it.
- INV-15: single-sourced the marker regex (one canonical value + a
  detector-parity assertion that every embedded copy is byte-identical).
- release.yml Publish summary counts what buildx actually pushed (bake
  --metadata-file), not the static target list.
- registry-tag-probe test enumerates the bake DEFAULT group (scoped), matching
  the guard's enumeration.
- added scripts/tests/secret-scan.test.sh (fake-docker mutation suite).

release.yml: added a post-push step running scripts/verify-image-pins.sh so a
release cannot complete while shipping an unpublished image pin (script owned
by the deploy agent; referenced by path from .release-tooling).

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r3): single-source every deploy version pin + close deploy majors/minors

B-1/B-1b: dist/docker-compose{,.local}.yml, dist/.env.example and
ibm_cloud_bnk_forge.sh hard-pinned 4.0.0, a tag that has never been published,
while Helm pinned 3.1.6 -- two shipped paths naming two versions, one of which
does not exist. Bring all of them under sync-version-artifacts.sh (new PIN/
DISTENV readers+writers, --check, --list) so every pin derives from VERSION
(3.1.6, which exists) and the release re-stamps them atomically via the existing
--write $NEW; drop the dist/ disclaimer. Add scripts/verify-image-pins.sh (+
selftest) that resolves every shipped compose image: pin against the registry and
fails on manifest unknown, wired post-push in the release job.

M-1..M-5: reword NOTES/compose/chart comments that asserted post-guard behaviour
as already-true on the pre-guard pinned image (they land with the guard-carrying
release); NOTES now leads with the required ALLOWED_ORIGINS override; dist/README
and the install guide stop recommending latest/3.1.6 and the keys-file cat the
pinned image does not write; install.sh strips quotes and rejects the known-
default MCP passwords so the "MCP not active" warning fires instead of a green
lie; add deploy-version-lockstep + helm-known-defaults-lockstep selftests.

Deploy minors: MCP_USERNAME "do not set" made consistent across docs/Makefile;
chart ENCRYPTION_KEY now emits a valid Fernet key; DEFAULT_ADMIN_* added to the
ibm P3 backend env; secrets.mcpUsername defaults to "mcp" on null; ibm mapfile ->
portable while-read.

Verified: sync --check exit 0; --write round-trip moves every pin and restores;
helm lint/template clean (default + origin override); script selftests green;
bash -n + shellcheck clean.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r4): unify the at-rest encryption key (B-3), fail-close key marker (M-1), row-matches-client (M-2) + credential minors

B-3: validate_production gated ENCRYPTION_KEY while a second, independent generator
in core/encryption.py produced the real at-rest Fernet key unchecked — setting
ENCRYPTION_KEY (the documented remedy: token_hex(16), not even a Fernet key) turned
the gate green while encryption auto-generated a different key. Unify: one key file
(_encryption_key_path == core.encryption.ENCRYPTION_KEY_FILE), one generator. When
ENCRYPTION_KEY is set it is VALIDATED as a real Fernet key (fail clearly if not),
written to that file with a .operator marker, and consumed by core.encryption and
services.backup_service; the provenance flag reflects the value that actually
protects data. Never clobber an operator-marked key on a mismatch. config.py:319
and .env.example now print the Fernet recipe.

M-1: _persist_or_load_key required a regular-FILE marker (os.path.isfile, not
os.path.exists — a directory no longer counts) and treats "marker present, key file
absent" as a provisioning error: generate but do NOT persist, so the stale-marker
rotation gesture can never heal into auto=False on the next boot.

M-2 (regression this PR introduced): ensure_service_user normalised the username
before lookup/create, so MCP_SERVICE_USERNAME=MCP seeded 'mcp' while the client
sends the raw 'MCP' and authenticate_user matched exactly -> login denied. Create/
reconcile under the RAW value (what the client sends); the disable_stale skip keys
on the same raw value; only the reserved-name guard normalises. Fixed the false
"Matches the Helm chart lower|trim" docstring.

Credential minors: non-vacuous localhost-CORS test (valid MCP password so only the
CORS branch fails) + wildcard is now an exact origin-list entry, not a substring;
new DPU-websocket must-change/unresolvable-user tests mirroring the k8s twin;
middleware unresolvable-JWT-subject-refused and exact-vs-suffix exempt-path tests;
corrected the denylist copy count (4th copy in dist/install.sh); corrected v2_155's
rationale (v2_154 is new in this diff, not "already shipped"); documented why
ensure_service_user's adoption branch is kept.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r4): close deploy-surface blocker + majors (B1, M7-M12) and deploy minors

B-1: a fresh install of the pinned 3.1.6 bundle seeded an admin nobody could log
in as. `${DEFAULT_ADMIN_PASSWORD:-}` delivers the var present-and-empty; 3.1.6's
`DEFAULT_ADMIN_PASSWORD: str = "changeme"` is then overridden by "" and the login
schema rejects "" (422). The prescribed `${VAR}` form does NOT omit on this compose
(map interpolation still renders ""); the working omit-when-unset form is a map
entry with NO value (passthrough / `docker run -e KEY` semantics). Converted
DEFAULT_ADMIN_PASSWORD to passthrough across all four compose paths (dist base+local,
root base+local, IBM embedded). Swept the class: JWT_SECRET_KEY/ENCRYPTION_KEY also
converted (3.1.6 uses `if KEY is None`, so "" is used literally, not auto-generated);
MCP_SERVICE_PASSWORD kept as `${...:-...}` because omitting it restores 3.1.6's known
default "mcp-service-changeme". Added a commented DEFAULT_ADMIN_PASSWORD block to
dist/.env.example. Verified via `docker compose config` + real container env both
directions (unset -> omitted; set in .env -> forwarded).

M-7: chart checksum/secret did not change on mcp-password rotation (3 renders, 3
generated passwords, one identical checksum). Made all generate/rotate fallbacks
deterministic (deriveSecret, release-seeded) so the Secret is stable across renders
and includes, and hash the RENDERED Secret so the annotation tracks every resolved
value. Now stable across renders, identical across the 4 deployments, and it flips
when any resolved value changes.

M-8: dist/install.sh credential guard failed open on `changeme ` (whitespace). Trim
leading/trailing whitespace around the quote-strip before the known-default compare.

M-9: removed newly-added forward-dated 4.0.0 prose (install-guide.html x2, DOCKER.md).

M-10: default helm install crashlooped (production + localhost). Added a render-time
guard mirroring backend validate_production (fail on wildcard under staging/production,
localhost under production); defaulted ALLOWED_ORIGINS to empty so the bare render
boots (empty is neither wildcard nor localhost). `helm lint` and bare `helm template`
stay green; the guard fires with a clear message on a real fatal posture.

M-11: portable in-place sed in the IBM installer (`sed -i.bak … && rm`), both sites.

M-12: dist/ no longer ships published default DB/redis creds on host networking.
install.sh generates strong POSTGRES_PASSWORD/REDIS_PASSWORD on the fresh .env (like
Helm/IBM); .env.example ships them empty; a pre-existing default triggers a warning.

Deploy minors: extended deploy-version-lockstep.test.sh to the bnk-operator chart
(appVersion + image.tag) and dist/VERSION; brought dist/VERSION under
sync-version-artifacts.sh (--check/--list/--write green); reworded install.sh's MCP
UNHEALTHY assertion to match what the pinned image actually reports; added
scripts/tests/ibm-compose-drift.test.sh freezing the credential/hardening env so the
IBM embedded compose and dist/docker-compose.yml cannot silently diverge.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r4): make the pin verifier reachable pre-push, de-vacuum the parity/self-test harnesses, and close the release minors

B-2 (blocker): scripts/verify-image-pins.sh could never pass where the release job
runs it — from the 4-file sparse .release-tooling checkout that holds no compose
file (ROOT resolved there), with REGISTRY/VERSION handed in as env vars the script
reads only as flags, and the whole step wired AFTER the tag/Release/push/signing.
Fixes, end to end:
  - add a consistency mode (--expect-version) that asserts every shipped first-party
    pin already renders to $NEW without a registry probe, and run it as the PRIMARY
    PRE-push gate in release-final and release-manual (before anything irreversible);
  - fix the post-push existence step to pass REGISTRY/VERSION as FLAGS and the compose
    files explicitly by --file (they live at the tag checkout at the workspace root),
    keeping it as a secondary confirmation;
  - widen the default file set to include the IBM Cloud installer's embedded compose;
  - add a dryrun-release-tooling job that rebuilds the exact publish-job layout and
    exercises both invocations against a fake probe, and gate release-publish on it,
    so a step that cannot execute is caught before it is wired ahead of a signature.
  The sibling registry-tag-probe.sh / registry-overwrite-guard.sh read nothing
  $ROOT-relative outside the sparse set, so they are unaffected.

M-3: detector-parity.test.sh enumerated the marker copies with the very token that
drifts, so a copy that drifted in the token vanished from enumeration (drifting
:96-97 dropped the count 5->3 yet stayed green). Enumerate by position/count instead:
an exact per-file canonical count plus a stable-anchor site scan that flags any
drifted site even under a compensating add.

M-4: the filesystem self-test loop checked only a non-empty enumeration and each
file's exit 0, so a test gutted to a no-op passed and deleting 7 of 8 stayed green.
It now requires each file to emit PASS lines, no FAIL line, and an ALL PASS terminal
marker, plus a count floor derived from git's tracked *.test.sh set (detector-parity
was conformed to that output convention).

M-5: release-rc created and pushed the RC tag before the fail-closed notes step;
the tag is now created locally, notes generated, then the tag pushed.

M-6: added mutation-tested coverage for this PR's four previously-uncovered lint
fixes (the PR-title lint, the pending-message lint, the RANGE fail-closed branch,
and the skip-checks trailer rule).

LEAD: the anti-vacuity staging floor derived the count from a stale literal while
--list grew to 8 paths; both sites now derive it from --list and require every listed
path to stage, and the stale comments are corrected.

Release minors: scope the release-bot commit-lint exemption to the range tip (a
forged release subject buried mid-range is no longer exempt) and add a REACHABLE
published-history exemption anchored to the last release tag so a mis-anchored marker
in unamendable history cannot red the release; add fixtures for the untested registry
probe/guard arms (5xx, unexpected code, unknown status, bake-parse failure);
ancestry-filter the LAST_FINAL tag queries so a tag on another branch cannot skew the
notes range; reconcile the empty-RANGE handling (commit-lint now fails closed on an
explicit empty RANGE, ci.yml leaves it unset); give the pre-push hook a clear message
and a fetch fallback when the remote tip is absent locally; derive the cosign
verify-identity org from REGISTRY instead of hardcoding it.

Flagged: gate Makefile push-customer-build through registry-overwrite-guard.sh, the
last documented push path that was still unguarded.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r4): keep M-7 checksum-tracks-rotation WITHOUT predictable secrets

The r4 deploy fix closed M-7 (checksum did not move on a secret rotation, so pods
never rolled) by making the generated fallbacks deterministic -- deriveSecret =
sha256(Release.Name|Namespace|fullname|purpose). Every input is public (they appear
in resource labels and the chart source), so that made the JWT signing key, the
at-rest Fernet key and the admin/mcp passwords COMPUTABLE by anyone who can read a
label -- a forge-any-token / decrypt-all-secrets exposure, strictly worse than the
cosmetic churn it fixed.

Revert generation to randAlphaNum (unpredictable) and instead hash the DETERMINISTIC
inputs that determine the Secret -- values.secrets, the persisted .data (reused via
lookup), plus a per-credential "rotating-from-default" marker for admin/mcp whose
persisted value is a known published default. That tracks every rotation (operator
edit, persisted-value change, rotate-away-from-default) so the pods roll, is stable
across renders including a bare no-cluster `helm template` (the hashed inputs carry
no randomness), and never derives a secret from public identity. deriveSecret removed.

New scripts/tests/helm-secret-checksum.test.sh locks all three: stable-across-renders,
changes-on-rotation, and generated-value-is-random -- so the determinism cannot return.

Verified: helm lint 0-failed; bare + override template OK; the M-10 render guard still
fires on production+localhost; the new selftest ALL PASS.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r4): self-review — close the encryption-key data-loss BLOCKER + deploy minors

A cold adversarial self-review (three auditors mirroring the reviewer's method) of
the r4 changeset found a data-loss BLOCKER we introduced this round, plus deploy
minors. Fixing before it ships.

B-3 (BLOCKER, data loss): the r4 encryption-key unification made ENCRYPTION_KEY env
OVERWRITE an existing at-rest key file on boot. Traced only to the "operator freshly
sets the key" consumer, never to (a) backup_service restore, which writes the backup's
key to that file, or (b) the r3->r4 upgrade population, whose data was encrypted under
a marker-less auto-gen FILE key (r3 core.encryption used the file regardless of env).
The first r4 boot clobbered those -> restored/existing data undecryptable (or bricked
the boot on a marker mismatch). A new test even LOCKED the clobber with a false premise.
Fix: the at-rest key FILE is the single source of truth. ENCRYPTION_KEY only SEEDS the
file when it is ABSENT and never overwrites an existing one; the gate reads the FILE's
.operator provenance. backup restore now drops the .operator marker so a restored key
passes the gate without a clobber. Rewrote the clobber-locking tests to lock the
no-clobber invariant; added upgrade-shape, production-fail-without-data-loss, and
restore-marker tests.

M-7 (deploy self-review): removed the rotation-marker from secretsChecksum — the
auditor proved it redundant (the same .data change already moves the digest; deleting
it left the test green) and its admin branch dead. Kept the input-hash; documented the
genuine trilemma (cluster-less-template-stable / tracks-generated-rotation /
unpredictable-secrets — pick two; determinism is the predictable-secret hole).

M-10: the render guard's wildcard check is now an exact comma-split entry, matching the
backend's `"*" in cors_origins` since r4, so a legitimate `https://*.example.com` is no
longer blocked; the localhost check stays a substring to match the backend.

.env.example: the admin-password template was an empty assignment that uncomments into
a lockout; it now carries a replace-me placeholder.

Verified: backend 8082 passed; config/encryption 66, backup 13; script-selftests all
pass; helm lint/template clean (subdomain-wildcard passes, bare '*'/prod+localhost
fail); ruff clean.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

* fix(#193 r5): fail closed instead of regenerating over an existing at-rest key (I-1)

bonnyr-f5 #193 round-5 closure blocker. B-3 wrote down the invariant "the at-rest key
file is the single source of truth; nothing overwrites it once it holds bytes" and
config.py honoured it -- but core.encryption.get_encryption_key() did not. A file
present but under 32 bytes (truncated / partial write / disk full / bad restore) logged
"Invalid key, regenerating" and OVERWROTE it with a fresh key, permanently destroying
the key any existing data was encrypted under -- silently, on a GREEN production boot,
because the intact .operator marker keeps validate_production passing.

Fix: the read path now fails CLOSED. A genuinely absent (or 0-byte) file still generates
and persists a new key. A file that HOLDS bytes is validated as a real Fernet key: valid
-> returned untouched; unusable -> SystemExit with a clear message, never regenerated. A
crashloop is recoverable; an overwritten key is not. This also closes r5 note #3 -- the
old `len >= 32` check accepted any blob and surfaced a mis-shaped key as a later cipher
error; it now Fernet-validates and says so plainly.

Locked by TestAtRestKeyFileNeverRegeneratedOverBytes: a 30-byte truncated key -> SystemExit
AND the original bytes survive on disk (recoverable); a valid key -> returned untouched; an
absent file -> generates a valid key.

Verified: backend 8086 passed; encryption unit tests 19 passed; ruff clean.

Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

---------

Co-authored-by: John Gruber <john.t.gruber@gmail.com>
@jgruberf5
jgruberf5 deleted the fix/ci-hygiene-gitleaks-shellcheck branch August 24, 2026 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants