Conversation
) * Integrate the #177 follow-up stack (#179 #180 #181 #182 #183 #186 #188) Consolidated landing of seven interdependent PRs whose shared credential and release/CI surfaces prevented merging in any order (see issue #192's conflict matrix). Merged in dependency order 179, 180, 181, 188, 186, 183, 182; the #186/#188 credential surface was reconciled once (single reserved-name guard; provenance + migrations + stale-disable combined with rotation + backend MCP wiring + threadpool). Squashed to one commit; per-PR history retained on the seven archived branches. Validated on the merged tree: ruff + mypy clean; 172 auth/credential/startup/ migration tests pass; single alembic head v2_155; openapi + frontend types fresh; helm lint/template and docker compose config green on all modes; version and detector self-tests green; commit-message lint clean. Closes #192. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193): reconcile the merged unset-MCP-password model and harden the credential seed Addresses mwiget's three CHANGES_REQUESTED blockers on the #177 integration PR (#193), plus follow-up findings from a max-effort review of the same credential surface. The merge chose #188's "unset MCP_SERVICE_PASSWORD -> disable" over #186's "unset -> generate", so the generate/rotate-on-unset code was left unreachable but still documented, and the release-notes footer was missing. BLOCKING 2 (core/config.py): the MCP_SERVICE_PASSWORD comment claimed ensure_service_user "generates a random secret and surfaces it once" when unset, contradicting the merged behaviour. Rewrote it to state the truth: when unset (or a known default) seed_auth_step calls disable_stale_service_user, leaving the mcp account disabled/unavailable until an operator configures a real password; a published default is refused and rotated out; the backend receives MCP_SERVICE_PASSWORD on every deploy mode and reconciles to it when set. Also removed the duplicate #186 block that sat above the wrong field. BLOCKING 3 (services/auth_service.py): removed the unreachable usable_password is None branches from ensure_service_user (the generate-on-create and rotate-on-unset paths). seed_auth_step only calls it under the _mcp_pw_usable gate, so password is never None/default in production. ensure_service_user now requires a usable password and only creates/reconciles with it (failing closed and loudly if handed an unusable one); the unset case is owned entirely by disable_stale_service_user. Dropped the now-dead _log_generated_service_password helper and the service-account token_urlsafe/_persist_generated_password calls (_persist_generated_password is still used by the admin seed). Kept the reserved-name guard, the provenance check, the adopt-a-published-default remediation, and disable_stale_service_user fully intact. Updated the affected unit tests (published-default/None now refused; added a reachable adopt-and-reconcile test; stale-row setup builds the legacy row directly) and fixed scripts/mcp_live_smoke.py, which pointed operators at /app/keys/initial_mcp_password, a file no reachable path writes. CR-1 (services/auth_service.py, seed_admin_user): fixed a concurrent-first-boot admin lockout. With DEFAULT_ADMIN_PASSWORD unset and 2+ api replicas, both generated different passwords and the loser overwrote the keys file while its INSERT rolled back, so the file and the committed row disagreed. The fresh seed now creates+flushes first (the loser's INSERT raises IntegrityError -> rollback, no file write) and persists the keys file only after winning but before commit, so the file can only ever hold the committed row's password. Added a losing-replica test. CR-5 (services/auth_service.py, _persist_generated_password): os.open's 0600 mode only applies on create, so a pre-existing 0644 file was truncated in place and kept 0644, writing the secret world-readable. Added os.fchmod(fd, 0o600) and a test that a pre-existing 0644 file is tightened to 0600. CR-2 (routes/k8s_websocket.py, dpus_websocket.py, benchmarks.py): the WS auth validators called the blocking sync token_user_state directly on the event loop. Moved it off the loop via run_in_threadpool, matching core/auth_middleware.py. Validation: make lint-backend clean; mypy core/ schemas/ unchanged; auth (57) + ws/benchmark/startup (79) suites pass; alembic heads single v2_155; helm lint/template OK and --set secrets.mcpPassword=changeme fails the render; docker compose config OK on all modes; extract-breaking-changes and compute_version_bump self-tests pass; lint-commit-markers clean. BREAKING CHANGE: MCP_SERVICE_PASSWORD is now required — the backend SystemExits under ENVIRONMENT=staging|production when it is unset or a known default; the shipped admin/changeme default is removed and rotated out on upgrade; the dist bundle renames MCP_USERNAME/MCP_PASSWORD to MCP_SERVICE_USERNAME/MCP_SERVICE_PASSWORD; and the Helm chart ships mcpUsername=mcp with a generated mcpPassword instead of admin/changeme. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193): close bonnyr-f5's credential/backend/helm/dist/docs blockers Follow-up to the #177 integration on pr177-integration, addressing the CREDENTIAL/BACKEND/HELM/DIST/DOCS half of bonnyr-f5's BLOCK on PR #193. B1 (SECURITY): ensure_service_user no longer adopts any human account whose password is a known default. The adoption exception is now scoped to v2_155's exact backfill fingerprint (username=='mcp' AND email=='mcp@bnk-forge.local'), matching the migration's own conservative rule, and must_change_password is no longer cleared on an adopted row. Adds tests proving a human operator/changeme row (and a wrong-email mcp row) is REFUSED, not taken over. B2/B2b: all five compose files (root + dist docker-compose{,.local}.yml and the IBM embedded compose) honor legacy MCP_USERNAME/MCP_PASSWORD as aliases for MCP_SERVICE_USERNAME/MCP_SERVICE_PASSWORD on both backend and mcp env, so an existing customer .env keeps working after upgrade. Docs (dist/README.md, dist/.env.example, user-pack/install-guide.html) document MCP_SERVICE_PASSWORD as canonical with MCP_PASSWORD honored as a legacy alias. B3: ENVIRONMENT is plumbed to the backend in every compose file, so an operator who sets ENVIRONMENT=staging|production actually reaches config.py's MCP fail-fast. Helm already routes ENVIRONMENT=production onto api/worker/beat. M1: the unset-MCP behavior stays "disabled" (#188 over #186); added an explicit deliberate-consolidation comment at the decision point. M2: disable_stale_service_user skips the about-to-be-reconciled row and the "no usable MCP_SERVICE_PASSWORD" warning is conditional on a configured password, so a correctly-configured install no longer logs a false warning or commits an inactive MCP window on every boot. M7: the Helm mcp deployment and the IBM installer's mcp service now run the exec auth-probe healthcheck (python -m bnk_forge_mcp.healthcheck) instead of a bare tcpSocket probe. M8: the chart fails the render for a reserved mcpUsername (admin), mirroring the mcpPassword guard, and NOTES.txt/values.yaml call it out. Minors: deterministic checksum/secret via a shared helper (stable across renders, identical across api/worker/beat/mcp); vestigial _persist_generated_password filename docstring; false "backend generates its own secret" rationale corrected in compose/helpers/ibm; mcp_live_smoke.py #186/#188 attribution; e2e/config.py changeme default note; .env.example :58/:73/:94 fixes; unified BNK_FORGE_VERSION to latest across dist. Validation: ruff clean; typecheck-backend (core/ schemas/) Success 38 files; 199 auth/credential/startup/ws/migration tests pass incl. new B1 tests; helm lint + template stable checksums, --set secrets.mcpUsername=admin and secrets.mcpPassword=changeme both FAIL; docker compose config on all five modes shows the backend receiving MCP_SERVICE_PASSWORD (via either alias) and ENVIRONMENT. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193): close bonnyr-f5 CI/RELEASE/SCRIPTS blockers, majors, and minors Blockers: - B4 registry-tag-probe.sh: replace GNU-only BRE alternation (\|) with portable `sed -nE (access_token|token)` so the token parse works on BSD/ macOS sed; on BSD the empty token classified every image `unknown` and routed the operator into FORCE_LATEST=1, overwriting immutable :VERSION manifests (INV-24). Reproduced under `sed --posix`, fix proven to parse T2. - B5 registry-tag-probe.test.sh had no caller and was red on BSD. Both the Makefile script-selftests target and ci.yml's script-selftests job now enumerate and run every scripts/tests/*.test.sh, failing on an empty enumeration or any non-zero rc. - B6 lint-commit-markers.sh: replace the spoofable committer-identity exemption (GitHub <noreply@github.com> + single parent) with an unspoofable "already reachable from origin/main|origin/staging" check; lint the PR title (PR_TITLE via env) on pull_request events; split the rules so machine/already-merged is exempt for the marker rule but the spurious-major rule always applies. Majors: - M3 release.yml overwrite guard: derive the vacuity floor from an independent source (docker-bake.hcl default group, sourced from the workflow-ref tooling) and assert the probe's exit status before trusting its output, so an unavailable probe fails closed instead of "safe". - M4 (INV-31): generate release notes and run the registry existence-probe BEFORE the irreversible push in release-final/release-manual (new shared scripts/registry-overwrite-guard.sh); release-publish keeps its own in-critical-section re-check. - M5 make script-selftests now runs the INV-15 detector-parity diff (extracted to scripts/tests/detector-parity.test.sh) so local == CI. - M6 extractor self-test runs unconditionally with anti-vacuity assertions (ok lines + END marker), no longer gated on grepping its own --self-test. Minors: stale cross-PR comments in release.yml and extract-breaking-changes.sh; removed the duplicate Makefile version-check target; `git add dist/VERSION` no longer swallows failures; first-ever-release notes range fixed; CHANGELOG insertion asserts a non-no-op before committing; refreshed .trivyignore CVE-2026-7598 review deadline; removed e2e-tests.yml dead `|| true`; documented the new Docker dependency in the pre-push hook. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r2): close credential/backend/helm/dist blockers — drop the username alias, make ENVIRONMENT=production satisfiable, and fix the mcp liveness probe bonnyr-f5 round-2 BLOCK, credential/backend/helm/dist/docs half. B-1 (INV-12): the compose files aliased the SERVICE username (MCP_SERVICE_USERNAME:-${MCP_USERNAME:-mcp}); a legacy .env with MCP_USERNAME=admin resolved it to `admin`, and against the guardless image `latest` still points at, the old ensure_service_user rewrites the human admin row to `changeme` every boot. Drop the username alias across all five compose files + the ibm embedded compose (keep the harmless password alias), and default BNK_FORGE_VERSION to 4.0.0 (the release this tree becomes, first image with the guards) instead of `latest`, so a compose file can never hand the new credential contract to a pre-guard image. B-2: ENVIRONMENT=production reaches validate_production, which also gates on JWT_SECRET_KEY / ENCRYPTION_KEY / ALLOWED_ORIGINS — none of which were deliverable from a compose install, so the switch bricked the backend. Plumb all three into every x-backend-env anchor (four compose files + ibm) and document them in the env examples; treat an empty ("" from ${VAR:-}) key as unset in config.py so the plumbed empty default auto-generates rather than passing as a real empty key. _persist_or_load_key now flags only keys WE generated as auto_generated (sidecar .autogen marker), so an operator-provisioned key on the volume validates while a fresh prod boot still fail-fasts permanently. M-4: mcp liveness returned non-zero when the BACKEND was unreachable, so k8s restarted the pod for a dependency outage. Move the auth-probe to readiness only; liveness is tcpSocket. Fix mcp-server/README + mcp_live_smoke hints to name only the vars each process actually reads (container: BNK_FORGE_*, backend: MCP_SERVICE_*). Minors: refuse a known-default DEFAULT_ADMIN_PASSWORD on fresh seed + helm adminPassword fail-guard; guard secrets.yaml mcpUsername nil with kindIs "invalid"; make the Python reserved-name check case-insensitive/trim to match Helm; neutralise the hash when disabling a stale service account; correct the benchmarks.py JWT-gate comment; surface an empty MCP_SERVICE_PASSWORD in install.sh; fix the .env.example "No .env file is needed!" contradiction. Tests: config B-2 satisfiability + provenance-marker tests; seed_auth_step against an admin-still-holds-changeme upgrade DB; case-insensitive reserved-name and disable-hash-neutralisation cases. ruff clean, mypy clean, 4831 unit pass, helm lint/template green, docker compose config verified on all modes. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193): close r2 CI/RELEASE blockers B-3, M-1..M-3 + scripts/release minors B-3 (commit-lint exemptions): key the already-merged exemption on the range BASE (github.event.before), not the post-push tip, so a new skip-CI marker on a push to main/staging is caught while a genuinely-already-merged base commit stays exempt; replace the self-settable `^release: ` subject exemption with release.yml's own version+trailing-skip fingerprint. M-1 (spurious-major rule): redefine rule 2 as the exact complement of the detectors, sourced from the shared predicate, so it flags only a marker the detectors would MISS (never dash-bullet, markdown-bold or indented shapes); give it the same already-merged exemption; and lint inputs.release_notes through the script before it becomes a release commit/tag. M-2 (overwrite guard floor): derive the vacuity floor from `docker buildx bake --print default | jq '.group.default.targets | length'`, scoped to the default group, so a second bake group no longer wedges the release; separate bake-file parse failures from registry-unreachable in the messaging. Single-source the policy: release-publish and make push-images now call the one guard. M-3 (portability): drop bash-4 mapfile from the probe test; rebuild the compute self-test newline expansion with awk to avoid the bash-3.2 parameter-expansion cliff, so make script-selftests runs under stock macOS bash 3.2. Detector single-sourced into scripts/lib/breaking-change-detect.sh (compute, extract, lint all source it); detector-parity test asserts the wiring; added mutation tests for the lint rules and the overwrite guard. Minors: fix version-consistency misdiagnosis of a column-0 YAML comment; correct the compute/extract parity docstrings and the docker-bake four-push-paths note; wire artifact-network-self-test into ci-gates; make the pre-push hook migration message reachable under set -e; omit the false provenance buildStartedOn; filter the release CI-status poll by commit SHA. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193): seed the re-enable-guard test's default-hash row directly The round-2 disable_stale_service_user hash-scrub (bonnyr-f5 #193 minor) collided with the re-enable guard's own regression test: _seed_disabled_default_mcp built its "disabled while holding the published default" state BY CALLING disable_stale, which now scrubs the hash -- so holds_known_default_password was false and the PUT re-enable was allowed (200) instead of refused (400). The guard defends a row taken inactive by a path that LEAVES the credential intact (a manual operator PUT), not one disable_stale scrubbed. Seed that state directly (set is_active=False on the default-hash row) so the guard's real scenario is exercised; assert the default hash survives the seed. Corrected the now-stale guard comment in routes/auth.py that still claimed disable "only flips is_active". Neutralisation and its asserting tests are unchanged. Verified: TestServiceAccountReEnableGuard 2/2 pass; the three affected auth files (test_startup_seed_auth, component/test_auth_service, integration/test_routes_auth) 97/97 pass; ruff clean. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r3): fail-close key provenance (B-2) + credential/auth minors Own the round-3 CREDENTIAL/AUTH findings. B-2 (BLOCKER): _persist_or_load_key classified a marker-less key file as operator-provided, so every upgrade keys volume (key present, no marker) let SEC-006's fail-fast pass OPEN on an auto-generated JWT/ENCRYPTION secret in production. Invert the sentinel: a marker-less key now classifies AUTO-GENERATED (fail closed); an operator asserts provenance with an explicit <filename>.operator opt-out marker. No marker is written on generation, which also removes the second trigger (a partial marker write can no longer downgrade provenance). Regression tests seed the PREVIOUS-RELEASE on-disk shape and assert validate_production raises under ENVIRONMENT=production. Minors: - Write the key via os.open(0o600)+fchmod so the secret is never briefly 0644. - Single-source the MCP known-default denylist: delete the local tuple in auth_service and use core.config.MCP_KNOWN_DEFAULT_PASSWORDS (comment notes the helm copy is deploy-owned). - Correct holds_known_default_password docstring (disable_stale now scrubs the hash; this guard covers the other disable paths). - Reconcile ENCRYPTION_KEY docs with reality: the keys-file is the source of truth for at-rest crypto; the env var only drives the production gate (encryption.py comment + .env.example). - Clarify the v2_155 custom-username remedy in disable_stale docstring. Test-gaps: - Normalise the service username (trim/casefold) at the reconcile lookup and the disable skip filter, so " mcp "/"MCP" reconciles the existing mcp row instead of minting a second service account and disabling the live one. - Honest seed_admin_user log/logic under DEFAULT_ADMIN_MUST_CHANGE=false (no more "must change on first login" when no gate was applied). - disable_stale_service_user(skip_username=...) leaves the live row wholly untouched (no inactive window), variant included. - db.commit() failure after the keys file is written leaves a retriable state (published default still authenticates, orphan file password does not). All owned-suite tests pass; ruff clean. Each fix reproduced then mutation-tested. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r3): close the release/CI blocker + major + every release/CI minor M-6 (blocker): commit-lint no longer reds unamendable merge history. - rule 2 now flags ONLY a mis-anchored DECLARATIVE marker (the colon form); a colonless marker-shaped PROSE line the detectors treat as inert (an already-merged body such as "- <MARKER> footer in the body ...") is no longer flagged, so the push-to-main range (before..head, which INCLUDES the PR merge-base) goes green without a history rewrite. Detection of a real mis-anchored marker is unchanged. - deleted the already-merged exemption as dead code: base..head excludes the base by construction, so no scanned commit can ever be an ancestor of it. Removed the BEFORE derivation, _already_merged, tests B3.2/M1.5, and the ~20-line header claim. The release-bot exemption stays. - rule 2 now scans the whole body via _under_detected_markers and reports EVERY mis-anchored marker, not just the first. M-7 (major): secret-scan no longer false-fails a delete-only range. A delete-only commit has rev-list count > 0 but gitleaks scans 0 (it scans added content), so the count-based backstop is replaced by a range- resolvability check plus gitleaks' exit status. Release/CI minors: - release-bot fingerprint single-sourced to scripts/lib/is-release-bot-subject.sh; release.yml's inline copy byte-locked by a parity self-test; dropped the false unforgeability claim and documented the residual honestly. - registry-overwrite-guard: added a fail-closed default arm for an unrecognised/empty probe status (+ malformed/empty test scenarios). - Makefile push-images: FORCE_LATEST now overrides ONLY the recency guard; a new FORCE_OVERWRITE overrides ONLY the immutable-tag guard; fixed the missing-jq remediation text. - registry-tag-probe: the network arm now matches the real doubled "000000" curl-failure shape (was dead code); test fixture reproduces it. - INV-15: single-sourced the marker regex (one canonical value + a detector-parity assertion that every embedded copy is byte-identical). - release.yml Publish summary counts what buildx actually pushed (bake --metadata-file), not the static target list. - registry-tag-probe test enumerates the bake DEFAULT group (scoped), matching the guard's enumeration. - added scripts/tests/secret-scan.test.sh (fake-docker mutation suite). release.yml: added a post-push step running scripts/verify-image-pins.sh so a release cannot complete while shipping an unpublished image pin (script owned by the deploy agent; referenced by path from .release-tooling). Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r3): single-source every deploy version pin + close deploy majors/minors B-1/B-1b: dist/docker-compose{,.local}.yml, dist/.env.example and ibm_cloud_bnk_forge.sh hard-pinned 4.0.0, a tag that has never been published, while Helm pinned 3.1.6 -- two shipped paths naming two versions, one of which does not exist. Bring all of them under sync-version-artifacts.sh (new PIN/ DISTENV readers+writers, --check, --list) so every pin derives from VERSION (3.1.6, which exists) and the release re-stamps them atomically via the existing --write $NEW; drop the dist/ disclaimer. Add scripts/verify-image-pins.sh (+ selftest) that resolves every shipped compose image: pin against the registry and fails on manifest unknown, wired post-push in the release job. M-1..M-5: reword NOTES/compose/chart comments that asserted post-guard behaviour as already-true on the pre-guard pinned image (they land with the guard-carrying release); NOTES now leads with the required ALLOWED_ORIGINS override; dist/README and the install guide stop recommending latest/3.1.6 and the keys-file cat the pinned image does not write; install.sh strips quotes and rejects the known- default MCP passwords so the "MCP not active" warning fires instead of a green lie; add deploy-version-lockstep + helm-known-defaults-lockstep selftests. Deploy minors: MCP_USERNAME "do not set" made consistent across docs/Makefile; chart ENCRYPTION_KEY now emits a valid Fernet key; DEFAULT_ADMIN_* added to the ibm P3 backend env; secrets.mcpUsername defaults to "mcp" on null; ibm mapfile -> portable while-read. Verified: sync --check exit 0; --write round-trip moves every pin and restores; helm lint/template clean (default + origin override); script selftests green; bash -n + shellcheck clean. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r4): unify the at-rest encryption key (B-3), fail-close key marker (M-1), row-matches-client (M-2) + credential minors B-3: validate_production gated ENCRYPTION_KEY while a second, independent generator in core/encryption.py produced the real at-rest Fernet key unchecked — setting ENCRYPTION_KEY (the documented remedy: token_hex(16), not even a Fernet key) turned the gate green while encryption auto-generated a different key. Unify: one key file (_encryption_key_path == core.encryption.ENCRYPTION_KEY_FILE), one generator. When ENCRYPTION_KEY is set it is VALIDATED as a real Fernet key (fail clearly if not), written to that file with a .operator marker, and consumed by core.encryption and services.backup_service; the provenance flag reflects the value that actually protects data. Never clobber an operator-marked key on a mismatch. config.py:319 and .env.example now print the Fernet recipe. M-1: _persist_or_load_key required a regular-FILE marker (os.path.isfile, not os.path.exists — a directory no longer counts) and treats "marker present, key file absent" as a provisioning error: generate but do NOT persist, so the stale-marker rotation gesture can never heal into auto=False on the next boot. M-2 (regression this PR introduced): ensure_service_user normalised the username before lookup/create, so MCP_SERVICE_USERNAME=MCP seeded 'mcp' while the client sends the raw 'MCP' and authenticate_user matched exactly -> login denied. Create/ reconcile under the RAW value (what the client sends); the disable_stale skip keys on the same raw value; only the reserved-name guard normalises. Fixed the false "Matches the Helm chart lower|trim" docstring. Credential minors: non-vacuous localhost-CORS test (valid MCP password so only the CORS branch fails) + wildcard is now an exact origin-list entry, not a substring; new DPU-websocket must-change/unresolvable-user tests mirroring the k8s twin; middleware unresolvable-JWT-subject-refused and exact-vs-suffix exempt-path tests; corrected the denylist copy count (4th copy in dist/install.sh); corrected v2_155's rationale (v2_154 is new in this diff, not "already shipped"); documented why ensure_service_user's adoption branch is kept. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r4): close deploy-surface blocker + majors (B1, M7-M12) and deploy minors B-1: a fresh install of the pinned 3.1.6 bundle seeded an admin nobody could log in as. `${DEFAULT_ADMIN_PASSWORD:-}` delivers the var present-and-empty; 3.1.6's `DEFAULT_ADMIN_PASSWORD: str = "changeme"` is then overridden by "" and the login schema rejects "" (422). The prescribed `${VAR}` form does NOT omit on this compose (map interpolation still renders ""); the working omit-when-unset form is a map entry with NO value (passthrough / `docker run -e KEY` semantics). Converted DEFAULT_ADMIN_PASSWORD to passthrough across all four compose paths (dist base+local, root base+local, IBM embedded). Swept the class: JWT_SECRET_KEY/ENCRYPTION_KEY also converted (3.1.6 uses `if KEY is None`, so "" is used literally, not auto-generated); MCP_SERVICE_PASSWORD kept as `${...:-...}` because omitting it restores 3.1.6's known default "mcp-service-changeme". Added a commented DEFAULT_ADMIN_PASSWORD block to dist/.env.example. Verified via `docker compose config` + real container env both directions (unset -> omitted; set in .env -> forwarded). M-7: chart checksum/secret did not change on mcp-password rotation (3 renders, 3 generated passwords, one identical checksum). Made all generate/rotate fallbacks deterministic (deriveSecret, release-seeded) so the Secret is stable across renders and includes, and hash the RENDERED Secret so the annotation tracks every resolved value. Now stable across renders, identical across the 4 deployments, and it flips when any resolved value changes. M-8: dist/install.sh credential guard failed open on `changeme ` (whitespace). Trim leading/trailing whitespace around the quote-strip before the known-default compare. M-9: removed newly-added forward-dated 4.0.0 prose (install-guide.html x2, DOCKER.md). M-10: default helm install crashlooped (production + localhost). Added a render-time guard mirroring backend validate_production (fail on wildcard under staging/production, localhost under production); defaulted ALLOWED_ORIGINS to empty so the bare render boots (empty is neither wildcard nor localhost). `helm lint` and bare `helm template` stay green; the guard fires with a clear message on a real fatal posture. M-11: portable in-place sed in the IBM installer (`sed -i.bak … && rm`), both sites. M-12: dist/ no longer ships published default DB/redis creds on host networking. install.sh generates strong POSTGRES_PASSWORD/REDIS_PASSWORD on the fresh .env (like Helm/IBM); .env.example ships them empty; a pre-existing default triggers a warning. Deploy minors: extended deploy-version-lockstep.test.sh to the bnk-operator chart (appVersion + image.tag) and dist/VERSION; brought dist/VERSION under sync-version-artifacts.sh (--check/--list/--write green); reworded install.sh's MCP UNHEALTHY assertion to match what the pinned image actually reports; added scripts/tests/ibm-compose-drift.test.sh freezing the credential/hardening env so the IBM embedded compose and dist/docker-compose.yml cannot silently diverge. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r4): make the pin verifier reachable pre-push, de-vacuum the parity/self-test harnesses, and close the release minors B-2 (blocker): scripts/verify-image-pins.sh could never pass where the release job runs it — from the 4-file sparse .release-tooling checkout that holds no compose file (ROOT resolved there), with REGISTRY/VERSION handed in as env vars the script reads only as flags, and the whole step wired AFTER the tag/Release/push/signing. Fixes, end to end: - add a consistency mode (--expect-version) that asserts every shipped first-party pin already renders to $NEW without a registry probe, and run it as the PRIMARY PRE-push gate in release-final and release-manual (before anything irreversible); - fix the post-push existence step to pass REGISTRY/VERSION as FLAGS and the compose files explicitly by --file (they live at the tag checkout at the workspace root), keeping it as a secondary confirmation; - widen the default file set to include the IBM Cloud installer's embedded compose; - add a dryrun-release-tooling job that rebuilds the exact publish-job layout and exercises both invocations against a fake probe, and gate release-publish on it, so a step that cannot execute is caught before it is wired ahead of a signature. The sibling registry-tag-probe.sh / registry-overwrite-guard.sh read nothing $ROOT-relative outside the sparse set, so they are unaffected. M-3: detector-parity.test.sh enumerated the marker copies with the very token that drifts, so a copy that drifted in the token vanished from enumeration (drifting :96-97 dropped the count 5->3 yet stayed green). Enumerate by position/count instead: an exact per-file canonical count plus a stable-anchor site scan that flags any drifted site even under a compensating add. M-4: the filesystem self-test loop checked only a non-empty enumeration and each file's exit 0, so a test gutted to a no-op passed and deleting 7 of 8 stayed green. It now requires each file to emit PASS lines, no FAIL line, and an ALL PASS terminal marker, plus a count floor derived from git's tracked *.test.sh set (detector-parity was conformed to that output convention). M-5: release-rc created and pushed the RC tag before the fail-closed notes step; the tag is now created locally, notes generated, then the tag pushed. M-6: added mutation-tested coverage for this PR's four previously-uncovered lint fixes (the PR-title lint, the pending-message lint, the RANGE fail-closed branch, and the skip-checks trailer rule). LEAD: the anti-vacuity staging floor derived the count from a stale literal while --list grew to 8 paths; both sites now derive it from --list and require every listed path to stage, and the stale comments are corrected. Release minors: scope the release-bot commit-lint exemption to the range tip (a forged release subject buried mid-range is no longer exempt) and add a REACHABLE published-history exemption anchored to the last release tag so a mis-anchored marker in unamendable history cannot red the release; add fixtures for the untested registry probe/guard arms (5xx, unexpected code, unknown status, bake-parse failure); ancestry-filter the LAST_FINAL tag queries so a tag on another branch cannot skew the notes range; reconcile the empty-RANGE handling (commit-lint now fails closed on an explicit empty RANGE, ci.yml leaves it unset); give the pre-push hook a clear message and a fetch fallback when the remote tip is absent locally; derive the cosign verify-identity org from REGISTRY instead of hardcoding it. Flagged: gate Makefile push-customer-build through registry-overwrite-guard.sh, the last documented push path that was still unguarded. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r4): keep M-7 checksum-tracks-rotation WITHOUT predictable secrets The r4 deploy fix closed M-7 (checksum did not move on a secret rotation, so pods never rolled) by making the generated fallbacks deterministic -- deriveSecret = sha256(Release.Name|Namespace|fullname|purpose). Every input is public (they appear in resource labels and the chart source), so that made the JWT signing key, the at-rest Fernet key and the admin/mcp passwords COMPUTABLE by anyone who can read a label -- a forge-any-token / decrypt-all-secrets exposure, strictly worse than the cosmetic churn it fixed. Revert generation to randAlphaNum (unpredictable) and instead hash the DETERMINISTIC inputs that determine the Secret -- values.secrets, the persisted .data (reused via lookup), plus a per-credential "rotating-from-default" marker for admin/mcp whose persisted value is a known published default. That tracks every rotation (operator edit, persisted-value change, rotate-away-from-default) so the pods roll, is stable across renders including a bare no-cluster `helm template` (the hashed inputs carry no randomness), and never derives a secret from public identity. deriveSecret removed. New scripts/tests/helm-secret-checksum.test.sh locks all three: stable-across-renders, changes-on-rotation, and generated-value-is-random -- so the determinism cannot return. Verified: helm lint 0-failed; bare + override template OK; the M-10 render guard still fires on production+localhost; the new selftest ALL PASS. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r4): self-review — close the encryption-key data-loss BLOCKER + deploy minors A cold adversarial self-review (three auditors mirroring the reviewer's method) of the r4 changeset found a data-loss BLOCKER we introduced this round, plus deploy minors. Fixing before it ships. B-3 (BLOCKER, data loss): the r4 encryption-key unification made ENCRYPTION_KEY env OVERWRITE an existing at-rest key file on boot. Traced only to the "operator freshly sets the key" consumer, never to (a) backup_service restore, which writes the backup's key to that file, or (b) the r3->r4 upgrade population, whose data was encrypted under a marker-less auto-gen FILE key (r3 core.encryption used the file regardless of env). The first r4 boot clobbered those -> restored/existing data undecryptable (or bricked the boot on a marker mismatch). A new test even LOCKED the clobber with a false premise. Fix: the at-rest key FILE is the single source of truth. ENCRYPTION_KEY only SEEDS the file when it is ABSENT and never overwrites an existing one; the gate reads the FILE's .operator provenance. backup restore now drops the .operator marker so a restored key passes the gate without a clobber. Rewrote the clobber-locking tests to lock the no-clobber invariant; added upgrade-shape, production-fail-without-data-loss, and restore-marker tests. M-7 (deploy self-review): removed the rotation-marker from secretsChecksum — the auditor proved it redundant (the same .data change already moves the digest; deleting it left the test green) and its admin branch dead. Kept the input-hash; documented the genuine trilemma (cluster-less-template-stable / tracks-generated-rotation / unpredictable-secrets — pick two; determinism is the predictable-secret hole). M-10: the render guard's wildcard check is now an exact comma-split entry, matching the backend's `"*" in cors_origins` since r4, so a legitimate `https://*.example.com` is no longer blocked; the localhost check stays a substring to match the backend. .env.example: the admin-password template was an empty assignment that uncomments into a lockout; it now carries a replace-me placeholder. Verified: backend 8082 passed; config/encryption 66, backup 13; script-selftests all pass; helm lint/template clean (subdomain-wildcard passes, bare '*'/prod+localhost fail); ruff clean. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 * fix(#193 r5): fail closed instead of regenerating over an existing at-rest key (I-1) bonnyr-f5 #193 round-5 closure blocker. B-3 wrote down the invariant "the at-rest key file is the single source of truth; nothing overwrites it once it holds bytes" and config.py honoured it -- but core.encryption.get_encryption_key() did not. A file present but under 32 bytes (truncated / partial write / disk full / bad restore) logged "Invalid key, regenerating" and OVERWROTE it with a fresh key, permanently destroying the key any existing data was encrypted under -- silently, on a GREEN production boot, because the intact .operator marker keeps validate_production passing. Fix: the read path now fails CLOSED. A genuinely absent (or 0-byte) file still generates and persists a new key. A file that HOLDS bytes is validated as a real Fernet key: valid -> returned untouched; unusable -> SystemExit with a clear message, never regenerated. A crashloop is recoverable; an overwritten key is not. This also closes r5 note #3 -- the old `len >= 32` check accepted any blob and surfaced a mis-shaped key as a later cipher error; it now Fernet-validates and says so plainly. Locked by TestAtRestKeyFileNeverRegeneratedOverBytes: a 30-byte truncated key -> SystemExit AND the original bytes survive on disk (recoverable); a valid key -> returned untouched; an absent file -> generates a valid key. Verified: backend 8086 passed; encryption unit tests 19 passed; ruff clean. Claude-Session: https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4 --------- Co-authored-by: John Gruber <john.t.gruber@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes
stagingtomainto cut v4.0.0.What this carries
One commit:
ab2916c9— the #193 integration of the #177 follow-up stack (#179 #180 #181 #182 #183#186 #188), which closed #192. Reviewed over six rounds; final round verdict MERGE with no open
findings.
v4.0.0-rc.1was cut from this exact commit.Why 4.0.0
Two commits in
v3.1.6..stagingcarry deliberateBREAKING CHANGE:footers:ab2916c9—MCP_SERVICE_PASSWORDis now required; the backendSystemExits underENVIRONMENT=staging|productionwhen it is unset or a known default. The shippedadmin/changemedefault is removed and rotated out on upgrade,MCP_USERNAME/MCP_PASSWORDare renamed to
MCP_SERVICE_USERNAME/MCP_SERVICE_PASSWORD, and the Helm chart shipsmcpUsername=mcpwith a generatedmcpPassword.7ece9b04— the container runner refuses named users;USER nonrootmust becomeUSER 65532.scripts/compute_version_bump.shagainst the simulated merge result:No manual version edit is included, by design —
sync-version-artifacts.shis the single writerand the release job runs
--write 4.0.0across all 9 paths.VERSIONis a derived artifact;compute_version_bump.shnever reads it back.Merge instructions
Use a merge commit — not squash. A merge commit keeps
ab2916c9's body, and therefore itsBREAKING CHANGE:footer, inside thev3.1.6..HEADrange the version derivation scans. It alsomatches the precedent on
main(Merge pull request #177 from f5devcentral/staging).Do not retitle this PR to start with
release:— the release workflow's loop guard fails therun on that prefix.
What merging does
Merging is the publish action. On the resulting push to
main,.github/workflows/release.yml:guard→preflight(polls CI for this SHA, up to 45 min)release-final— bump 9 paths to 4.0.0, updateCHANGELOG.md, commit + tag, refuse to overwritean existing tag,
verify-image-pins.sh --expect-version 4.0.0, push commit + tag, create theGitHub Release
release-publish— 7 images toghcr.io/f5devcentral, cosign signatures, SBOM, provenanceKnown context
main's branch protection (lock_branch, required PR reviews) is temporarily relaxed so theworkflow's
git push origin maincan land — the identical promotion failed on 2026-08-19(run 32313394044) at exactly
that step. Protection is restored immediately after the release run completes. Durable fixes are
tracked in #196 (scoped bypass) and #197 (stop pushing to
mainat all) — competing designs.Pre-existing
v3.1.7-rc.2..rc.9tags are mis-numbered: they were cut before #178 taught thederivation to read commit bodies, so they under-numbered work that had already gone breaking.
They are inert — derivation baselines on the last final tag — and cleaning them up is optional.