Stop shipping a default admin credential; generate it and enforce rotation

[jgruberf5]

Fixes #184 — the pre-4.0.0 release blocker.

The problem

Every fresh deployment seeded admin / changeme, and it was a live, API-reachable credential:

• The password is published in this repo (config.py, NOTES.txt, compose).
• must_change_password was returned in the login response but never enforced server-side — get_current_user/require_role never consulted it — so a client could skip the change-password screen and call every endpoint directly with the seed credential.
• gitleaks can't catch it: a runtime-seeded default + an unenforced flag, not a committed secret.

The fix

• DEFAULT_ADMIN_PASSWORD → None (never a hardcoded value), mirroring the existing JWT_SECRET_KEY/ENCRYPTION_KEY handling.
• seed_admin_user generates a strong random password when unset and logs it once; the account stays must_change_password.
• Server-side enforcement in get_current_user: refuse every endpoint except change-password / me / logout until the password is rotated — for both JWT and API-token auth.
• Helm: generates an admin-password secret (reused across upgrades) and injects DEFAULT_ADMIN_PASSWORD into the backend pods; values.yaml gains adminPassword: "" (empty → generated).
• Docs sweep: README, INSTALLATION, DEPLOYMENT, the Helm NOTES, the user-pack install guide, and the adr424 compose header now explain retrieving the generated password (backend logs for compose, the secret for Helm) and that rotation is enforced.

Testing

• Seed generates a random password when unset — the published changeme no longer authenticates — and uses an explicit one when set.
• A must-change user is refused (403) on a protected endpoint, can still reach /me and change-password, and is unblocked after changing it; a normal user is not gated.
• 99 auth tests pass, ruff clean, OpenAPI unchanged, helm lint clean.

Scope

The admin credential (the critical, API-reachable one). The mcp service account ships the same class of default (MCP_SERVICE_PASSWORD = "mcp-service-changeme", and that account is role=admin) — its fix is entangled with the chart's mcp-password wiring, so it's tracked as a follow-up rather than bundled here.

Fixes #184

https://claude.ai/code/session_01UpRYiFserdBE5ESHn759N4

[mwiget]

This is the right diagnosis and mostly the right fix -- must_change_password returned in the login response but never consulted server-side is exactly the kind of gap that reads as "handled" for years. The seed-generation half I have no objection to. The enforcement half has two holes, and one of them leaves the most dangerous surface in the product wide open.

Verified good

• DEFAULT_ADMIN_PASSWORD: str | None = None mirrors the existing JWT_SECRET_KEY/ENCRYPTION_KEY shape, and secrets.token_urlsafe(18) is a sound generator.
• The Helm block matches the four secrets above it exactly and reuses the value across upgrades, so nothing rotates under a running install.
• The integration test is the right test: login as a must-change user, assert 403 on a protected endpoint, assert /me and change-password still work, assert the same endpoint works after the change. Plus a regression guard that an ordinary user isn't gated.
• The docs sweep is thorough -- README, INSTALLATION, DEPLOYMENT, NOTES.txt, the user-pack guide and the adr424 compose header all stop saying changeme.

Blocking 1 -- WebSockets bypass the gate completely

_enforce_password_change lives in get_current_user, but the WebSocket routes never go through it. k8s_websocket.py::_validate_ws_token (and the identical one in dpus_websocket.py) decodes the JWT and checks the role claim -- it never loads the User row, so must_change_password cannot be consulted:

payload = decode_token(token)
role = payload.get("role", "")
if role not in ("admin", "operator", "viewer"):
    ...
return True

So with the seed credential and its perfectly valid token, a must-change admin still reaches:

endpoint	what it grants
/ws/k8s/clusters/{id}/pods/{pod}/exec	interactive shell in a cluster pod
/ws/k8s/clusters/{id}/pods/{pod}/logs/follow	pod log stream
/ws/projects/{p}/dpus/{d}/serial-console	DPU serial console
/ws/projects/{p}/dpus/{d}/ssh-bmc / ssh-os	SSH to the BMC / host OS

Every one of those is strictly more powerful than the REST endpoints the gate does block. The PR body says "a client could skip the change-password screen and call every endpoint directly with the seed credential" -- that sentence is still true, and it's now true of pod exec and BMC SSH specifically while /api/auth/users is refused.

The fix wants the flag checked where the user is loaded, not where the request is routed: _validate_ws_token has to resolve the token subject to a User and refuse when must_change_password is set. A shared helper both paths call would keep them from drifting again.

Blocking 2 -- the suffix match is bypassable

path.endswith(suffix) tests the raw URL, and the repo has one route with a :path converter -- state_viewer.py:269, @router.get("/module/{module_id}/resource/{resource_address:path}") under prefix /api/state. :path matches slashes, so the tail is attacker-chosen. Reproduced with a minimal FastAPI app using this exact gate and route shape:

403  /api/auth/users
403  /api/state/module/1/resource/aws_instance.web
200  /api/state/module/1/resource/x/auth/me   -> {"gate":"EXEMPT","resource_address":"x/auth/me"}

A must-change user reads terraform state -- which routinely carries sensitive outputs -- by appending /auth/me to the resource address.

Match the resolved route instead of the raw path; FastAPI has already done the routing by the time this dependency runs:

_PASSWORD_CHANGE_EXEMPT_ROUTES = {
    "/api/auth/change-password",
    "/api/auth/me",
    "/api/auth/logout",
}

route = request.scope.get("route")
if route is not None and route.path in _PASSWORD_CHANGE_EXEMPT_ROUTES:
    return

That is exact rather than suffix-based, and it can't be widened by anything in the URL. It also fails closed if route is missing.

Non-blocking

• benchmarks.py::_require_agent_auth is the same shape as the WS check -- role claim out of the JWT, no User load -- so the agent write endpoints are ungated too. Lower impact than pod exec, but it's the same fix.
• Logging the generated password at WARNING is the only channel compose has, and the "shown once" framing is right -- worth noting in the code that it lands in any log aggregator the deployment ships to, and that Helm avoids it entirely because the env var is always set.
• Conflicts with #180 (approved) in helm/bnk-forge/templates/secrets.yaml and values.yaml -- both add a $...Pass block in the same place. This branch is based on pre-#180 staging, so its values.yaml still reads mcpPassword: changeme; after the rebase that line is already handled.
• MCP_SERVICE_PASSWORD = "mcp-service-changeme" on a role=admin account is the same vulnerability with the same blast radius, and I don't see an open issue for it -- #184 is this one, #185 is branch protection. Worth filing before this closes #184, so it doesn't leave with the box ticked.

[mwiget]

endswith on the raw URL is bypassable, because one route in the repo takes an attacker-chosen tail: state_viewer.py:269,

@router.get("/module/{module_id}/resource/{resource_address:path}", dependencies=[Depends(require_viewer)])

under prefix="/api/state". The :path converter matches slashes, so anything can follow. I built a minimal FastAPI app with this exact gate and route shape:

403  /api/auth/users
403  /api/state/module/1/resource/aws_instance.web
200  /api/state/module/1/resource/x/auth/me   -> {"gate":"EXEMPT","resource_address":"x/auth/me"}

Terraform state is a reasonable thing to want behind the gate -- outputs land there.

FastAPI has already resolved the route by the time this dependency runs, so match the route template rather than the URL:

_PASSWORD_CHANGE_EXEMPT_ROUTES = {
    "/api/auth/change-password",
    "/api/auth/me",
    "/api/auth/logout",
}

route = request.scope.get("route")
if route is not None and route.path in _PASSWORD_CHANGE_EXEMPT_ROUTES:
    return

Exact match, nothing in the URL can widen it, and a missing route falls through to the refusal rather than past it.

Worth a test alongside the two you added -- a must-change user hitting a :path route with /auth/me appended should be 403. That's the case that would have caught this.

[mwiget]

The gate lives here, but the WebSocket routes never call get_current_user. k8s_websocket.py::_validate_ws_token -- and its twin in dpus_websocket.py -- authenticates off the JWT payload alone:

payload = decode_token(token)
role = payload.get("role", "")
if role not in ("admin", "operator", "viewer"):
    await websocket.close(code=4401, ...)
    return False
return True

No User row is loaded, so must_change_password is never in scope. A seeded admin who has not rotated the password holds a fully valid token and still reaches:

• /ws/k8s/clusters/{cluster_id}/pods/{pod_name}/exec — interactive shell in a cluster pod
• /ws/k8s/clusters/{cluster_id}/pods/{pod_name}/logs/follow
• /ws/projects/{project_id}/dpus/{dpu_id}/serial-console
• /ws/projects/{project_id}/dpus/{dpu_id}/ssh-bmc and /ssh-os

/api/auth/users is refused while pod exec is not, which inverts the risk ordering the gate is meant to impose.

Because the check needs the DB row, the natural shape is a shared helper the WS validators also call -- resolve payload["sub"] to a User, refuse on must_change_password, and have both _validate_ws_tokens go through it. Keeping it in one place is what stops the two paths drifting apart again, which is how this gap arose in the first place.

[mwiget]

Blocker 2 is fixed and I re-tested it. Blocker 1 is untouched -- k8s_websocket.py, dpus_websocket.py and benchmarks.py are byte-identical to the previous head, so a must-change admin still gets pod exec and BMC SSH with the seed credential.

Blocker 2 -- fixed

Exact-path matching on request.url.path is sound here, and I was wrong to imply scope["route"] was the only way: if the raw path equals /api/auth/me, the router sends it to that endpoint, so exact equality can't be widened by a :path tail. Re-ran the harness against your new logic:

200  /api/auth/me
403  /api/auth/users
403  /api/state/module/1/resource/x/auth/me            <- was 200
404  /api/state/module/1/resource/../../../api/auth/me <- never reaches the exempt branch
403  /api/state/module/1/resource/x/api/auth/me

The rstrip("/") is harmless -- a trailing-slash variant either redirects or 404s, so nothing gets past on it.

You're also right that there's no server-side logout: the only logout in backend/routes/ after this commit is your own comment. Dropping it rather than leaving a dead exemption in a security gate is the better call.

Blocker 1 -- still open

$ git diff 3c158a0c 9d0e607c --stat -- backend/routes/k8s_websocket.py       backend/routes/dpus_websocket.py backend/routes/benchmarks.py
                                        # no output

_validate_ws_token is unchanged and still authenticates off the JWT claims alone:

payload = decode_token(token)
role = payload.get("role", "")
if role not in ("admin", "operator", "viewer"):
    ...
return True

No User row, so no must_change_password. /ws/k8s/clusters/{id}/pods/{pod}/exec, /logs/follow, and the three DPU consoles (serial-console, ssh-bmc, ssh-os) all still accept a seed-credential token. As it stands this PR refuses GET /api/auth/users to a user it will hand an interactive shell inside a cluster pod, which is the wrong way round.

The check has to move to where the user is loaded. A small shared helper -- resolve payload["sub"] to a User, refuse when must_change_password -- called from both _validate_ws_tokens keeps the REST and WS paths from drifting apart again, which is how the original gap arose.

Everything else from the last review stands unchanged: benchmarks.py::_require_agent_auth has the same shape (non-blocking), the #180 conflict in secrets.yaml/values.yaml is still there, and MCP_SERVICE_PASSWORD still wants an issue before this one closes #184.

[mwiget]

Good -- and better than what I suggested. Exact equality on the raw path is sound on its own: if request.url.path is /api/auth/me, the router has sent the request to that endpoint, so no :path tail can produce a match. Re-ran my harness with this logic:

200  /api/auth/me
403  /api/auth/users
403  /api/state/module/1/resource/x/auth/me            <- was 200
404  /api/state/module/1/resource/../../../api/auth/me
403  /api/state/module/1/resource/x/api/auth/me

rstrip("/") doesn't open anything either -- a trailing-slash variant redirects or 404s rather than reaching a handler.

And dropping /auth/logout is right: the only logout left in backend/routes/ is the comment on line 119. A dead entry in a security allowlist is the kind of thing that gets copied into the next one.

[mwiget]

Still the blocker -- unchanged at this head:

$ git diff 3c158a0c 9d0e607c --stat -- backend/routes/k8s_websocket.py \
      backend/routes/dpus_websocket.py backend/routes/benchmarks.py
                                        # no output

_validate_ws_token never loads the User, so must_change_password can't be consulted, and these stay reachable with the seed credential:

• /ws/k8s/clusters/{cluster_id}/pods/{pod_name}/exec
• /ws/k8s/clusters/{cluster_id}/pods/{pod_name}/logs/follow
• /ws/projects/{project_id}/dpus/{dpu_id}/serial-console
• /ws/projects/{project_id}/dpus/{dpu_id}/ssh-bmc and /ssh-os

The gate now refuses GET /api/auth/users to a user it will still give an interactive shell in a cluster pod and SSH to a BMC. That ordering is backwards, and it's the specific claim in the PR body -- "a client could skip the change-password screen and call every endpoint directly" -- still holding for the endpoints that matter most.

Because the flag lives on the row rather than in the token, this can't be fixed at the routing layer; it needs the same lookup get_current_user does. One helper both WS validators call, refusing on must_change_password, is what stops the two paths diverging again.

[jgruberf5]

Blocker 1 fixed — you were right, and it's the dangerous one: refusing /api/auth/users while handing out a pod shell is exactly backwards.

The gate now lives where the User is loaded, shared between REST and WS. New token_requires_password_change(token) in auth_service resolves the JWT sub to a User row (via get_user_from_token) and returns must_change_password; both k8s_websocket._validate_ws_token and dpus_websocket._validate_ws_token route through it, so /pods/.../exec, /logs/follow, and the DPU serial-console / ssh-bmc / ssh-os all refuse a seed-credential token now. One helper, one gate — the REST and WS paths can't drift apart again.

Tests: the helper is true for a must-change user, false for a normal one, false on a garbage token; and the :path-bypass case you flagged (/api/state/module/1/resource/x/auth/me) is now pinned at 403 alongside the two REST enforcement tests.

Also from the thread:

• Retrieval hardened per the discussion: the generated password is written to /app/keys/initial_admin_password (mode 600, on the persisted keys volume) and the log carries a pointer, not the secret — a one-time boot log is easy to miss and logging plaintext is the aggregation risk a prior review flagged. Docs/scripts (.env.example, the e2e step_login docstring, mcp smoke, the install echoes) updated to match.
• benchmarks.py::_require_agent_auth — noted; agent tokens aren't the human must-change accounts, but I'll fold the same shared-helper treatment in if you'd like it in this PR.
• MCP_SERVICE_PASSWORD and the Sync Helm chart tag / appVersion / package.json with VERSION — PR #177 Blocker 2 #180 secrets.yaml/values.yaml conflict — opening the MCP follow-up issue now so this doesn't close Default admin credential (admin/changeme) is live and API-reachable on every fresh deployment #184 on a half-fixed class, and the Sync Helm chart tag / appVersion / package.json with VERSION — PR #177 Blocker 2 #180 overlap reconciles at merge.

(Transparency: a test-generated initial_admin_password slipped into one pushed commit; I amended it out — 0 occurrences on the branch now — and added backend/keys/ to .gitignore plus a tmp-KEYS_DIR fixture so it can't recur.)

[mwiget]

Blocker 1 is closed the right way -- one helper, both WS validators calling it, three unit tests plus the :path regression test I asked for. Nearly there. One thing in the new helper needs changing, and it's a few lines.

Fixed

• token_requires_password_change loads the User row instead of trusting claims, and both _validate_ws_tokens call it, so pod exec, log follow, DPU serial console and BMC/OS SSH now refuse a seed-credential token.
• test_path_route_with_auth_me_suffix_is_not_exempted pins the exact bypass I reproduced -- that is the test that would have caught it.
• KEYS_DIR is the established variable (core/config.py:20, same /app/keys default, same volume as the JWT and encryption keys), so the file lands where operators already look, and .gitignore covers backend/keys/. Moving the secret out of the log line and leaving a pointer is better than what I would have settled for.

Blocking -- the new gate fails open

except Exception:
    return False

False means "no password change owed", so every failure resolving the user is an allow. The docstring's justification -- "an invalid/expired token is refused by the caller" -- is true of decode_token, but that is not the only thing that raises here. get_user_from_token also raises UnauthorizedError for:

• "User not found" -- the account was deleted
• "Account is disabled" -- the account was deactivated

Both are swallowed into "allowed". So a deleted or deactivated admin keeps pod exec and BMC SSH until their JWT expires. That is not a regression -- the WS path was claims-only before this PR, so it was already true -- but the lookup is now right there, and discarding its answer is a choice. The third case, a transient DB error re-opening the gate, is new.

Distinguish "resolved, no change owed" from "could not resolve" and let the caller refuse on the second:

def token_user_state(token: str) -> User | None:
    # Resolve the JWT's user, or None if it cannot be resolved.
    from database import get_db_context
    try:
        with get_db_context() as db:
            return get_user_from_token(db, token)
    except Exception:
        return None

user = token_user_state(token)
if user is None or user.must_change_password:
    await websocket.close(code=4401, reason="Unauthorized")
    return False

Same shape, fails closed, and it brings the WS paths in line with get_current_user -- which does refuse a disabled account -- rather than leaving them permanently more permissive. The existing test_false_on_garbage_token becomes "returns None", and a deactivated-user case is worth adding beside it.

Non-blocking

• open(pw_path, "w") then os.chmod(0o600) creates the file at 0666 & ~umask first -- typically 0644 -- and narrows it after. Small window, container-local, but it is a plaintext credential: os.open(pw_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) never widens it. os.makedirs matches what config.py already does, so I would leave that alone.
• The fallback that logs the plaintext when the write fails is the right call -- better than locking the operator out -- and it cannot fire under Helm, where DEFAULT_ADMIN_PASSWORD is always set and generated stays False.
• Nothing removes the file after first login. Fine given the account is must_change_password, and the log line says to delete it; just worth knowing it persists on the volume.
• Still standing: the #180 conflict in secrets.yaml/values.yaml, and MCP_SERVICE_PASSWORD wanting an issue before this closes #184.

[mwiget]

return False is "no password change owed", so this turns every resolution failure into an allow.

The docstring's reasoning covers decode_token, but that is not the only raiser on this path. get_user_from_token raises UnauthorizedError for two states the caller has not already screened:

user = db.query(User).filter(User.username == username).first()
if not user:
    raise UnauthorizedError("User not found")
if not user.is_active:
    raise UnauthorizedError("Account is disabled")

Both land here and become "allowed" -- so a deleted or deactivated admin keeps pod exec, log follow and BMC/OS SSH until the JWT expires. Not a regression (the WS path was claims-only before), but the row is loaded now and this discards what it says. A transient DB error re-opening the gate is new.

Returning the user rather than a bool lets the caller fail closed and fixes all three at once:

def token_user_state(token: str) -> User | None:
    # Resolve the JWT's user, or None if it cannot be resolved.
    from database import get_db_context
    try:
        with get_db_context() as db:
            return get_user_from_token(db, token)
    except Exception:
        return None

user = token_user_state(token)
if user is None or user.must_change_password:
    await websocket.close(code=4401, reason="Unauthorized")
    return False

That also puts the WS paths on the same footing as get_current_user, which does refuse a disabled account today. test_false_on_garbage_token becomes an is None assertion, and a deactivated-user test belongs next to it.

[mwiget]

Non-blocking: the file is created before it is narrowed. open(pw_path, "w") uses 0666 & ~umask -- 0644 under the usual container umask -- and the chmod on the next line closes it after. A plaintext admin credential exists world-readable for that window.

fd = os.open(pw_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, "w") as fh:
    fh.write(seed_password + "\n")

Never wider than 0600, and O_EXCL means a pre-existing file is an error rather than a silent overwrite -- which is what you want for a write that only ever happens on first boot.

os.makedirs(keys_dir, exist_ok=True) matches what core/config.py:56 already does for the JWT and encryption keys, so I would leave that consistent rather than special-case it here.

[jgruberf5]

Fixed — you're right, and it's a worse hole than "not a regression" implies: a disabled admin holding pod exec until token expiry is exactly the account you'd have just disabled to cut off access.

token_requires_password_change(...) -> bool is replaced by token_user_state(token) -> User | None — returns the row on success, None on any resolution failure (invalid/expired token, deleted account, disabled account, transient DB error). Both WS validators now refuse on user is None or user.must_change_password, so the gate fails closed and lines up with get_current_user — the WS paths are no longer permanently more permissive than REST.

Tests reworked to the new shape, and I added the two cases the fail-open was hiding: test_none_for_deactivated_account and test_none_for_deleted_account both resolve to None → WS refuses.

Non-blocking adopted: the password file is now created with os.open(pw_path, O_WRONLY|O_CREAT|O_TRUNC, 0o600), so the plaintext never sits at 0644 between create and chmod. I used O_TRUNC rather than O_EXCL so a stale file from a prior seed is overwritten cleanly instead of dropping to the log fallback — same never-widen guarantee.

Standing items acknowledged: the #180 secrets.yaml/values.yaml conflict (merge-time rebase), and MCP_SERVICE_PASSWORD now has its own issue — #187 — so this doesn't close #184 on a half-fixed class. The plaintext-fallback and the file persisting past first login you're fine with; noted.

[mwiget]

Both of my blockers are fixed, and I checked the fixes rather than reading them. CI has gone red though, and the failures are worth being precise about: they are the old contract failing, not the new code being wrong, and the fix belongs in the tests.

Fixed

• The gate fails closed. token_user_state returns User | None, both validators refuse on None or must_change_password. The tests you added cover exactly the cases I named -- deactivated account, deleted account, garbage token -- so a disabled admin no longer keeps pod exec until their JWT expires.
• File mode. os.open(..., O_CREAT | O_TRUNC, 0o600) never widens. Using O_TRUNC rather than the O_EXCL I suggested is the better call and I'd keep it: O_EXCL would hard-fail on a stale file left by a prior seed on a reused volume, which locks the operator out of the very thing the file exists to give them.

I also checked the trap this refactor invites, since it would have been an ugly one: token_user_state now returns the ORM instance and the caller reads must_change_password after the session has closed. That works only because get_db_context closes without committing, so the loaded column stays readable on the detached instance -- verified against a minimal SQLAlchemy repro. Worth a line in the docstring, because if anyone later adds a db.commit() to get_db_context, expire_on_commit=True expires that attribute and every WebSocket starts failing closed with no obvious cause.

Blocking -- CI is red

FAILED tests/component/test_k8s_websocket.py::TestValidateWsToken::test_valid_admin_token_returns_true    - assert False is True
FAILED tests/component/test_k8s_websocket.py::TestValidateWsToken::test_valid_operator_token_returns_true - assert False is True
FAILED tests/component/test_k8s_websocket.py::TestValidateWsToken::test_valid_viewer_token_returns_true   - assert False is True
3 failed, 3140 passed

These three mint create_access_token({"sub": "testadmin", "role": "admin"}) with no User row behind it, which is precisely the claims-only contract this PR is removing. token_user_state can't resolve testadmin, returns None, and the validator refuses -- correct behaviour, stale test.

Fix the tests, not the gate. Relaxing token_user_state back toward fail-open to make them pass would put the hole straight back.

I checked whether any real token would break the same way, since that would be a different and much worse story. It won't: the only subjects that aren't usernames are the agent tokens --

create_access_token({"agent_id": agent.id, "role": "agent", "sub": agent.name}, ...)   # benchmark_agent_provision_service
create_access_token({"sub": "forge-builtin-agent", "role": "agent"}, ...)              # startup_steps

-- and both validators reject role == "agent" before they ever reach the lookup (("admin","operator","viewer") for k8s, ("admin","operator") for dpus). Every token that gets as far as token_user_state was minted at /api/auth/login from a real user.username. So this is genuinely fixture-only.

While you're in that file: it currently tests the helper in isolation and the validator only for the reject paths. A case at the _validate_ws_token level -- real must-change user refused, real normal user allowed -- would pin the wiring between the two, which is the part that actually regressed here.

Still standing

• benchmarks.py::_require_agent_auth reads the role claim without loading a row (non-blocking, lower impact).
• Conflicts with #180 (approved) in helm/bnk-forge/templates/secrets.yaml and values.yaml.
• MCP_SERVICE_PASSWORD = "mcp-service-changeme" on a role=admin account still has no tracking issue -- worth filing before this closes #184.

[mwiget]

This is right, and I verified the non-obvious part: token_user_state returns the ORM instance from inside with get_db_context() as db:, and must_change_password is read here, after that session has closed.

It works — but only because get_db_context ends with a bare db.close() and never commits, so the loaded column value survives on the detached instance. I confirmed the exact pattern against SQLAlchemy 2.0:

must_change_password after close -> True

Worth a sentence in the token_user_state docstring, because the failure mode is nasty and remote: add a db.commit() to get_db_context at any point in the future and expire_on_commit=True expires this attribute, the access raises DetachedInstanceError, the surrounding except Exception catches it, and every WebSocket — pod exec, log follow, all three DPU consoles — starts refusing with 4401 and no diagnostic. Failing closed is the right direction, but nobody would find that from the symptom.

Returning a plain tuple or a small dataclass ((exists: bool, must_change: bool)) read inside the session would sidestep it entirely, if you'd rather not rely on the detached-read.

[mwiget]

The test fix is exactly right -- you updated the tests to the new contract rather than relaxing the gate, which was the thing I most wanted not to happen. The two cases you added on top (test_must_change_user_refused, test_token_for_missing_user_refused) are the ones that pin the wiring between token_user_state and _validate_ws_token, which is precisely what regressed.

CI is still red, but only on lint, and it's mechanical:

=== Backend Lint (ruff) ===
I001 [*] Import block is un-sorted or un-formatted
  --> tests/component/test_k8s_websocket.py:62:9
  --> tests/component/test_k8s_websocket.py:77:9
  --> tests/component/test_k8s_websocket.py:91:9
Found 3 errors.

All three are the same shape -- a second from services.auth_service import ... added a couple of lines below the first, which breaks the sorted block. Folding them into one import clears it:

        from routes.k8s_websocket import _validate_ws_token
        from services.auth_service import create_access_token, create_user

        create_user(db, "testadmin", "testadmin@t.com", "pw", role="admin", must_change_password=False)
        db.commit()

I001 carries [*], so ruff check --fix backend/tests/component/test_k8s_websocket.py will do all three. The two new tests further down already use the combined form, so it's only the three you edited in place.

Nothing else outstanding from my side on the code -- the fail-closed gate, the 0600 file and the test coverage all check out. Once lint is green and the component suite goes with it, I'll approve.

Still worth carrying (unchanged, none of it blocking):

• A line in the token_user_state docstring about the detached-instance read, since a future db.commit() in get_db_context would silently take every WebSocket down.
• benchmarks.py::_require_agent_auth reads the role claim without loading a row.
• The secrets.yaml / values.yaml overlap with #180 (approved) and now #188 as well -- three PRs editing the same generated-secret block, so the rebase order matters.

[jgruberf5]

CI is green again — the three TestValidateWsToken failures were exactly the stale claims-only contract you called out, fixed in the tests, not the gate (0486344):

• test_valid_{admin,operator,viewer}_token_returns_true now create a real User(..., must_change_password=False) and commit before minting the token, so they exercise the new resolve-a-real-row path instead of the removed claims-only one.
• Added the wiring cases you asked for at the _validate_ws_token level: test_must_change_user_refused (real must-change user → refused) and test_token_for_missing_user_refused (real token, no row → refused). 36 passed locally.

And the detached-instance trap you flagged is now documented (5392e16): token_user_state's docstring spells out that the caller reads must_change_password after the session closes, which is only safe because get_db_context closes without committing — so a future db.commit() there (with expire_on_commit=True) would silently make every WebSocket fail closed. Left O_TRUNC as-is per your point about stale-file lockout.

Still-standing items: benchmarks.py::_require_agent_auth is tracked separately (non-blocking); the mcp credential class is now issue #187 fixed in #188; the #180 secrets.yaml/values.yaml overlap is a known rebase.

[mwiget]

Lint is green and the component suite passed, so the WS test fix landed correctly. P3 · Integration Tests · Backend is now red for the same fixture reason one layer up -- same class, same remedy.

FAILED tests/integration/test_routes_k8s_websocket.py::TestPodExecAuth::test_exec_valid_viewer_token_accepted   - starlette.websockets.WebSocketDisconnect
FAILED tests/integration/test_routes_k8s_websocket.py::TestPodExecAuth::test_exec_valid_operator_token_accepted - starlette.websockets.WebSocketDisconnect
2 failed, 752 passed

The cause is neat once you line it up:

def _make_token(role: str = "admin") -> str:
    return create_access_token(data={"sub": f"test{role}", "role": role})

so the three tests mint testadmin, testviewer, testoperator -- and sample_user (conftest.py:220) creates exactly one user, testadmin, with must_change_password=False. That is precisely why the admin case passes and the other two don't: testviewer and testoperator have no row, token_user_state returns None, and the validator refuses. Correct behaviour, stale fixture -- same as the component tests, and again the fix belongs in the tests rather than the gate.

Note _make_token("operator") gives testoperator, not the testop you created in the component suite, so it needs its own row.

Sweep for the rest

I checked every other test that drives a WS validator, so this shouldn't need a third round:

file	verdict
tests/component/test_k8s_websocket.py	clean — every sub it mints now has a row, and the one that doesn't ("sub": "user", role guest) is the invalid-role test, rejected before the lookup
tests/integration/test_routes_k8s_websocket.py	the two failures above
tests/integration/test_routes_benchmarks.py	unaffected — the agent WS uses its own _agent_ws_authorized, which this PR doesn't touch
tests/integration/test_routes_operator_ws.py	unaffected — /ws/operator uses a registration token, not _validate_ws_token

One gap worth knowing rather than fixing here: there are no tests for dpus_websocket._validate_ws_token at all (backend/tests/ has nothing matching dpus). It got the identical fail-closed change, so serial-console / ssh-bmc / ssh-os auth is currently unpinned in either direction. Not this PR's debt to pay, but it's the reason the k8s side caught two rounds of fixture drift and the DPU side caught none — there was nothing there to catch it.

Everything else on this PR is settled from my side: fail-closed gate, exact-path exemption with its regression test, 0600 password file, and the docstring on the detached-instance trap. Once these two fixtures have rows I'll approve.

[jgruberf5]

Fixed — same class, same remedy one layer up (f83474b). test_exec_valid_viewer_token_accepted / test_exec_valid_operator_token_accepted now pull in sample_viewer_user / sample_operator_user, so testviewer and testoperator have real rows and token_user_state resolves them; the admin case already had its row via sample_user. And yes — I used the role-specific fixtures whose usernames match _make_token's test{role} (testviewer/testoperator), not the component suite's testop. Full TestPodExecAuth passes (5); whole-tree ruff check . clean. CI re-running now.

Thanks for the sweep table — I confirmed the same: the only files touching _validate_ws_token are the two k8s ones, benchmarks/operator-ws use their own authorizers, and the component guest case is rejected pre-lookup.

On the real gap you named: dpus_websocket._validate_ws_token got the identical fail-closed change with zero tests, so serial-console / ssh-bmc / ssh-os auth is unpinned in both directions. Agreed it's not this PR's debt — filed #189 to add the DPU-side coverage (must-change refused, missing-row refused, valid allowed) so the DPU side can't drift silently the way the k8s side did.

[mwiget]

The fixtures are fixed and the suite proves it. Approving at 241c768e.

P3 · Integration Tests · Backend: 754 passed, 0 failed (job 96416511755, success). The previous run on this branch was 2 failed, 752 passed — so the count moved 752 → 754 with the failures gone, which is exactly the two tests I flagged and nothing else changing behavior. f83474b4 wires sample_viewer_user / sample_operator_user into test_exec_valid_{viewer,operator}_token_accepted; I confirmed those fixtures exist at backend/tests/conftest.py:237 and :254 and create testoperator / testviewer with must_change_password=False, which is what _make_token's sub = f"test{role}" needs to resolve.

Worth saying plainly: those two failures were the gate working. _validate_ws_token now refuses a token whose subject has no User row, and a rowless valid-signature token is precisely what an attacker holding a JWT for a deleted account would present. A test suite that kept passing through that change would have been the bad outcome.

241c768e — the doc sweep is thorough and the commands in it are correct. Seven surfaces updated (Makefile ×2, Developers Guide ×2, dist/README.md, dist/install.sh, docs/INSTALLATION.md, vm-bnk-forge/README.md), and I checked the two claims a reader would actually type:

• docker exec bnk-forge-backend cat /app/keys/initial_admin_password — container_name: bnk-forge-backend at docker-compose.yml:115 ✅, and /app/keys is the volume-backed mount bnk-forge-keys:/app/keys ✅, so it survives a restart rather than living only in a dead container's layer.
• The vm-bnk-forge/README.md security section now says "generated admin password" instead of "comes up with admin / changeme" — the right place to fix it, since that paragraph is specifically warning about attaching a public IP.

Scoping the MCP credential lines in dist/.env.example out to #188 is correct; that file's remaining changeme belongs to the MCP scheme, not this one.

I swept for anything the sweep missed. The remaining changeme literals are scripts/e2e/config.py:184 and tests/e2e/config/test-config.ts:15-18 — both are harness defaults, and scripts/e2e/steps.py:173 plus tests/e2e/E2E_STRATEGY.md:30 already tell the operator to deploy with DEFAULT_ADMIN_PASSWORD set, so they're coherent rather than stale. E2E isn't wired into ci.yml, so nothing breaks either way.

Everything I raised across this PR is now settled: fail-closed token_user_state with the detached-instance docstring, exact-path exemption plus its regression test, the 0600-at-creation password file, and now the WS fixtures. The DPU-side gap I flagged — dpus_websocket._validate_ws_token got the identical change with no tests either way — is tracked as #189, which is the right place for it.

One note for merge order: #183's install-guide rewrite documents this PR's DEFAULT_ADMIN_PASSWORD / /app/keys/initial_admin_password scheme, and that code exists only here. Merging #186 before #183 keeps the customer guide honest at every point; the other order opens a window where it describes a file that doesn't exist yet. I've said the same on #183.

[jgruberf5]

@bonnyr-f5 round 2 — both bypasses were real and survived my round-1 fix. Fixed + verified (116 auth/ws/benchmark tests) in 7b4e234d.

• Middleware failed OPEN on None: you're exactly right — decode_token already succeeded, so None is the fail-closed case. Now refuses on None, matching the WS validators.
• Agent bearer path never gated: _require_agent_bearer now resolves the human-role token's user and refuses a must-change one (regression test: real must-change admin → 400). Agent/service tokens keep role-based behaviour.
• E2E logged in with changeme: added DEFAULT_ADMIN_MUST_CHANGE (default True; prod never disables the gate); the ephemeral e2e stack seeds a known non-default admin with it off, and the two test configs match.
• The rotation-undo + admin-clobber class is closed by Stop shipping the MCP service default credential; make Helm use the mcp account #188's provenance fix (below).

[bonnyr-f5]

Review: BLOCK

Round 3, cold re-audit of 7b4e234 against origin/staging 4a52ed4 — full 39-file diff, fresh
context, no access to round 2's findings or to your replies. All execution in a git archive sandbox.

A lot of this is now genuinely solid, and I attacked it rather than reading it:

• The generated-password mechanism is sound. secrets.token_urlsafe(18), file mode verified 0o600
  by os.stat, the plaintext is never logged, and it is stable across restarts.
• The exemption matching is exact, not prefix-based. Seven crafted bypass attempts — trailing
  slashes, .., case variants, query strings, a longer path sharing an exempt prefix — all correctly
  403/307.
• Round 2's INV-17 finding is fixed: DEFAULT_ADMIN_PASSWORD now reaches the process in all five
  deployment modes.
• A full route-tree walk (656 route entries) found the 12 middleware-only /api routes all → 403, and I
  classified all 20 middleware-exempt routes individually.
• helm lint/template clean, INV-13 hasKey guard present, ruff clean, mypy errors identical to
  base, and 8,851 backend tests pass.

What blocks is where the sweep stopped, not the design.

BLOCKER 1 — must_change_password = True does not invalidate a credential that is already published

_rotate_known_default_admin in backend/services/auth_service.py. Its own docstring identifies the
hazard exactly:

/api/auth/change-password (exempt from the gate) is exactly the endpoint that lets that credential
persist. On every boot, if the admin still authenticates with a known shipped default, force a password
change so the seed credential can no longer be used to reach the API.

Then the remedy is admin.must_change_password = True — which depends on that same exempt endpoint. The
three pieces:

• PASSWORD_CHANGE_EXEMPT_PATHS contains /api/auth/change-password (it must, or nobody could rotate).
• change_password verifies current_password against the stored hash — i.e. against changeme.
• changeme is published in dist/README.md:67, user-pack/install-guide.html:271 and
  scripts/ibm_cloud_bnk_forge.sh:705.

So on any pre-#184 install, after this boot-time remediation runs:

POST /api/auth/login          admin / changeme                    -> 200
POST /api/auth/change-password  current=changeme  new=<attacker>  -> 200
POST /api/auth/login          admin / <attacker>                  -> 200
GET  /api/projects            (as owned admin)                    -> 200

Whoever reaches login first wins, and the current value is in three public files. The legitimate operator
is simultaneously locked out, with no migration and no out-of-band retrieval path. The flag makes the
account require a rotation; it does not make the known credential unusable.

Class fix: a mitigation must remove the capability, not request its removal. On detecting a known default
on an existing row, overwrite the hash with a fresh random secret and surface it the same way a fresh
install does (/app/keys/initial_admin_password, mode 0600, logged once) — or deactivate the account and
require an explicit operator action. Setting a flag is not a remediation for a public secret.

BLOCKER 2 — DEFAULT_ADMIN_MUST_CHANGE reaches no process, and that breaks the release gate

This PR adds two settings and plumbs one. DEFAULT_ADMIN_MUST_CHANGE has exactly three references:

.github/workflows/e2e-tests.yml:116   DEFAULT_ADMIN_MUST_CHANGE: "false"   # sets it
backend/core/config.py:108            declares it
backend/services/auth_service.py:250  reads it

It is absent from all three compose files and from helm/bnk-forge/templates/_helpers.tpl, while its
sibling DEFAULT_ADMIN_PASSWORD is wired into all four. With no env_file anywhere, the workflow's value
never reaches the container:

$ DEFAULT_ADMIN_PASSWORD=… DEFAULT_ADMIN_MUST_CHANGE=false docker compose config
DEFAULT_ADMIN_PASSWORD    = 'e2e-Admin-Pass-1'
DEFAULT_ADMIN_MUST_CHANGE = <ABSENT>          env_file = <none>

Simulating that exact container env: e2e login → 200 (login is exempt), then GET /api/projects → 403,
/api/system/info → 403, /api/clusters → 403. e2e-tests.yml runs on push: tags: v* and its own
header calls itself required for release, so this breaks the release gate rather than just a test. Add the
variable to the three compose files and _helpers.tpl alongside its sibling.

Major — INV-20 fail-open, reintroduced by its own fix

backend/routes/benchmarks.py:134:

agent_user = token_user_state(token)
if agent_user is not None:
    try:
        enforce_password_change(request.url.path, agent_user)
    ...

token_user_state's documented contract is that the caller refuses on None — fail closed. There is
no else, so None means "skip the gate": fail open. Proven with POST /api/benchmarks/agents using
a nonexistent user's admin JWT → 201.

This is the same class as the round-2 finding at :107, re-committed 27 lines from the site that was
fixed. Three of four token_user_state call sites uphold the contract; this one doesn't. The sweep needs
to enumerate every call site and be re-run after the fix — a fix is a new region of code, not a
verified one.

Major — Helm and dist/ still wire MCP to admin/changeme, so MCP is dead on both after this PR

helm/bnk-forge/values.yaml and dist/.env.example still point the MCP service at the human admin
account with the published default, while this PR changes what that account's password is. Result: MCP
authentication fails on both paths.

I'm not filing this as a defect against this PR — config.py:109-111 declares the deferral
explicitly ("its fix is entangled with the chart's mcp-password wiring — tracked separately"), and the fix
is #188's. I'm recording it because it is the same observation from the other side of the #186/#188 split:
each PR's tests pass on its own tree, and the tree that ships is the merged one. See the cross-PR note
below.

Major — ensure_service_user has no identity check here and clears this PR's own gate

At this head it reconciles by name with must_change_password=False, so it can clear the gate this PR
introduces. #188 replaces this with a provenance check; flagging it so the two don't diverge during the
merge.

Major — customer docs tell operators to grep the logs for a password the code deliberately doesn't log

Three docs instruct retrieving the initial password from container logs. The implementation writes it to
/app/keys/initial_admin_password and specifically avoids logging the plaintext — which is the right
call. Point the docs at the file. Helm NOTES.txt also hands over a credential that doesn't work on
upgrade, and INSTALLATION.md lists the same command twice under "either".

Cross-PR — #186 and #188 must land as one tested tree

git merge-tree reports content conflicts in 9 files, every one on the credential surface:

.env.example                                docker-compose.yml
backend/tests/component/test_auth_service.py  helm/bnk-forge/templates/_helpers.tpl
dist/docker-compose.yml                     helm/bnk-forge/templates/secrets.yaml
docker-compose.local.yml                    helm/bnk-forge/values.yaml
                                            scripts/ibm_cloud_bnk_forge.sh

Both suites pass separately; no tree containing both has ever been built or tested, and that is the
tree that ships. Please resolve once on an integration branch and run the auth suites, helm template,
and docker compose config over all five compose files against the merged result.

Also note #183 and scripts/ibm_cloud_bnk_forge.sh:708 both document
/app/keys/initial_admin_password, which exists only here — so this PR is a hard prerequisite for
both, and if it slips while they land, they ship instructions for a file that doesn't exist having
deleted the accurate warning they replaced.

Review Assessment

• Verdict: BLOCK
• Audit SHA: 7b4e234
• Cold Audit Performed: Yes
• Invariants Verified: INV-6 (clean), INV-7 (n/a — no migration), INV-10 (clean — 656 routes walked), INV-11 (violated), INV-12 (deferred to Stop shipping the MCP service default credential; make Helm use the mcp account #188), INV-13 (upheld), INV-16 (clean), INV-17 (violated — DEFAULT_ADMIN_MUST_CHANGE), INV-18 (violated — name-based reconcile), INV-20 (violated at benchmarks.py:134), new INV-26 (violated)
• Git & Harness Cleanliness: Clean

Findings & Action Items

• Major (Blockers):
  • auth_service.py _rotate_known_default_admin: flagging does not invalidate a published credential — randomize the hash or deactivate the account
  • config.py:108 + 3 compose files + _helpers.tpl: DEFAULT_ADMIN_MUST_CHANGE reaches no process; the release e2e gate 403s on every protected route
  • routes/benchmarks.py:134: if agent_user is not None: with no else — fail-open against the helper's contract; proven 201
• Minor (Non-blocking):
  • ensure_service_user reconciles by name and clears the new gate — reconcile with Stop shipping the MCP service default credential; make Helm use the mcp account #188's provenance check
  • Three docs + Helm NOTES.txt point at log output for a password that is written to a file by design
  • Land as one integration branch with Stop shipping the MCP service default credential; make Helm use the mcp account #188 (9-file conflict, all credential surface)
• Nits:
  • INSTALLATION.md duplicate command under "either"
  • Confirm the new e2e-Admin-Pass-1 literals are acceptable to the secret scanner once Wire gitleaks / shellcheck / script self-test CI gates — PR #177 Major #182 lands

[jgruberf5]

@bonnyr-f5 Thanks for the round-3 cold audit — every blocker reproduced and is fixed at 6c1bde0. Verification below was run against a fresh py3.11 venv (deps from requirements*.txt).

BLOCKER 1 — must_change_password=True didn't invalidate a published credential ✅ fixed

Confirmed the attack path in code first: /api/auth/change-password is in PASSWORD_CHANGE_EXEMPT_PATHS, and change_password verifies current_password against the stored hash (still changeme). So a flag alone left the published default usable to rotate the account.

_rotate_known_default_admin now removes the capability instead of requesting its removal: on detecting a known shipped default on an existing admin row it overwrites the hash with a fresh secrets.token_urlsafe(18), keeps must_change_password=True, and surfaces the new secret exactly like a fresh install — I extracted the fresh-install path into a shared _persist_generated_admin_password() (mode-0600 /app/keys/initial_admin_password, pointer logged, plaintext never logged) and both call sites now use it. The published changeme stops authenticating the moment this runs, and rotation is idempotent across boots (the second boot's verify("changeme", …) is False).

Test evidence (strengthened test_rotates_existing_admin_still_on_a_known_default + new test_rotation_is_idempotent_across_boots):

authenticate_user(db, "admin", "changeme")  -> raises UnauthorizedError   # capability removed
initial_admin_password exists, != "changeme", and authenticates           # fresh secret surfaced

BLOCKER 2 — DEFAULT_ADMIN_MUST_CHANGE reached no process ✅ fixed

Plumbed it the same way DEFAULT_ADMIN_PASSWORD is (the INV-17 fix you confirmed reaches all five modes): added ${DEFAULT_ADMIN_MUST_CHANGE:-true} to docker-compose.yml, docker-compose.local.yml, dist/docker-compose.yml, and DEFAULT_ADMIN_MUST_CHANGE (from .Values.secrets.adminMustChange, default true) to _helpers.tpl (+ values.yaml). Default is secure "true"; the e2e workflow's "false" now reaches the container.

Your exact repro, now passing:

$ DEFAULT_ADMIN_PASSWORD=e2e-Admin-Pass-1 DEFAULT_ADMIN_MUST_CHANGE=false docker compose config
DEFAULT_ADMIN_MUST_CHANGE: "false"      # was <ABSENT>
DEFAULT_ADMIN_PASSWORD:    e2e-Admin-Pass-1

helm template renders "true" by default and "false" on --set secrets.adminMustChange=false; helm lint clean. (Used a direct .Values ref + quote, not | default true, precisely so a bool false doesn't collapse back to the default.)

Major INV-20 — fail-open in benchmarks.py:134 ✅ fixed

Confirmed: if agent_user is not None: had no else, so a validly-signed admin/operator token resolving to no live User skipped the gate. token_user_state's contract is fail closed, so None now → 400 AGENT_AUTH_INVALID. This makes all four call sites consistent (middleware raises Unauthorized, dpus/k8s WS both close 4401 on None). New regression test_register_rejects_nonexistent_user_token proves a phantom-user admin JWT → 400. I also updated two older agent-bearer tests that asserted the old permissive behavior to seed a real User row (a human curl token always corresponds to one — that's how it was minted).

Minor — docs pointing at logs for a file-only password ✅ fixed

DEPLOYMENT.md and INSTALLATION.md no longer tell operators to grep logs for a password that is written only to /app/keys/initial_admin_password; the INSTALLATION.md duplicate command under "either" is now distinct compose/Helm retrieval; Helm NOTES.txt notes the install secret no longer authenticates after the first-login change.

Deferred / handed to siblings (unchanged here, by your call)

• ensure_service_user name-based reconcile and the Helm/dist MCP admin/changeme wiring remain Stop shipping the MCP service default credential; make Helm use the mcp account #188's provenance-check + mcp-password work — not touched here.
• The Stop shipping a default admin credential; generate it and enforce rotation #186/Stop shipping the MCP service default credential; make Helm use the mcp account #188 9-file credential-surface conflict still needs a single integration branch: resolve once, then run the auth suites + helm template + docker compose config over all five compose files on the merged tree. Fix registry/cosign/upgrade metadata + USER-gate authoring guide — PR #177 Majors #183 and ibm_cloud_bnk_forge.sh:708 still depend on this PR for /app/keys/initial_admin_password.

Verification

• ruff check . — clean
• mypy core/ schemas/ — Success: no issues found in 38 source files
• Auth suites: test_auth_service.py, test_benchmark_agent_auth.py — 61 passed; test_auth_service_pure.py, test_auth_middleware.py, test_routes_auth.py (unit+integration), test_auth_contracts.py — 60 passed; test_routes_benchmarks.py — 188 passed.
• helm lint clean; docker compose config verified on all three compose files.

[bonnyr-f5]

Review: BLOCK

Round 4, cold re-audit of 6c1bde0 against origin/staging 4a52ed4 — whole diff (39 files,
+742/-72, 14 commits), fresh context, re-audited as new code. The core mechanism is sound and I
want to say so first: the upgrade rotation genuinely destroys the old hash (INV-11/INV-26 hold for
the admin row), the must-change gate is wired at both HTTP auth-resolution points including the
API-token branch the tests miss, and the generated credential is strong (24 chars, token_urlsafe(18),
per-call, os.open(..., 0o600) with no 0644 window) and appears in no diff hunk, compose file,
Helm value, doc, or fixture. What blocks the PR is the surface it did not sweep.

BLOCKER 1 — a second shipped default admin credential stays live and is exempt from the new gate

backend/core/config.py:113 + backend/startup_steps.py:228-233: the mcp/mcp-service-changeme
account (role=admin, must_change_password=False) is re-forced every boot and is not covered by
the new must-change gate. Execution-proven: login 200, then 200 on /api/auth/users,
/api/system/process-metrics, and /api/projects with REQUIRE_AUTH=true. Removing the human
default while leaving a second admin-role default live doesn't close issue #184's class. (This is
#188's remit, but at this PR's ref it is live — see the cross-PR note.)

BLOCKER 2 — every shipped distro still points MCP at admin/changeme

helm/bnk-forge/values.yaml:35-36, dist/docker-compose.yml:364-365,
dist/docker-compose.local.yml:128, dist/.env.example:28-30, scripts/ibm_cloud_bnk_forge.sh:584-585
— two of which this PR edits. Those credentials can no longer authenticate once the default is
removed, and even setting the real password still 403s every tool call. Helm proven: helm template renders mcp-username: "admin", mcp-password: "changeme"; helm lint green. MCP is
broken on every install as shipped.

Major findings

• auth_service.py:232-277 + NOTES.txt:21-25 / docs/DEPLOYMENT.md:60-64 /
  INSTALLATION.md:128-140 — Helm upgrade rotation generates its own secret and ignores
  DEFAULT_ADMIN_PASSWORD, so the Secret the docs tell you to read holds a value that never
  authenticated; NOTES.txt asserts it "holds the ORIGINAL install password". (Live-cluster half
  UNPROVEN — lookup is inert offline.)
• auth_service.py:271-275,322-326 — docs say "the plaintext is never logged"; on an unwritable
  /app/keys (which entrypoint.sh:14-18 only warns about) the password is logged at WARNING.
  Proven — captured …could not persist to disk): M-7xI1jYSpEZpJb7LG5AJkDS on both the seed and
  rotation paths. No test reaches the branch.
• user-pack/install-guide.html:271,274 — the customer guide says grep "GENERATED password"
  in the logs; the normal path logs only a file path, and the stale "the default password is
  well-known" callout survives 3 lines below.
• helm/.../secrets.yaml:33 (+ :5,11,17,23) — the new hasKey guard fixes the missing-key
  case but still errors when the Secret has no .data map; the four pre-existing keys share it
  (4-topology synthetic chart proven).
• helm/.../_helpers.tpl:112 — adminMustChange: null renders value: and "false" renders
  "\"false\""; both make Settings() raise → backend crashloop (render + pydantic parse proven).

Minors (selected)

• backend/routes/benchmarks.py:1505-1511 — INV-10 residue: 5 of 6 JWT-resolving entry points
  gated; /ws/benchmarks/agents/{id} accepts a must-change admin (default true mitigates).
• auth_service.py:253-269 — read-then-write rotation with no row lock: two api replicas can
  desync the DB password from the file → permanent admin lockout (INV-8, code-proven absence of lock).
• auth_service.py:253 — remediation keyed on the name admin (INV-18 inverted); another
  role=admin row on changeme survives.
• docs/DEPLOYMENT.md:257-259 (INV-17) — runbook names MCP_USERNAME/MCP_PASSWORD, which the
  default compose stack does not read.

Cross-PR (BLOCK-class — for the integration owner)

git merge-tree 6c1bde0 <#188> → 9 conflicts across the whole credential surface (.env.example,
the three compose files, _helpers.tpl, secrets.yaml, values.yaml, ibm_cloud_bnk_forge.sh, the
shared auth test). Each PR's gates pass on its own tree; no tree containing both has been built or
tested, and that is the tree that ships. Land #186 + #188 on one integration branch, resolve once,
then run the auth suites + helm template + docker compose config over all five compose files
against the merged tree.

Review Assessment

• Verdict: BLOCK
• Audit SHA: 6c1bde0
• Cold Audit Performed: Yes
• Invariants Verified: INV-8, INV-10, INV-11, INV-12, INV-18, INV-23 VIOLATED; INV-13, INV-17, INV-26 PARTIAL; INV-4, INV-5, INV-16, INV-20 UPHELD
• Git & Harness Cleanliness: Clean

Findings & Action Items

• Major (Blockers):
  • config.py:113+startup_steps.py:228-233: 2nd default admin credential (mcp/mcp-service-changeme) stays live, exempt from the gate
  • values.yaml:35-36 + 4 distro files: every shipped distro points MCP at admin/changeme → MCP broken on every install
• Minor (Non-blocking): DEPLOYMENT/NOTES read a Secret that never authenticated; password logged on unwritable /app/keys; install-guide grep wrong + stale callout; secrets.yaml .data-nil; _helpers.tpl:112 crashloop; benchmarks WS gate; no rotation lock; name-keyed remediation
• Nits: duplicate KEYS_DIR resolution; MCP service account still role=admin; +2 DB round-trips per /api request

[jgruberf5]

@bonnyr-f5 — round-4 blockers addressed in 4e62648. Reproduced each in code first, then fixed and verified.

BLOCKER 1 — second shipped admin default (mcp / mcp-service-changeme) is now dead

Reproduced (pre-fix): startup_steps.seed_auth_step → ensure_service_user(db, "mcp", settings.MCP_SERVICE_PASSWORD) ran unconditionally every boot with the published default MCP_SERVICE_PASSWORD="mcp-service-changeme" (config.py:113), creating/reconciling a role=admin, must_change_password=False row. enforce_password_change only refuses must_change_password=True, so the account was gate-exempt and authenticate_user(db, "mcp", "mcp-service-changeme") succeeded — a live, publicly-known admin credential, the exact #184 class.

Fix (same shape as the admin remediation):

• MCP_SERVICE_PASSWORD now defaults to None (config.py) — no published value in source.
• ensure_service_user (auth_service.py) treats a known published default (mcp-service-changeme/changeme) or an absent password as "no usable secret": a fresh row gets a secrets.token_urlsafe(18) secret surfaced like the admin seed (mode-0600 initial_mcp_password + one-time pointer log); an existing row still holding a published default is overwritten with a fresh random secret; a row already holding a generated/operator secret is left intact (idempotent, no reboot churn). A genuine operator-set MCP_SERVICE_PASSWORD still reconciles.

Mutation test (the published value cannot authenticate): new TestEnsureServiceUser cases assert authenticate_user(db, "mcp", "mcp-service-changeme") raises UnauthorizedError after (a) fresh seed with the default, (b) None seed, and (c) an upgrade row that authenticated before the call and is dead after it; plus idempotency and operator-override cases. All green.

BLOCKER 2 — distros pointing MCP at admin/changeme: fixed #186's share, left #188's

Your framing is right that the surface splits. I fixed the parts that are mechanical env wiring and are broken because #186 removed admin/changeme:

• dist/docker-compose.yml, dist/docker-compose.local.yml, dist/.env.example, scripts/ibm_cloud_bnk_forge.sh: repointed BNK_FORGE_USERNAME/PASSWORD from ${MCP_USERNAME:-admin}/${MCP_PASSWORD:-changeme} to ${MCP_SERVICE_USERNAME:-mcp}/${MCP_SERVICE_PASSWORD} — the mcp service account, no shipped default. (The IBM script already mints a random MCP secret; it now feeds MCP_SERVICE_PASSWORD, so backend + MCP share it and MCP actually authenticates.)
• Root docker-compose.yml/docker-compose.local.yml: dropped the now-dead :-mcp-service-changeme fallback.

Verified: docker compose config on the changed files renders BNK_FORGE_USERNAME: mcp, BNK_FORGE_PASSWORD: "" (empty until the operator sets it); setting MCP_SERVICE_PASSWORD=realsecret renders it through.

What I deliberately did NOT touch — it's #188's, and I can prove it: the Helm mcp-username/mcp-password provenance. helm template still renders mcp-username: "admin" / mcp-password: "changeme" from values.yaml. Making Helm MCP work is not an env swap — it requires generating an mcp secret in secrets.yaml and injecting MCP_SERVICE_USERNAME/MCP_SERVICE_PASSWORD into the backend from that same secret so the seeded account and the MCP client resolve to one value. That is verbatim issue #188's item 3 ("_helpers.tpl injects MCP_SERVICE_USERNAME/PASSWORD into the backend from the same secret the MCP server reads"), as is the validate_production fail-fast (#188 item 1). Doing it here would collide head-on with #188 — which you already flag as a 9-conflict merge across exactly secrets.yaml/values.yaml/_helpers.tpl.

Also fixed (Major, #186's own code)

_helpers.tpl DEFAULT_ADMIN_MUST_CHANGE: adminMustChange: null rendered a bare value: (pydantic rejects the empty string → crashloop). | default true can't be used (sprig default treats bool false as empty and would flip an intentional false). Guarded with kindIs "invalid": nil → secure "true", explicit false still → "false", true → "true". Verified across all three via helm template; helm lint clean.

Test / tooling output

• 143 passed — test_auth_service, test_routes_auth, test_benchmark_agent_auth, test_k8s_websocket, test_routes_k8s_websocket (isolated py3.11 venv).
• ruff check and mypy core/config.py — clean.
• helm template/helm lint and docker compose config (all changed compose files) — green.

Cross-PR note (#188)

#188 declares MCP_SERVICE_PASSWORD → None, the compose fallback drop, and the Helm mcp provenance. This PR now also sets MCP_SERVICE_PASSWORD → None and drops the compose fallbacks (unavoidable to kill the live default), so those hunks will conflict with #188 — resolve on the integration branch, keeping #188's Helm mcp-secret generation + backend injection as the authoritative provenance. I did not touch that Helm wiring or validate_production.

Not merging — leaving that to the integration owner.

[jgruberf5]

@bonnyr-f5 — round-4 Major findings addressed on head 3ade3fac (builds on the BLOCKER/_helpers.tpl fixes in 4e626480). Each was reproduced first, fixed, and covered by a test; the plaintext-log guard is mutation-tested. Isolated py3.11 venv (uv). Counts below are actual.

Majors

1. auth_service.py:271-275,322-326 — plaintext logged on unwritable /app/keys

Reproduced (unwritable keys dir = a file used as a directory, NotADirectoryError): all three paths logged the generated plaintext, e.g. Seeded admin user 'admin' with GENERATED password (could not persist to disk): -W1DuNbvWyaaJADH15x8-FeA, the rotation path, and the mcp service path.

Fix: _persist_generated_password now raises GeneratedCredentialPersistError (the message never contains the secret) instead of returning (path, False) and letting callers log the plaintext. Every path persists BEFORE it commits the DB row, so a failure leaves nothing committed and the next boot retries cleanly (no admin lockout, no half-rotated row). seed_auth_step catches it and raises SystemExit — which escapes main.py's best-effort except Exception — so the process refuses to start rather than run with an unretrievable credential. The docs' "the plaintext is never logged" is now true.

Mutation test (TestUnwritableKeysDirNeverLeaksPlaintext, 4 tests): patch token_urlsafe to a sentinel, force an unwritable keys dir, assert the sentinel is in no log record and not in the exception, on the admin-seed, admin-rotate, mcp-seed and mcp-rotate paths. Verified effective: reintroducing logger.warning("...", password) fails all 4; removing it passes. seed_auth_step fail-closed→SystemExit also has a test.

2. auth_service.py:232-277 + NOTES.txt / docs/DEPLOYMENT.md / docs/INSTALLATION.md — secret provenance

The chart wires the admin-password Secret → DEFAULT_ADMIN_PASSWORD, which the backend honored only at first-seed. On upgrade-from-changeme the rotation generated its own secret and wrote it to the keys volume, so the Secret the docs cite never authenticated — NOTES.txt's "holds the ORIGINAL install password" was wrong for that path.

Fix: the upgrade rotation now rotates TO DEFAULT_ADMIN_PASSWORD when it is set and not a published default (mirrors how ensure_service_user honors MCP_SERVICE_PASSWORD), so the Secret/env the docs name is authoritative on upgrade too; otherwise it generates + writes the keys-file. Rotating to a published default (DEFAULT_ADMIN_PASSWORD=changeme) is refused. Single documented rule now: DEFAULT_ADMIN_PASSWORD set → that value (Helm: the Secret), no keys-file; unset → generated keys-file. NOTES.txt, DEPLOYMENT.md and INSTALLATION.md rewritten to it. Tests: test_rotation_honors_default_admin_password_when_set (Secret authenticates, no keys-file written) and test_rotation_refuses_to_rotate_to_a_published_default.

3. user-pack/install-guide.html:271,274 — wrong retrieval + stale callout

grep "GENERATED password" targeted a value the code writes to a file and never logs. Fix: point at docker compose exec backend cat /app/keys/initial_admin_password (with the DEFAULT_ADMIN_PASSWORD alternative), and replace the stale "the default password is well-known" callout (obsolete since #184) with a one-time-bootstrap-credential note.

4. helm/.../secrets.yaml:33 (+ :5,11,17,23) — .data-nil render error

Reproduced on a synthetic chart driving $existing: an existing Secret with no .data map errors index of untyped nil (the earlier hasKey guard hit it too — hasKey of nil). Fix: normalize .data to an empty dict once ({{- if and $existing $existing.data -}}{{- $data = $existing.data -}}{{- end -}}) and route all five keys through hasKey $data .... Proven across 4 topologies (no existing / full .data / .data missing admin-password / .data nil) — all render; the nil case that previously errored now generates fresh secrets. helm lint + helm template on the real chart: green.

(already fixed in 4e626480) _helpers.tpl:112 — DEFAULT_ADMIN_MUST_CHANGE

Verified the kindIs "invalid" fix renders correctly: unset→"true", false→"false", null→"true"; pydantic parses each. No further change.

Minors (quick, clearly-correct)

• benchmarks.py:1505-1511 (INV-10) — the agent WS Layer-2 branch validated only the JWT signature. Reproduced: a must-change human admin token returned None (admitted). Fix: for a non-agent role, resolve token_user_state and close 4001 if it owes a change or no longer resolves; agent-role tokens still connect. 4 new unit tests.
• auth_service.py:253-269 (INV-8) — added SELECT FOR UPDATE to the admin-rotation and service-reconcile row reads, so two api replicas can't desync the stored hash from the keys-file (no-op on SQLite). All suites green.
• docs/DEPLOYMENT.md:257-259,357 (INV-17) — the runbook named MCP_USERNAME=admin/MCP_PASSWORD, contradicting this PR's own compose (which reads MCP_SERVICE_*; MCP runs its own mcp account). Rewritten to MCP_SERVICE_PASSWORD.

Deferred (with reasoning)

• Chart mcp-password provenance across the 9-file surface (.env.example, the three compose files, _helpers.tpl, secrets.yaml, values.yaml, ibm_cloud_bnk_forge.sh, the shared auth test) — genuinely Stop shipping the MCP service default credential; make Helm use the mcp account #188. The related stale MCP_USERNAME/MCP_PASSWORD prose in mcp-server/README.md, docs/E2E-CRITICAL-004_MCP_SANITY.md and the user-pack .env.example table sits on that same surface; touching it here would collide with Stop shipping the MCP service default credential; make Helm use the mcp account #188's migration.
• Name-keyed remediation (INV-18) — the only role=admin credentials this project ships are admin (rotated) and the mcp service account (handled in ensure_service_user); any other role=admin/changeme row is operator-created, outside the shipped-default class, and generalizing has keys-file-naming implications better decided deliberately. Deferred rather than fixed blindly.

Evidence summary

• Suites: test_auth_service 49 passed, test_startup_steps 10 passed, test_benchmark_agent_auth 28 passed (+ test_routes_auth 24, test_routes_benchmarks green in the consolidated 118-passed run). Mutation-verified the plaintext-log guard.
• helm lint 0 failed; helm template clean; 4-topology synthetic proof for secrets.yaml.
• No compose files changed (so no docker compose config needed).

CI is running on 3ade3fac. Not merging.

[bonnyr-f5]

Review: BLOCK

Round 5, cold re-audit of 3ade3fa against origin/staging 4a52ed4. Auditor had no prior-round
context. All three blockers re-verified by me before posting.

Merges standalone: mechanically yes (rc=0, 0 conflicts). Functionally no —
backend/core/config.py:110-116 defers "the chart's mcp-password provenance" to #188, so merged
alone this PR removes the shipped MCP default that was the only thing making the backend and the MCP
client agree, without plumbing the replacement.

The core of this PR is correct, and it was proven by execution

Credit where it's earned — the round-4 Majors are genuinely closed:

• No plaintext leak. os.open(..., 0o600) at create time, pointer-only logging, fail-closed
  SystemExit on an unwritable keys dir. 163 tests green including the four sentinel tests and the
  mode-0600 assertion. An independent grep of every logging site found no plaintext path.
• The must-change gate is unbypassable. All 642 /api (method, path) pairs swept with a live
  must-change admin JWT — only 9 not refused, and every one is exempt by design
  (change-password, me, login, health, the three agent-public ingest paths that
  _require_agent_bearer gates, two registration-token paths). The API-token branch is gated too
  (probed: 403). All three JWT WS validators fail closed on None and on
  must_change_password.
• The upgrade path removes the capability, not just the flag (INV-26 upheld): the hash is
  overwritten, so changeme can no longer drive the gate-exempt /api/auth/change-password. Same
  for the mcp row.
• INV-13 fixed as a class (nil-.data folded once, all five keys hasKey-guarded), and the
  DEFAULT_ADMIN_MUST_CHANGE sprig-default trap correctly avoided across all three render states.

Every blocker below is in the second account and its distribution — not in the admin rotation
this PR set out to fix.

BLOCKER-1 · MCP_SERVICE_PASSWORD reaches the MCP client on 5 channels and the backend on ZERO

INV-17 — and the Origin line for that invariant already names #186, for DEFAULT_ADMIN_PASSWORD.
That instance was fixed and the class was re-committed one variable over.

Verified on this ref: MCP_SERVICE_PASSWORD appears in the root compose only under the mcp
service —

service mcp:  line 471:  BNK_FORGE_PASSWORD: ${MCP_SERVICE_PASSWORD}

— and nowhere in the backend's environment. docker compose config confirms the backend receives
only DEFAULT_ADMIN_*; probing settings.MCP_SERVICE_PASSWORD yields None, so the backend
generates a random secret while the client is handed the operator's .env value. They can never
agree, and no .env value fixes it. The base ref worked only because both halves defaulted to
mcp-service-changeme.

Consequences: six operator-facing files now state a false claim, and make mcp-readiness — the
banner's own "recommended next step" — fails on every install.

BLOCKER-2 · the chart still ships mcpUsername: admin / mcpPassword: changeme

helm/bnk-forge/values.yaml:35-36, rendered into the release Secret via templates/secrets.yaml:56-57
(proven by helm template). A shipped default admin credential on a public repo, in the PR titled
"stop shipping a default admin credential", pointing MCP at the very account this PR invalidates.

On attribution, to be precise: these lines exist at base (4a52ed4:helm/bnk-forge/values.yaml:28-29)
and this PR does not touch them. So it isn't a regression — it's an incomplete sweep of the class
the PR names in its own title. Given that title, I think it belongs in this PR rather than in #188.

BLOCKER-3 · ensure_service_user has no identity check, and BLOCKER-2 makes it live

INV-18, unfixed here. Verified: grep -nE 'RESERVED|reserved|ValueError|is_service_account' over
auth_service.py at this ref returns nothing. The function resolves its target purely by
User.username == username, with role: str = "admin" as a default parameter.

So pointing it at admin — exactly the username the chart still ships — rewrites the human admin
row, clears must_change_password, and forces role=admin / is_active=True. Probe output:

old generated admin password still works?  NO  -- human admin credential destroyed
mcp secret now authenticates as admin?     YES role=admin must_change=False

For contrast, #188 does add the guard (_RESERVED_HUMAN_USERNAMES = frozenset({"admin"}) →
raise ValueError). That's the perverse coupling: fixing BLOCKER-1 makes BLOCKER-3 live on Helm,
and the protection lives in a different PR. These two must land together, or the reserved-name
refusal needs to come back into this one.

Major · a blocking sync DB query inside async def dispatch

The gate adds a synchronous DB lookup on every authenticated request (previously only for rare
bnk_ tokens), duplicating the lookup get_current_user already performs. No registry home for
this class — proposing one: an auth gate added to middleware must not add a per-request DB round
trip that a downstream dependency already makes.

Minors

• with_for_update() cannot lock a non-existent row, so both create paths are unserialised
  despite the comment claiming otherwise (INV-8).
• _agent_ws_authorized Layer 1 returns before the new gate — latent, since no minting path
  produces a non-agent token carrying an agent_id claim.
• No test covers the API-token branch.
• Three stale env-var docs; no upgrade note for existing installs whose admin is silently rotated.

Swept as instructed, out of scope, pre-existing

dist/.env.example:22,25 ship live POSTGRES_PASSWORD / REDIS_PASSWORD defaults, with those
services host-bound in the server topology — a second class of shipped default. Pre-existing, not
yours, flagging it because the sweep found it.

The admin-rotation core here is the best-verified work in the series. The blockers are all about the
mcp account's plumbing and the chart, and BLOCKER-1 + BLOCKER-3 need to land with #188 rather than
ahead of it.

[bonnyr-f5]

Cross-PR merge-order constraints for this series are now tracked in #192.

Relevant here: this PR and #188 conflict on 16 files under both strategies, across the whole credential surface and both auth test modules — so the safety net merges at the same time as the code it guards. There is also a coupling worth noting: the reserved-name refusal that stops ensure_service_user retargeting the human admin row lives only in #188, so closing this PR's wiring gap without that guard present makes the retarget live on Helm. Land these two together, or #188 first with this PR rebased onto it.

[jgruberf5]

@bonnyr-f5 — thanks for the round-5 cold re-audit. You're right on every blocker; they were all in the mcp account and its distribution, not the admin core. Pushed f778478f. Here's what changed and, per your integration-branch point, exactly where #186 stops and #188 begins.

BLOCKER-1 — backend now receives MCP_SERVICE_PASSWORD on every deploy mode

Root cause confirmed exactly as you described: the var reached the mcp client on 5 channels and the backend on zero (config.py has no env_file), so the backend generated a random secret while the client used the operator value — no .env could reconcile them.

Fix — plumbed MCP_SERVICE_USERNAME/MCP_SERVICE_PASSWORD into the backend everywhere it reads env:

• docker-compose.yml, docker-compose.local.yml, dist/docker-compose.yml x-backend-env anchors; dist/docker-compose.local.yml inherits via compose's env-map merge (verified).
• scripts/ibm_cloud_bnk_forge.sh embedded compose anchor (it already writes a per-install MCP_SERVICE_PASSWORD to .env).
• Helm shared-env in _helpers.tpl, sourced from the release Secret's mcp-username/mcp-password keys — identical to mcp.yaml, so backend and client bind to the same secret.

Evidence:

$ MCP_SERVICE_PASSWORD=test-mcp-pw docker compose -f docker-compose.yml config | grep -A40 bnk-forge-backend | grep MCP_SERVICE
      MCP_SERVICE_PASSWORD: test-mcp-pw
      MCP_SERVICE_USERNAME: mcp

helm template now renders MCP_SERVICE_PASSWORD (valueFrom mcp-password) on api/worker/beat. This directly answers your standalone-merge critique ("removes the shipped default without plumbing the replacement") — the replacement is now plumbed on both compose and Helm.

BLOCKER-2 — chart no longer ships a default admin credential

You called it: given the PR title, this is #186's. values.yaml now ships mcpUsername: mcp and mcpPassword: ""; secrets.yaml generates mcp-password per-install and reuses it across upgrades (the exact admin-password pattern already in that file). No default renders:

$ helm template t helm/bnk-forge | grep mcp-
  mcp-username: "mcp"
  mcp-password: "EJRQQnVl8lIBFreWQeWD"   # random per-install, was "changeme"
$ helm template t helm/bnk-forge --set secrets.mcpPassword=op-secret | grep mcp-password
  mcp-password: "op-secret"              # operator value still honoured

BLOCKER-3 — ensure_service_user reserved-name guard

Reproduced your attack, then fixed it in this PR's code (you offered "or the reserved-name refusal needs to come back into this one"). Added _RESERVED_HUMAN_USERNAMES = frozenset({"admin"}) with a fail-closed raise ValueError before any lookup. I deliberately kept the identifier identical to #188's guard so the integration branch is a no-op merge, not a conflict. seed_auth_step is a FATAL step, so pointing a service account at admin now refuses boot rather than silently rewriting the human admin row. New tests prove the human admin's hash + must-change gate survive and the mcp secret cannot authenticate as admin.

Major — blocking sync DB query in async def dispatch

Agreed and clearly ours. _verify_api_token and token_user_state both open a sync Session; on the JWT branch that ran on the event-loop thread for every authenticated request. Both are now await run_in_threadpool(...), so the blocking round-trip is off-loaded and dispatch stays non-blocking.

Minors

• API-token branch coverage (your minor): added two middleware tests — must-change bnk_ token → 403, settled → pass — which also exercise the new threadpool path.
• INV-8 create-path: you're right that with_for_update() can't lock a non-existent row. Corrected the comment to be honest: the create path is serialised by the username UNIQUE constraint (losing racer's INSERT → IntegrityError → fail-closed retry that converges), and startup is single-replica in every shipped topology; the row lock is what matters on the reconcile/rotate paths at scale.
• _agent_ws_authorized Layer-1 ordering — left as-is; latent exactly as you noted (no minting path produces a non-agent token carrying an agent_id).
• dist/.env.example POSTGRES/REDIS defaults — pre-existing, out of Stop shipping a default admin credential; generate it and enforce rotation #186's scope, agree it's worth a separate issue.

#186 / #188 integration split (explicit, per your repeated point)

Everything above makes #186 standalone-secure: no shipped default credential renders on any distro, the backend reconciles to the same secret the client uses, and the reserved-name guard is live. The shared ~credential surface with #188 is auth_service.py (the guard — same identifier), values.yaml, secrets.yaml, and _helpers.tpl. Because the guard name matches and the chart changes complete the class #186 already owns, the integration branch should reconcile, not conflict. If #188's mcp-secret work restructures those keys differently, that reconciliation is the one place it happens — flag it there and I'll rebase.

Verification

• 192 passed across the auth/middleware/route suites (isolated py3.11 uv venv), incl. 4 new tests. test_routes_benchmarks + test_k8s_websocket gate suites green (36 passed).
• Mutation-checked: disabling the reserved-name guard → both BLOCKER-3 tests fail; dropping the API-token gate → the 403 test fails. Both caught.
• helm lint + helm template clean; docker compose config green on base, local overlay, dist base, dist overlay, and the ibm embedded compose.

Not merging — leaving that to you and the #188 sequencing.

[jgruberf5]

Superseded by the consolidated integration PR #193 (branch pr177-integration), which merges this and the other six #177 follow-up PRs in the dependency order from #192 and passes full CI (CI Gate green). Per #192, these seven share the credential and release/CI surfaces and could not merge in arbitrary order, so they land together via #193.

Closing unmerged, not abandoning: the branch is retained and all review history stays on this page for reference. See #193 for the integrated, validated result and #192 for the cross-PR conflict analysis.