🤖 feat: activate the CoderAIGatewayKey controller by ThomasK33 · Pull Request #346 · coder/coder-k8s

ThomasK33 · 2026-10-14T08:09:48Z

K10 of #156: the operator now runs the CoderAIGatewayKey controller, and the install bundle ships its CRD. This turns on AI Gateway key management in real installs, so it follows the activation rules of #315 (plan G3).

K10a (#341) is on main. I merged main into this branch, so its diff is only the K10b change.

What changes

Registration. controllerapp.SetupControllers registers CoderAIGatewayKeyReconciler. It uses the manager's API reader for live reads of the output Secret and of the reserved-name sources, and the real clock.
CRD and RBAC. The CRD moves from config/crd/dormant/ to config/crd/bases/ and into dist/install.yaml. The CI guard step for the dormant CRD is gone. The RBAC markers add get, list, watch and patch on coderaigatewaykeys (finalizer merge patch), get and update on coderaigatewaykeys/status, and update on coderaigatewaykeys/finalizers. There is no update on the object itself: the controller writes only its status. The secrets rule does not change (plan G2).
The allowKeys gate is live. The "no effect yet" sentences are gone from spec.aiGateway and AIGatewaySpec.allowKeys. The kind description now states the deletion delay, the retries and the no-control-plane case, instead of "revokes at once". The CRDs, dist/install.yaml and the reference docs are regenerated.
CI. The full Kind E2E path filter now includes internal/controller/coderaigatewaykey*, owned_secret.go and reserved_secret_names.go.
Docs. There is a new how-to, docs/how-to/manage-ai-gateway-keys.md (in the mkdocs nav), and a sample, config/samples/coder_v1alpha1_coderaigatewaykey.yaml. The index, deploy, architecture and getting-started pages name the new kind. The getting-started cleanup deletes keys before the control plane. The how-to covers:
- the trust level of a gateway key, and who may write these objects;
- the prerequisites, including RK13: node clocks within 2 minutes of each other, and the default 60 s API server request timeout that the settle window depends on;
- the Secret shape, gateway reload, every condition reason, rotation, Secret drift, the kill switch with both cases that need a second spec edit, and the stalls;
- the deletion flow, a warning to delete key objects before their namespace, the stall table with the 60 s and 10-minute retries, ControlPlaneGone and the bounded pass;
- every way out of a stuck finalizer: fix Coder or the operator user, delete the control plane, or remove the finalizer with a test-first JSON patch and then delete the keys by name;
- the known limits, and the RBAC table.

The K9 notes items (o) to (u)

Item	Disposition
(o) The bounded pass during control-plane deletion can leave one late key	Documented (how-to, "Delete a key"). Accepted: the pass must end so that a namespace deletion finishes. The RK7 sweeper follow-up covers leftovers.
(p) The bounded-pass log names the keys that it listed but did not delete	Fixed in #341.
(q) Retry a failed operator-access read once while the control plane is being deleted	Accepted as is: `CoderAIProvider` does the same, the log names the key IDs, and a retry would delay the namespace deletion that the bounded pass exists for.
(r) After a kill switch, unlisted retiring keys stay in status until a new current key is listed	Accepted: no create checks the 3-key limit while there is no current key, so they block nothing. The next LIST that lists the new key drops them.
(s) A LIST that lists the current key but omits a retiring key drops it	Documented (how-to, "Known limits"). Accepted: Coder v2.37.3 answers the LIST unpaginated and unfiltered, so only a proxy can cause it.
(t) The re-arm after a mismatch stays gated in the same generation	Tested in #341.
(u) The gate is lost when the status write right after a mismatch fails	Accepted: this is older than K9a, and the marker plus the settle window bound it to one more create, whose key the sweep deletes.

Tests

TestSetupControllersStartsEveryController: a manager with every controller of the app starts against envtest, and its caches sync. This covers duplicate field indexes and served kinds. It runs serially, and its cache sees only an empty namespace of its own.
TestCoderAIGatewayKeyAllowKeysGate on a live object: without the block, NotAllowed and zero Coder calls. With false, the same. With true, Ready. Back to false with a new trigger, NotAllowed: no rotation, and the key stays in the Secret and in Coder. This replaces the K2-era wording of TestControlPlaneAIGatewayAllowKeysLeavesDeploymentUnchanged. That test stays, because the field must still not roll the Coder Deployment.

Size: about 350 changed lines without generated files (dist/install.yaml, the CRDs, docs/reference/api/*). About 200 of them are the how-to.

Local gates on the head:

make verify-vendor, make build, make lint, make test, make test-integration.
go tool govulncheck ./....
go test -race ./internal/controller/....
make codegen, make build-installer and make docs-reference leave no drift.
Docs: make docs-check (mkdocs build --strict), plus markdownlint-cli2 0.18.1 and cspell 8.19.4, as in the docs workflow.
actionlint v1.7.10.

The full Kind E2E run and the licensed dogfood run on this exact head are in the G3 evidence comment. Dogfood found one gap: a namespace deletion leaves the keys in Coder, because Kubernetes deletes the control plane at the same time. The how-to now warns about it and gives the cleanup. A controller-side fix is tracked in #353.

Refs #156

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high • Cost: $136.24

Change-Id: I9e7b6f5ea04db940c6bd982d0a861a9a811ee0f6 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: Id9d090afd3e29026d578116a592057b1ab6f007a Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: I32890aa15aec7a173c3da1edcb0731302f8b2cde Signed-off-by: Thomas Kosiewski <tk@coder.com>

chatgpt-codex-connector · 2026-10-14T08:10:03Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T12:01:19.763518Z	`dd94392`	Manual request
🔒 Security Review	✅ Completed 2026-10-14T12:04:07.726216Z	`dd94392`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

ThomasK33 · 2026-10-14T08:10:09Z

@codex security review

chatgpt-codex-connector · 2026-10-14T08:15:19Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 4641a82031

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Change-Id: Idfe28ab7654a28103d15ae2e1a0faa806238c98f Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 · 2026-10-14T09:30:19Z

@codex review

ThomasK33 · 2026-10-14T09:30:21Z

@codex security review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 812167e302

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-14T09:38:55Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 812167e302

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…er test Change-Id: If7ef13f439aac8f7de5c432cf7ba1f8f45613d38 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 · 2026-10-14T10:11:47Z

@codex review

ThomasK33 · 2026-10-14T10:11:50Z

@codex security review

chatgpt-codex-connector · 2026-10-14T10:14:49Z

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 1a211c072b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-14T10:17:25Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 1a211c072b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

ThomasK33 · 2026-10-14T11:27:23Z

G3 on the head `1a211c0`

CI full Kind E2E: run 38547828976, job 115799178542 passed in 480 s on this head. The full Kind steps ran, and the "Skip full E2E on pull requests" step was skipped. The first head, 4641a820, also passed (job 115750663774).

Image identity: I built the image from git archive of this head. Its ID sha256:6e28334ce34f… equals the node image and the running manager container image. The operator ran with --app=controller, with the CRD and RBAC applied from this head.

Setup: a Kind cluster (kindest/node v1.32.0) running Coder v2.37.2 (the CI image digest) with a Premium license. aibridge was entitled and enabled, and GET /api/v2/ai-gateway/keys answered 200. The checks below read Coder's own request metrics and audit log, and the ai_gateway_keys table.

Step	Result	Coder writes
1. `allowKeys` absent	`NotAllowed` (Stalled), no Secret, no key in Coder	0
2. `allowKeys: true`	`Ready`. The Secret holds a 43-character key, the annotation `key-name` is `ck8s-<uid>-0`, the owner is the object.	1 POST
3. Scheduled rotation (status `currentKey.issuedAt` moved back 1 day)	Key 1 is current. Key 0 retires with `Ready=True`, `WaitingForGracePeriod`.	1 POST
4. Requested rotation (`trigger: r1`) while key 0 retires	Key 2 is current. Keys 0 and 1 retire.	1 POST
5. A third rotation (`trigger: r2`)	`RotationBlocked` with `Ready=True`, naming key 0	0
6. Secret drift (value overwritten)	Re-issue: key 3 is current, and 3 keys retire	1 POST
7. Second drift inside the grace period	`SecretConflict` (Stalled), no new key	0
8. Grace periods end	The retire step deletes keys 0-2, one per reconcile. When the conflict window ends, the drift is re-issued as key 4.	3 DELETE, 1 POST
9. Kill switch (current key deleted in Coder)	`KeyDeletedExternally` (Stalled), still the same 20 s later	0
10. Re-arm (new trigger)	Key 5 is current, `Ready=True`	1 POST
11. Delete (normal path)	A delete within 5 minutes of the last create shows `DeletingKeys` ("deletes its keys in Coder at …") and is gone after 296 s. Every key is deleted, and the Secret is gone.	2 DELETE
12. Delete during control-plane deletion (control plane held Terminating, access working)	The object is gone in 1 s, inside its settle window, after one bounded pass	1 DELETE
13. Namespace deletion (second control plane in `coder2`)	The namespace is gone in 11 s, but the key stays in Coder (see below)	0

Stuck finalizer, following the how-to as written (Coder scaled to 0, then the object deleted):

Check	Result
Condition	`CoderUnavailable`, "list AI Gateway keys: … connection refused", finalizer kept
JSON patch with a wrong index (`test` on `/metadata/finalizers/1`)	Refused, no change
The documented `test` + `remove` patch at index 0	`patched`, object gone
Deleting the keys whose name starts with `ck8s-<UID>-`	1 leftover key found and deleted (204), 0 left

Leak scan: the operator log, all recordings and every CoderAIGatewayKey YAML hold no 43-character alphanumeric run. The recordings show only a sha256 prefix of each Secret value.

Finding: namespace deletion leaves the keys in Coder (step 13). Kubernetes deletes the control plane and the CoderAIGatewayKey together.

In one run the control plane was gone first, so the release took the ControlPlaneGone path.
In the run on this head, the bounded pass ran while Coder's pod was already down. Its LIST got "connection refused", and the finalizer went as designed.
Either way the key stays valid in Coder, and the log names its ID. CoderAIProvider behaves the same.
Head 812167e3 added a warning to the how-to: delete the CoderAIGatewayKey objects first, then the namespace, with the cleanup command. The known limits now list this case too.
A controller-side fix, such as a sweeper (plan RK7) or blocking control-plane deletion while key objects exist, is a follow-up decision, not part of this PR.

Environment notes (not code):

The shared host hit its 64 GiB cgroup limit several times (OOM kills, three host restarts). Twice this restarted the Coder database or the Kind node.
Once the control plane object disappeared between my runs. I did not find the cause, and none of my scripts deleted it in that window.
I recreated it on the same database and re-ran every step on this head.
Steps 1-13 above come from runs at 10:33-11:24Z on 1a211c07. A full earlier pass on 4641a820 (08:35-09:29Z) showed the same results.

The video joins the six terminal recordings, rendered with idle gaps capped at 2 s, so waits look shorter than they were.

k10-dogfood.mp4

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

…ctivate Change-Id: If233f06bb871de790124f63b03f343998ab9ebc7 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 · 2026-10-14T11:58:11Z

@codex review

ThomasK33 · 2026-10-14T11:58:13Z

@codex security review

chatgpt-codex-connector · 2026-10-14T12:01:18Z

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: dd943924a2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-14T12:04:06Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: dd943924a2

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

ThomasK33 added 3 commits October 14, 2026 07:32

fix: trust only Coder's own 404 when deleting AI Gateway keys

6a14d5b

Change-Id: I9e7b6f5ea04db940c6bd982d0a861a9a811ee0f6 Signed-off-by: Thomas Kosiewski <tk@coder.com>

feat: activate the CoderAIGatewayKey controller

2353e1c

Change-Id: Id9d090afd3e29026d578116a592057b1ab6f007a Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs: allow the AI Gateway chart and reloader words

4641a82

Change-Id: I32890aa15aec7a173c3da1edcb0731302f8b2cde Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs: delete AI Gateway keys before their namespace

812167e

Change-Id: Idfe28ab7654a28103d15ae2e1a0faa806238c98f Signed-off-by: Thomas Kosiewski <tk@coder.com>

chatgpt-codex-connector Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread internal/controller/controllerapp_setup_test.go Outdated

Comment thread dist/install.yaml Outdated

fix: describe the AI Gateway key deletion delay and isolate the manag…

1a211c0

…er test Change-Id: If7ef13f439aac8f7de5c432cf7ba1f8f45613d38 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 mentioned this pull request Oct 14, 2026

Namespace deletion leaves AI Gateway keys and AI providers in Coder #353

Open

Base automatically changed from feat/ai-gateway-key-delete-404 to main October 14, 2026 11:47

Merge remote-tracking branch 'origin/main' into feat/ai-gateway-key-a…

dd94392

…ctivate Change-Id: If233f06bb871de790124f63b03f343998ab9ebc7 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 added this pull request to the merge queue Oct 14, 2026

Merged via the queue into main with commit 3172d66 Oct 14, 2026
13 checks passed

ThomasK33 deleted the feat/ai-gateway-key-activate branch October 14, 2026 12:30

This was referenced Oct 14, 2026

🤖 fix: hold a deleted CoderControlPlane while AI Gateway keys and providers still use it #362

Merged

🤖 feat: disable the CoderMCPServer config when the control plane withdraws its opt-in #360

Merged

Conversation

ThomasK33 commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

The K9 notes items (o) to (u)

Tests

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

ThomasK33 commented Oct 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026

🛡️ Codex Security Review

Uh oh!

ThomasK33 commented Oct 14, 2026

Uh oh!

ThomasK33 commented Oct 14, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026

🛡️ Codex Security Review

Uh oh!

ThomasK33 commented Oct 14, 2026

Uh oh!

ThomasK33 commented Oct 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026

🛡️ Codex Security Review

Uh oh!

ThomasK33 commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

G3 on the head 1a211c0

Uh oh!

ThomasK33 commented Oct 14, 2026

Uh oh!

ThomasK33 commented Oct 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026

🛡️ Codex Security Review

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ThomasK33 commented Oct 14, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading

ThomasK33 commented Oct 14, 2026 •

edited

Loading

G3 on the head `1a211c0`