🤖 fix: hold a deleted CoderControlPlane while AI Gateway keys and providers still use it

[ThomasK33]

Part 1 of #353. When you delete a CoderControlPlane while CoderAIGatewayKey or CoderAIProvider objects still use it, the control plane now waits for them for up to 10 minutes. Coder keeps running in that time, so they can delete their keys and providers in Coder. Before this change, Kubernetes garbage-collected the Coder Deployment at once, and the objects released their finalizers with ControlPlaneGone, leaving the keys and providers valid in Coder.

Design note (#353)

What a hold can and cannot fix. A namespace deletion removes every object in the namespace in parallel: the Coder Deployment, ReplicaSets, Pods, the Service and the operator token Secret. A finalizer on the control plane keeps only the control plane object, not Coder, so no hold can keep Coder reachable during a namespace deletion. The dogfood on #346 saw both outcomes: the control plane gone first, or Coder already refusing connections. So the fix has three parts:

1. This PR: the control-plane hold (option 2 of Namespace deletion leaves AI Gateway keys and AI providers in Coder #353). It covers a direct control-plane delete, plus item q and the release metric.
2. Next PR: a cleanup record per object that survives the namespace. A Ready control plane on the same Coder deployment (deployment_id from /api/v2/buildinfo) later sweeps what a namespace deletion left behind. The record type (a small cluster-scoped CRD, or a ConfigMap) is a maintainer decision. I will ask it separately.
3. Then: the key half of the sweep, and item u.

If Coder's database is deleted with the namespace, nothing leaks. The leak matters only when the database survives, for example when it is external or shared.

What changes

1. The hold.
   • finalizeWorkspaceRBAC keeps coder.com/workspace-rbac-cleanup while a CoderAIGatewayKey or CoderAIProvider in the namespace that references the control plane is being deleted and still has its cleanup finalizer. A dependent that is not being deleted holds it only for 30 s after the control plane deletion (controlPlaneLiveDependentWait), which covers one command that deletes both.
   • It does not hold when the namespace is being deleted (Coder's pods go anyway, and a hold would only delay the namespace), when it is deleted in the foreground (Kubernetes deletes the Coder Deployment first), or after 10 minutes from deletionTimestamp (controlPlaneDependentHold).
   • After the 10 minutes, it stays for a grace period of 30 s plus 10 s per remaining dependent of the larger kind (at most 5 minutes), so each dependent runs its bounded pass while Coder still runs. One worker per kind runs these passes in turn.
   • It removes the managed workspace RBAC before it checks the hold, so Coder loses its Kubernetes access at once. Only the Deployment stays during the hold.
   • While it waits, it requeues every 10 s, and the removal of a dependent requeues it at once. A delete-only watch on both kinds makes that happen.
   • The log line Waiting for the objects that use this control plane to delete their resources in Coder lists the dependents.
2. Dependents take the normal path while the control plane holds for them (item q).
   • The key and provider delete paths use the one bounded pass only when the control plane does not hold, as computed by the shared controlPlaneHolds.
   • Under the hold they retry as usual. A failed operator-access read or a 502 keeps the finalizer and retries, instead of releasing at once. A key waits for its settle window as usual.
3. Metric. coder_k8s_cleanup_released_without_delete_total{kind} counts every release without a delete in Coder, next to the existing log line. It counts after the finalizer patch succeeds, and a retry that then deletes in Coder does not count. The repo uses no Kubernetes Events, so this PR adds none.
4. Docs.
   • Both how-tos describe the hold, its 10-minute bound and the metric.
   • The "delete the control plane" way out of a stuck key finalizer now first waits for the hold.
   • The provider how-to says to delete objects before their namespace.
5. CI. The full Kind E2E path filter includes internal/controller/controlplane_hold.go.

go.mod lists github.com/prometheus/client_golang as a direct requirement, because the metric uses it. It was already vendored.

Tests

• TestControlPlaneWaitsForDependents:
  • "dependents": a deleted control plane stays while its key and provider exist (requeue 10 s), stays after the key goes, and goes after the provider goes. A key of another control plane does not hold it.
  • "namespace deletion": with the namespace Terminating, it goes at once.
• TestCoderAIGatewayKeyDeleteUnderControlPlaneHold:
  1. A deleted key waits for the settle window (DeletingKeys, requeue 5 minutes) instead of the bounded pass.
  2. Without operator access it shows OperatorAccessNotReady with a 60 s retry, and keeps the keys.
  3. When access comes back, it deletes every key and releases.
• TestCoderAIProviderDeleteUnderControlPlaneHold: a 502 on DELETE keeps the finalizer (CoderUnavailable, 60 s). The next reconcile deletes the provider and releases.
• TestReleasedWithoutDeleteMetric: a ControlPlaneGone release increments the counter.

Mutations that the tests catch:

• no namespace check in the hold;
• dependents not filtered by controlPlaneRef;
• no hold in finalizeWorkspaceRBAC;
• the key's bounded pass under the hold;
• the provider's bounded pass under the hold.

Size: about 150 production lines (with docs) and 180 test lines.

Local gates on the head: make verify-vendor, make build, make lint, make test, make test-integration, go tool govulncheck ./..., go test -race ./internal/controller/..., make codegen, make build-installer and make docs-reference with no drift, make docs-check, and actionlint.

Refs #353

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high • Cost: $151.58

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T14:47:52.029860Z	49e6fdc	Manual request
🔒 Security Review	✅ Completed 2026-10-14T14:47:45.333225Z	49e6fdc	Manual request

Security findings

Advisory findings (3)

• 🟡 Wait for every dependent's bounded cleanup pass · Medium · Resolved
• 🟡 Revoke workspace RBAC before holding the control plane · Medium · Resolved
• 🟡 Bound grace to dependents that still need cleanup · Medium

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: b457bee9eb

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b457bee9eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1874d474e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Dogfood on head 1874d47

I ran both checks in a licensed Kind cluster (Kubernetes v1.32.0, Coder v2.37.2, AI Bridge on). The operator image is built from this head: sha256:a29af0fd4247f19fcd671f40acc09536080ff8943343b6bfffb1914eef7e63b1. The running pod reports the same image ID.

Check	Result
1. Delete the CoderControlPlane right after a CoderAIGatewayKey and a CoderAIProvider are Ready (normal delete)	Pass. The operator held the control plane for 299 s and logged both dependents. Coder deleted the key (ai_gateway_keys 1 → 0) and the provider (not-deleted rows 1 → 0). 0 releases without a delete. No metric samples.
2. Same, with kubectl delete --cascade=foreground	No hold, as the fix for thread 4271056328 intends. The control plane was gone in 3 s. Both objects released their finalizers without a delete, because Kubernetes removed the operator access first (why: CoderControlPlane is being deleted without operator access). The key and the provider stay in Coder. The metric counts exactly 1 per kind, which is the overcount fix (thread 4271056316).

Check 2 is the expected limit of this PR: a hold cannot help when Kubernetes deletes Coder's Deployment and token first. The docs say so. The cleanup records planned for #353 cover this case.

Recording of check 1:

normal-delete.mp4

Recording of check 2:

foreground-delete.mp4

Check 1 transcript

head=1874d474e23617acf4c7f24cb9d1e3d23ddb5aec image=sha256:a29af0fd4247f19fcd671f40acc09536080ff8943343b6bfffb1914eef7e63b1
### 1. A key and a provider that use control plane coder  (13:13:22Z)
coderaigatewaykey.coder.com/gateway created
coderaiprovider.coder.com/openai created
CoderAIGatewayKey/gateway: Ready=True Ready
CoderAIProvider/openai: Ready=True Applied
Coder database: ai_gateway_keys=1 ai_providers=1 (soft-deleted=2)
### 2. Delete the control plane right away (the key's settle window is still open)  (13:13:26Z)
codercontrolplane.coder.com "coder" deleted from coder namespace
coderaigatewaykey.coder.com "gateway" deleted from coder namespace
coderaiprovider.coder.com "openai" deleted from coder namespace
+1s CoderControlPlane/coder:   deleting, finalizers=["coder.com/workspace-rbac-cleanup"] | CoderAIGatewayKey/gateway: Ready=False DeletingKeys  deleting, finalizers=["coder.com/ai-gateway-key-cleanup"] |  | coder pods=1
+299s  |  |  | coder pods=0
control plane gone after 299 s
### 3. Result  (13:18:26Z)
Coder database: ai_gateway_keys=0 ai_providers=0 (soft-deleted=3)
hold log: dependents=["CoderAIGatewayKey/gateway", "CoderAIProvider/openai"]
releases without a delete in this run: 0
coder_k8s_cleanup_released_without_delete_total: no samples (no release without a delete)

Check 2 transcript

head=1874d474e23617acf4c7f24cb9d1e3d23ddb5aec image=sha256:a29af0fd4247f19fcd671f40acc09536080ff8943343b6bfffb1914eef7e63b1
### 1. A key and a provider that use control plane coder  (13:19:05Z)
coderaigatewaykey.coder.com/gateway created
coderaiprovider.coder.com/openai created
CoderAIGatewayKey/gateway: Ready=True Ready
CoderAIProvider/openai: Ready=True Applied
Coder database: ai_gateway_keys=1 ai_providers=1 (soft-deleted=3)
### 2. Delete the control plane with foreground cascading (no hold expected: Kubernetes removes the Deployment first)  (13:19:07Z)
codercontrolplane.coder.com "coder" deleted from coder namespace
coderaigatewaykey.coder.com "gateway" deleted from coder namespace
coderaiprovider.coder.com "openai" deleted from coder namespace
+1s CoderControlPlane/coder:   deleting, finalizers=["foregroundDeletion"] | coderaigatewaykey/gateway: gone | coderaiprovider/openai: gone | coder pods=1
+3s codercontrolplane/coder: gone | coderaigatewaykey/gateway: gone | coderaiprovider/openai: gone | coder pods=0
control plane gone after 3 s
### 3. Result  (13:19:11Z)
Coder database: ai_gateway_keys=1 ai_providers=1 (soft-deleted=3)
releases without a delete in this run: 2
coder_k8s_cleanup_released_without_delete_total{kind="CoderAIGatewayKey"} 1
coder_k8s_cleanup_released_without_delete_total{kind="CoderAIProvider"} 1

The scripts never print the key or the provider API key. I scanned both transcripts for sk- and ck8s- key values.

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

[chatgpt-codex-connector]

🛡️ Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: 1874d474e2

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ThomasK33]

@codex review

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ad8cc53b4b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: ad8cc53b4b

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ThomasK33]

Dogfood on head ad8cc53 (round 2 fixes)

Same licensed Kind cluster as the comment above. The operator image is built from this head: sha256:15e52d1122bdba31c72ba8672c60df184d8648aae272450a3073626e09280218. The running pod reports the same image ID. Before each check I removed the key and provider that the foreground check above left in Coder, so each check starts from 0 rows.

Check	Result
1. Delete the CoderControlPlane, the CoderAIGatewayKey and the CoderAIProvider in one go	Pass. The control plane held for 298 s (the key's settle window) and logged both dependents. Coder deleted the key and the provider (ai_gateway_keys 1 → 0, providers 1 → 0). 0 releases without a delete.
2. Delete only the CoderControlPlane, keep the key and the provider (thread 4271268565)	Pass. The live dependents held the control plane for 33 s (30 s wait plus the 10 s poll), not 10 minutes. Deleting them afterwards released both without a delete, so they stay in Coder, and the metric counts 1 per kind. The how-tos now say to delete the objects together with the control plane.

The recordings of check 1 and then check 2 follow at the end of this comment.

Check 1 transcript

head=ad8cc53b4b65c020123916c92799a166e274438b image=sha256:15e52d1122bdba31c72ba8672c60df184d8648aae272450a3073626e09280218
### 1. A key and a provider that use control plane coder  (13:47:40Z)
coderaigatewaykey.coder.com/gateway created
coderaiprovider.coder.com/openai created
CoderAIGatewayKey/gateway: Ready=True Ready
CoderAIProvider/openai: Ready=True Applied
Coder database: ai_gateway_keys=1 ai_providers=1 (soft-deleted=4)
### 2. Delete the control plane right away (the key's settle window is still open)  (13:47:42Z)
codercontrolplane.coder.com "coder" deleted from coder namespace
coderaigatewaykey.coder.com "gateway" deleted from coder namespace
coderaiprovider.coder.com "openai" deleted from coder namespace
+1s CoderControlPlane/coder:   deleting, finalizers=["coder.com/workspace-rbac-cleanup"] | CoderAIGatewayKey/gateway: Ready=False DeletingKeys  deleting, finalizers=["coder.com/ai-gateway-key-cleanup"] | coderaiprovider/openai: gone | coder pods=1
+298s codercontrolplane/coder: gone | coderaigatewaykey/gateway: gone | coderaiprovider/openai: gone | coder pods=1
control plane gone after 298 s
### 3. Result  (13:52:40Z)
Coder database: ai_gateway_keys=0 ai_providers=0 (soft-deleted=5)
hold log: dependents=["CoderAIGatewayKey/gateway", "CoderAIProvider/openai"]
releases without a delete in this run: 0
coder_k8s_cleanup_released_without_delete_total: no samples (no release without a delete)

Check 2 transcript

head=ad8cc53b4b65c020123916c92799a166e274438b image=sha256:15e52d1122bdba31c72ba8672c60df184d8648aae272450a3073626e09280218
### 1. A key and a provider that use control plane coder  (13:53:06Z)
coderaigatewaykey.coder.com/gateway created
coderaiprovider.coder.com/openai created
CoderAIGatewayKey/gateway: Ready=True Ready
CoderAIProvider/openai: Ready=True Applied
Coder database: ai_gateway_keys=1 ai_providers=1 (soft-deleted=5)
### 2. Delete only the control plane (the key and the provider stay): they hold it for 30 s only  (13:53:08Z)
codercontrolplane.coder.com "coder" deleted from coder namespace
+1s CoderControlPlane/coder:   deleting, finalizers=["coder.com/workspace-rbac-cleanup"] | CoderAIGatewayKey/gateway: Ready=False ControlPlaneNotReady | CoderAIProvider/openai: Ready=False ControlPlaneNotReady | coder pods=1
+33s codercontrolplane/coder: gone | CoderAIGatewayKey/gateway: Ready=False ControlPlaneNotReady | CoderAIProvider/openai: Ready=False ControlPlaneNotReady | coder pods=0
control plane gone after 33 s
### 3. Result (the key and the provider still exist; deleting them now releases without a delete)  (13:53:41Z)
CoderAIGatewayKey/gateway: Ready=False ControlPlaneNotReady
CoderAIProvider/openai: Ready=False ControlPlaneNotReady
coderaigatewaykey.coder.com "gateway" deleted from coder namespace
coderaiprovider.coder.com "openai" deleted from coder namespace
Coder database: ai_gateway_keys=1 ai_providers=1 (soft-deleted=5)
hold log: dependents=["CoderAIGatewayKey/gateway", "CoderAIProvider/openai"]
releases without a delete in this run: 2
coder_k8s_cleanup_released_without_delete_total{kind="CoderAIGatewayKey"} 1
coder_k8s_cleanup_released_without_delete_total{kind="CoderAIProvider"} 1

The scripts never print the key or the provider API key.

Recordings:

r3-normal-delete.mp4

r3-live-dependents.mp4

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

[ThomasK33]

@codex review

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: f667afb00e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Dogfood on head f667afb (workspace RBAC fix)

Same licensed Kind cluster. The operator image is built from this head: sha256:ccff8ac421efba888e74990b4d9cd919cfa952b02737e9658603f6488b5526d2, and the running operator pod reports that image ID.

Check: delete the CoderControlPlane, the CoderAIGatewayKey and the CoderAIProvider in one go, and count the workspace RoleBindings of the control plane (label coder.com/control-plane=coder).

Result: pass. The RoleBinding count went from 1 to 0 within 1 s of the deletion, while the control plane still held (297 s, the key's settle window). Coder still deleted the key and the provider (ai_gateway_keys 1 → 0, providers 1 → 0). 0 releases without a delete.

Transcript

head=f667afb00e28990a16b7fa8b2baa6420fcdff44e image=sha256:ccff8ac421efba888e74990b4d9cd919cfa952b02737e9658603f6488b5526d2
### 1. A key and a provider that use control plane coder  (14:16:46Z)
coderaigatewaykey.coder.com/gateway created
coderaiprovider.coder.com/openai created
CoderAIGatewayKey/gateway: Ready=True Ready
CoderAIProvider/openai: Ready=True Applied
Coder database: ai_gateway_keys=1 ai_providers=1 (soft-deleted=6)
workspace RoleBindings of control plane coder: 1
### 2. Delete the control plane right away (the key's settle window is still open)  (14:16:50Z)
codercontrolplane.coder.com "coder" deleted from coder namespace
coderaigatewaykey.coder.com "gateway" deleted from coder namespace
coderaiprovider.coder.com "openai" deleted from coder namespace
+1s CoderControlPlane/coder:   deleting, finalizers=["coder.com/workspace-rbac-cleanup"] | CoderAIGatewayKey/gateway: Ready=False DeletingKeys  deleting, finalizers=["coder.com/ai-gateway-key-cleanup"] | coderaiprovider/openai: gone | coder pods=1 | workspace RoleBindings=0
+297s codercontrolplane/coder: gone | coderaigatewaykey/gateway: gone | coderaiprovider/openai: gone | coder pods=1 | workspace RoleBindings=0
control plane gone after 297 s
### 3. Result  (14:21:48Z)
Coder database: ai_gateway_keys=0 ai_providers=0 (soft-deleted=7)
hold log: dependents=["CoderAIGatewayKey/gateway", "CoderAIProvider/openai"]
releases without a delete in this run: 0
coder_k8s_cleanup_released_without_delete_total: no samples (no release without a delete)

Recording:

r4-rbac-hold.mp4

---

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

[chatgpt-codex-connector]

🛡️ Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: f667afb00e

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ThomasK33]

@codex review

[ThomasK33]

@codex security review

[chatgpt-codex-connector]

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 49e6fdc0dc

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 49e6fdc0dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".