CoderTemplateTest: test a template version like a Kubernetes Job

Goal: a CRD that creates a throwaway workspace from a template version, waits until its agent is healthy, records Succeeded or Failed, and deletes the workspace.

Uses: gate template promotion in GitOps (Argo CD and Flux read the condition); nightly template checks.

First slice: provisioning and agent health only. Running a command inside the workspace is a later, separate security slice.

Open points: cleanup after controller crashes (finalizer plus labels), timeouts, which Coder user owns the test workspace, cost of leaked workspaces.

Acceptance: envtest for the state machine and cleanup; Kind dogfood with a passing and a failing template; docs.

Owner: maintainer desk.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: medium

test: match template names case-insensitively in the fake Coder

test: accept repeated delete builds and cover pending cancel in the f…

test: set CanceledAt when the fake Coder cancels a running build

feat(api): validate CoderTemplateTest names at admission

feat(api): reject Coder's reserved names in CoderTemplateTest.spec.te…

🤖 fix: use the internal Coder URL and recheck the deadline after lookups

🤖 fix: check default organization roles and fail fast on a wrong owner

🤖 fix: check every trusted Coder answer, user status, and deprecation

🤖 fix: recheck the deadline after create and release proven-clean tests

🤖 fix: clamp the jittered 429 backoff to its cap

🤖 fix: tighten provenance and liveness on the confirm path

🤖 fix: assert every provenance pin before confirming reads

🤖 fix: name CoderTemplateTest agents by ID in status

🤖 docs: complete the CoderTemplateTest how-to

🤖 docs: remove only the test finalizer, and fix two how-to reasons

🤖 docs: say top-level agents for NoAgents, and stop nightly Job retries

🤖 docs: split WorkspaceNameConflict by WorkspaceDeleted, and explain …

🤖 docs: check the active version before a promotion retry, and budget…

🤖 docs: complete the CoderTemplateTest how-to

🤖 docs: remove only the test finalizer, and fix two how-to reasons

🤖 docs: say top-level agents for NoAgents, and stop nightly Job retries

🤖 docs: split WorkspaceNameConflict by WorkspaceDeleted, and explain …

🤖 docs: complete the CoderTemplateTest how-to

🤖 docs: remove only the test finalizer, and fix two how-to reasons

🤖 docs: say top-level agents for NoAgents, and stop nightly Job retries

🤖 docs: split WorkspaceNameConflict by WorkspaceDeleted, and explain …

Done: `CoderTemplateTest`

CoderTemplateTest is active on main. A test creates a throwaway workspace from one template version, waits until every top-level agent is connected and ready, records Succeeded or Failed, and deletes the workspace. Argo CD and Flux can read the result through the health rules in config/gitops/.

The acceptance criteria of this issue are met:

envtest covers the state machine and cleanup, including restart replay, namespace teardown, and a lost control plane.
Kind E2E runs a passing template and failing ones (agent failure, bad parameter), plus restart, immutability, dry run, TTL, tester key count, and namespace deletion while Coder is unreachable.
Docs: the how-to, the GitOps page, troubleshooting, the API reference, and a sample.

Pull requests (all merged into `main`)

PR	Content
#194, #195	Test-local stateful fake Coder
#196	Dormant API types
#197	Dormant reconciler skeleton
#199	Owner, template, and version lookups
#200	Create request behind a marker
#201	Confirm an uncertain create with provenance
#202	Workspace readiness
#203	Cleanup decisions and finalizer release
#204	Workspace delete
#205	TTL after a finished test
#206	Claimed prebuilds in provenance
#207	Activation blockers
#208	Kind E2E fixture with a real agent
#210	Activation of the controller
#211	Remaining Kind E2E phases
#212	GitOps health rules and page
#213	Complete how-to, troubleshooting, and sample

Changes to the original plan

Lost control plane. If the referenced CoderControlPlane is gone or being deleted, the controller releases the finalizer at once (ControlPlaneGone). Deleting a namespace never deadlocks on a test.
Escape hatch. coder.com/deletion-policy: retain (Lifecycle policy annotations for controller-managed Coder objects #151) releases a stuck test without a Coder call. It works only when the control plane sets spec.templateTests.allowRetain: true, because any test creator can set the annotation. Without the opt-in, an admin removes the finalizer by hand.
Uncertain creates. After the settle window, the controller releases a test as NotCreated only if no workspace exists and the operator user can see every workspace. A workspace without proof of ownership keeps the finalizer (OwnershipUnknown).
404 is not deletion. Only Coder's 410 proves that a workspace is deleted. A 404 can mean lost read access, so the test waits.
Agents. An agent that disconnects after it was ready makes the test wait, not fail. Devcontainer sub-agents do not count.
Dormant owners. Coder creates API users as dormant and refuses their agents, so the controller refuses a dormant owner (OwnerNotEligible).
Tester keys. Keys do not accumulate. The Kind E2E counted 0 tester API keys before and after three tests.
Scope cuts. CI does not run Argo CD or Flux. Unit tests run the Lua rule with gopher-lua and the Flux expressions with cel-go. The GitOps page lists what a promotion step must check, but has no runnable promotion example (docs: add a runnable promotion example for CoderTemplateTest GitOps gating #216).

Known limits and follow-ups

🤖 Namespace deletion hangs: the aggregated API answers LIST with 503 when the namespace has no control plane #209 is fixed by 🤖 fix: return an empty namespaced LIST where no Coder backend serves the namespace #214 (merged as b567e86). A namespace without a control plane now finishes deleting, because the aggregated API answers its LIST with an empty list. The Kind E2E now waits until the namespace is gone.
🤖 Aggregated API: decide two edge cases of the empty namespaced LIST (#209 follow-up) #215: an open edge case of that empty LIST (a standalone server without Coder credentials), a follow-up of 🤖 fix: return an empty namespaced LIST where no Coder backend serves the namespace #214.
🤖 Aggregated API watches never hold: LIST has no resourceVersion and watch-list returns 400 #217: watches on the aggregated API never hold, because LIST has no resourceVersion and a watch-list request gets 400. This predates the CoderTemplateTest work.
🤖 Use the internal Coder URL for provisioner keys and the aggregated API when TLS is on #198: with TLS on, the CoderProvisioner controller and the aggregated API call Coder at the HTTPS service URL, which can fail TLS verification. CoderTemplateTest itself uses the internal HTTP URL, so its operator token crosses the cluster network without encryption. The how-to says so.
CoderTemplateTest deferred items #218 tracks the items that the plan deferred, one checklist line each, with the trigger to revisit it. The annotation work stays in Lifecycle policy annotations for controller-managed Coder objects #151, and the runnable promotion example in docs: add a runnable promotion example for CoderTemplateTest GitOps gating #216.

Closing as complete.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

Navigation Menu

Repository navigation

CoderTemplateTest: test a template version like a Kubernetes Job #152

27 remaining items

Done: `CoderTemplateTest`

Pull requests (all merged into `main`)

Changes to the original plan

Known limits and follow-ups

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CoderTemplateTest: test a template version like a Kubernetes Job #152

Description

Activity

27 remaining items

ThomasK33 commented on Oct 3, 2026

Done: CoderTemplateTest

Pull requests (all merged into main)

Changes to the original plan

Known limits and follow-ups

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Done: `CoderTemplateTest`

Pull requests (all merged into `main`)