From 80c823b08694d972262ee4af3bca3f2556e10878 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Tue, 13 Oct 2026 15:23:05 +0000 Subject: [PATCH 1/3] =?UTF-8?q?=F0=9F=A4=96=20docs:=20clarify=20log=20foll?= =?UTF-8?q?ow=20end=20and=20Unchanged,=20document=20the=20Coder=20rate=20l?= =?UTF-8?q?imit=20for=20logs=20(Refs=20#193)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Change-Id: I5fb09db063086f3125cbeba4c657b025039eded9 Signed-off-by: Thomas Kosiewski --- docs/how-to/start-stop-workspaces.md | 4 ++-- docs/reference/aggregated-api-behavior.md | 8 +++++--- internal/aggregated/coder/errors_test.go | 5 +++++ internal/aggregated/storage/workspace_log_follow_test.go | 4 ++++ internal/aggregated/storage/workspace_log_test.go | 8 ++++++++ 5 files changed, 24 insertions(+), 5 deletions(-) diff --git a/docs/how-to/start-stop-workspaces.md b/docs/how-to/start-stop-workspaces.md index 7c008ae61..d70270417 100644 --- a/docs/how-to/start-stop-workspaces.md +++ b/docs/how-to/start-stop-workspaces.md @@ -66,7 +66,7 @@ The response is a `CoderWorkspaceTransition`. Read `status.outcome`: - `Queued`: the server queued a build. `status.buildID` and `status.buildNumber` name it. - `InProgress`: a build of the same kind is already pending or running. Nothing was queued. -- `Unchanged`: the workspace is already started or stopped. Nothing was queued. +- `Unchanged`: the latest build is a start (for `start`) or a stop (for `stop`), and it succeeded. Nothing was queued. If the latest build of that kind failed or was canceled, the request queues a new build and returns `Queued`. - `WouldQueue`: only with `dryRun=All`. A real request would queue a build. A repeated request does not queue a second build, so you can retry a start or stop that is still in progress. A start while a stop build runs, or a stop while a start build runs, returns `409`. @@ -81,7 +81,7 @@ kubectl get coderworkspace.aggregation.coder.com acme.alice.dev -n coder \ -o jsonpath='{.status.latestBuildStatus}{"\n"}' ``` -- The follow ends when the build ends. If the build has already ended, it prints the whole log and returns at once. +- The follow ends when the build ends, or earlier when it reaches the size or time limit of a log request. Then send the request again. See the [log limits](../reference/aggregated-api-behavior.md#workspace-build-log). If the build has already ended, it prints the whole log and returns at once. - The status is `running` after a start and `stopped` after a stop. - Without `follow=true`, the log request returns the log so far. Add `limitBytes=` to read at most `n` bytes. - Build logs can contain secrets that Terraform printed. Share them with care. diff --git a/docs/reference/aggregated-api-behavior.md b/docs/reference/aggregated-api-behavior.md index 9fc092295..ad95a124d 100644 --- a/docs/reference/aggregated-api-behavior.md +++ b/docs/reference/aggregated-api-behavior.md @@ -17,7 +17,7 @@ The aggregated API server serves `coderworkspaces`, `codertemplates` and `codert | [Template versions](#template-versions) | Read-only: `get` and `list`, no watch. Reads never download template source. | | [Promote a template version](#promote-a-template-version) | `codertemplates//promote` (`create`) makes a version active. A rollback promotes an older version. It needs its own RBAC grant. `dryRun=All` previews it. An unconfirmed result returns `503`, and a concurrent change returns `409`. | | [Template builds](#template-builds) | Create and update with `spec.files` wait until Coder completes the import. The full request must complete within 34 seconds. | -| [Coder timeouts](#coder-timeouts) | A Coder call that runs out of time returns `504` on every resource. A create that gets a broken `2xx` answer returns `504` or `500`. After a `504` on a write, re-read before you retry. | +| [Coder timeouts](#coder-timeouts) | A Coder call that runs out of time returns `504` on every resource. A create that gets a broken `2xx` answer returns `504` or `500`. After a `504` on a write, or a `500` on a create, re-read before you retry. | | [Workspace build log](#workspace-build-log) | `coderworkspaces/log` returns the latest build log as text. It needs its own RBAC grant and has fixed size, time, and concurrency limits. | | [Start and stop](#start-and-stop) | `coderworkspaces/start` and `coderworkspaces/stop` start or stop a workspace. Each needs its own RBAC grant. A repeated request does not queue a second build while one is active. | @@ -393,6 +393,8 @@ The server limits each log request: Snapshots and `follow=true` streams use the same open-request slots. A user with 4 open `follow=true` streams gets `429` on a snapshot too. +Every log request also uses the Coder API rate limit of the operator token, which all aggregated API calls share. Coder allows 512 requests per minute for each user and request path by default, and the Coder deployment can change it with `CODER_API_RATE_LIMIT`. A log request makes 2 Coder requests: one that reads the workspace and one that reads the log. With `follow=true`, it makes 1 more request that opens the live stream, and it sends no more requests while the stream is open. When Coder rate-limits the operator token, the server returns `429` with the message of Coder and without `Retry-After`. See [Cost and limits of reads](#cost-and-limits-of-reads). + ## Start and stop `POST …/namespaces//coderworkspaces//start` starts a workspace. `POST …/coderworkspaces//stop` stops it: @@ -430,8 +432,8 @@ The server reads the latest build of the workspace and decides from its transiti - `Queued`: the server queued a new build. The build fields name the new build. - `InProgress`: a build of the requested kind is already pending or running. The server queued nothing. The build fields name that build. -- `Unchanged`: the workspace is already started or stopped. The server queued nothing. -- `409 Conflict`: another build is active. The message names that build. Retry after it ends. `kubectl get --raw "$API/acme.alice.dev/log?follow=true"` ends when the build ends. +- `Unchanged`: the latest build has the requested kind and succeeded. The server queued nothing. +- `409 Conflict`: another build is active. The message names that build. Retry after it ends. `kubectl get --raw "$API/acme.alice.dev/log?follow=true"` ends when the build ends, or earlier at a [log limit](#workspace-build-log). Other status codes: diff --git a/internal/aggregated/coder/errors_test.go b/internal/aggregated/coder/errors_test.go index f4ec3696b..897a894cc 100644 --- a/internal/aggregated/coder/errors_test.go +++ b/internal/aggregated/coder/errors_test.go @@ -116,6 +116,11 @@ func TestMapCoderError(t *testing.T) { if !apierrors.IsTooManyRequests(err) { t.Fatalf("expected TooManyRequests, got %v", err) } + // The docs promise Coder's message and no Retry-After, so clients do not retry on + // their own against the shared operator-token limit. + if delay, ok := apierrors.SuggestsClientDelay(err); ok || delay != 0 || !strings.Contains(err.Error(), "rate limited") { + t.Fatalf("expected Coder's message and no Retry-After, got %v (delay %d)", err, delay) + } }, }, { diff --git a/internal/aggregated/storage/workspace_log_follow_test.go b/internal/aggregated/storage/workspace_log_follow_test.go index 3c3b7848d..c425df145 100644 --- a/internal/aggregated/storage/workspace_log_follow_test.go +++ b/internal/aggregated/storage/workspace_log_follow_test.go @@ -123,6 +123,10 @@ func TestWorkspaceLogFollowStreamsLiveEntries(t *testing.T) { if got := ff.after.Load(); got != entries[1].ID { t.Fatalf("follow continued after %d, want the last snapshot entry %d", got, entries[1].ID) } + // The documented cost of a follow: the 2 snapshot requests and 1 that opens the live stream. + if got := f.requests.Load(); got != 3 { + t.Fatalf("a follow must make 3 Coder requests, got %d", got) + } close(next) rest, err := io.ReadAll(lines) if err != nil || string(rest) != entries[3].Text()+"\n" { diff --git a/internal/aggregated/storage/workspace_log_test.go b/internal/aggregated/storage/workspace_log_test.go index 294abd546..2ebfc1529 100644 --- a/internal/aggregated/storage/workspace_log_test.go +++ b/internal/aggregated/storage/workspace_log_test.go @@ -37,6 +37,7 @@ const ( type logFakeCoder struct { server *httptest.Server buildID uuid.UUID + requests atomic.Int32 // every request, of any kind lookups atomic.Int32 logFetches atomic.Int32 logsHandler atomic.Pointer[http.HandlerFunc] @@ -56,6 +57,7 @@ func newLogFakeCoder(t testing.TB) *logFakeCoder { } f.setLogs(nil) f.server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + f.requests.Add(1) parts := strings.Split(strings.Trim(r.URL.Path, "/"), "/") switch { case len(parts) == 6 && parts[2] == "users" && parts[4] == "workspace": @@ -214,6 +216,12 @@ func TestWorkspaceLogSnapshot(t *testing.T) { if !strings.HasPrefix(got, "2026-10-02T10:00:00Z [info] [provisioner|Planning infrastructure] line 1\n") { t.Fatalf("unexpected line format: %q", got) } + // The documented cost of a snapshot (docs/reference/aggregated-api-behavior.md): one workspace + // lookup and one log read, against the Coder rate limit of the operator token. + if f.requests.Load() != 2 || f.lookups.Load() != 1 || f.logFetches.Load() != 1 { + t.Fatalf("a snapshot must make 2 Coder requests (1 lookup, 1 log read), got %d (%d lookups, %d log reads)", + f.requests.Load(), f.lookups.Load(), f.logFetches.Load()) + } for _, empty := range []string{"[]", "null"} { f.setLogsHandler(func(w http.ResponseWriter, _ *http.Request) { _, _ = io.WriteString(w, empty) }) From 772fcd285e87791ceec1fb8485afc4dea94d9f84 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Tue, 13 Oct 2026 15:29:35 +0000 Subject: [PATCH 2/3] =?UTF-8?q?=F0=9F=A4=96=20docs:=20scope=20the=20follow?= =?UTF-8?q?=20retry=20advice=20and=20the=20follow=20request=20count=20to?= =?UTF-8?q?=20the=20size=20limit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Change-Id: I67ed066a043d6e28ef09423a2439cda29b45fc13 Signed-off-by: Thomas Kosiewski --- docs/how-to/start-stop-workspaces.md | 2 +- docs/how-to/troubleshooting.md | 2 +- docs/reference/aggregated-api-behavior.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/how-to/start-stop-workspaces.md b/docs/how-to/start-stop-workspaces.md index d70270417..9bf964c84 100644 --- a/docs/how-to/start-stop-workspaces.md +++ b/docs/how-to/start-stop-workspaces.md @@ -81,7 +81,7 @@ kubectl get coderworkspace.aggregation.coder.com acme.alice.dev -n coder \ -o jsonpath='{.status.latestBuildStatus}{"\n"}' ``` -- The follow ends when the build ends, or earlier when it reaches the size or time limit of a log request. Then send the request again. See the [log limits](../reference/aggregated-api-behavior.md#workspace-build-log). If the build has already ended, it prints the whole log and returns at once. +- The follow ends when the build ends, or earlier at a [log limit](../reference/aggregated-api-behavior.md#workspace-build-log). After the time limit, send the request again: it prints the existing log again and continues to follow. After a size limit, a new request prints the same first part of the log again and does not follow, because this API cannot return the rest. If the build has already ended, it prints the whole log and returns at once. - The status is `running` after a start and `stopped` after a stop. - Without `follow=true`, the log request returns the log so far. Add `limitBytes=` to read at most `n` bytes. - Build logs can contain secrets that Terraform printed. Share them with care. diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md index 738490b94..38fe213cb 100644 --- a/docs/how-to/troubleshooting.md +++ b/docs/how-to/troubleshooting.md @@ -209,7 +209,7 @@ See [Promote a template version](../reference/aggregated-api-behavior.md#promote - `429` that says `You've been rate limited`: Coder rate-limited the operator token, which every aggregated API call uses. Coder allows 512 requests per minute by default (`CODER_API_RATE_LIMIT`). - `504`: Coder did not answer in time before the response started. Either one Coder call took longer than the Coder request timeout (30 seconds by default), or the time limit of the request ran out: 60 seconds for a snapshot, or 25 minutes with `follow=true`. Check that Coder is healthy, then try again. - A `Warning` that says the server cut the log: the log is larger than the server limit. See [Workspace build log](../reference/aggregated-api-behavior.md#workspace-build-log). -- A `follow=true` stream ends before the build ends: the stream reached the 4 MiB response limit or the 25-minute time limit. The server sends no `Warning` for a cut in the live part of a stream. Send a new request with `follow=true`. It sends the existing entries again. +- A `follow=true` stream ends before the build ends: the stream reached the 4 MiB response limit or the 25-minute time limit. The server sends no `Warning` for a cut in the live part of a stream. After the time limit, send a new request with `follow=true`. It sends the existing entries again and continues to follow. After the size limit, a new request returns the same first 4 MiB and does not follow, because this API cannot return the rest of the log. ## `coderworkspaces/start` or `/stop` returns `400`, `403`, `409`, or `504` diff --git a/docs/reference/aggregated-api-behavior.md b/docs/reference/aggregated-api-behavior.md index ad95a124d..43cc4029c 100644 --- a/docs/reference/aggregated-api-behavior.md +++ b/docs/reference/aggregated-api-behavior.md @@ -393,7 +393,7 @@ The server limits each log request: Snapshots and `follow=true` streams use the same open-request slots. A user with 4 open `follow=true` streams gets `429` on a snapshot too. -Every log request also uses the Coder API rate limit of the operator token, which all aggregated API calls share. Coder allows 512 requests per minute for each user and request path by default, and the Coder deployment can change it with `CODER_API_RATE_LIMIT`. A log request makes 2 Coder requests: one that reads the workspace and one that reads the log. With `follow=true`, it makes 1 more request that opens the live stream, and it sends no more requests while the stream is open. When Coder rate-limits the operator token, the server returns `429` with the message of Coder and without `Retry-After`. See [Cost and limits of reads](#cost-and-limits-of-reads). +Every log request also uses the Coder API rate limit of the operator token, which all aggregated API calls share. Coder allows 512 requests per minute for each user and request path by default, and the Coder deployment can change it with `CODER_API_RATE_LIMIT`. A log request makes 2 Coder requests: one that reads the workspace and one that reads the log. With `follow=true`, it makes 1 more request that opens the live stream, unless the existing log already reaches a size limit. It sends no more requests while the stream is open. When Coder rate-limits the operator token, the server returns `429` with the message of Coder and without `Retry-After`. See [Cost and limits of reads](#cost-and-limits-of-reads). ## Start and stop From b508ad13ac5a0058e03ca376f94a5d99c52583a3 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Tue, 13 Oct 2026 15:35:07 +0000 Subject: [PATCH 3/3] =?UTF-8?q?=F0=9F=A4=96=20docs:=20base=20the=20follow?= =?UTF-8?q?=20retry=20advice=20on=20whether=20the=20existing=20log=20fits?= =?UTF-8?q?=20the=20limits?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Change-Id: I25a9aa1b0292e5b68ae7c5aec6a6db09cf35acf5 Signed-off-by: Thomas Kosiewski --- docs/how-to/start-stop-workspaces.md | 2 +- docs/how-to/troubleshooting.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/how-to/start-stop-workspaces.md b/docs/how-to/start-stop-workspaces.md index 9bf964c84..4016b8d62 100644 --- a/docs/how-to/start-stop-workspaces.md +++ b/docs/how-to/start-stop-workspaces.md @@ -81,7 +81,7 @@ kubectl get coderworkspace.aggregation.coder.com acme.alice.dev -n coder \ -o jsonpath='{.status.latestBuildStatus}{"\n"}' ``` -- The follow ends when the build ends, or earlier at a [log limit](../reference/aggregated-api-behavior.md#workspace-build-log). After the time limit, send the request again: it prints the existing log again and continues to follow. After a size limit, a new request prints the same first part of the log again and does not follow, because this API cannot return the rest. If the build has already ended, it prints the whole log and returns at once. +- The follow ends when the build ends, or earlier at a [log limit](../reference/aggregated-api-behavior.md#workspace-build-log). To continue, send the request again. It prints the existing log again. It continues to follow only if the existing log fits within the size limits. Otherwise it prints only the first part and ends, because this API cannot return the rest. If the build has already ended, it prints the whole log and returns at once. - The status is `running` after a start and `stopped` after a stop. - Without `follow=true`, the log request returns the log so far. Add `limitBytes=` to read at most `n` bytes. - Build logs can contain secrets that Terraform printed. Share them with care. diff --git a/docs/how-to/troubleshooting.md b/docs/how-to/troubleshooting.md index 38fe213cb..a400cab39 100644 --- a/docs/how-to/troubleshooting.md +++ b/docs/how-to/troubleshooting.md @@ -209,7 +209,7 @@ See [Promote a template version](../reference/aggregated-api-behavior.md#promote - `429` that says `You've been rate limited`: Coder rate-limited the operator token, which every aggregated API call uses. Coder allows 512 requests per minute by default (`CODER_API_RATE_LIMIT`). - `504`: Coder did not answer in time before the response started. Either one Coder call took longer than the Coder request timeout (30 seconds by default), or the time limit of the request ran out: 60 seconds for a snapshot, or 25 minutes with `follow=true`. Check that Coder is healthy, then try again. - A `Warning` that says the server cut the log: the log is larger than the server limit. See [Workspace build log](../reference/aggregated-api-behavior.md#workspace-build-log). -- A `follow=true` stream ends before the build ends: the stream reached the 4 MiB response limit or the 25-minute time limit. The server sends no `Warning` for a cut in the live part of a stream. After the time limit, send a new request with `follow=true`. It sends the existing entries again and continues to follow. After the size limit, a new request returns the same first 4 MiB and does not follow, because this API cannot return the rest of the log. +- A `follow=true` stream ends before the build ends: the stream reached the 4 MiB response limit or the 25-minute time limit. The server sends no `Warning` for a cut in the live part of a stream. Send a new request with `follow=true`. It sends the existing entries again. It continues to follow only if the existing entries fit within the size limits of a log request. Otherwise it returns only the first part and ends, because this API cannot return the rest of the log. ## `coderworkspaces/start` or `/stop` returns `400`, `403`, `409`, or `504`