From 05ec4e33134d9f04045e14a6e020cf6dc4d49c2f Mon Sep 17 00:00:00 2001 From: TheMeinerLP Date: Fri, 21 Aug 2026 13:03:35 +0200 Subject: [PATCH 1/2] fix(chart): let every component finish starting before liveness judges it 0.6.0 added the OpenTelemetry import graph, and `link` outgrew its liveness budget: 55 seconds from container start to 'Link service listening' on 200m of CPU, against `initialDelaySeconds + periodSeconds x failureThreshold` = 65 seconds. It was SIGKILLed at 137 with the process healthy and about to serve. Observed in production immediately after the 0.6.0 rollout. `worker` has had a startupProbe since the model download made the same point more loudly; `bot` and `link` never got one, because until now nothing they did at import took long enough to notice. Both get one now, with five minutes of budget against the 55 seconds observed -- being generous costs a later notice for a process genuinely wedged at startup, and being tight costs a crash loop on a cold node. The chart job now asserts every component has a startupProbe and that its budget exceeds its liveness budget, since a startupProbe shorter than liveness buys nothing and looks like it does. Counter-checked: removing link's stanza makes the assertion fail. --- .github/workflows/chart.yml | 24 ++++++++++ charts/sturnus/templates/bot-deployment.yaml | 4 ++ charts/sturnus/values.yaml | 46 ++++++++++++++++++++ docs/operations.md | 16 +++++++ 4 files changed, 90 insertions(+) diff --git a/.github/workflows/chart.yml b/.github/workflows/chart.yml index 091bbe5..319512c 100644 --- a/.github/workflows/chart.yml +++ b/.github/workflows/chart.yml @@ -111,6 +111,30 @@ jobs: f'{name} mounts a ReadWriteOnce claim {sorted(claims & rwo_claims)} ' f'but its strategy is {strategy!r}, not Recreate') + # Every component gets a startupProbe, and its budget must exceed + # its liveness budget. Without one, liveness starts counting from + # `initialDelaySeconds` while the process is still importing, and + # a slow start is indistinguishable from a hung one. That is not + # hypothetical: 0.6.0 added the OpenTelemetry import graph, `link` + # took 55 seconds to reach "listening" on 200m of CPU, and its own + # liveness probe SIGKILLed it at 137 -- with the process healthy + # and about to serve. + for name, dep in deployments.items(): + container = dep['spec']['template']['spec']['containers'][0] + startup = container.get('startupProbe') + check(startup is not None, + f'{name} has no startupProbe, so liveness judges it while it is ' + f'still starting') + if startup is None: + continue + startup_budget = startup.get('periodSeconds', 10) * startup.get('failureThreshold', 3) + live = container.get('livenessProbe') or {} + live_budget = (live.get('initialDelaySeconds', 0) + + live.get('periodSeconds', 10) * live.get('failureThreshold', 3)) + check(startup_budget > live_budget, + f'{name} allows {startup_budget}s to start but liveness would have ' + f'killed it after {live_budget}s; the startupProbe buys nothing') + # podLabels must reach every pod, and must never reach a selector: # selectors are immutable on a live Deployment. for name, dep in deployments.items(): diff --git a/charts/sturnus/templates/bot-deployment.yaml b/charts/sturnus/templates/bot-deployment.yaml index 1e09969..b465f50 100644 --- a/charts/sturnus/templates/bot-deployment.yaml +++ b/charts/sturnus/templates/bot-deployment.yaml @@ -84,6 +84,10 @@ spec: - name: health containerPort: {{ .Values.healthPort }} protocol: TCP + {{- with .Values.bot.startupProbe }} + startupProbe: + {{- toYaml . | nindent 12 }} + {{- end }} {{- with .Values.bot.livenessProbe }} livenessProbe: {{- toYaml . | nindent 12 }} diff --git a/charts/sturnus/values.yaml b/charts/sturnus/values.yaml index b8e858a..185a312 100644 --- a/charts/sturnus/values.yaml +++ b/charts/sturnus/values.yaml @@ -232,6 +232,29 @@ bot: # documented `max_session_hours` ceiling plus upload time with margin; size # this to the deployed `max_session_hours` if that default ever changes. terminationGracePeriodSeconds: 21600 + # Startup is slower than it looks, and on 0.6.0 it outgrew the liveness + # budget: a `link` pod took 55 seconds from container start to "Link + # service listening" and was SIGKILLed at 137 by its own liveness probe, + # which allows 5 + 3x20 seconds and then gives up. + # + # The time is import time, not work: the OpenTelemetry SDK and its + # exporters are a large import graph, and `bot` has 1 CPU of CPU to do + # it with. A startupProbe is exactly the instrument for that -- liveness + # and readiness are not evaluated at all until it succeeds, so a slow + # start cannot be mistaken for a hung process, and once it succeeds the + # normal cadence takes over. `worker` has had one since the model + # download made the same point more loudly. + # + # 5 minutes of budget, which is far more than the 55 seconds observed: + # the cost of being generous is that a process genuinely wedged at + # startup is noticed later, and the cost of being tight is a crash loop + # on a cold node. Only one of those is a working deployment. + startupProbe: + httpGet: + path: /healthz + port: health + periodSeconds: 10 + failureThreshold: 30 livenessProbe: httpGet: path: /healthz @@ -277,6 +300,29 @@ link: limits: cpu: 200m memory: 128Mi + # Startup is slower than it looks, and on 0.6.0 it outgrew the liveness + # budget: a `link` pod took 55 seconds from container start to "Link + # service listening" and was SIGKILLed at 137 by its own liveness probe, + # which allows 5 + 3x20 seconds and then gives up. + # + # The time is import time, not work: the OpenTelemetry SDK and its + # exporters are a large import graph, and `link` has 200m of CPU to do + # it with. A startupProbe is exactly the instrument for that -- liveness + # and readiness are not evaluated at all until it succeeds, so a slow + # start cannot be mistaken for a hung process, and once it succeeds the + # normal cadence takes over. `worker` has had one since the model + # download made the same point more loudly. + # + # 5 minutes of budget, which is far more than the 55 seconds observed: + # the cost of being generous is that a process genuinely wedged at + # startup is noticed later, and the cost of being tight is a crash loop + # on a cold node. Only one of those is a working deployment. + startupProbe: + httpGet: + path: /healthz + port: health + periodSeconds: 10 + failureThreshold: 30 livenessProbe: httpGet: path: /healthz diff --git a/docs/operations.md b/docs/operations.md index f387339..c9cd495 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -527,6 +527,22 @@ stored is actually what is running. ## 5. Troubleshooting +**A pod is SIGKILLed at 137 shortly after starting, with `connection +refused` on its probes.** The process is not crashing; its liveness probe +is killing it while it is still importing. Look at the gap between the +container's `startedAt` and its first "listening" log line: on 0.6.0 a +`link` pod took 55 seconds on 200m of CPU, against a liveness budget of +`initialDelaySeconds + periodSeconds x failureThreshold` = 65 seconds. + +Every component has a `startupProbe` for exactly this, and liveness and +readiness are not evaluated at all until it succeeds. If you see this +again, the startup budget has been outgrown rather than the process broken +-- raise `.startupProbe.failureThreshold`, or give the component +more CPU so it imports faster. `.github/workflows/chart.yml` asserts that +every component's startup budget exceeds its liveness budget, so the two +cannot drift apart silently. + + **`/queue`, the admin command for most of what follows.** `QueueCog` (`src/sturnus/infrastructure/discord/queue_cog.py`) adds three subcommands, all admin-gated the same way `/setup` and `/config` are (`require_admin`, From 135a74ee3bcd19e9d6370ebe5166890f431ba6cb Mon Sep 17 00:00:00 2001 From: TheMeinerLP Date: Fri, 21 Aug 2026 19:51:58 +0200 Subject: [PATCH 2/2] fix(chart): render link's startupProbe, which values declared but nothing emitted The previous commit added `link.startupProbe` to values.yaml and the block to `bot-deployment.yaml`, but never to `link-deployment.yaml` -- so the value sat there configuring nothing and `link` kept being judged by liveness while it was still importing the OTel SDK. Which is the exact failure the commit set out to fix, on the exact component whose 55-second start motivated it. Found by the assertion that same commit added. It did its job on its own author. --- charts/sturnus/templates/link-deployment.yaml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/charts/sturnus/templates/link-deployment.yaml b/charts/sturnus/templates/link-deployment.yaml index c975fac..908efbf 100644 --- a/charts/sturnus/templates/link-deployment.yaml +++ b/charts/sturnus/templates/link-deployment.yaml @@ -60,6 +60,10 @@ spec: - name: health containerPort: {{ .Values.healthPort }} protocol: TCP + {{- with .Values.link.startupProbe }} + startupProbe: + {{- toYaml . | nindent 12 }} + {{- end }} {{- with .Values.link.livenessProbe }} livenessProbe: {{- toYaml . | nindent 12 }}