fix(chart): let every component finish starting before liveness judges it

.github/workflows/chart.yml

@@ -111,6 +111,30 @@ jobs:
                         f'{name} mounts a ReadWriteOnce claim {sorted(claims & rwo_claims)} '
                         f'but its strategy is {strategy!r}, not Recreate')
 
+          # Every component gets a startupProbe, and its budget must exceed
+          # its liveness budget. Without one, liveness starts counting from
+          # `initialDelaySeconds` while the process is still importing, and
+          # a slow start is indistinguishable from a hung one. That is not
+          # hypothetical: 0.6.0 added the OpenTelemetry import graph, `link`
+          # took 55 seconds to reach "listening" on 200m of CPU, and its own
+          # liveness probe SIGKILLed it at 137 -- with the process healthy
+          # and about to serve.
+          for name, dep in deployments.items():
+              container = dep['spec']['template']['spec']['containers'][0]
+              startup = container.get('startupProbe')
+              check(startup is not None,
+                    f'{name} has no startupProbe, so liveness judges it while it is '
+                    f'still starting')
+              if startup is None:
+                  continue
+              startup_budget = startup.get('periodSeconds', 10) * startup.get('failureThreshold', 3)
+              live = container.get('livenessProbe') or {}
+              live_budget = (live.get('initialDelaySeconds', 0)
+                             + live.get('periodSeconds', 10) * live.get('failureThreshold', 3))
+              check(startup_budget > live_budget,
+                    f'{name} allows {startup_budget}s to start but liveness would have '
+                    f'killed it after {live_budget}s; the startupProbe buys nothing')
+
           # podLabels must reach every pod, and must never reach a selector:
           # selectors are immutable on a live Deployment.
           for name, dep in deployments.items():

charts/sturnus/templates/bot-deployment.yaml

@@ -84,6 +84,10 @@ spec:
             - name: health
               containerPort: {{ .Values.healthPort }}
               protocol: TCP
+          {{- with .Values.bot.startupProbe }}
+          startupProbe:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
           {{- with .Values.bot.livenessProbe }}
           livenessProbe:
             {{- toYaml . | nindent 12 }}

charts/sturnus/templates/link-deployment.yaml

@@ -60,6 +60,10 @@ spec:
             - name: health
               containerPort: {{ .Values.healthPort }}
               protocol: TCP
+          {{- with .Values.link.startupProbe }}
+          startupProbe:
+            {{- toYaml . | nindent 12 }}
+          {{- end }}
           {{- with .Values.link.livenessProbe }}
           livenessProbe:
             {{- toYaml . | nindent 12 }}

charts/sturnus/values.yaml

@@ -232,6 +232,29 @@ bot:
   # documented `max_session_hours` ceiling plus upload time with margin; size
   # this to the deployed `max_session_hours` if that default ever changes.
   terminationGracePeriodSeconds: 21600
+  # Startup is slower than it looks, and on 0.6.0 it outgrew the liveness
+  # budget: a `link` pod took 55 seconds from container start to "Link
+  # service listening" and was SIGKILLed at 137 by its own liveness probe,
+  # which allows 5 + 3x20 seconds and then gives up.
+  #
+  # The time is import time, not work: the OpenTelemetry SDK and its
+  # exporters are a large import graph, and `bot` has 1 CPU of CPU to do
+  # it with. A startupProbe is exactly the instrument for that -- liveness
+  # and readiness are not evaluated at all until it succeeds, so a slow
+  # start cannot be mistaken for a hung process, and once it succeeds the
+  # normal cadence takes over. `worker` has had one since the model
+  # download made the same point more loudly.
+  #
+  # 5 minutes of budget, which is far more than the 55 seconds observed:
+  # the cost of being generous is that a process genuinely wedged at
+  # startup is noticed later, and the cost of being tight is a crash loop
+  # on a cold node. Only one of those is a working deployment.
+  startupProbe:
+    httpGet:
+      path: /healthz
+      port: health
+    periodSeconds: 10
+    failureThreshold: 30
   livenessProbe:
     httpGet:
       path: /healthz
@@ -277,6 +300,29 @@ link:
     limits:
       cpu: 200m
       memory: 128Mi
+  # Startup is slower than it looks, and on 0.6.0 it outgrew the liveness
+  # budget: a `link` pod took 55 seconds from container start to "Link
+  # service listening" and was SIGKILLed at 137 by its own liveness probe,
+  # which allows 5 + 3x20 seconds and then gives up.
+  #
+  # The time is import time, not work: the OpenTelemetry SDK and its
+  # exporters are a large import graph, and `link` has 200m of CPU to do
+  # it with. A startupProbe is exactly the instrument for that -- liveness
+  # and readiness are not evaluated at all until it succeeds, so a slow
+  # start cannot be mistaken for a hung process, and once it succeeds the
+  # normal cadence takes over. `worker` has had one since the model
+  # download made the same point more loudly.
+  #
+  # 5 minutes of budget, which is far more than the 55 seconds observed:
+  # the cost of being generous is that a process genuinely wedged at
+  # startup is noticed later, and the cost of being tight is a crash loop
+  # on a cold node. Only one of those is a working deployment.
+  startupProbe:
+    httpGet:
+      path: /healthz
+      port: health
+    periodSeconds: 10
+    failureThreshold: 30
   livenessProbe:
     httpGet:
       path: /healthz

docs/operations.md

@@ -527,6 +527,22 @@ stored is actually what is running.
 
 ## 5. Troubleshooting
 
+**A pod is SIGKILLed at 137 shortly after starting, with `connection
+refused` on its probes.** The process is not crashing; its liveness probe
+is killing it while it is still importing. Look at the gap between the
+container's `startedAt` and its first "listening" log line: on 0.6.0 a
+`link` pod took 55 seconds on 200m of CPU, against a liveness budget of
+`initialDelaySeconds + periodSeconds x failureThreshold` = 65 seconds.
+
+Every component has a `startupProbe` for exactly this, and liveness and
+readiness are not evaluated at all until it succeeds. If you see this
+again, the startup budget has been outgrown rather than the process broken
+-- raise `<component>.startupProbe.failureThreshold`, or give the component
+more CPU so it imports faster. `.github/workflows/chart.yml` asserts that
+every component's startup budget exceeds its liveness budget, so the two
+cannot drift apart silently.
+
+
 **`/queue`, the admin command for most of what follows.** `QueueCog`
 (`src/sturnus/infrastructure/discord/queue_cog.py`) adds three subcommands,
 all admin-gated the same way `/setup` and `/config` are (`require_admin`,