Skip to content

fix(chart): let every component finish starting before liveness judges it - #52

Open
TheMeinerLP wants to merge 1 commit into
mainfrom
fix/chart-startup-probes
Open

fix(chart): let every component finish starting before liveness judges it#52
TheMeinerLP wants to merge 1 commit into
mainfrom
fix/chart-startup-probes

Conversation

@TheMeinerLP

Copy link
Copy Markdown
Contributor

Found in production, minutes after the 0.6.0 rollout.

exitCode 137 (SIGKILL), reason Error
Liveness probe failed: connect: connection refused
container startedAt 10:58:03 — "Link service listening" at 10:58:58

The process was not crashing. It was still importing, and its own liveness probe killed it: initialDelaySeconds 5 + periodSeconds 20 x failureThreshold 3 = 65 seconds, against 55 seconds to reach listening.

The time is import time, not work — 0.6.0 adds the OpenTelemetry SDK and its exporters, which is a large import graph, and link has 200m of CPU to do it with. worker survived because it has 4 CPUs and a startupProbe, which it has had since the model download made the same point more loudly. bot and link never got one, because until now nothing they did at import took long enough to notice.

Both get one now. Liveness and readiness are not evaluated at all until a startupProbe succeeds, which is exactly the instrument for "slow to start" versus "hung".

Five minutes of budget against the 55 seconds observed. Being generous costs a later notice for a process genuinely wedged at startup; being tight costs a crash loop on a cold node. Only one of those is a working deployment.

The regression guard matters as much as the fix

.github/workflows/chart.yml now asserts that every component has a startupProbe and that its budget exceeds its liveness budget — because a startupProbe shorter than liveness buys nothing while looking like it does. Rendered:

startup budget liveness budget
bot 300 s 70 s
link 300 s 65 s
worker 1200 s 60 s

Counter-checked: removing link's stanza from the template makes the assertion fail.

Also documented in docs/operations.md §5, since the symptom — 137 with connection refused — reads like a crash and is not one.

Current production state

Both link pods are running; one restarted once and has been stable since. This is not an outage, it is fragility: any restart on a cold node can exceed the budget again, and bot is exposed too at 1 CPU.

…s it

0.6.0 added the OpenTelemetry import graph, and `link` outgrew its
liveness budget: 55 seconds from container start to 'Link service
listening' on 200m of CPU, against `initialDelaySeconds + periodSeconds x
failureThreshold` = 65 seconds. It was SIGKILLed at 137 with the process
healthy and about to serve. Observed in production immediately after the
0.6.0 rollout.

`worker` has had a startupProbe since the model download made the same
point more loudly; `bot` and `link` never got one, because until now
nothing they did at import took long enough to notice. Both get one now,
with five minutes of budget against the 55 seconds observed -- being
generous costs a later notice for a process genuinely wedged at startup,
and being tight costs a crash loop on a cold node.

The chart job now asserts every component has a startupProbe and that its
budget exceeds its liveness budget, since a startupProbe shorter than
liveness buys nothing and looks like it does. Counter-checked: removing
link's stanza makes the assertion fail.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant