fix(chart): let every component finish starting before liveness judges it - #52
Open
TheMeinerLP wants to merge 1 commit into
Open
fix(chart): let every component finish starting before liveness judges it#52TheMeinerLP wants to merge 1 commit into
TheMeinerLP wants to merge 1 commit into
Conversation
…s it 0.6.0 added the OpenTelemetry import graph, and `link` outgrew its liveness budget: 55 seconds from container start to 'Link service listening' on 200m of CPU, against `initialDelaySeconds + periodSeconds x failureThreshold` = 65 seconds. It was SIGKILLed at 137 with the process healthy and about to serve. Observed in production immediately after the 0.6.0 rollout. `worker` has had a startupProbe since the model download made the same point more loudly; `bot` and `link` never got one, because until now nothing they did at import took long enough to notice. Both get one now, with five minutes of budget against the 55 seconds observed -- being generous costs a later notice for a process genuinely wedged at startup, and being tight costs a crash loop on a cold node. The chart job now asserts every component has a startupProbe and that its budget exceeds its liveness budget, since a startupProbe shorter than liveness buys nothing and looks like it does. Counter-checked: removing link's stanza makes the assertion fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found in production, minutes after the 0.6.0 rollout.
The process was not crashing. It was still importing, and its own liveness probe killed it:
initialDelaySeconds 5 + periodSeconds 20 x failureThreshold 3= 65 seconds, against 55 seconds to reach listening.The time is import time, not work — 0.6.0 adds the OpenTelemetry SDK and its exporters, which is a large import graph, and
linkhas 200m of CPU to do it with.workersurvived because it has 4 CPUs and a startupProbe, which it has had since the model download made the same point more loudly.botandlinknever got one, because until now nothing they did at import took long enough to notice.Both get one now. Liveness and readiness are not evaluated at all until a startupProbe succeeds, which is exactly the instrument for "slow to start" versus "hung".
Five minutes of budget against the 55 seconds observed. Being generous costs a later notice for a process genuinely wedged at startup; being tight costs a crash loop on a cold node. Only one of those is a working deployment.
The regression guard matters as much as the fix
.github/workflows/chart.ymlnow asserts that every component has a startupProbe and that its budget exceeds its liveness budget — because a startupProbe shorter than liveness buys nothing while looking like it does. Rendered:Counter-checked: removing link's stanza from the template makes the assertion fail.
Also documented in
docs/operations.md§5, since the symptom — 137 withconnection refused— reads like a crash and is not one.Current production state
Both
linkpods are running; one restarted once and has been stable since. This is not an outage, it is fragility: any restart on a cold node can exceed the budget again, andbotis exposed too at 1 CPU.