Repository navigation
Make zoekt-webserver probes tunable - #983
Merged
Merged
Conversation
devdinu
changed the title
devdinu
marked this pull request as ready for review
October 26, 2026 18:35
The zoekt-webserver readiness probe is hardcoded to failureThreshold 3 and periodSeconds 5, so a pod has 15s of grace. On instances with large monorepos that is not enough. Loading the shards for a many-shard repo stalls /healthz past the 5s timeout, and the pod is dropped from the indexed-search service while it is still healthy. Observed on a 138-shard repo, where /healthz went from 3-5ms to 245ms under shard load with no container restarts. The liveness probe already allows for this with failureThreshold 10 and periodSeconds 60, which is why only readiness flaps. Readiness never got the same treatment. Makes the three timing fields overridable per instance. Defaults match the previously hardcoded values, so rendered output is unchanged unless a value is set. - [x] Update [changelog](https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/charts/sourcegraph/CHANGELOG.md) - Rendered the StatefulSet from main and from this branch with default values. Output is identical - Rendered with an override and confirmed only the readiness probe changes, liveness untouched - helm lint and helm unittest pass
The zoekt-webserver liveness probe is hardcoded to failureThreshold 10, periodSeconds 60, and timeoutSeconds 5, giving a container 12 minutes of grace. Where the index greatly exceeds the container memory limit, page cache fills to the cgroup ceiling and reclaim stalls /healthz past the 5s timeout for minutes at a time. Consecutive failures then restart a container that is still serving. Makes the four timing fields overridable per instance. Defaults match the previously hardcoded values, so rendered output is unchanged unless a value is set. Keep any override wider than zoekt's in-process watchdog so the watchdog stays the first line of defence. ### Test plan - Rendered the StatefulSet from main and from this branch with default values. Output is identical - Rendered with partial overrides of each probe and confirmed the other probe is unaffected - helm lint and helm unittest pass
devdinu
force-pushed
the
fix/zoekt-readiness-probe-tunable
branch
from
October 27, 2026 22:59
ae6e89d to
965a7c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
zoekt-webserverprobes are hardcoded. Readiness allows 15s of grace (failureThreshold: 3,periodSeconds: 5) and liveness allows 12 minutes (failureThreshold: 10,periodSeconds: 60), both withtimeoutSeconds: 5. On instances where/healthzstalls past that timeout, readiness drops a healthy pod out of theindexed-searchservice and liveness restarts a container that is still serving. Neither window can be adjusted per instance.Two causes seen in production. Loading the shards for a many-shard repo stalls
/healthzduring startup. Separately, where the on-disk index greatly exceeds the container memory limit, page cache fills to the cgroup ceiling and reclaim stalls every process in the cgroup for a sustained fraction of wall-clock time.Changes
indexedSearch.readinessProbewithfailureThreshold,periodSeconds, andtimeoutSecondsindexedSearch.livenessProbewithfailureThreshold,initialDelaySeconds,periodSeconds, andtimeoutSeconds3/5/5and10/120/60/5), so rendered output is unchanged unless a value is setLiveness remains a backup to zoekt's in-process watchdog, as set up in #850. Any override should keep the probe window wider than
ZOEKT_WATCHDOG_ERRORSxZOEKT_WATCHDOG_TICKso the watchdog detects a wedged process first.Raising a threshold is a mitigation, not a fix. It keeps a stalling pod in service rather than cycling it, but where the index outsizes the memory limit the real remedy is more replicas or more memory per pod.
ref EPD2-432
Test Plan
mainand from this branch with default values. Output is byte-identical, so existing deployments are unaffected on upgradehelm lintand the fullhelm unittestsuite pass (206 tests, 35 suites)Alert - Slack Investigation with restart trend: https://sourcegraph.slack.com/archives/C0A4S5X3L8P/p1793039912710159?thread_ts=1793033284.231829&cid=C0A4S5X3L8P