From ff66e85802a639e1aa4754e0b1bbf89161d49c12 Mon Sep 17 00:00:00 2001 From: Dinesh Kumar <5300055+devdinu@users.noreply.github.com> Date: Thu, 1 Oct 2026 16:32:27 -0700 Subject: [PATCH 1/2] fix/indexed-search: make zoekt readiness probe tunable The zoekt-webserver readiness probe is hardcoded to failureThreshold 3 and periodSeconds 5, so a pod has 15s of grace. On instances with large monorepos that is not enough. Loading the shards for a many-shard repo stalls /healthz past the 5s timeout, and the pod is dropped from the indexed-search service while it is still healthy. Observed on a 138-shard repo, where /healthz went from 3-5ms to 245ms under shard load with no container restarts. The liveness probe already allows for this with failureThreshold 10 and periodSeconds 60, which is why only readiness flaps. Readiness never got the same treatment. Makes the three timing fields overridable per instance. Defaults match the previously hardcoded values, so rendered output is unchanged unless a value is set. - [x] Update [changelog](https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/charts/sourcegraph/CHANGELOG.md) - Rendered the StatefulSet from main and from this branch with default values. Output is identical - Rendered with an override and confirmed only the readiness probe changes, liveness untouched - helm lint and helm unittest pass --- charts/sourcegraph/CHANGELOG.md | 1 + charts/sourcegraph/README.md | 1 + .../indexed-search.StatefulSet.yaml | 6 +-- .../indexedSearchReadinessProbe_test.yaml | 47 +++++++++++++++++++ charts/sourcegraph/values.yaml | 8 ++++ 5 files changed, 60 insertions(+), 3 deletions(-) create mode 100644 charts/sourcegraph/tests/indexedSearchReadinessProbe_test.yaml diff --git a/charts/sourcegraph/CHANGELOG.md b/charts/sourcegraph/CHANGELOG.md index 11acb065..b1a4e441 100644 --- a/charts/sourcegraph/CHANGELOG.md +++ b/charts/sourcegraph/CHANGELOG.md @@ -10,6 +10,7 @@ Use `**BREAKING**:` to denote a breaking change - Added an `outbound-http-proxy` example, which routes outbound traffic from frontend, worker, and migrator through an HTTP proxy with `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` - Fixed the `codeintel-db` PersistentVolumeClaim being renamed by `codeIntelDB.name`, which left the StatefulSet pointing at a `codeintel-db` claim that no longer existed and stuck the pod in `Pending`. The PVC is now always named `codeintel-db`, matching the other components and the documented behaviour of `*.name`. If you worked around this by creating a `codeintel-db` PVC by hand, label it `app.kubernetes.io/managed-by: Helm` and annotate it with `meta.helm.sh/release-name` and `meta.helm.sh/release-namespace` before upgrading so Helm can adopt it. [#612](https://github.com/sourcegraph/deploy-sourcegraph-helm/issues/612) +- Added `indexedSearch.readinessProbe` (`failureThreshold`, `periodSeconds`, `timeoutSeconds`) to tune the `zoekt-webserver` readiness probe, which was previously hardcoded. Defaults are unchanged (`3`/`5`/`5`). Raise `failureThreshold` on instances with large monorepos, where loading a many-shard repo can stall `/healthz` past `timeoutSeconds` and drop a healthy pod out of the `indexed-search` service. - Added opt-in Redis config management through `redisCache.config.enabled` and `redisStore.config.enabled` (both default to `false`). Default deployments retain their image configs, custom mounts, authentication, and memory settings. When enabled, the chart mounts `/etc/redis/redis.conf` and supports `existingConfig`, `maxmemory`, `maxmemoryRatio`, `maxmemoryPolicy`, and `additionalConfig`. See [Enabling Redis config management](README.md#enabling-redis-config-management) before opting in, especially when using custom images or Secret-mounted configs. - With Redis config management enabled, the chart sizes `maxmemory` at 75% of `resources.limits.memory` to leave room for overhead; this reduces, but does not eliminate, OOM risk. Ratios must be greater than 0 and less than 1. Auto-sizing is skipped in local development mode or when the limit is absent or unrecognised. At the default 7Gi limit, opting in lowers the effective cap from 6GiB to 5.25GiB: `redis-cache` evicts earlier, but `redis-store` uses `noeviction`, so its write-error ceiling drops too. Set `redisStore.config.maxmemory: 6gb` to keep the old ceiling if the pod has sufficient memory. - `searcher.storageSize` now accepts any Kubernetes quantity (e.g. `150G`, `1Ti`, `1.5Ti`, `500000M`) when deriving `SEARCHER_CACHE_SIZE_MB` and `SYMBOLS_CACHE_SIZE_MB`. Previously anything other than `Gi` silently rendered both as `0`, which disables cache eviction and lets the cache volume fill up. Unparseable values, or values below `3M`, now fail the render unless `searcher.autoCacheSize` is enabled. The derived size now matches searcher's own auto-sizing arithmetic (45% of the volume in decimal megabytes), so existing `Gi` deployments see a slightly higher value on upgrade, e.g. the default `26Gi` renders `12562` instead of `11700`. diff --git a/charts/sourcegraph/README.md b/charts/sourcegraph/README.md index 2506226e..aa0aabf1 100644 --- a/charts/sourcegraph/README.md +++ b/charts/sourcegraph/README.md @@ -224,6 +224,7 @@ In addition to the documented values, all services also support the following va | indexedSearch.name | string | `"indexed-search"` | Name used by resources. Does not affect service names or PVCs. | | indexedSearch.podDisruptionBudget | object | `{}` | Pod disruption budget for `indexed-search`. Configure either `minAvailable` or `maxUnavailable` to enable it. | | indexedSearch.podSecurityContext | object | `{"fsGroup":101,"fsGroupChangePolicy":"OnRootMismatch"}` | Security context for the `indexed-search` pod, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) | +| indexedSearch.readinessProbe | object | `{"failureThreshold":3,"periodSeconds":5,"timeoutSeconds":5}` | Readiness probe tuning for the `zoekt-webserver` container. Raise `failureThreshold` on instances with large monorepos: loading a many-shard repo can stall `/healthz` past `timeoutSeconds`, briefly dropping the pod from the service even though it is healthy. | | indexedSearch.replicaCount | int | `1` | Number of `indexed-search` pod | | indexedSearch.resources | object | `{"limits":{"cpu":"2","memory":"4G"},"requests":{"cpu":"500m","memory":"2G"}}` | Resource requests & limits for the `zoekt-webserver` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) | | indexedSearch.serviceAccount.create | bool | `false` | Enable creation of ServiceAccount for `indexed-search` | diff --git a/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml b/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml index 561afe8c..3842f5b5 100644 --- a/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml +++ b/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml @@ -73,13 +73,13 @@ spec: periodSeconds: 60 failureThreshold: 10 readinessProbe: - failureThreshold: 3 + failureThreshold: {{ .Values.indexedSearch.readinessProbe.failureThreshold }} httpGet: path: /healthz port: grpc scheme: HTTP - periodSeconds: 5 - timeoutSeconds: 5 + periodSeconds: {{ .Values.indexedSearch.readinessProbe.periodSeconds }} + timeoutSeconds: {{ .Values.indexedSearch.readinessProbe.timeoutSeconds }} {{- if not .Values.sourcegraph.localDevMode }} resources: {{- toYaml .Values.indexedSearch.resources | nindent 10 }} diff --git a/charts/sourcegraph/tests/indexedSearchReadinessProbe_test.yaml b/charts/sourcegraph/tests/indexedSearchReadinessProbe_test.yaml new file mode 100644 index 00000000..12f48a6f --- /dev/null +++ b/charts/sourcegraph/tests/indexedSearchReadinessProbe_test.yaml @@ -0,0 +1,47 @@ +suite: indexedSearchReadinessProbe +release: + name: sourcegraph + namespace: sourcegraph +templates: +- indexed-search/indexed-search.StatefulSet.yaml +tests: +- it: should use the default readiness probe tuning + asserts: + - equal: + path: spec.template.spec.containers[0].readinessProbe.failureThreshold + value: 3 + - equal: + path: spec.template.spec.containers[0].readinessProbe.periodSeconds + value: 5 + - equal: + path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds + value: 5 +- it: should allow overriding readiness probe tuning + set: + indexedSearch: + readinessProbe: + failureThreshold: 6 + periodSeconds: 10 + timeoutSeconds: 8 + asserts: + - equal: + path: spec.template.spec.containers[0].readinessProbe.failureThreshold + value: 6 + - equal: + path: spec.template.spec.containers[0].readinessProbe.periodSeconds + value: 10 + - equal: + path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds + value: 8 +- it: should leave the liveness probe untouched when readiness is overridden + set: + indexedSearch: + readinessProbe: + failureThreshold: 6 + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.failureThreshold + value: 10 + - equal: + path: spec.template.spec.containers[0].livenessProbe.periodSeconds + value: 60 diff --git a/charts/sourcegraph/values.yaml b/charts/sourcegraph/values.yaml index c5128c7f..29fafbdd 100644 --- a/charts/sourcegraph/values.yaml +++ b/charts/sourcegraph/values.yaml @@ -582,6 +582,14 @@ indexedSearch: replicaCount: 1 # -- Pod disruption budget for `indexed-search`. Configure either `minAvailable` or `maxUnavailable` to enable it. podDisruptionBudget: {} + # -- Readiness probe tuning for the `zoekt-webserver` container. Raise + # `failureThreshold` on instances with large monorepos: loading a many-shard + # repo can stall `/healthz` past `timeoutSeconds`, briefly dropping the pod + # from the service even though it is healthy. + readinessProbe: + failureThreshold: 3 + periodSeconds: 5 + timeoutSeconds: 5 # -- Resource requests & limits for the `zoekt-webserver` container, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) resources: From 965a7c5f247cef3535575de3f9c07a5c322d1b0c Mon Sep 17 00:00:00 2001 From: Dinesh Kumar <5300055+devdinu@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:17:22 -0700 Subject: [PATCH 2/2] fix/indexed-search: make zoekt liveness probe tunable The zoekt-webserver liveness probe is hardcoded to failureThreshold 10, periodSeconds 60, and timeoutSeconds 5, giving a container 12 minutes of grace. Where the index greatly exceeds the container memory limit, page cache fills to the cgroup ceiling and reclaim stalls /healthz past the 5s timeout for minutes at a time. Consecutive failures then restart a container that is still serving. Makes the four timing fields overridable per instance. Defaults match the previously hardcoded values, so rendered output is unchanged unless a value is set. Keep any override wider than zoekt's in-process watchdog so the watchdog stays the first line of defence. ### Test plan - Rendered the StatefulSet from main and from this branch with default values. Output is identical - Rendered with partial overrides of each probe and confirmed the other probe is unaffected - helm lint and helm unittest pass --- charts/sourcegraph/CHANGELOG.md | 1 + charts/sourcegraph/README.md | 1 + .../indexed-search.StatefulSet.yaml | 8 +-- .../indexedSearchLivenessProbe_test.yaml | 68 +++++++++++++++++++ charts/sourcegraph/values.yaml | 11 +++ 5 files changed, 85 insertions(+), 4 deletions(-) create mode 100644 charts/sourcegraph/tests/indexedSearchLivenessProbe_test.yaml diff --git a/charts/sourcegraph/CHANGELOG.md b/charts/sourcegraph/CHANGELOG.md index b1a4e441..32b7d4e0 100644 --- a/charts/sourcegraph/CHANGELOG.md +++ b/charts/sourcegraph/CHANGELOG.md @@ -11,6 +11,7 @@ Use `**BREAKING**:` to denote a breaking change - Added an `outbound-http-proxy` example, which routes outbound traffic from frontend, worker, and migrator through an HTTP proxy with `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` - Fixed the `codeintel-db` PersistentVolumeClaim being renamed by `codeIntelDB.name`, which left the StatefulSet pointing at a `codeintel-db` claim that no longer existed and stuck the pod in `Pending`. The PVC is now always named `codeintel-db`, matching the other components and the documented behaviour of `*.name`. If you worked around this by creating a `codeintel-db` PVC by hand, label it `app.kubernetes.io/managed-by: Helm` and annotate it with `meta.helm.sh/release-name` and `meta.helm.sh/release-namespace` before upgrading so Helm can adopt it. [#612](https://github.com/sourcegraph/deploy-sourcegraph-helm/issues/612) - Added `indexedSearch.readinessProbe` (`failureThreshold`, `periodSeconds`, `timeoutSeconds`) to tune the `zoekt-webserver` readiness probe, which was previously hardcoded. Defaults are unchanged (`3`/`5`/`5`). Raise `failureThreshold` on instances with large monorepos, where loading a many-shard repo can stall `/healthz` past `timeoutSeconds` and drop a healthy pod out of the `indexed-search` service. +- Added `indexedSearch.livenessProbe` (`failureThreshold`, `initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`) to tune the `zoekt-webserver` liveness probe, which was previously hardcoded. Defaults are unchanged (`10`/`120`/`60`/`5`). Raise `failureThreshold` or `timeoutSeconds` where the index greatly exceeds the container memory limit: page-cache reclaim stalls `/healthz` past `timeoutSeconds`, and consecutive failures restart a container that is still serving. Keep the resulting window wider than zoekt's in-process watchdog (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog remains the first line of defence. - Added opt-in Redis config management through `redisCache.config.enabled` and `redisStore.config.enabled` (both default to `false`). Default deployments retain their image configs, custom mounts, authentication, and memory settings. When enabled, the chart mounts `/etc/redis/redis.conf` and supports `existingConfig`, `maxmemory`, `maxmemoryRatio`, `maxmemoryPolicy`, and `additionalConfig`. See [Enabling Redis config management](README.md#enabling-redis-config-management) before opting in, especially when using custom images or Secret-mounted configs. - With Redis config management enabled, the chart sizes `maxmemory` at 75% of `resources.limits.memory` to leave room for overhead; this reduces, but does not eliminate, OOM risk. Ratios must be greater than 0 and less than 1. Auto-sizing is skipped in local development mode or when the limit is absent or unrecognised. At the default 7Gi limit, opting in lowers the effective cap from 6GiB to 5.25GiB: `redis-cache` evicts earlier, but `redis-store` uses `noeviction`, so its write-error ceiling drops too. Set `redisStore.config.maxmemory: 6gb` to keep the old ceiling if the pod has sufficient memory. - `searcher.storageSize` now accepts any Kubernetes quantity (e.g. `150G`, `1Ti`, `1.5Ti`, `500000M`) when deriving `SEARCHER_CACHE_SIZE_MB` and `SYMBOLS_CACHE_SIZE_MB`. Previously anything other than `Gi` silently rendered both as `0`, which disables cache eviction and lets the cache volume fill up. Unparseable values, or values below `3M`, now fail the render unless `searcher.autoCacheSize` is enabled. The derived size now matches searcher's own auto-sizing arithmetic (45% of the volume in decimal megabytes), so existing `Gi` deployments see a slightly higher value on upgrade, e.g. the default `26Gi` renders `12562` instead of `11700`. diff --git a/charts/sourcegraph/README.md b/charts/sourcegraph/README.md index aa0aabf1..6a6ad8d1 100644 --- a/charts/sourcegraph/README.md +++ b/charts/sourcegraph/README.md @@ -221,6 +221,7 @@ In addition to the documented values, all services also support the following va | indexedSearch.env | object | the chart will add some default environment values | Environment variables for the `zoekt-webserver` container | | indexedSearch.image.defaultTag | string | `@sha256:`, pinned by each release | Docker image tag for the `zoekt-webserver` image | | indexedSearch.image.name | string | `"indexed-searcher"` | Docker image name for the `zoekt-webserver` image | +| indexedSearch.livenessProbe | object | `{"failureThreshold":10,"initialDelaySeconds":120,"periodSeconds":60,"timeoutSeconds":5}` | Liveness probe tuning for the `zoekt-webserver` container. Raise `failureThreshold` or `timeoutSeconds` where memory pressure stalls `/healthz` for minutes at a time, so the container is not killed while it is still serving. Keep the window wider than zoekt's in-process watchdog (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog stays the first line of defence. | | indexedSearch.name | string | `"indexed-search"` | Name used by resources. Does not affect service names or PVCs. | | indexedSearch.podDisruptionBudget | object | `{}` | Pod disruption budget for `indexed-search`. Configure either `minAvailable` or `maxUnavailable` to enable it. | | indexedSearch.podSecurityContext | object | `{"fsGroup":101,"fsGroupChangePolicy":"OnRootMismatch"}` | Security context for the `indexed-search` pod, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) | diff --git a/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml b/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml index 3842f5b5..0b1b6317 100644 --- a/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml +++ b/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml @@ -68,10 +68,10 @@ spec: path: /healthz port: grpc scheme: HTTP - initialDelaySeconds: 120 - timeoutSeconds: 5 - periodSeconds: 60 - failureThreshold: 10 + initialDelaySeconds: {{ .Values.indexedSearch.livenessProbe.initialDelaySeconds }} + timeoutSeconds: {{ .Values.indexedSearch.livenessProbe.timeoutSeconds }} + periodSeconds: {{ .Values.indexedSearch.livenessProbe.periodSeconds }} + failureThreshold: {{ .Values.indexedSearch.livenessProbe.failureThreshold }} readinessProbe: failureThreshold: {{ .Values.indexedSearch.readinessProbe.failureThreshold }} httpGet: diff --git a/charts/sourcegraph/tests/indexedSearchLivenessProbe_test.yaml b/charts/sourcegraph/tests/indexedSearchLivenessProbe_test.yaml new file mode 100644 index 00000000..075822eb --- /dev/null +++ b/charts/sourcegraph/tests/indexedSearchLivenessProbe_test.yaml @@ -0,0 +1,68 @@ +suite: indexedSearchLivenessProbe +release: + name: sourcegraph + namespace: sourcegraph +templates: +- indexed-search/indexed-search.StatefulSet.yaml +tests: +- it: should use the default liveness probe tuning + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.failureThreshold + value: 10 + - equal: + path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds + value: 120 + - equal: + path: spec.template.spec.containers[0].livenessProbe.periodSeconds + value: 60 + - equal: + path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds + value: 5 +- it: should allow overriding liveness probe tuning + set: + indexedSearch: + livenessProbe: + failureThreshold: 20 + initialDelaySeconds: 300 + periodSeconds: 30 + timeoutSeconds: 15 + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.failureThreshold + value: 20 + - equal: + path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds + value: 300 + - equal: + path: spec.template.spec.containers[0].livenessProbe.periodSeconds + value: 30 + - equal: + path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds + value: 15 +- it: should leave the readiness probe untouched when liveness is overridden + set: + indexedSearch: + livenessProbe: + failureThreshold: 20 + asserts: + - equal: + path: spec.template.spec.containers[0].readinessProbe.failureThreshold + value: 3 + - equal: + path: spec.template.spec.containers[0].readinessProbe.periodSeconds + value: 5 + - equal: + path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds + value: 5 +- it: should keep the liveness probe target and scheme unchanged + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet.path + value: /healthz + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet.port + value: grpc + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet.scheme + value: HTTP diff --git a/charts/sourcegraph/values.yaml b/charts/sourcegraph/values.yaml index 29fafbdd..6a7825df 100644 --- a/charts/sourcegraph/values.yaml +++ b/charts/sourcegraph/values.yaml @@ -582,6 +582,17 @@ indexedSearch: replicaCount: 1 # -- Pod disruption budget for `indexed-search`. Configure either `minAvailable` or `maxUnavailable` to enable it. podDisruptionBudget: {} + # -- Liveness probe tuning for the `zoekt-webserver` container. Raise + # `failureThreshold` or `timeoutSeconds` where memory pressure stalls + # `/healthz` for minutes at a time, so the container is not killed while it + # is still serving. Keep the window wider than zoekt's in-process watchdog + # (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog stays + # the first line of defence. + livenessProbe: + failureThreshold: 10 + initialDelaySeconds: 120 + periodSeconds: 60 + timeoutSeconds: 5 # -- Readiness probe tuning for the `zoekt-webserver` container. Raise # `failureThreshold` on instances with large monorepos: loading a many-shard # repo can stall `/healthz` past `timeoutSeconds`, briefly dropping the pod