Make zoekt-webserver probes tunable

charts/sourcegraph/CHANGELOG.md

@@ -10,6 +10,8 @@ Use `**BREAKING**:` to denote a breaking change
 
 - Added an `outbound-http-proxy` example, which routes outbound traffic from frontend, worker, and migrator through an HTTP proxy with `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`
 - Fixed the `codeintel-db` PersistentVolumeClaim being renamed by `codeIntelDB.name`, which left the StatefulSet pointing at a `codeintel-db` claim that no longer existed and stuck the pod in `Pending`. The PVC is now always named `codeintel-db`, matching the other components and the documented behaviour of `*.name`. If you worked around this by creating a `codeintel-db` PVC by hand, label it `app.kubernetes.io/managed-by: Helm` and annotate it with `meta.helm.sh/release-name` and `meta.helm.sh/release-namespace` before upgrading so Helm can adopt it. [#612](https://github.com/sourcegraph/deploy-sourcegraph-helm/issues/612)
+- Added `indexedSearch.readinessProbe` (`failureThreshold`, `periodSeconds`, `timeoutSeconds`) to tune the `zoekt-webserver` readiness probe, which was previously hardcoded. Defaults are unchanged (`3`/`5`/`5`). Raise `failureThreshold` on instances with large monorepos, where loading a many-shard repo can stall `/healthz` past `timeoutSeconds` and drop a healthy pod out of the `indexed-search` service.
+- Added `indexedSearch.livenessProbe` (`failureThreshold`, `initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`) to tune the `zoekt-webserver` liveness probe, which was previously hardcoded. Defaults are unchanged (`10`/`120`/`60`/`5`). Raise `failureThreshold` or `timeoutSeconds` where the index greatly exceeds the container memory limit: page-cache reclaim stalls `/healthz` past `timeoutSeconds`, and consecutive failures restart a container that is still serving. Keep the resulting window wider than zoekt's in-process watchdog (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog remains the first line of defence.
 - Added opt-in Redis config management through `redisCache.config.enabled` and `redisStore.config.enabled` (both default to `false`). Default deployments retain their image configs, custom mounts, authentication, and memory settings. When enabled, the chart mounts `/etc/redis/redis.conf` and supports `existingConfig`, `maxmemory`, `maxmemoryRatio`, `maxmemoryPolicy`, and `additionalConfig`. See [Enabling Redis config management](README.md#enabling-redis-config-management) before opting in, especially when using custom images or Secret-mounted configs.
 - With Redis config management enabled, the chart sizes `maxmemory` at 75% of `resources.limits.memory` to leave room for overhead; this reduces, but does not eliminate, OOM risk. Ratios must be greater than 0 and less than 1. Auto-sizing is skipped in local development mode or when the limit is absent or unrecognised. At the default 7Gi limit, opting in lowers the effective cap from 6GiB to 5.25GiB: `redis-cache` evicts earlier, but `redis-store` uses `noeviction`, so its write-error ceiling drops too. Set `redisStore.config.maxmemory: 6gb` to keep the old ceiling if the pod has sufficient memory.
 - `searcher.storageSize` now accepts any Kubernetes quantity (e.g. `150G`, `1Ti`, `1.5Ti`, `500000M`) when deriving `SEARCHER_CACHE_SIZE_MB` and `SYMBOLS_CACHE_SIZE_MB`. Previously anything other than `Gi` silently rendered both as `0`, which disables cache eviction and lets the cache volume fill up. Unparseable values, or values below `3M`, now fail the render unless `searcher.autoCacheSize` is enabled. The derived size now matches searcher's own auto-sizing arithmetic (45% of the volume in decimal megabytes), so existing `Gi` deployments see a slightly higher value on upgrade, e.g. the default `26Gi` renders `12562` instead of `11700`.

charts/sourcegraph/README.md

@@ -221,9 +221,11 @@ In addition to the documented values, all services also support the following va
 | indexedSearch.env | object | the chart will add some default environment values | Environment variables for the `zoekt-webserver` container |
 | indexedSearch.image.defaultTag | string | `<appVersion>@sha256:<digest>`, pinned by each release | Docker image tag for the `zoekt-webserver` image |
 | indexedSearch.image.name | string | `"indexed-searcher"` | Docker image name for the `zoekt-webserver` image |
+| indexedSearch.livenessProbe | object | `{"failureThreshold":10,"initialDelaySeconds":120,"periodSeconds":60,"timeoutSeconds":5}` | Liveness probe tuning for the `zoekt-webserver` container. Raise `failureThreshold` or `timeoutSeconds` where memory pressure stalls `/healthz` for minutes at a time, so the container is not killed while it is still serving. Keep the window wider than zoekt's in-process watchdog (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog stays the first line of defence. |
 | indexedSearch.name | string | `"indexed-search"` | Name used by resources. Does not affect service names or PVCs. |
 | indexedSearch.podDisruptionBudget | object | `{}` | Pod disruption budget for `indexed-search`. Configure either `minAvailable` or `maxUnavailable` to enable it. |
 | indexedSearch.podSecurityContext | object | `{"fsGroup":101,"fsGroupChangePolicy":"OnRootMismatch"}` | Security context for the `indexed-search` pod, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) |
+| indexedSearch.readinessProbe | object | `{"failureThreshold":3,"periodSeconds":5,"timeoutSeconds":5}` | Readiness probe tuning for the `zoekt-webserver` container. Raise `failureThreshold` on instances with large monorepos: loading a many-shard repo can stall `/healthz` past `timeoutSeconds`, briefly dropping the pod from the service even though it is healthy. |
 | indexedSearch.replicaCount | int | `1` | Number of `indexed-search` pod |
 | indexedSearch.resources | object | `{"limits":{"cpu":"2","memory":"4G"},"requests":{"cpu":"500m","memory":"2G"}}` | Resource requests & limits for the `zoekt-webserver` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) |
 | indexedSearch.serviceAccount.create | bool | `false` | Enable creation of ServiceAccount for `indexed-search` |

charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml

@@ -68,18 +68,18 @@ spec:
             path: /healthz
             port: grpc
             scheme: HTTP
-          initialDelaySeconds: 120
-          timeoutSeconds: 5
-          periodSeconds: 60
-          failureThreshold: 10
+          initialDelaySeconds: {{ .Values.indexedSearch.livenessProbe.initialDelaySeconds }}
+          timeoutSeconds: {{ .Values.indexedSearch.livenessProbe.timeoutSeconds }}
+          periodSeconds: {{ .Values.indexedSearch.livenessProbe.periodSeconds }}
+          failureThreshold: {{ .Values.indexedSearch.livenessProbe.failureThreshold }}
         readinessProbe:
-          failureThreshold: 3
+          failureThreshold: {{ .Values.indexedSearch.readinessProbe.failureThreshold }}
           httpGet:
             path: /healthz
             port: grpc
             scheme: HTTP
-          periodSeconds: 5
-          timeoutSeconds: 5
+          periodSeconds: {{ .Values.indexedSearch.readinessProbe.periodSeconds }}
+          timeoutSeconds: {{ .Values.indexedSearch.readinessProbe.timeoutSeconds }}
         {{- if not .Values.sourcegraph.localDevMode }}
         resources:
           {{- toYaml .Values.indexedSearch.resources | nindent 10 }}

charts/sourcegraph/tests/indexedSearchLivenessProbe_test.yaml

@@ -0,0 +1,68 @@
+suite: indexedSearchLivenessProbe
+release:
+  name: sourcegraph
+  namespace: sourcegraph
+templates:
+- indexed-search/indexed-search.StatefulSet.yaml
+tests:
+- it: should use the default liveness probe tuning
+  asserts:
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.failureThreshold
+      value: 10
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds
+      value: 120
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.periodSeconds
+      value: 60
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds
+      value: 5
+- it: should allow overriding liveness probe tuning
+  set:
+    indexedSearch:
+      livenessProbe:
+        failureThreshold: 20
+        initialDelaySeconds: 300
+        periodSeconds: 30
+        timeoutSeconds: 15
+  asserts:
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.failureThreshold
+      value: 20
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds
+      value: 300
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.periodSeconds
+      value: 30
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds
+      value: 15
+- it: should leave the readiness probe untouched when liveness is overridden
+  set:
+    indexedSearch:
+      livenessProbe:
+        failureThreshold: 20
+  asserts:
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.failureThreshold
+      value: 3
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.periodSeconds
+      value: 5
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
+      value: 5
+- it: should keep the liveness probe target and scheme unchanged
+  asserts:
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.httpGet.path
+      value: /healthz
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.httpGet.port
+      value: grpc
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.httpGet.scheme
+      value: HTTP

charts/sourcegraph/tests/indexedSearchReadinessProbe_test.yaml

@@ -0,0 +1,47 @@
+suite: indexedSearchReadinessProbe
+release:
+  name: sourcegraph
+  namespace: sourcegraph
+templates:
+- indexed-search/indexed-search.StatefulSet.yaml
+tests:
+- it: should use the default readiness probe tuning
+  asserts:
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.failureThreshold
+      value: 3
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.periodSeconds
+      value: 5
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
+      value: 5
+- it: should allow overriding readiness probe tuning
+  set:
+    indexedSearch:
+      readinessProbe:
+        failureThreshold: 6
+        periodSeconds: 10
+        timeoutSeconds: 8
+  asserts:
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.failureThreshold
+      value: 6
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.periodSeconds
+      value: 10
+  - equal:
+      path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds
+      value: 8
+- it: should leave the liveness probe untouched when readiness is overridden
+  set:
+    indexedSearch:
+      readinessProbe:
+        failureThreshold: 6
+  asserts:
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.failureThreshold
+      value: 10
+  - equal:
+      path: spec.template.spec.containers[0].livenessProbe.periodSeconds
+      value: 60

charts/sourcegraph/values.yaml

@@ -582,6 +582,25 @@ indexedSearch:
   replicaCount: 1
   # -- Pod disruption budget for `indexed-search`. Configure either `minAvailable` or `maxUnavailable` to enable it.
   podDisruptionBudget: {}
+  # -- Liveness probe tuning for the `zoekt-webserver` container. Raise
+  # `failureThreshold` or `timeoutSeconds` where memory pressure stalls
+  # `/healthz` for minutes at a time, so the container is not killed while it
+  # is still serving. Keep the window wider than zoekt's in-process watchdog
+  # (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog stays
+  # the first line of defence.
+  livenessProbe:
+    failureThreshold: 10
+    initialDelaySeconds: 120
+    periodSeconds: 60
+    timeoutSeconds: 5
+  # -- Readiness probe tuning for the `zoekt-webserver` container. Raise
+  # `failureThreshold` on instances with large monorepos: loading a many-shard
+  # repo can stall `/healthz` past `timeoutSeconds`, briefly dropping the pod
+  # from the service even though it is healthy.
+  readinessProbe:
+    failureThreshold: 3
+    periodSeconds: 5
+    timeoutSeconds: 5
   # -- Resource requests & limits for the `zoekt-webserver` container,
   # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
   resources: