diff --git a/charts/sourcegraph/CHANGELOG.md b/charts/sourcegraph/CHANGELOG.md index 11acb065..32b7d4e0 100644 --- a/charts/sourcegraph/CHANGELOG.md +++ b/charts/sourcegraph/CHANGELOG.md @@ -10,6 +10,8 @@ Use `**BREAKING**:` to denote a breaking change - Added an `outbound-http-proxy` example, which routes outbound traffic from frontend, worker, and migrator through an HTTP proxy with `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` - Fixed the `codeintel-db` PersistentVolumeClaim being renamed by `codeIntelDB.name`, which left the StatefulSet pointing at a `codeintel-db` claim that no longer existed and stuck the pod in `Pending`. The PVC is now always named `codeintel-db`, matching the other components and the documented behaviour of `*.name`. If you worked around this by creating a `codeintel-db` PVC by hand, label it `app.kubernetes.io/managed-by: Helm` and annotate it with `meta.helm.sh/release-name` and `meta.helm.sh/release-namespace` before upgrading so Helm can adopt it. [#612](https://github.com/sourcegraph/deploy-sourcegraph-helm/issues/612) +- Added `indexedSearch.readinessProbe` (`failureThreshold`, `periodSeconds`, `timeoutSeconds`) to tune the `zoekt-webserver` readiness probe, which was previously hardcoded. Defaults are unchanged (`3`/`5`/`5`). Raise `failureThreshold` on instances with large monorepos, where loading a many-shard repo can stall `/healthz` past `timeoutSeconds` and drop a healthy pod out of the `indexed-search` service. +- Added `indexedSearch.livenessProbe` (`failureThreshold`, `initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`) to tune the `zoekt-webserver` liveness probe, which was previously hardcoded. Defaults are unchanged (`10`/`120`/`60`/`5`). Raise `failureThreshold` or `timeoutSeconds` where the index greatly exceeds the container memory limit: page-cache reclaim stalls `/healthz` past `timeoutSeconds`, and consecutive failures restart a container that is still serving. Keep the resulting window wider than zoekt's in-process watchdog (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog remains the first line of defence. - Added opt-in Redis config management through `redisCache.config.enabled` and `redisStore.config.enabled` (both default to `false`). Default deployments retain their image configs, custom mounts, authentication, and memory settings. When enabled, the chart mounts `/etc/redis/redis.conf` and supports `existingConfig`, `maxmemory`, `maxmemoryRatio`, `maxmemoryPolicy`, and `additionalConfig`. See [Enabling Redis config management](README.md#enabling-redis-config-management) before opting in, especially when using custom images or Secret-mounted configs. - With Redis config management enabled, the chart sizes `maxmemory` at 75% of `resources.limits.memory` to leave room for overhead; this reduces, but does not eliminate, OOM risk. Ratios must be greater than 0 and less than 1. Auto-sizing is skipped in local development mode or when the limit is absent or unrecognised. At the default 7Gi limit, opting in lowers the effective cap from 6GiB to 5.25GiB: `redis-cache` evicts earlier, but `redis-store` uses `noeviction`, so its write-error ceiling drops too. Set `redisStore.config.maxmemory: 6gb` to keep the old ceiling if the pod has sufficient memory. - `searcher.storageSize` now accepts any Kubernetes quantity (e.g. `150G`, `1Ti`, `1.5Ti`, `500000M`) when deriving `SEARCHER_CACHE_SIZE_MB` and `SYMBOLS_CACHE_SIZE_MB`. Previously anything other than `Gi` silently rendered both as `0`, which disables cache eviction and lets the cache volume fill up. Unparseable values, or values below `3M`, now fail the render unless `searcher.autoCacheSize` is enabled. The derived size now matches searcher's own auto-sizing arithmetic (45% of the volume in decimal megabytes), so existing `Gi` deployments see a slightly higher value on upgrade, e.g. the default `26Gi` renders `12562` instead of `11700`. diff --git a/charts/sourcegraph/README.md b/charts/sourcegraph/README.md index 2506226e..6a6ad8d1 100644 --- a/charts/sourcegraph/README.md +++ b/charts/sourcegraph/README.md @@ -221,9 +221,11 @@ In addition to the documented values, all services also support the following va | indexedSearch.env | object | the chart will add some default environment values | Environment variables for the `zoekt-webserver` container | | indexedSearch.image.defaultTag | string | `@sha256:`, pinned by each release | Docker image tag for the `zoekt-webserver` image | | indexedSearch.image.name | string | `"indexed-searcher"` | Docker image name for the `zoekt-webserver` image | +| indexedSearch.livenessProbe | object | `{"failureThreshold":10,"initialDelaySeconds":120,"periodSeconds":60,"timeoutSeconds":5}` | Liveness probe tuning for the `zoekt-webserver` container. Raise `failureThreshold` or `timeoutSeconds` where memory pressure stalls `/healthz` for minutes at a time, so the container is not killed while it is still serving. Keep the window wider than zoekt's in-process watchdog (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog stays the first line of defence. | | indexedSearch.name | string | `"indexed-search"` | Name used by resources. Does not affect service names or PVCs. | | indexedSearch.podDisruptionBudget | object | `{}` | Pod disruption budget for `indexed-search`. Configure either `minAvailable` or `maxUnavailable` to enable it. | | indexedSearch.podSecurityContext | object | `{"fsGroup":101,"fsGroupChangePolicy":"OnRootMismatch"}` | Security context for the `indexed-search` pod, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod) | +| indexedSearch.readinessProbe | object | `{"failureThreshold":3,"periodSeconds":5,"timeoutSeconds":5}` | Readiness probe tuning for the `zoekt-webserver` container. Raise `failureThreshold` on instances with large monorepos: loading a many-shard repo can stall `/healthz` past `timeoutSeconds`, briefly dropping the pod from the service even though it is healthy. | | indexedSearch.replicaCount | int | `1` | Number of `indexed-search` pod | | indexedSearch.resources | object | `{"limits":{"cpu":"2","memory":"4G"},"requests":{"cpu":"500m","memory":"2G"}}` | Resource requests & limits for the `zoekt-webserver` container, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) | | indexedSearch.serviceAccount.create | bool | `false` | Enable creation of ServiceAccount for `indexed-search` | diff --git a/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml b/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml index 561afe8c..0b1b6317 100644 --- a/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml +++ b/charts/sourcegraph/templates/indexed-search/indexed-search.StatefulSet.yaml @@ -68,18 +68,18 @@ spec: path: /healthz port: grpc scheme: HTTP - initialDelaySeconds: 120 - timeoutSeconds: 5 - periodSeconds: 60 - failureThreshold: 10 + initialDelaySeconds: {{ .Values.indexedSearch.livenessProbe.initialDelaySeconds }} + timeoutSeconds: {{ .Values.indexedSearch.livenessProbe.timeoutSeconds }} + periodSeconds: {{ .Values.indexedSearch.livenessProbe.periodSeconds }} + failureThreshold: {{ .Values.indexedSearch.livenessProbe.failureThreshold }} readinessProbe: - failureThreshold: 3 + failureThreshold: {{ .Values.indexedSearch.readinessProbe.failureThreshold }} httpGet: path: /healthz port: grpc scheme: HTTP - periodSeconds: 5 - timeoutSeconds: 5 + periodSeconds: {{ .Values.indexedSearch.readinessProbe.periodSeconds }} + timeoutSeconds: {{ .Values.indexedSearch.readinessProbe.timeoutSeconds }} {{- if not .Values.sourcegraph.localDevMode }} resources: {{- toYaml .Values.indexedSearch.resources | nindent 10 }} diff --git a/charts/sourcegraph/tests/indexedSearchLivenessProbe_test.yaml b/charts/sourcegraph/tests/indexedSearchLivenessProbe_test.yaml new file mode 100644 index 00000000..075822eb --- /dev/null +++ b/charts/sourcegraph/tests/indexedSearchLivenessProbe_test.yaml @@ -0,0 +1,68 @@ +suite: indexedSearchLivenessProbe +release: + name: sourcegraph + namespace: sourcegraph +templates: +- indexed-search/indexed-search.StatefulSet.yaml +tests: +- it: should use the default liveness probe tuning + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.failureThreshold + value: 10 + - equal: + path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds + value: 120 + - equal: + path: spec.template.spec.containers[0].livenessProbe.periodSeconds + value: 60 + - equal: + path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds + value: 5 +- it: should allow overriding liveness probe tuning + set: + indexedSearch: + livenessProbe: + failureThreshold: 20 + initialDelaySeconds: 300 + periodSeconds: 30 + timeoutSeconds: 15 + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.failureThreshold + value: 20 + - equal: + path: spec.template.spec.containers[0].livenessProbe.initialDelaySeconds + value: 300 + - equal: + path: spec.template.spec.containers[0].livenessProbe.periodSeconds + value: 30 + - equal: + path: spec.template.spec.containers[0].livenessProbe.timeoutSeconds + value: 15 +- it: should leave the readiness probe untouched when liveness is overridden + set: + indexedSearch: + livenessProbe: + failureThreshold: 20 + asserts: + - equal: + path: spec.template.spec.containers[0].readinessProbe.failureThreshold + value: 3 + - equal: + path: spec.template.spec.containers[0].readinessProbe.periodSeconds + value: 5 + - equal: + path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds + value: 5 +- it: should keep the liveness probe target and scheme unchanged + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet.path + value: /healthz + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet.port + value: grpc + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet.scheme + value: HTTP diff --git a/charts/sourcegraph/tests/indexedSearchReadinessProbe_test.yaml b/charts/sourcegraph/tests/indexedSearchReadinessProbe_test.yaml new file mode 100644 index 00000000..12f48a6f --- /dev/null +++ b/charts/sourcegraph/tests/indexedSearchReadinessProbe_test.yaml @@ -0,0 +1,47 @@ +suite: indexedSearchReadinessProbe +release: + name: sourcegraph + namespace: sourcegraph +templates: +- indexed-search/indexed-search.StatefulSet.yaml +tests: +- it: should use the default readiness probe tuning + asserts: + - equal: + path: spec.template.spec.containers[0].readinessProbe.failureThreshold + value: 3 + - equal: + path: spec.template.spec.containers[0].readinessProbe.periodSeconds + value: 5 + - equal: + path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds + value: 5 +- it: should allow overriding readiness probe tuning + set: + indexedSearch: + readinessProbe: + failureThreshold: 6 + periodSeconds: 10 + timeoutSeconds: 8 + asserts: + - equal: + path: spec.template.spec.containers[0].readinessProbe.failureThreshold + value: 6 + - equal: + path: spec.template.spec.containers[0].readinessProbe.periodSeconds + value: 10 + - equal: + path: spec.template.spec.containers[0].readinessProbe.timeoutSeconds + value: 8 +- it: should leave the liveness probe untouched when readiness is overridden + set: + indexedSearch: + readinessProbe: + failureThreshold: 6 + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.failureThreshold + value: 10 + - equal: + path: spec.template.spec.containers[0].livenessProbe.periodSeconds + value: 60 diff --git a/charts/sourcegraph/values.yaml b/charts/sourcegraph/values.yaml index c5128c7f..6a7825df 100644 --- a/charts/sourcegraph/values.yaml +++ b/charts/sourcegraph/values.yaml @@ -582,6 +582,25 @@ indexedSearch: replicaCount: 1 # -- Pod disruption budget for `indexed-search`. Configure either `minAvailable` or `maxUnavailable` to enable it. podDisruptionBudget: {} + # -- Liveness probe tuning for the `zoekt-webserver` container. Raise + # `failureThreshold` or `timeoutSeconds` where memory pressure stalls + # `/healthz` for minutes at a time, so the container is not killed while it + # is still serving. Keep the window wider than zoekt's in-process watchdog + # (`ZOEKT_WATCHDOG_ERRORS` x `ZOEKT_WATCHDOG_TICK`) so the watchdog stays + # the first line of defence. + livenessProbe: + failureThreshold: 10 + initialDelaySeconds: 120 + periodSeconds: 60 + timeoutSeconds: 5 + # -- Readiness probe tuning for the `zoekt-webserver` container. Raise + # `failureThreshold` on instances with large monorepos: loading a many-shard + # repo can stall `/healthz` past `timeoutSeconds`, briefly dropping the pod + # from the service even though it is healthy. + readinessProbe: + failureThreshold: 3 + periodSeconds: 5 + timeoutSeconds: 5 # -- Resource requests & limits for the `zoekt-webserver` container, # learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) resources: