diff --git a/content/en/docs/next/guides/resource-management/_index.md b/content/en/docs/next/guides/resource-management/_index.md index 25622ab1..d0a16d84 100644 --- a/content/en/docs/next/guides/resource-management/_index.md +++ b/content/en/docs/next/guides/resource-management/_index.md @@ -83,10 +83,9 @@ The seven short names that existed before the instance-type rename remain accept Legacy names are scheduled for removal in a future Cozystack release; new manifests should use the instance-type form. The Cozystack API server logs a deprecation warning whenever an app CR carries a legacy value, naming the suggested replacement. -A platform upgrade runs a one-shot migration (Migration 39) that walks every `HelmRelease.spec.values` and every app CR under `apps.cozystack.io/v1alpha1` and rewrites legacy values to their instance-type equivalents in place. The conversion is idempotent, best-effort, and never changes CPU or memory. +A platform upgrade runs a one-shot migration (Migration 39) that walks every `HelmRelease.spec.values` and every app CR under `apps.cozystack.io/v1alpha1` and rewrites legacy values to their instance-type equivalents in place. The conversion is idempotent, best-effort, and never changes CPU or memory. A failure on one object logs a warning and leaves the rest of the migration to complete. Set `MIGRATION_DRY_RUN=1` to preview the patches without applying them. -Cozystack presets are defined in an internal library -[`cozy-lib`](https://github.com/cozystack/cozystack/tree/main/packages/library/cozy-lib). The canonical reference, including the full size matrix and migration table, lives in [`docs/operations/resource-presets.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/resource-presets.md). +Cozystack presets are defined in an internal library [`cozy-lib`](https://github.com/cozystack/cozystack/tree/main/packages/library/cozy-lib). ### Defining Resources Explicitly diff --git a/content/en/docs/next/networking/gateway-classes.md b/content/en/docs/next/networking/gateway-classes.md new file mode 100644 index 00000000..dcb57cd7 --- /dev/null +++ b/content/en/docs/next/networking/gateway-classes.md @@ -0,0 +1,130 @@ +--- +title: "GatewayClass selection and TLS termination for tenant Gateways" +linkTitle: "GatewayClass selection" +description: "How a tenant picks a GatewayClass, what the class decides about TLS termination, and how to move a tenant between classes." +weight: 16 +--- + +A tenant that owns a Gateway (`tenant.spec.gateway: true`) gets one `Gateway` object bound to one `GatewayClass`. Which class that is decides who programs the dataplane and, with it, whether TLS is terminated by the Gateway itself or by something in front of it. This page covers the operator-facing surface: the platform values that bound the choice, the tenant field that makes it, what each resulting certificate mode implies, what a move between classes does to certificates and Secrets, and how the mechanism fails when the configured names and the installed classes disagree. + +The chart-level reference for the per-tenant Gateway — inheritance, listener shapes, the security model, ACME rate limits — is [`packages/extra/gateway/README.md`](https://github.com/cozystack/cozystack/blob/main/packages/extra/gateway/README.md). + +## Platform values + +Three keys under `gateway` in the platform values bound which class a tenant Gateway can land on: + +```yaml +gateway: + enabled: true + className: cilium + tenantSelectableClasses: [] + edgeTerminatedClasses: [] +``` + +- `className` — the class every tenant Gateway uses unless the tenant names another one. Defaults to the Cilium class the platform ships. +- `tenantSelectableClasses` — the additional class names a tenant may name for itself. Empty (the default) means a tenant cannot name anything but the platform default. +- `edgeTerminatedClasses` — the class names whose provider terminates TLS upstream of the Gateway. Membership in this list is the only thing that puts a tenant into edge mode. + +All three reach the per-tenant gateway chart through the platform values channel, as `_cluster.gateway-class-name`, `_cluster.gateway-tenant-classes` and `_cluster.gateway-edge-classes`. The two lists are joined with commas on the way, so a name containing a comma is rejected at platform render time rather than silently split into two names. + +`gateway.enabled` is a separate switch and not a prerequisite for any of this: it moves the platform's own published services from Ingress onto Gateway API and points the ACME `ClusterIssuer`s' HTTP-01 solver at the publishing tenant's Gateway. A tenant Gateway is created from `tenant.spec.gateway` alone. + +## How a tenant picks a class + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Tenant +metadata: + name: acme + namespace: tenant-root +spec: + gateway: true + gatewayClass: edge-tunnel +``` + +`edge-tunnel` throughout this page stands for the name of a GatewayClass whose controller terminates TLS outside the cluster and forwards cleartext to the Gateway. Substitute the name your own edge-terminating controller registers. + +The field is read only for a tenant that owns a Gateway. A tenant that inherits — `gateway` left unset or set to `false` — publishes through the nearest ancestor that owns one and runs on that ancestor's class, so its own `gatewayClass` has no effect. Leaving `gatewayClass` empty selects `gateway.className`. + +The set a tenant may name is `gateway.tenantSelectableClasses` plus the current `gateway.className`, so a tenant may always name the default explicitly. Anything else fails that tenant's own gateway release at render time, with a message naming the class and the allowed set, and reaches no other tenant. The allowlist exists because `Tenant` is a tenant-writable object while a `GatewayClass` is cluster-scoped: without it a tenant could attach its Gateway to any class the cluster registers, including one an admin keeps for internal-only traffic. + +## What the class decides: the certificate mode + +`TenantGateway.spec.certMode` is derived, never written by the tenant. The gateway chart resolves it in this order, first match winning: + +| Condition | `certMode` | What the controller issues | +| --- | --- | --- | +| the resolved class is in `gateway.edgeTerminatedClasses` | `edge` | nothing | +| `publishing.certificates.wildcardSecretName` is set | `existingSecret` | nothing; the operator's Secret is referenced | +| `publishing.certificates.solver: dns01` | `dns01` | one wildcard `Certificate` per tenant | +| otherwise | `http01` | one `Certificate` per published hostname, except the passthrough names below | + +Under `http01` a hostname held by a TLS-passthrough listener is the exception to the per-hostname rule. The hostname earns neither an HTTPS-terminate listener nor a `Certificate`, whether or not a `TLSRoute` is attached to that listener yet, because the backend presents its own certificate and the Gateway never holds a key for the name. The shipped `tlsPassthroughServices` defaults put `api`, `vm-exportproxy` and `cdi-uploadproxy` in that position on the publishing tenant's Gateway, for the names `publishing.exposedServices` lists while `gateway.enabled` is on, and the `Certificate` objects that covered them are collected on upgrade. No other tenant renders those listeners, so a child tenant's `HTTPRoute` on one of those names keeps its listener. `packages/extra/gateway/README.md` carries what that changes and the ways forward. + +Edge therefore wins over an operator-supplied wildcard Secret, which wins over the solver. In edge mode the solver, DNS-01 provider and issuer-name inputs are not read at all, so an unsupported value in any of them does not fail the tenant's release. `existingSecret` skips the same validations; only `http01` and `dns01` reject a bad solver, provider or issuer name. + +## What edge mode changes on the Gateway + +- The Gateway serves port 80 only. Its listeners are `edge` for `*.`, `edge-apex` for ``, and one `edge-child-*` per inheriting child tenant carrying that child's `*.`. There is no port-443 listener, and no hostname-less port-80 listener either. +- No per-tenant ACME `Issuer` and no `Certificate` are created. +- No http-to-https redirect `HTTPRoute` is created. The edge is expected to perform that redirect. +- `tlsPassthroughServices` is not written onto the `TenantGateway` and no passthrough listener is rendered, so the tenant Kubernetes API, VM export and CDI upload endpoints are not published through such a Gateway. The packages that own those endpoints still create their `TLSRoute` objects, which then match no listener on that Gateway. +- Application `HTTPRoute`s attach by hostname exactly as in the other modes — the edge listeners carry the same namespace selector the HTTPS listeners carry elsewhere. + +That last point has a blast radius worth stating on its own, and it applies only where those endpoints are published through the Gateway at all. Each of them renders its `TLSRoute` only when `gateway.enabled` is on and its name is still in `publishing.exposedServices`; at the default `gateway.enabled: false` there is no such route and nothing to lose. Where both hold, setting `gateway.className` to an edge-terminated class puts the publishing tenant on it too, and then the endpoints that go unpublished are the platform's own: the `TLSRoute` for `api.` is pinned to a `tls-api` listener, an edge Gateway has no such listener, so the tenant Kubernetes API stops being served — VM export and CDI upload go the same way. Nothing fails at render time, and the loss surfaces only on the routes themselves, which the controller flips to `Accepted=False` with reason `NoMatchingParent`. Putting only child tenants on the edge class avoids it, because the platform's routes keep pointing at the publishing tenant's Gateway. + +Listing a class in `gateway.edgeTerminatedClasses` is an assertion about that class, and nothing verifies it. Put the bundled `cilium` class in the list and its Gateways serve every application hostname over plain HTTP on a public address, with no redirect and no certificate. Only list a class whose provider really does terminate TLS in front of the Gateway. + +## Moving a tenant between classes + +Onto an edge-terminated class, on the next reconcile: + +| Object | What happens | +| --- | --- | +| the per-tenant ACME `Issuer` | deleted | +| per-listener and wildcard `Certificate` objects | deleted | +| the TLS Secrets those `Certificate`s issued | left in the namespace, unreferenced | +| the http-to-https redirect `HTTPRoute` | deleted | +| the replica of `publishing.certificates.wildcardSecretName` | deleted from that tenant's namespace | +| route status conditions the controller wrote earlier | an attached `TLSRoute` that declares hostnames is flipped to `Accepted=False`, reason `NoMatchingParent`; on a hostname-less `TLSRoute`, and on every attached `HTTPRoute`, the controller's own entry is removed instead | + +Every deletion is guarded by ownership: an object of the same name that the controller did not create is left in place. + +The TLS Secrets survive because cert-manager ships here with `enableCertificateOwnerRef: false`, so a `Certificate` does not own the Secret it issued into. That is how every mode transition behaves and is not something edge introduces. Removing the leftovers is an operator action. + +The wildcard replica is withdrawn because the namespace stops counting as a TLS termination point. Three things keep it anyway: a namespace that also owns an ingress controller still terminates locally; a namespace holding more than one `TenantGateway` keeps the replica while any of them is not on an edge-terminated class; and a namespace whose `TenantGateway` objects cannot be listed, or which carries the Gateway owner label but has no `TenantGateway` yet, keeps it as well, so that a transient read failure never withdraws a key a Gateway may still be serving. + +Moving back off an edge-terminated class restores whatever the new mode calls for: the ACME `Issuer` and its `Certificate`s under `http01` or `dns01`, the redirect route under any non-edge mode, and the wildcard replica whenever `publishing.certificates.wildcardSecretName` is set. The `TLSRoute` conditions go back too, since the passthrough listeners do: `http01` judges those routes again from the hostname claims it collects, while `dns01` and `existingSecret` collect none and so drop the entry rather than leave a withdrawal standing on a listener that is serving. `existingSecret` mints no `Issuer` and no `Certificate` of its own, so a move from `edge` into it brings back the replica and the redirect route and nothing else. + +The two halves of the move are not ordered against each other. The wildcard-secret controller watches the `TenantGateway` and prunes the replica as soon as `certMode` changes, while the `Gateway` itself is re-rendered when the tenant's HelmRelease reconciles, and the class controller reprograms after that. A tenant leaving `existingSecret` can therefore be briefly serving HTTPS listeners whose Secret has already gone. + +## When the configured names and the installed classes disagree + +Nothing in this mechanism checks that a class name corresponds to a `GatewayClass` the cluster actually has. Run `kubectl get gatewayclass` before putting a name into any of the three values. + +- **A name in `className` or `tenantSelectableClasses` that matches no installed class.** The tenant's release renders and the `Gateway` object is created, but no controller claims it, so it gets no address and no listener status. Nothing fails at render time; the symptom is the `TenantGateway` holding `Ready=False` with reason `GatewayNotAccepted`. +- **A misspelled name in `edgeTerminatedClasses`.** It matches no class, so edge mode never engages for it and nothing says so. A tenant whose TLS really is terminated upstream stays in whichever mode the solver and wildcard-secret settings select, and its Gateway keeps terminating TLS itself. +- **A class removed from `tenantSelectableClasses` while a tenant still names it.** That tenant's gateway release starts failing to render. Other tenants are unaffected. +- **`className` changed while a tenant pins the outgoing name.** The allowed set is the *current* default plus `tenantSelectableClasses`, so the pin stops being valid and that tenant's gateway release fails. Add the outgoing class to `tenantSelectableClasses` before changing the default, or update the tenant first. +- **A class whose `status.supportedFeatures` lists features but not `TLSRoute`.** A tenant Gateway that renders TLS-passthrough listeners on it reads `Ready=False` with reason `GatewayClassUnsupported`, naming the class, because nothing would serve those listeners. A class that lists no features at all is not judged, since implementations predating the field leave it empty. +- **The publishing tenant selecting a class that terminates TLS differently from `className`.** Refused at render time, in both directions. The cluster-wide ACME `ClusterIssuer`s point their HTTP-01 solver at the publishing tenant's Gateway, and whether that solver pins `sectionName: http` is decided from `gateway.className` alone — a tenant's own choice does not travel on that channel. A publishing tenant that moved itself onto an edge class would keep a solver pinned to the `http` listener its Gateway no longer renders, and every HTTP-01 certificate in the cluster would stop issuing with nothing in any status naming the cause. Change `gateway.className` instead. The publishing tenant is the namespace named by `publishing.ingressName`, `tenant-root` by default. +- **A comma or a non-string entry in either list.** The platform chart fails to render, naming the entry. + +## Checking what a tenant landed on + +```bash +kubectl get gatewayclass + +kubectl --namespace tenant-acme get tenantgateway cozystack \ + --output jsonpath='{.spec.gatewayClassName}{"\t"}{.spec.certMode}{"\n"}' + +kubectl --namespace tenant-acme get tenantgateway cozystack \ + --output jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{"\t"}{.reason}{"\n"}{end}' + +kubectl --namespace tenant-acme get gateway cozystack \ + --output jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.port}{"\t"}{.hostname}{"\n"}{end}' + +kubectl --namespace tenant-acme get certificate,issuer +``` + +A tenant on an edge-terminated class reports `certMode: edge`, shows port-80 listeners only, and has no `Certificate` or `Issuer` that this controller created. Leftover TLS Secrets from an earlier mode are not shown by the last command — list Secrets of type `kubernetes.io/tls` in the namespace to find those. diff --git a/content/en/docs/next/operations/configuration/system-memory-limits.md b/content/en/docs/next/operations/configuration/system-memory-limits.md new file mode 100644 index 00000000..9a554153 --- /dev/null +++ b/content/en/docs/next/operations/configuration/system-memory-limits.md @@ -0,0 +1,184 @@ +--- +title: "System Component Memory Limits" +linkTitle: "System memory limits" +description: "Why Cozystack sets memory limits on system components, how to tune them, and how to withhold or acknowledge a default." +weight: 45 +--- + +Cozystack defaults container memory in eligible system namespaces and sets explicit requests and limits on the node DaemonSets. This page explains why that is not merely resource hygiene, how to tune it, and what it deliberately does not cover. + +## Why a memory limit, and not a PriorityClass + +Talos Linux v1.12 introduced a userspace OOM handler, enabled by default, that reacts to memory pressure before the kernel OOM killer does. Its only input from the machine is its own config document; everything else it reads straight out of cgroupfs, and it holds no Kubernetes client at all. `PriorityClass` — `system-node-critical` included — therefore has no bearing on which cgroup it picks. + +Setting a priority on a system component to stop these kills looks like a fix and changes nothing about them. Take that claim narrowly, because priority is far from inert elsewhere: it still drives scheduler preemption, kubelet eviction ordering, and the `oom_score_adj` kubelet hands the kernel OOM killer. None of those three is what kills these pods. The one ordering input this handler has is the Kubernetes **QoS class**, inferred from the cgroup path rather than from any pod object — a different axis from priority, so a `system-node-critical` pod with no memory limit is an ordinary candidate like any other. + +The handler scores each pod cgroup with a CEL expression. The default in Talos v1.14.2, the version Cozystack currently ships for host nodes, is: + +``` +memory_max.hasValue() ? 0.0 : + {Besteffort: 1.0, Burstable: 0.5, Guaranteed: 0.0, Podruntime: 0.0, System: 0.0}[class] * + double(memory_current.orValue(0u)) +``` + +Any cgroup scoring zero is dropped from the candidate set entirely. A pod whose containers all carry a memory limit has `memory.max` set on its pod cgroup, scores `0.0`, and can never be selected. A pod without one stays a candidate no matter how little memory it is using. + +Three consequences follow, and all three are easy to get wrong: + +- Only a **limit** grants immunity. A memory **request** merely moves the pod from BestEffort to Burstable, which under the default `strictCgroupClassOrdering: true` means "killed second" rather than "not killed" — Burstable cgroups are considered only once no BestEffort one is eligible, and the score then breaks ties within the class. That setting arrived in v1.13.4; on v1.13.0 through v1.13.3 the score alone decided, so a large Burstable pod could outrank a small BestEffort one. +- **Every** container in the pod needs a limit, init containers included. Kubelet only sets pod-level `memory.max` when all of them have one, so a single limit-free sidecar puts the whole pod back in the candidate set. +- CPU limits are irrelevant here. The ranking expression is given the cgroup's path, its QoS class, and `memory.max`, `memory.current` and `memory.peak` — no CPU information of any kind. + +Victim selection is also decoupled from the trigger. The cgroup that caused the pressure and the cgroup that gets `SIGKILL`ed are unrelated by design. In practice the pods carrying limits are overwhelmingly tenant workloads — managed applications are sized through resource presets, and a tenant namespace with `resourceQuotas` configured gets a default limit of its own — while system components carried none until the change this page describes. That inverts the intended order: whatever fires the trigger, `metallb-speaker` or `linstor-satellite`, using a few dozen megabytes and entirely uninvolved, is the one killed for it, repeatedly, until the pressure subsides. On Talos v1.13.x a tenant workload thrashing against its own multi-gigabyte limit is enough to fire it, through the node-wide PSI clause described [below](#when-the-trigger-itself-is-the-problem). + +## What Cozystack does + +**A default LimitRange in eligible system namespaces.** The `cozystack-operator` maintains a `LimitRange` named `cozystack-system-defaults` in each namespace it reconciles whose name does not begin with `tenant-`, defaulting container memory for anything that declares none. It covers containers whose upstream chart exposes no `resources` knob. Namespaces with a foreign memory policy or an unresolved safety hold are excluded. + +**Explicit requests and limits on node DaemonSets.** Charts additionally set real values on the DaemonSets that run on every node — the cilium agent and its init containers, metallb speaker, the frr-k8s controller and its sidecars, the linstor satellite along with plunger and drbd-logger, virt-handler, fluent-bit, node-exporter, multus and its init container, velero node-agent, and all three kubevirt-csi-node containers. Requests there are fitted to observed usage, which is a real signal for the scheduler; the LimitRange cannot supply that, because it applies one number to every container in the namespace and so has to keep its default request deliberately tiny. kube-ovn is absent from that list because its vendored chart already sets both. + +The operator's LimitRange stops at the `tenant-` prefix, so tenant namespaces never receive it. A tenant namespace gets a default of its own only from the tenant chart's `tenant-range-limits`, which is rendered only when `resourceQuotas` is configured on the tenant and is empty by default. A tenant workload left without a memory limit therefore stays an eviction candidate, which is the upstream design working as intended and is what restores the ordering described above. + +## Tuning + +Two installer values, both empty by default so the operator's own defaults apply: + +| Value | Operator flag | Default | +|---|---|---| +| `cozystackOperator.systemNamespaceMemoryLimit` | `--system-namespace-memory-limit` | `32Gi` | +| `cozystackOperator.systemNamespaceMemoryRequest` | `--system-namespace-memory-request` | `32Mi` | + +The limit is a ceiling, not a reservation, and the default is deliberately far above any real working set. The handler discards a cgroup for carrying a `memory.max` at all, whatever its value, so what the default buys is immunity rather than a fitted ceiling. That is also why it is set so high: the scan described below sees *declared requests*, never a container's actual resident set, so a ceiling low enough to bind is a ceiling that can convert a rare pressure-driven kill into a deterministic one for any component whose real usage nobody measured. Raising it is close to free; lowering it is where the risk lives, and it is worth doing per component in that component's own chart before doing it fleet-wide here. + +**The limit must stay above the largest memory request in any system namespace.** A defaulted limit below a container's own request is rejected at admission, and the pod simply will not start. List the requests before lowering it — the operator marks exactly the namespaces it treats as system with `cozystack.io/system=true`, the same condition under which it creates the LimitRange: + +```bash +for ns in $(kubectl get ns -l cozystack.io/system=true -o jsonpath='{.items[*].metadata.name}'); do + kubectl get pods -n "$ns" -o json | jq -r --arg ns "$ns" ' + .items[] + | [.spec.containers[], (.spec.initContainers // [])[]][] + | select(.resources.requests.memory != null) + | "\($ns)\t\(.name)\t\(.resources.requests.memory)"' +done | sort -u +``` + +**Keep the request small.** The operator always pairs the default limit with a default request, because a `LimitRange` that sets `default` without `defaultRequest` makes each container's request equal its limit and reserves the whole ceiling at schedule time. Note that leaving `systemNamespaceMemoryRequest` empty does not produce that state: an empty installer value omits the flag, which selects the operator's own `32Mi`. The knob is there to be raised or lowered, and clearing it is not a way to switch the request off. + +The operator also refuses to start when the request exceeds the limit. A `LimitRange` whose `defaultRequest` is above its `default` is rejected by the API server, which would wedge namespace reconciliation for every system package, so the check happens at startup rather than at apply time. + +Setting the limit to `0` disables the feature, so the knob is reversible. Leaving it empty is not the same thing: that selects the operator's `32Gi` default. + +On the next Package reconcile after the feature is disabled, the operator lists LimitRanges cluster-wide by `app.kubernetes.io/managed-by: cozystack-package-controller` and removes every object it owns, including leftovers in namespaces no active Package targets. A same-named LimitRange without that ownership label is left alone. The same ownership boundary applies while the feature is enabled: a foreign object already holding the `cozystack-system-defaults` name is neither overwritten nor adopted, even if it contains only CPU or storage policy, so that namespace receives no Cozystack memory default until the name is free. + +## Withholding and acknowledging a default + +**An unsafe default stays withheld until a cluster administrator acknowledges it.** Before applying, the operator scans Deployments, StatefulSets, DaemonSets, CronJobs, Jobs and Pods, including init containers, for requests above the configured limit without an explicit limit. A visible blocker withdraws the managed LimitRange and records a hold on the Namespace. Removing a Pod or template does not release the hold: the same empty snapshot can occur between deletion and recreation by another controller. + +Lowering a previously applied default also creates a hold, even if no blocker is visible. LimitRanger persists its default on admitted Pods, so a stored limit cannot prove that the workload's source declares one. The operator remembers the previous limit independently of the LimitRange and does not infer permission to tighten from a Pod's `limits` field or `kubernetes.io/limit-ranger` annotation. + +The cluster-scoped Namespace carries `operator.cozystack.io/system-memory-state`, a JSON object with `version`, `lastLimit`, and, when held, `hold`, `target` and `reason`. This is operator-managed safety state. It survives controller restart, removal of the LimitRange, removal of the Package target, and disabling the feature. Do not remove or edit it to clear a hold: doing so discards the evidence protecting future Pod admission. Invalid state blocks new defaults until a cluster administrator repairs it. + +Inspect the hold and the required acknowledgement: + +```bash +kubectl get namespace -o json | jq -r ' + .metadata.annotations["operator.cozystack.io/system-memory-state"] + | fromjson | {reason, target, acknowledgement: (.hold + ":" + .target)}' +``` + +Before acknowledging, inspect the source of every affected workload, including custom resources and static-pod manifests that this scanner cannot read. Give the affected container an explicit limit at least as large as its request, lower its request only when appropriate, or raise the configured default. A currently running Pod's defaulted limit is not that source. Also check actual memory usage: admission safety does not make a small runtime ceiling safe. + +Copy the exact `acknowledgement` value reported for the current target into a separate annotation: + +```bash +kubectl annotate namespace \ + operator.cozystack.io/system-memory-ack=':' --overwrite +``` + +This requires permission to patch the cluster-scoped Namespace; a tenant's namespaced RoleBinding does not grant it. The operator grants no additional permission for acknowledgement. A token authorizes only its hold generation and canonical target quantity. Changing the configured target rotates the hold, even when returning to an earlier value. The operator consumes a matching token once; if a concrete request-above-default blocker still exists, the token is rejected and removed, and the hold stays. Acknowledgement never overrides a foreign LimitRange. + +After successful acknowledgement and a clean scan, the configured default is applied. Existing Pods keep their resources; the new default reaches later admissions. Package reconciles run after ordinary events and are additionally scheduled every five minutes, so acknowledgement and workload changes also progress on a quiet cluster, subject to controller queue and API availability. The reason and required token are visible on the Namespace and in operator logs: + +```bash +kubectl -n cozy-system logs deploy/cozystack-operator | grep "withholding the default container memory limit" +``` + +A hold remains after disable/re-enable, and changing the configured value while held requires acknowledging the new token. The operator also removes its LimitRange when no active Package targets the namespace, considering all Packages sharing it; the Namespace safety state remains for a later reinstall. Setting the limit to zero removes managed defaults even if their safety state is invalid. + +**A namespace another LimitRange already governs is left alone.** If a system namespace — `kube-system` included — already carries a LimitRange that says anything about memory, the operator writes nothing there and withdraws anything it wrote earlier. Any memory-bearing field counts, at either the `Container` or the `Pod` scope: a second `default` leaves the effective ceiling to the order `LimitRanger` happens to iterate them, a `max` below this default or a `min` above the default request rejects the pod outright, and a `maxLimitRequestRatio` rejects it too, since a 32Gi default against a 32Mi default request is a ratio of 1024:1. The `Pod` scope is included because the collision there is not decidable in advance at all — whether a per-container default breaches a pod-wide bound depends on how many containers a not-yet-created pod will have. LimitRanges that bound only cpu or storage say nothing about memory and coexist normally. Note that withdrawing hands the namespace to the administrator's policy in full: if that LimitRange defaults container memory below some container's own request, the resulting admission rejection is theirs to resolve, and it is one that would have occurred without Cozystack in the picture at all. + +A LimitRange only mutates at admission. Existing pods keep running without limits until they restart, so the protection lands progressively as workloads roll rather than the moment the setting is applied. + +## Verifying + +Confirm a pod cgroup actually carries `memory.max` — this is the property that matters, not the QoS class. Talos runs kubelet with the `cgroupfs` driver on a unified cgroup v2 hierarchy, so a pod cgroup is a directory named for the pod UID, dashes and all: + +```bash +POD_UID=$(kubectl get pod -n -o jsonpath='{.metadata.uid}') +NODE=$(kubectl get pod -n -o jsonpath='{.spec.nodeName}') +NODE_IP=$(kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') +talosctl -n "$NODE_IP" read "/sys/fs/cgroup/kubepods/burstable/pod$POD_UID/memory.max" +``` + +A byte count rather than the literal `max` means the pod is out of the victim set. Note that `talosctl -n` takes a machine address rather than a Kubernetes node name, which is why the address is looked up instead of being reused from `nodeName`. + +Mind the QoS segment of that path, and do not extrapolate it. BestEffort and Burstable pods live under `kubepods/besteffort/` and `kubepods/burstable/`, but there is no `guaranteed` directory: kubelet creates QoS-level cgroups for those two classes only, so a Guaranteed pod sits directly at `kubepods/pod`. To avoid guessing, list the pod directories two levels down instead: + +```bash +talosctl -n "$NODE_IP" ls -d 2 -t d /sys/fs/cgroup/kubepods +``` + +`talosctl read` wants an `os:admin` talosconfig and a single node per call. Reading the same file through `kubectl debug node/$NODE --profile=sysadmin` under `/host/sys/fs/cgroup` also works, but it depends on the kubelet, on scheduling, and on a namespace that admits a privileged pod — all shakier than the Talos API in precisely the situation that makes you run this check. + +Inspect what the handler has actually killed: + +```bash +talosctl -n get oomactions -o yaml +``` + +Each entry carries the score the victim was selected on, the command lines of the processes in it, and a dump of the trigger context it fired under. Ask for `-o yaml`: the default table shows only a score, and its `Time` column reads a field the spec never sets, so it renders empty. This is a ring buffer of the last 50 actions held in memory, so it does not survive a `machined` restart and is no substitute for an alert — but it is still the better of the two signals here, being structured and complete where the logs are neither. + +Once every eligible pod in a namespace carries a limit, the healthy signature is the controller still triggering under pressure and finding nothing it is allowed to kill: + +```bash +talosctl -n logs controller-runtime | grep OOMController +``` + +Both halves of that signature — `OOM controller triggered`, then `no eligible cgroup to kill` — are emitted at the default log level, so neither needs a debug flag. Note that the structured fields are encoded as JSON rather than logfmt, so the second line reads `no eligible cgroup to kill {"component": "controller-runtime", "controller": "runtime.OOMController", "ranked": 0}`. Grep for `"ranked"` or for the message text; `ranked=` matches nothing and reads as though the handler were not running. The trigger firing is not itself a fault. + +`talosctl dmesg` carries the same lines and is the more common reflex, but it is the weaker source: that path strips log levels and timestamps, truncates at 976 bytes, and suppresses the first four occurrences of each ranking error — precisely the lines worth having if scoring ever misbehaves. + +## What this does not cover + +**Kubernetes static pods.** `kube-apiserver`, `kube-controller-manager` and `kube-scheduler` are managed by Talos rather than by any chart. A `cozystack-system-defaults` LimitRange does reach `kube-system`, since `cozystack-scheduler` installs there, but it cannot touch them: a LimitRange defaults resources at API-server admission, and kubelet builds static pods straight from files on disk without ever passing through it. Sizing those is a Talos machine-config matter. Their mirror pods pass admission. An already persisted mirror Pod with an oversized request and no limit creates a hold, and lowering the previous default requires acknowledgement. A mirror Pod rejected before persistence provides no scan evidence; its absence does not mean the static manifest is safe. + +**Future Pods outside the scanned templates.** A custom controller can change its desired request without producing a Deployment, StatefulSet, DaemonSet, CronJob or Job. If its first replacement Pod is rejected at admission, there is no new Pod for this scanner to inspect. An ownerless ReplicaSet at zero replicas has the same gap. A `RequestsOnly` VPA can likewise introduce an unseen request after defaulting. The hold prevents automatic lowering and automatic recovery after an observed blocker, but does not validate arbitrary future custom-resource changes. Configure explicit resources for those workloads and inspect their admission errors. + +**A handful of vendored containers with no upstream `resources` knob** — the four frr-k8s `cp-*` init containers, the kube-ovn `hostpath-init` and `install-cni` init containers, and `cozy-proxy`, whose chart exposes no resources value at all. These are covered by the namespace LimitRange and nothing else. That is enough for OOM immunity, since the defaulted limit is what puts `memory.max` on the cgroup, but their request is then the generic default rather than a figure fitted to measured usage, so the scheduler gets a weaker signal for them than for the DaemonSets above. Lifting that needs changes upstream. + +**Optional components** `hami` and `kilo` carry no chart-level values, on the grounds that inventing numbers for components with no usage measurements behind them is worse than the blanket default. + +## When the trigger itself is the problem + +Giving system components limits stops them being *victims*. It does not stop the handler *triggering*, and a node under genuine sustained pressure will keep firing. The v1.14.2 default trigger is: + +``` +(multiply_qos_vectors(d_qos_memory_full_total, {System: 8.0, Podruntime: 4.0}) > 3000.0 && + multiply_qos_vectors(qos_memory_full_avg10, {System: 1.0, Podruntime: 1.0}) > 5.0 && + time_since_trigger > duration("5s")) +``` + +It fires only on memory stalls in the `System` and `Podruntime` cgroups, so pressure confined to pods does not trip it. It reached that shape over four changes: [siderolabs/talos#12602](https://github.com/siderolabs/talos/pull/12602) replaced the original global-PSI trigger with the per-QoS form in v1.13.0, [#12632](https://github.com/siderolabs/talos/pull/12632) added the `qos_memory_full_avg10 > 5.0` conjunct to make it less sensitive, [#13725](https://github.com/siderolabs/talos/pull/13725) added the `5s` cooldown in v1.13.6, and [#13895](https://github.com/siderolabs/talos/pull/13895) dropped a second clause, `memory_full_avg10 > 75.0 && time_since_trigger > duration("10s")`, in v1.14.0. That clause was a cause-blind global-PSI backstop: a single workload thrashing inside its own cgroup limit could drive root `memory_full` above 75 while the node had free RAM. Any node still running a v1.13 release carries it, tenant Kubernetes workers included. A tenant worker's machine config is rendered by the `kubernetes-nodes` chart, which accepts no extra documents, so the workaround below is for host nodes only. + +On a v1.13 host node the backstop can be removed without upgrading, through an `OOMConfig` machine-config document that sets the v1.14 trigger — at the cost of the last-resort guard before the kernel OOM killer takes over. The document also accepts `cgroupRankingExpression`, `strictCgroupClassOrdering` and `sampleInterval`; anything left out keeps its default, so overriding the trigger alone is enough here: + +```yaml +apiVersion: v1alpha1 +kind: OOMConfig +triggerExpression: |- + (multiply_qos_vectors(d_qos_memory_full_total, {System: 8.0, Podruntime: 4.0}) > 3000.0 && + multiply_qos_vectors(qos_memory_full_avg10, {System: 1.0, Podruntime: 1.0}) > 5.0 && + time_since_trigger > duration("5s")) +``` + +Prefer fixing the workload that is generating the pressure. Persistent triggering on v1.14 means the system and runtime cgroups keep stalling for memory, and capping the pod that is growing into the node's memory addresses the cause rather than the symptom. On v1.13 it can also mean a pod sitting at its own memory ceiling and reclaiming constantly, which raising that pod's limit addresses. diff --git a/content/en/docs/next/operations/services/backup-classes.md b/content/en/docs/next/operations/services/backup-classes.md index 01b73c93..8c9e1f8d 100644 --- a/content/en/docs/next/operations/services/backup-classes.md +++ b/content/en/docs/next/operations/services/backup-classes.md @@ -26,9 +26,12 @@ Tenants reference `cozy-default` from `BackupJob`, `Plan`, and `RestoreJob` reso | `apps.cozystack.io/Etcd` | etcd-operator snapshot | `strategy.backups.cozystack.io/Etcd` `cozy-default-etcd` | | `apps.cozystack.io/RabbitMQ` | RabbitMQ definitions (management API) | `strategy.backups.cozystack.io/Rabbitmq` `cozy-default-rabbitmq` | | `apps.cozystack.io/Redis` | RDB dump Job (sentinel-discovered master) | `strategy.backups.cozystack.io/Redis` `cozy-default-redis` | +| `apps.cozystack.io/Valkey` | RDB dump Job (sentinel-discovered master) | `strategy.backups.cozystack.io/Redis` `cozy-default-redis` | | `apps.cozystack.io/VMInstance` | Velero + kubevirt-velero-plugin | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vminstance` | | `apps.cozystack.io/VMDisk` | Velero | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vmdisk` | +Valkey runs on the same RedisFailover operator as Redis, so `cozy-default-redis` serves both: the driver hands the template `.Release.Prefix` (`redis-` or `valkey-`, after the app's Kind), from which it builds the Sentinel Service and the auth and CA Secret names. A Valkey backup restores only into a Valkey app, a Redis backup only into a Redis app. The Valkey chart has no TLS option, so a Valkey dump always connects in plaintext. + ### Shipped but NOT bound (admin opt-in required) | Application Kind | Driver | Strategy CR | @@ -214,6 +217,8 @@ The same trap sits one release earlier, on the Secret everything else depends on Convergence is driven instead by the controller's default-objects gate (`backupStorage.reconcileDefaultObjects`, on by default), which repairs both, in order. While the credentials Secret is absent or carries no bucket name, the gate stamps `reconcile.fluxcd.io/forceAt` + `requestedAt` on the **bucket's** `bucket--system` HelmRelease. Once the bucket name resolves it checks that every object `cozy-default` routes to exists and stamps the same pair on the **`backupstrategy-controller`** HelmRelease, forcing the real Helm upgrade that re-runs the lookups. The two are throttled independently (one force per 5 minutes each), so the second is not delayed by the first. Expect the objects within a minute or two of the bucket becoming ready. +Each default `Strategy` CR also waits for its own CRD. The chart ships the `strategy.backups.cozystack.io` CRDs as templates in the same release, and Helm resolves every object's kind before it applies any, so a CR rendered next to a CRD the apiserver does not serve yet would fail the whole release, CRDs included. Each strategy template therefore renders only once its CRD is `Established`. That holds the CRs back on the first render of every install, including an external-S3 one where the bucket name is known up front, and on the upgrade that first ships a new strategy kind: that revision creates the CRDs, and the gate's forced upgrade above renders the CRs. + The gate does not force a suspended HelmRelease — helm-controller ignores both annotations while `spec.suspend` is true, so it logs the skip and leaves the force counter alone. A release left suspended (`cozyhr suspend`) is therefore never repaired until it is resumed. Watch it with: @@ -360,7 +365,7 @@ If a deployment runs against an external S3 (no SeaweedFS), set `backupStorage.p > > **Two platform-level changes ride along with this migration (they are required for the plugin to run):** > -> - **CloudNativePG operator upgraded** (`postgres-operator` chart). A minor operator upgrade for the whole fleet; the CNPG CRDs move with it. This also fixes a latent bug where the operator image was newer than its bundled CRDs (`status.instanceID.sessionID` pruned), which made the operator believe the instance manager restarted and fail **every** backup cluster-wide with `instance manager was restarted during backup`. The vendored chart ships the CNPG CRDs as ordinary release manifests (gated by `crds.create`), so a normal `helm upgrade` updates them together with the operator — the `helm.sh/resource-policy: keep` annotation only prevents their deletion, not updates. The historical skew came from the `image.tag` pin outrunning the vendored chart's CRDs, which this PR removes; no out-of-band CRD apply is needed unless the HelmRelease itself is held back. +> - **CloudNativePG operator upgraded** (`postgres-operator` chart). A minor operator upgrade for the whole fleet; the CNPG CRDs move with it. This also fixes a latent bug where the operator image was newer than its bundled CRDs (`status.instanceID.sessionID` pruned), which made the operator believe the instance manager restarted and fail **every** backup cluster-wide with `instance manager was restarted during backup`. The vendored chart ships the CNPG CRDs as ordinary release manifests (gated by `crds.create`), so a normal `helm upgrade` updates them together with the operator — the `helm.sh/resource-policy: keep` annotation only prevents their deletion, not updates. The historical skew came from the `image.tag` pin outrunning the vendored chart's CRDs; that pin has since been removed, so no out-of-band CRD apply is needed unless the HelmRelease itself is held back. > - **LINSTOR scheduler admission webhook must not strip native-sidecar fields.** Older `linstor-scheduler` extender builds (≤ v0.3.5's predecessors) decoded every pod through a stale vendored Kubernetes API and re-encoded it, stripping `initContainers[].restartPolicy` cluster-wide — which breaks the plugin's restartable barman sidecar ("startupProbe forbidden without restartPolicy=Always"). The platform already ships the fixed webhook (`linstor-scheduler` re-vendored to upstream chart 0.3.1 / extender v0.3.6, which stopped stripping unknown pod fields), so no action is needed on an up-to-date platform; just do not hold the `linstor-scheduler` package back on an older version when rolling out the plugin. > > The same `useSystemBucket` opt-in applies to ClickHouse — see [ClickHouse: opt-in to the system bucket](#clickhouse-opt-in-to-the-system-bucket). When `useSystemBucket: true` is set on ClickHouse, the legacy `-backup` CronJob, credential Secret, and backup script are no longer rendered (they are mutually exclusive with the platform flow); migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default`. diff --git a/content/en/docs/next/operations/services/monitoring/distributed-tracing.md b/content/en/docs/next/operations/services/monitoring/distributed-tracing.md new file mode 100644 index 00000000..8980043a --- /dev/null +++ b/content/en/docs/next/operations/services/monitoring/distributed-tracing.md @@ -0,0 +1,128 @@ +--- +title: "Distributed tracing" +linkTitle: "Distributed tracing" +description: "Send traces from your workloads, what each managed engine can emit, and how tenants share one traces store." +weight: 37 +--- + +Distributed tracing is the third observability signal in Cozystack, alongside metrics (VictoriaMetrics) and logs (VictoriaLogs). This guide explains what the platform provides, how to send traces from your workloads, what each managed engine can and cannot emit on its own, and how tenants can share one traces store instead of running their own. The design is [cozystack/community `design-proposals/distributed-tracing`](https://github.com/cozystack/community/tree/main/design-proposals/distributed-tracing). + +## What the platform provides + +A tenant's Monitoring application stands up two pieces once tracing is configured: + +- A **VictoriaTraces backend**, one per `tracingStorages` entry (a `VTCluster` in `cluster` mode, a `VTSingle` in `single` mode), with a Grafana Jaeger datasource named `vtraces-` for viewing its spans. +- A **per-tenant OpenTelemetry Collector**, the app-facing OTLP ingest gateway, reachable in the tenant namespace at: + - **`otel-traces:4317`**: OTLP over gRPC + - **`otel-traces:4318`**: OTLP over HTTP + +The collector applies head sampling (10% by default) and optional attribute redaction, then forwards spans to the backend. It lives in your tenant namespace, so traffic from your workloads to it stays inside the namespace. With the default per-tenant backend the spans stay there too; with [shared-central tracing](#shared-central-tracing) the collector forwards them to a store in tenant-root. + +What the platform does **not** do is emit spans for you: producing spans is the workload's job. The platform delivers the transport, storage and tenancy; the *depth* of what shows up is a property of how each application or engine is instrumented. + +## Sending traces from your application (client-side) + +This is the primary path and it works for any workload with OpenTelemetry instrumentation (an OTel SDK in your code, or an auto-instrumentation agent). You do **not** hard-code the endpoint: point your workload at the in-namespace collector through the standard OpenTelemetry environment variables: + +```yaml +env: + # Send to the per-tenant collector over OTLP/HTTP (use :4317 for gRPC). + - name: OTEL_EXPORTER_OTLP_ENDPOINT + value: "http://otel-traces:4318" + - name: OTEL_EXPORTER_OTLP_PROTOCOL + value: "http/protobuf" + # Name this service and the environment it runs in. + - name: OTEL_SERVICE_NAME + value: "my-app" + - name: OTEL_RESOURCE_ATTRIBUTES + value: "deployment.environment.name=production" +``` + +The variables are defined by the OpenTelemetry specification, and the official SDKs and agents read them, but support for each variable differs per language: check your SDK against the [specification compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md). For a JVM workload you can add the OpenTelemetry Java agent (`-javaagent:/otel/opentelemetry-javaagent.jar`), and the same variables drive it with no code changes. To use gRPC instead, set the endpoint to `http://otel-traces:4317` **and** `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`; with [shared-central tracing](#shared-central-tracing), stay on HTTP, since spans refused over the tenant's write rate are retried only over HTTP, and keep each export request below 3 MiB. + +You do not need to tag your tenant: the backend a tenant writes to is its own, and in shared-central mode the platform attributes every span to its tenant itself. `service.namespace` is for grouping your own services (for example by team), not for the tenant. + +Once spans arrive, they are visible in Grafana under **Explore**, in the traces datasource. + +> **Correlation is not wired yet.** Cross-signal pivots, span to logs by `trace_id` and span to RED metrics, are part of the design but not configured: the traces datasource carries no links to logs or metrics, and no `spanmetrics` connector produces span metrics. Traces are viewable on their own until that follow-up work lands. + +## What each managed engine can emit + +Tracing depth differs per engine, because a sidecar cannot see *inside* a server process. Most managed servers do **not** emit OTLP spans themselves; their tracing is client-side (instrument the application that talks to them). + +| Engine | Server-side OTLP spans? | How you get traces | +| --- | --- | --- | +| **ClickHouse** | Partial: native span *table*, no push, disabled | The server can record internal query spans into `system.opentelemetry_span_log`, which the Cozystack chart removes (see below). Client-side propagation gives end-to-end traces. | +| **PostgreSQL** | No (extension-gated) | Statement-level spans need the `pg_tracing` extension, which is not in the `sharedPreloadLibraries` allowlist, and `shared_preload_libraries` cannot be set through `parameters`. Use client-side instrumentation. | +| **Kafka** | No (brokers) | Brokers do not emit request spans; instrument producers/consumers (client-side). JVM clients can use the OTel Java agent. | +| **RabbitMQ** | No (non-OTLP) | The `rabbitmq_tracing` plugin emits RabbitMQ-format events, not OTLP. Instrument publishers/consumers (client-side). | +| **NATS** | No | Server message-tracing emits NATS-format events, not OTLP. Instrument clients (context is propagated in message headers). | +| **MariaDB** | No | No server-side OTLP. Instrument the application (client-side). | +| **Redis** | No | No server-side OTLP. Instrument the application (client-side). | + +The takeaway: for every engine here, the way to get a useful request trace is to **instrument the application** that issues the queries or messages and point it at `otel-traces` as shown above. The engine then appears as a span in your application's trace when context is propagated. + +## ClickHouse: internal query spans + +ClickHouse is the one engine that produces spans natively: it can record query-execution spans into the `system.opentelemetry_span_log` system table and take part in a trace whose context a client propagates through the W3C `traceparent` header. The Cozystack ClickHouse chart removes that log in its server configuration, alongside the other system logs it disables, and the ClickHouse application exposes no setting to turn it back on, so these spans are not available today. + +Even with the log enabled, ClickHouse has **no native OTLP push**: getting those spans into a collector, and so into Grafana, needs a shipper that reads `system.opentelemetry_span_log` and forwards OTLP, which neither Vector nor Fluent Bit does. Both are follow-up work; end-to-end request traces come from client-side instrumentation, as for every other engine. + +## Sampling, redaction, and tenancy + +- **Sampling:** the collector head-samples traces at 10% by default (`tracingCollector.samplingPercentage`). The decision is made per trace ID, so a kept trace keeps all of its spans and several collector replicas agree on it. A busy application keeps about one trace in ten, so the backend is not overwhelmed. +- **Redaction (PII):** span attributes can carry sensitive data (SQL text with literals, parameters, connection strings). List such keys in `tracingCollector.redactAttributes` to drop them before export. +- **Tenancy:** with the default per-tenant backend, spans never leave the tenant namespace, and only the tenant's own Grafana reads them. With shared-central tracing they are stored in tenant-root; see below for how tenants are kept apart there. + +## Shared-central tracing + +Instead of a VictoriaTraces backend per tenant, tenants can send their spans to one store in tenant-root. Applications do not change: they keep sending to `otel-traces` in their own namespace, and the collector forwards to the shared store. + +### Hosting the store (tenant-root) + +tenant-root opts in with `tracingCentralHost: true` on its Monitoring application, next to a `tracingStorages` entry named `generic` in `cluster` mode; that store is then shared with every tenant that opts in. A vmauth in tenant-root fronts it, and every central tenant's collector and Grafana go through that vmauth. + +All central tenants share the store's disk and retention with tenant-root, and nothing reserves space per tenant. Every traces store is bounded by disk usage: without `retentionDiskUsageBytes` it drops its oldest day once its volume is 80% full. VictoriaTraces always keeps the last two days, though, so the disk cap alone does not stop a fast writer from filling the volume. + +There is no per-tenant allow-list: with hosting on, every tenant that sets `tracingCentral` is admitted, nested tenants included. A tenant-root that needs to limit who shares its store keeps `tracingCentralHost` off. + +### The per-tenant write rate + +Because the disk cap alone does not bound a fast writer, each central tenant is also held to a write rate, which the platform sets with `monitoring.tracingCentralTenantBytesPerSecond` in the platform package, as a whole number of bytes per second up to 1099511627776 (1 TiB/s); any other value fails the platform render. By default it is about 97 KiB/s, at which one tenant writes at most the disk cap of a default 10Gi store over the two days, so raise it with the store. The rate is counted uncompressed and before the collector samples, so the store receives only the sampled share of it and keeps that compressed. A change reaches every tenant's collector through the `cozystack-values` Secret, and upgrades every release in every tenant once. + +The collector Cozystack ships is built with a rate-limit processor, and each central tenant's collector enforces the rate, split across its replicas: a sender whose connection stays on one replica gets that replica's share. Data above the rate is refused. An OTLP/HTTP exporter gets a 429 and retries; an OTLP/gRPC exporter gets `RESOURCE_EXHAUSTED` without the `RetryInfo` the OTLP specification requires for a retry, so it drops those spans. A request larger than 3 MiB, decompressed, is refused with a 400, which exporters do not retry. Applications of central tenants should therefore export over HTTP, in batches below 3 MiB. + +The limiter keeps its state in each collector pod's memory, and every pod starts with a full allowance, so a pod start lets through above the rate up to 5 MiB, or one second of the pod's share of the rate when that is larger. Shared-central tracing therefore allows at most 10 collector replicas, and the tenant's Monitoring render fails above that. The rate bounds each tenant, not their sum: several tenants writing at their limits together can still exceed the cap, so size the volume for the tenants you admit. + +### Opting a tenant in + +Set `tracingCentral: true` on the Tenant, which needs `monitoring: true` on the same tenant. The tenant chart then renders the egress rules the collector and Grafana need to reach the vmauth, and switches the tenant's Monitoring to the shared store. If tenant-root does not host the store, the tenant's Monitoring render fails with a message naming the prerequisite, rather than dropping spans. + +In the tenant's Grafana the shared store appears as the `traces-central` datasource. The tenant's own local stores, if it lists any, keep their datasources, so traces stored before the switch stay readable until the tenant removes those entries. + +### Isolation and the trust boundary + +The store itself authorizes nothing; it takes the tenant from request headers. Each central tenant gets two VMUsers that pin one account, derived from its namespace name and UID: the collector's, which may only write, and its Grafana's, which may only read, each with its own Secret (`traces-central-credentials` and `traces-central-reader-credentials`). vmauth replaces any account a client sends. Tenant pods can reach only the vmauth proxy port, and only from pods carrying the collector's or Grafana's label, with the tenant's credentials. So tenants sharing the store cannot read or write each other's traces. + +Within its own account, a tenant writes only through its collector, so the write rate holds. Grafana's login cannot write, so a write sent through the Grafana datasource proxy, which vmauth would otherwise route to the store, finds no route. The collector and Grafana images are the platform's and cannot be set from the Monitoring application, and of a Secret named in `oidc.customConfig.secretRef` only its `auth.ini` key is mounted into Grafana, so the collector's password does not reach a pod the tenant can shape. + +tenant-root is inside the store's trust boundary. Its workloads reach the store directly, and one that learns a tenant's account can read and write it. Whoever tenant-root admits to its Grafana can read every account through tenant-root's own datasource for the store, which does not go through vmauth. Who that is follows tenant-root Grafana's OIDC mode: with `System`, tenant-root's own groups, which already hold the same access level in every tenant under it; with `CustomConfig` or `None`, whoever that configuration admits. + +### Lifecycle + +- **The account follows the namespace.** It is derived from the namespace name and UID, so a re-created namespace, a restored one included, starts a new account and does not see the spans stored before. +- **Turning central on** keeps the tenant's local stores and their datasources. Right after it, spans the collector sends before the vmauth has loaded the tenant's login get a 401, which the exporter does not retry, so those spans are dropped. The same happens after the tenant's `traces-central-credentials` Secret is re-created with a new password; after `traces-central-reader-credentials` is, the tenant's Grafana gets a 401 on reads until the vmauth loads the new login. +- **Turning central off** removes the tenant's login to the shared store. The spans it wrote there stay until the store's retention drops them and cannot be read from the tenant while it is off; turning it on again on the same namespace makes them readable again. Spans sent between turning it off and the collector switching back to a local store may be dropped, since the egress rule to the shared store goes first. +- **Deleting a tenant** leaves its spans in the shared store until retention drops them. + +### Stopping hosting + +Turn `tracingCentral` off on every central tenant and let its Monitoring reconcile before tenant-root stops hosting. tenant-root's render refuses to stop hosting while a central tenant's VMUsers remain and lists the tenants still holding them. Removing tenant-root's Monitoring altogether is not a render and is not refused; until the central tenants turn central off, every Monitoring render of theirs fails, their metrics, logs and alerting changes included. + +The two refusals read each other's objects when they render, so a tenant turning central on while tenant-root stops hosting can pass both. That tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning the tenant's `tracingCentral` off instead stops the silent drop, and turning it on again is then refused until tenant-root hosts. + +## Prerequisites + +- The tenant's Monitoring application with a `tracingStorages` entry, or shared-central tracing turned on for the tenant, and `tracingCollector.enabled` left on (the default). +- For shared-central tracing: tenant-root hosting the store, as described above, and at most 10 collector replicas (`tracingCollector.replicas`). +- A workload with OpenTelemetry instrumentation (SDK or auto-instrumentation agent). diff --git a/content/en/docs/next/operations/services/object-storage/capacity-for-backups.md b/content/en/docs/next/operations/services/object-storage/capacity-for-backups.md new file mode 100644 index 00000000..2f42e46f --- /dev/null +++ b/content/en/docs/next/operations/services/object-storage/capacity-for-backups.md @@ -0,0 +1,42 @@ +--- +title: "SeaweedFS capacity for backups" +linkTitle: "Capacity for backups" +description: "Size a SeaweedFS instance for continuous WAL archiving: what a bucket needs, the alerts that watch the instance, and how to grow it." +weight: 30 +--- + +A SeaweedFS instance starts with two volume servers of `volume.size: 10Gi` each and `replicationFactor: 2`, so every byte it stores lands on both servers and the instance holds a little under 10 GiB of objects. That is enough for occasional backups and restores. It is not enough for continuous WAL archiving from a Postgres cluster that writes steadily: WAL is archived around the clock and kept for the whole retention window, so the bucket grows until the window is full, and an instance sized by default runs out long before then. + +A volume server stops taking writes in one of two ways. Once its free disk falls below `minFreeSpacePercent` (5%), which it checks every minute, every volume on it turns read-only and it takes no new volume, so every bucket with a copy there stops at once. Before that, it can run out of slots: the volumes that already exist keep taking writes until each reaches `master.volumeSizeLimitMB`, a bucket whose volumes are all full needs a new one, and a server with no room left for another volume of the size limit cannot host it. When no server can, the master logs `Not enough data nodes found` and every upload into that bucket fails. The slots can run out while the disk is still half empty: a volume server counts every volume it holds at the full size limit, so the nearly empty volumes of other buckets keep one bucket from growing. `barman-cloud-wal-archive` then fails on every segment, WAL segments stay on the Postgres volume, and the database goes down when that volume is full. + +## Estimating what a bucket needs + +For each Postgres cluster that archives into the instance, the bucket holds the WAL of the whole retention window plus every base backup inside it: + +```text +bucket ≈ WAL per day × retention days + database size × (base backups in the window + 1) +``` + +The WAL rate of a running cluster, in bytes per day before compression, is roughly the number of segments it archived over a day times the 16 MiB segment size. The count also includes the small history and backup-label files, and the sum covers every pod that was primary during the day: + +```promql +sum by (namespace, job) (increase(cnpg_pg_stat_archiver_archived_count{namespace=""}[1d])) * 16 * 1024 * 1024 +``` + +The platform's default backup strategy compresses both WAL and base backups with gzip, so the stored size is lower, by an amount that depends on the data; a cluster backed up to an object store of its own choosing stores them uncompressed unless its ObjectStore says otherwise. The default `retentionPolicy` is `30d`, so a cluster that writes 4 GB of WAL a day needs about 120 GB of WAL alone before its base backups are counted. + +Each volume server must hold the sum of those buckets, since at `replicationFactor: 2` with two servers each server carries a full copy. Size `volume.size` so that this sum stays below 95% of it, and leave room on top for every bucket that is still growing: at `replicationFactor: 2` a grow claims three volumes of `master.volumeSizeLimitMB` on each server. + +## Alerts that watch the instance + +The generic `KubePersistentVolumeFillingUp` alert does not cover this. It pages below 3% free, while a volume server stops at 5% or, out of slots, with its disk half empty, and its early warning needs the disk to keep filling, which it stops doing once writes fail. Each instance carries its own rules instead: + +- `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less than 10% of its disk left before the `minFreeSpacePercent` reserve. +- `SeaweedFSVolumeServerFull` (warning): a volume server cannot host another volume, because it is at the reserve, where all its volumes are read-only, or has no slot left, where buckets whose volumes on it are full stop taking writes. +- `SeaweedFSWritesFailing` (critical): the master has failed to place a write in every 5-minute window for 15 minutes. Other writes may still succeed in between. The usual cause is a bucket with no writable volume left that the master cannot grow, whose uploads all fail; a steady stream of newly created buckets, each failing a few picks while its first volumes grow, looks the same. + +On the Postgres side, `LastFailedArchiveTime` fires while WAL archiving keeps failing, and `KubePersistentVolumeFillingUp` fires as the WAL backlog fills the Postgres volume. + +## Growing an instance + +Raise `volume.size` on the SeaweedFS application. The upgrade expands each volume server's PVC in place, which needs a StorageClass that allows volume expansion. Postgres keeps retrying the segments that waited on its volume, so archiving catches up once the master can grow the bucket again. diff --git a/content/en/docs/next/operations/storage-class-immutability.md b/content/en/docs/next/operations/storage-class-immutability.md new file mode 100644 index 00000000..43d7c6f9 --- /dev/null +++ b/content/en/docs/next/operations/storage-class-immutability.md @@ -0,0 +1,50 @@ +--- +title: "storageClass is immutable after creation" +linkTitle: "storageClass immutability" +description: "Why storageClass cannot be changed after creation, where the chart schema enforces it, and which applications it covers." +weight: 170 +--- + +Every stateful Cozystack application that exposes a `storageClass` parameter declares it immutable in the chart schema (`x-kubernetes-validations: [{rule: "self == oldSelf"}]`). This document explains the contract and where it is enforced. + +## Why + +`storageClass` binds the StatefulSet's PVC template. Kubernetes fixes `PersistentVolumeClaim.spec.storageClassName` at PVC creation time — editing `volumeClaimTemplates[].spec.storageClassName` on an existing StatefulSet does not retroactively migrate any data. A user who edits `storageClass` on an existing resource may believe their data is being moved to the new class; it is not. Locking the field at the schema layer makes that contract explicit. + +## What enforces it + +| Consumer | Behavior | +| --- | --- | +| Cozystack UI (`packages/system/dashboard/images/console`) | Reads the CEL rule from the chart's `openAPISchema` and renders `storageClass` as a disabled, helper-text-annotated field on edit forms. Save-time overlay reinstates the original value before PUT. | +| `kubectl edit` / `kubectl patch` against the cozystack aggregated apiserver | **Currently accepted** — the apiserver does not yet evaluate CEL rules embedded in `ApplicationDefinition.openAPISchema`. The change passes through but does not propagate to existing PVCs (see "Why" above), so no data corruption is possible. Apiserver enforcement is tracked in cozystack/cozystack#2657. | +| `kubectl edit` / `kubectl patch` against native CRDs | Enforced today by the standard apiextensions apiserver. | + +## Apps covered + +The following charts annotate at least one `storageClass` field as immutable: + +- `clickhouse`, `foundationdb`, `harbor`, `kafka` (Kafka broker + KRaft controller), `kubernetes`, `kubernetes-nodes`, `mariadb`, `mongodb`, `nats`, `openbao`, `opensearch`, `postgres`, `qdrant`, `rabbitmq`, `redis`, `valkey`, `vm-disk`. + +The `kubernetes-nodes` chart annotates both its per-pool `storageClass` (the worker node system-disk class, defaulted `replicated`) and its `cluster` field immutable. `storageClass` binds the same PVC-template contract as every other chart in the table, so the plain `self == oldSelf` rule applies for the reason in "Why" above. `cluster` is immutable for a different reason: it wires the pool's CAPI objects to one parent cluster `kubernetes-`, and repointing a live pool at another cluster would orphan its running worker VMs rather than migrate them. + +## `vm-disk` also freezes `source` + +The `vm-disk` chart annotates `source` immutable in addition to `storageClass`, for a mechanism distinct from the PVC-template reasoning above. A `vm-disk` renders a single `DataVolume`, and `DataVolume.spec` is immutable in CDI: once the DataVolume exists, editing `spec.source` (or `spec.storage.storageClassName`) has no effect. The chart therefore looks the existing DataVolume up and reuses its stored `spec`, so an edit to `.Values.source` would otherwise render as a silent no-op while the HelmRelease still reported success — the CR and the cluster diverge with no diagnostic (cozystack/cozystack#2985). + +Because the aggregated apiserver does not yet evaluate the CEL rule (see the table above, cozystack/cozystack#2657), `vm-disk` adds a second, chart-level guard that is the substantive enforcement today: `templates/dv.yaml` fails the render when the requested `storageClass` or `source` differs from what the disk was created with. Both checks compare the operator's input recorded in an annotation at creation — `vm-disk.cozystack.io/storage-class` and `vm-disk.cozystack.io/source` — written only at creation and preserved verbatim afterward, never overwritten with an unapplied edit. The storage-class annotation is written even for the cluster default (`""`), and its key presence — not its value — marks a recorded class, so a disk created with `""` is still caught if later changed to a concrete class. Recording the input rather than reading the live object matters on both axes: the image-clone PVC name carries a chart-version prefix (`vm-image-` was renamed to `vm-default-images-`) that drifts across upgrades with no operator edit, and the live PVC's `storageClassName` on an adopted or foreign DataVolume may legitimately differ from the CR's default without being a change. + +Disks created before these annotations existed (or out-of-band) are handled conservatively so a working release never deadlocks. For `source`, such a disk is compared against its stored `spec.source` directly, but only when that stored source is a shape the chart itself produces (`blank`, `http` with just a `url`, `upload`, or a `pvc` with just `name`/`namespace`). An adopted DataVolume using `spec.sourceRef`, or a source type the chart cannot render (`registry`, `snapshot`, `imageio`, `http` with a `secretRef`, a foreign-namespace `pvc`), has no comparable provenance and is left alone rather than failed on a mismatch. The image-clone exemption fires only when the stored source is a `cozy-public` `pvc` **and** the request is itself an image — the one case where the `vm-image-` → `vm-default-images-` prefix drift is indistinguishable from a real image-name change; a change away from an image source to another type is still caught. For `storageClass` there is no safe live fallback (the CR default need not reflect an adopted disk's real class), so an un-annotated disk is simply not guarded here; apiserver CEL (cozystack/cozystack#2657) is the eventual guard for those. To actually change `storageClass` or `source`, delete and recreate the disk; size growth is the one mutable dimension and is handled separately by `templates/pvc-resize-hook.yaml`. + +The `storageClass` annotation records the effective value applied at creation, including when it came from the chart's `values.yaml` default (`replicated`) rather than an explicit operator value. Applications created through the aggregated apiserver have the default stamped into the CR, so it stays stable; but a direct HelmRelease consumer that omits `storageClass` records the chart default of the day, so changing that default in a future chart version would make every such untouched release start failing the render until its CR pins the old value or the disk is recreated. Treat the `storageClass` default as a frozen contract for the same reason the `source` serialization above is. + +The `source` annotation is a byte-for-byte `toJson` of `.Values.source`, so it is coupled to the exact serialization of the source schema. Adding a subfield with a schema default (which the apiserver would stamp onto every CR) would make the recorded and requested JSON diverge on an untouched disk; such a change needs an annotation-migration story rather than a bare schema bump. + +If you add a new stateful chart that exposes `storageClass`, annotate it the same way: + +```yaml +## @param {string} storageClass - StorageClass used to store the data. +## @immutable +storageClass: "" +``` + +For fields declared inside a `@typedef`, place `## @immutable` on the line below the `## @field` it should attach to. diff --git a/content/en/docs/v1.6/operations/storage-class-immutability.md b/content/en/docs/v1.6/operations/storage-class-immutability.md new file mode 100644 index 00000000..cc07ed27 --- /dev/null +++ b/content/en/docs/v1.6/operations/storage-class-immutability.md @@ -0,0 +1,38 @@ +--- +title: "storageClass is immutable after creation" +linkTitle: "storageClass immutability" +description: "Why storageClass cannot be changed after creation, where the chart schema enforces it, and which applications it covers." +weight: 170 +--- + +Every stateful Cozystack application that exposes a `storageClass` parameter declares it immutable in the chart schema (`x-kubernetes-validations: [{rule: "self == oldSelf"}]`). This document explains the contract and where it is enforced. + +## Why + +`storageClass` binds the StatefulSet's PVC template. Kubernetes fixes `PersistentVolumeClaim.spec.storageClassName` at PVC creation time — editing `volumeClaimTemplates[].spec.storageClassName` on an existing StatefulSet does not retroactively migrate any data. A user who edits `storageClass` on an existing resource may believe their data is being moved to the new class; it is not. Locking the field at the schema layer makes that contract explicit. + +## What enforces it + +| Consumer | Behavior | +| --- | --- | +| Cozystack UI (`packages/system/dashboard/images/console`) | Reads the CEL rule from the chart's `openAPISchema` and renders `storageClass` as a disabled, helper-text-annotated field on edit forms. Save-time overlay reinstates the original value before PUT. | +| `kubectl edit` / `kubectl patch` against the cozystack aggregated apiserver | **Currently accepted** — the apiserver does not yet evaluate CEL rules embedded in `ApplicationDefinition.openAPISchema`. The change passes through but does not propagate to existing PVCs (see "Why" above), so no data corruption is possible. Apiserver enforcement is tracked in cozystack/cozystack#2657. | +| `kubectl edit` / `kubectl patch` against native CRDs | Enforced today by the standard apiextensions apiserver. | + +## Apps covered + +The following charts annotate at least one `storageClass` field as immutable: + +- `clickhouse`, `foundationdb`, `harbor`, `http-cache`, `kafka` (Kafka + ZooKeeper), `kubernetes` (top-level only — see below), `mariadb`, `mongodb`, `nats`, `openbao`, `opensearch`, `postgres`, `qdrant`, `rabbitmq`, `redis`, `vm-disk`. + +The per-node-group `kubernetes.nodeGroups[name].storageClass` field is intentionally NOT annotated immutable: it is optional and undefaulted, so a strict `self == oldSelf` rule would block any future attempt to set it on an existing node group. If a richer ratcheting form (`!has(oldSelf.storageClass) || self.storageClass == oldSelf.storageClass`) becomes available, the annotation should be added back. + +If you add a new stateful chart that exposes `storageClass`, annotate it the same way: + +```yaml +## @param {string} storageClass - StorageClass used to store the data. +## @immutable +storageClass: "" +``` + +For fields declared inside a `@typedef`, place `## @immutable` on the line below the `## @field` it should attach to.