Skip to content

docs: move the guides that only live in the cozystack repo onto the site #763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
myasnikovdaniil wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: move the guides that only live in the cozystack repo onto the site #763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions content/en/docs/next/guides/resource-management/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,9 @@ The seven short names that existed before the instance-type rename remain accept

Legacy names are scheduled for removal in a future Cozystack release; new manifests should use the instance-type form. The Cozystack API server logs a deprecation warning whenever an app CR carries a legacy value, naming the suggested replacement.

A platform upgrade runs a one-shot migration (Migration 39) that walks every `HelmRelease.spec.values` and every app CR under `apps.cozystack.io/v1alpha1` and rewrites legacy values to their instance-type equivalents in place. The conversion is idempotent, best-effort, and never changes CPU or memory.
A platform upgrade runs a one-shot migration (Migration 39) that walks every `HelmRelease.spec.values` and every app CR under `apps.cozystack.io/v1alpha1` and rewrites legacy values to their instance-type equivalents in place. The conversion is idempotent, best-effort, and never changes CPU or memory. A failure on one object logs a warning and leaves the rest of the migration to complete. Set `MIGRATION_DRY_RUN=1` to preview the patches without applying them.

Cozystack presets are defined in an internal library
[`cozy-lib`](https://github.com/cozystack/cozystack/tree/main/packages/library/cozy-lib). The canonical reference, including the full size matrix and migration table, lives in [`docs/operations/resource-presets.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/resource-presets.md).
Cozystack presets are defined in an internal library [`cozy-lib`](https://github.com/cozystack/cozystack/tree/main/packages/library/cozy-lib).


### Defining Resources Explicitly
Expand Down
130 changes: 130 additions & 0 deletions content/en/docs/next/networking/gateway-classes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
title: "GatewayClass selection and TLS termination for tenant Gateways"
linkTitle: "GatewayClass selection"
description: "How a tenant picks a GatewayClass, what the class decides about TLS termination, and how to move a tenant between classes."
weight: 16
---

A tenant that owns a Gateway (`tenant.spec.gateway: true`) gets one `Gateway` object bound to one `GatewayClass`. Which class that is decides who programs the dataplane and, with it, whether TLS is terminated by the Gateway itself or by something in front of it. This page covers the operator-facing surface: the platform values that bound the choice, the tenant field that makes it, what each resulting certificate mode implies, what a move between classes does to certificates and Secrets, and how the mechanism fails when the configured names and the installed classes disagree.

The chart-level reference for the per-tenant Gateway — inheritance, listener shapes, the security model, ACME rate limits — is [`packages/extra/gateway/README.md`](https://github.com/cozystack/cozystack/blob/main/packages/extra/gateway/README.md).

## Platform values

Three keys under `gateway` in the platform values bound which class a tenant Gateway can land on:

```yaml
gateway:
enabled: true
className: cilium
tenantSelectableClasses: []
edgeTerminatedClasses: []
```

- `className` — the class every tenant Gateway uses unless the tenant names another one. Defaults to the Cilium class the platform ships.
- `tenantSelectableClasses` — the additional class names a tenant may name for itself. Empty (the default) means a tenant cannot name anything but the platform default.
- `edgeTerminatedClasses` — the class names whose provider terminates TLS upstream of the Gateway. Membership in this list is the only thing that puts a tenant into edge mode.

All three reach the per-tenant gateway chart through the platform values channel, as `_cluster.gateway-class-name`, `_cluster.gateway-tenant-classes` and `_cluster.gateway-edge-classes`. The two lists are joined with commas on the way, so a name containing a comma is rejected at platform render time rather than silently split into two names.

`gateway.enabled` is a separate switch and not a prerequisite for any of this: it moves the platform's own published services from Ingress onto Gateway API and points the ACME `ClusterIssuer`s' HTTP-01 solver at the publishing tenant's Gateway. A tenant Gateway is created from `tenant.spec.gateway` alone.

## How a tenant picks a class

```yaml
apiVersion: apps.cozystack.io/v1alpha1
kind: Tenant
metadata:
name: acme
namespace: tenant-root
spec:
gateway: true
gatewayClass: edge-tunnel
```

`edge-tunnel` throughout this page stands for the name of a GatewayClass whose controller terminates TLS outside the cluster and forwards cleartext to the Gateway. Substitute the name your own edge-terminating controller registers.

The field is read only for a tenant that owns a Gateway. A tenant that inherits — `gateway` left unset or set to `false` — publishes through the nearest ancestor that owns one and runs on that ancestor's class, so its own `gatewayClass` has no effect. Leaving `gatewayClass` empty selects `gateway.className`.

The set a tenant may name is `gateway.tenantSelectableClasses` plus the current `gateway.className`, so a tenant may always name the default explicitly. Anything else fails that tenant's own gateway release at render time, with a message naming the class and the allowed set, and reaches no other tenant. The allowlist exists because `Tenant` is a tenant-writable object while a `GatewayClass` is cluster-scoped: without it a tenant could attach its Gateway to any class the cluster registers, including one an admin keeps for internal-only traffic.

## What the class decides: the certificate mode

`TenantGateway.spec.certMode` is derived, never written by the tenant. The gateway chart resolves it in this order, first match winning:

| Condition | `certMode` | What the controller issues |
| --- | --- | --- |
| the resolved class is in `gateway.edgeTerminatedClasses` | `edge` | nothing |
| `publishing.certificates.wildcardSecretName` is set | `existingSecret` | nothing; the operator's Secret is referenced |
| `publishing.certificates.solver: dns01` | `dns01` | one wildcard `Certificate` per tenant |
| otherwise | `http01` | one `Certificate` per published hostname, except the passthrough names below |

Under `http01` a hostname held by a TLS-passthrough listener is the exception to the per-hostname rule. The hostname earns neither an HTTPS-terminate listener nor a `Certificate`, whether or not a `TLSRoute` is attached to that listener yet, because the backend presents its own certificate and the Gateway never holds a key for the name. The shipped `tlsPassthroughServices` defaults put `api`, `vm-exportproxy` and `cdi-uploadproxy` in that position on the publishing tenant's Gateway, for the names `publishing.exposedServices` lists while `gateway.enabled` is on, and the `Certificate` objects that covered them are collected on upgrade. No other tenant renders those listeners, so a child tenant's `HTTPRoute` on one of those names keeps its listener. `packages/extra/gateway/README.md` carries what that changes and the ways forward.

Edge therefore wins over an operator-supplied wildcard Secret, which wins over the solver. In edge mode the solver, DNS-01 provider and issuer-name inputs are not read at all, so an unsupported value in any of them does not fail the tenant's release. `existingSecret` skips the same validations; only `http01` and `dns01` reject a bad solver, provider or issuer name.

## What edge mode changes on the Gateway

- The Gateway serves port 80 only. Its listeners are `edge` for `*.<apex>`, `edge-apex` for `<apex>`, and one `edge-child-*` per inheriting child tenant carrying that child's `*.<child-apex>`. There is no port-443 listener, and no hostname-less port-80 listener either.
- No per-tenant ACME `Issuer` and no `Certificate` are created.
- No http-to-https redirect `HTTPRoute` is created. The edge is expected to perform that redirect.
- `tlsPassthroughServices` is not written onto the `TenantGateway` and no passthrough listener is rendered, so the tenant Kubernetes API, VM export and CDI upload endpoints are not published through such a Gateway. The packages that own those endpoints still create their `TLSRoute` objects, which then match no listener on that Gateway.
- Application `HTTPRoute`s attach by hostname exactly as in the other modes — the edge listeners carry the same namespace selector the HTTPS listeners carry elsewhere.

That last point has a blast radius worth stating on its own, and it applies only where those endpoints are published through the Gateway at all. Each of them renders its `TLSRoute` only when `gateway.enabled` is on and its name is still in `publishing.exposedServices`; at the default `gateway.enabled: false` there is no such route and nothing to lose. Where both hold, setting `gateway.className` to an edge-terminated class puts the publishing tenant on it too, and then the endpoints that go unpublished are the platform's own: the `TLSRoute` for `api.<root-host>` is pinned to a `tls-api` listener, an edge Gateway has no such listener, so the tenant Kubernetes API stops being served — VM export and CDI upload go the same way. Nothing fails at render time, and the loss surfaces only on the routes themselves, which the controller flips to `Accepted=False` with reason `NoMatchingParent`. Putting only child tenants on the edge class avoids it, because the platform's routes keep pointing at the publishing tenant's Gateway.

Listing a class in `gateway.edgeTerminatedClasses` is an assertion about that class, and nothing verifies it. Put the bundled `cilium` class in the list and its Gateways serve every application hostname over plain HTTP on a public address, with no redirect and no certificate. Only list a class whose provider really does terminate TLS in front of the Gateway.

## Moving a tenant between classes

Onto an edge-terminated class, on the next reconcile:

| Object | What happens |
| --- | --- |
| the per-tenant ACME `Issuer` | deleted |
| per-listener and wildcard `Certificate` objects | deleted |
| the TLS Secrets those `Certificate`s issued | left in the namespace, unreferenced |
| the http-to-https redirect `HTTPRoute` | deleted |
| the replica of `publishing.certificates.wildcardSecretName` | deleted from that tenant's namespace |
| route status conditions the controller wrote earlier | an attached `TLSRoute` that declares hostnames is flipped to `Accepted=False`, reason `NoMatchingParent`; on a hostname-less `TLSRoute`, and on every attached `HTTPRoute`, the controller's own entry is removed instead |

Every deletion is guarded by ownership: an object of the same name that the controller did not create is left in place.

The TLS Secrets survive because cert-manager ships here with `enableCertificateOwnerRef: false`, so a `Certificate` does not own the Secret it issued into. That is how every mode transition behaves and is not something edge introduces. Removing the leftovers is an operator action.

The wildcard replica is withdrawn because the namespace stops counting as a TLS termination point. Three things keep it anyway: a namespace that also owns an ingress controller still terminates locally; a namespace holding more than one `TenantGateway` keeps the replica while any of them is not on an edge-terminated class; and a namespace whose `TenantGateway` objects cannot be listed, or which carries the Gateway owner label but has no `TenantGateway` yet, keeps it as well, so that a transient read failure never withdraws a key a Gateway may still be serving.

Moving back off an edge-terminated class restores whatever the new mode calls for: the ACME `Issuer` and its `Certificate`s under `http01` or `dns01`, the redirect route under any non-edge mode, and the wildcard replica whenever `publishing.certificates.wildcardSecretName` is set. The `TLSRoute` conditions go back too, since the passthrough listeners do: `http01` judges those routes again from the hostname claims it collects, while `dns01` and `existingSecret` collect none and so drop the entry rather than leave a withdrawal standing on a listener that is serving. `existingSecret` mints no `Issuer` and no `Certificate` of its own, so a move from `edge` into it brings back the replica and the redirect route and nothing else.

The two halves of the move are not ordered against each other. The wildcard-secret controller watches the `TenantGateway` and prunes the replica as soon as `certMode` changes, while the `Gateway` itself is re-rendered when the tenant's HelmRelease reconciles, and the class controller reprograms after that. A tenant leaving `existingSecret` can therefore be briefly serving HTTPS listeners whose Secret has already gone.

## When the configured names and the installed classes disagree

Nothing in this mechanism checks that a class name corresponds to a `GatewayClass` the cluster actually has. Run `kubectl get gatewayclass` before putting a name into any of the three values.

- **A name in `className` or `tenantSelectableClasses` that matches no installed class.** The tenant's release renders and the `Gateway` object is created, but no controller claims it, so it gets no address and no listener status. Nothing fails at render time; the symptom is the `TenantGateway` holding `Ready=False` with reason `GatewayNotAccepted`.
- **A misspelled name in `edgeTerminatedClasses`.** It matches no class, so edge mode never engages for it and nothing says so. A tenant whose TLS really is terminated upstream stays in whichever mode the solver and wildcard-secret settings select, and its Gateway keeps terminating TLS itself.
- **A class removed from `tenantSelectableClasses` while a tenant still names it.** That tenant's gateway release starts failing to render. Other tenants are unaffected.
- **`className` changed while a tenant pins the outgoing name.** The allowed set is the *current* default plus `tenantSelectableClasses`, so the pin stops being valid and that tenant's gateway release fails. Add the outgoing class to `tenantSelectableClasses` before changing the default, or update the tenant first.
- **A class whose `status.supportedFeatures` lists features but not `TLSRoute`.** A tenant Gateway that renders TLS-passthrough listeners on it reads `Ready=False` with reason `GatewayClassUnsupported`, naming the class, because nothing would serve those listeners. A class that lists no features at all is not judged, since implementations predating the field leave it empty.
- **The publishing tenant selecting a class that terminates TLS differently from `className`.** Refused at render time, in both directions. The cluster-wide ACME `ClusterIssuer`s point their HTTP-01 solver at the publishing tenant's Gateway, and whether that solver pins `sectionName: http` is decided from `gateway.className` alone — a tenant's own choice does not travel on that channel. A publishing tenant that moved itself onto an edge class would keep a solver pinned to the `http` listener its Gateway no longer renders, and every HTTP-01 certificate in the cluster would stop issuing with nothing in any status naming the cause. Change `gateway.className` instead. The publishing tenant is the namespace named by `publishing.ingressName`, `tenant-root` by default.
- **A comma or a non-string entry in either list.** The platform chart fails to render, naming the entry.

## Checking what a tenant landed on

```bash
kubectl get gatewayclass

kubectl --namespace tenant-acme get tenantgateway cozystack \
--output jsonpath='{.spec.gatewayClassName}{"\t"}{.spec.certMode}{"\n"}'

kubectl --namespace tenant-acme get tenantgateway cozystack \
--output jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{"\t"}{.reason}{"\n"}{end}'

kubectl --namespace tenant-acme get gateway cozystack \
--output jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.port}{"\t"}{.hostname}{"\n"}{end}'

kubectl --namespace tenant-acme get certificate,issuer
```

A tenant on an edge-terminated class reports `certMode: edge`, shows port-80 listeners only, and has no `Certificate` or `Issuer` that this controller created. Leftover TLS Secrets from an earlier mode are not shown by the last command — list Secrets of type `kubernetes.io/tls` in the namespace to find those.
Loading
Loading