Skip to content

feat(service): secrets by reference, web probes, and a release job - #1

Closed
sylvesterdamgaard wants to merge 1 commit into
mainfrom
feat/secrets-health-release
Closed

sylvesterdamgaard wants to merge 1 commit into
mainfrom
feat/secrets-health-release

Conversation

@sylvesterdamgaard

Copy link
Copy Markdown
Contributor

Three things a production service (cbox-id's cbox.yaml) needs that a ServiceSpec could not say.

Secrets by reference — ServiceSpec::$secretRefs, a list of SecretReference(variable, secret, key). Compiles to valueFrom.secretKeyRef on every container of the service: web, every process, runs and the release (they all derive from the web container). The Secret is the operator's; nothing here creates, updates or prunes it, and its value is never in the spec or anything compiled. Never optional — a missing Secret/key leaves the pod in CreateContainerConfigError. Refused at compile time: invalid variable/Secret/key names, a variable claimed by a reference and by env/envSecret/a binding/another reference, and a reference to <service>-env or <service>-registry.

Probes — ServiceSpec::$health, a HealthCheck(readinessPath, livenessPath, tokenVariable). Web Deployment only:

  • startupProbe: httpGet liveness path, every 2s, 150 failures (5 min to boot)
  • livenessProbe: httpGet liveness path, every 10s, timeout 3s, 3 failures
  • readinessProbe: httpGet readiness path, or with a token an exec probe: printf 'Authorization: Bearer %s\n' "${VAR}" | curl -fsS -o /dev/null --max-time 4 -H @- 'http://127.0.0.1:PORT/path'. An httpGet probe can only send static headers, so the token would otherwise be in the pod spec. Header via stdin keeps it out of argv. Needs sh + curl in the image (every Cbox base image has both). Refused when the token variable is not set by the spec, and on the CRIU scale-to-zero tier.

Process Deployments and run Jobs now also strip startupProbe.

The release — ServiceSpec::$release (argv) + ReleaseJobCompiler (Contracts\ReleaseCompiler). A run Job named <service>-release-<hash of its pod template>: same release twice = same Job (no double migration), new image/env/command = new Job (Job templates are immutable). Empty while suspended. Kept out of the service's manifest set on purpose, so the applier can sequence: apply the set without Deployments/scalers → run the Job to completion → on failure stop, on success apply the full set. Documented in docs/core-concepts/secrets-probes-releases.md.

Golden files

Existing goldens are byte-for-byte unchanged (no new field is set in them). New: tests/Golden/service-production.golden.yaml — a cbox-id-shaped service (2 replicas, queue + scheduler, a Postgres binding, two Secret references, the token readiness probe) followed by its release Job. What it records that is new: the startupProbe/livenessProbe/readinessProbe on the web container only, the secretKeyRef entries at cbox-id-app on all three Deployments, and the cbox-id-release-<hash> Job. I could not run kubeconform locally (no image cached and I did not pull one); the CI schema step covers tests/Golden/*.yaml, so it will validate the new file.

Gates

pint --test, phpstan (max), pest (308 passed), license-check, audit — all green locally.

Consumers: cbox-engine (ProjectManifestReader → secrets/health/release in cbox.yaml, and the deploy sequencing) follows in a separate PR that needs this released as 0.12.0.

Three things a production service needs that a spec could not say.

Secrets by reference. ServiceSpec::$secretRefs (SecretReference: variable,
secret, key) compiles to valueFrom.secretKeyRef on every container of the
service, at a Secret the operator created. Unlike $envSecret, the value
never passes through the spec or anything compiled. Never optional: a
missing Secret or key holds the pod in CreateContainerConfigError, which
is the right failure. Names Kubernetes would refuse, a variable claimed
twice, and a reference to a Secret this compiler writes are refused here.

Probes. ServiceSpec::$health (HealthCheck) compiles startup + liveness
(httpGet on the liveness path) and readiness for the web Deployment only.
A readiness token compiles to an exec probe that reads the variable from
the container's own environment and pipes the header to curl -H @-: an
httpGet probe can only carry static headers, which would put the token
in the pod spec.

The release. ServiceSpec::$release and ReleaseJobCompiler compile the
service's release command as a run Job named for its pod template's hash,
kept OUT of the service's manifest set so an applier can run it to
completion before the Deployments roll. The sequence is documented in
docs/core-concepts/secrets-probes-releases.md.

Existing goldens are byte-for-byte unchanged; service-production is new.
@sylvesterdamgaard

Copy link
Copy Markdown
Contributor Author

Closed: Cbox ID's production deployment is defined in cbox-infra (infra/kubernetes/cbox-id), not cbox.yaml, so the platform doesn't need these keys for it. Branch kept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant