fix(backup): trust store, failed-enable revert, cluster-scoped repositories — with a server E2E tier

[vishr]

Three faults on the backup path, and the end-to-end tier that found them.

Why a second tier

Every engine in the Docker suite is built with transport.NewLocal(), and internal/transport/transport.go says so outright: local docker stands in for the deploy host. So ob's real transport had never run end to end, and neither had anything that needs a machine — bootstrap on a host with no container runtime, systemd timers, the host trust store, or what teardown leaves behind.

This adds a tier that makes the substitution nowhere. The test process runs on the workstation and reaches a disposable Lima guest over real SSH as root, driving the compiled binary rather than internal engine APIs, so the approval artifacts, the confirmations and the exit codes are part of what is tested. The guest arrives bare, so ob bootstrap is on the critical path rather than assumed.

Same two commands on an Apple Silicon Mac and on a GitHub runner; the image is pinned by digest per architecture, so neither emulates.

• TestServerLifecycle — bootstrap, plan, approve, deploy, status, exec, logs, audit, second release, rollback, service apply, secrets list, job plan and run, doctor, the eight backup subcommands, destroy. One machine, in order. 17 of 28 commands, from none. ~200s warm.
• TestServerProbes — what is left behind when things go wrong: a failed enablement, a repeated one, a redeployed database, teardown with and without volumes.

Two of the probes found bugs on their first run.

The fixes

The host trust store never reached the container (#88). wal-g runs inside the driver's image and postgres:18 carries no certificate authorities, so every upload to the HTTPS endpoint an s3-compatible target must declare failed with x509: certificate signed by unknown authority — after the base backup, a quarter of an hour in, with archiving already on. The bundle now travels the way the binary does, through the directory already mounted read-only into the container, and the wrapper exports it under the three names that can do the verifying. A host with no bundle is refused before archiving is turned on.

A failed enablement left archiving on (#88). PostgreSQL retains every segment it cannot ship, so a command that reported failure established an unbounded disk commitment. Enablement now walks the same pending-then-disabled transitions ob backup disable walks — but only when that run is what turned archiving on, so a service that was already protected is left as it was.

Repositories were shared across clusters (#91). WalgPrefix was <app>_<service> with no cluster identity, and every fresh cluster starts its WAL at 000000010000000000000001. A database redeployed against a target that still held the previous one's history wrote object names that already existed with different bytes; wal-g refuses the overwrite, PostgreSQL archives in order, and the chain stopped at the first collision while enable reported success. ob destroy --volumes removes the cluster and leaves the repository — that is the point of a backup — so the sequence is ordinary. The prefix now carries PostgreSQL's system identifier, recorded at enablement and used by rendering, status, backup and restore alike. An empty generation names the legacy layout and stays readable; a moved repository is reported rather than silently rebased.

Backup timers survived ob destroy (#92). RemoveSchedules matched ob-<app>- while backup timers are named ob-backup-<app>-…, so two timers and four unit files outlived the release directory they invoked. The namespace split stays — it exists because a deploy once deleted every scheduled backup — and teardown now sweeps both.

Enablement also quiesces the archiver before the base backup. Nothing at enable time can undo a collision written by a cluster that no longer exists, but the condition is detected and refused rather than establishing a chain that is already broken, and that failure routes through the revert above.

Verification

Red before each fix, green after, on a real machine — not a mock.

• TestServerLifecycle 22/22 including backup restore returns the data
• TestServerProbes 7/7 including the redeploy regression that defines backup: a redeployed database collides with the previous cluster's WAL history #91
• 1475 unit tests across 19 packages, golangci-lint 0 issues, actionlint clean, generated docs current

For #88 specifically, the unfixed build fails in 14m1s against the 14m25s in the original report, with pg_stat_archiver reading 0 archived | 1 failed where the report read 0 | 13.

Worth knowing before merging

Historical generations remain recoverable. Status discovers every cluster generation in the repository, and status, drill, and restore accept --generation to select one explicitly. A successful historical restore records that recovered binding. The recovery guide documents the loss-of-volume and loss-of-host sequence.

e2e/testdata/postgres/secrets/age.key is a committed private key. It decrypts only the fixture credentials for a MinIO the test starts and deletes on a throwaway guest, and the fixture README says so — but it is a real key in the repository and worth an explicit decision.

Automation executes both E2E tiers. Docker runs on PRs; real-server lifecycle and probe jobs run in parallel before release, nightly, and on demand.

Tier allocation is deliberate. Fast repository, native, and Docker checks stay on PRs. The Lima/SSH suite gates publication and keeps nightly/manual coverage without adding about sixteen minutes to review feedback.

Three findings about the environment came out of building this and are recorded in #44 rather than fixed here: Ubuntu's docker.io ships no buildx while plan.go requires it and preflight.go never checks for it; Ubuntu's packaged buildx accepts --format and ignores it, so ob blames the registry; and tag pinning walks every child manifest in the index, which throttles against public registries.

Closes #88
Closes #91
Closes #92