feat(backup): executable PostgreSQL point-in-time recovery, and the spec rename around it - #80
Merged
Conversation
Protection was fully described and entirely inert. The project schema
accepted a policy, the lifecycle catalogue declared postgres qualified,
the artifact set, locks, fences, journal, and scheduled-runner machinery
were all built and unit-tested — and none of it had a production caller.
`ob validate` accepted a 15-minute-RPO PITR policy and `ob preview`
rendered a plain `postgres:17` with no archive command and no repository.
A database could be declared protected and be backed up by nothing.
This wires the vertical slice end to end for the postgres driver:
- a derived image (images/postgres-pgbackrest) carrying pgBackRest,
whose entrypoint assembles pgBackRest's configuration from the
credential file on the host;
- rendering that turns an established policy into archive_mode,
an archive_command, and a repository configuration, gated on the
durable lifecycle state rather than on the policy's presence;
- `ob backup enable`, which pins the protected image by registry
digest, restarts the service under it, creates the stanza, proves
the archive path, and does not report success until a base backup
exists;
- `ob backup create` and `ob backup status`, the latter reading what
the repository can recover rather than what the project claims.
archive_timeout is derived from maximum_data_loss, which is what makes
the declared RPO a bound rather than an aspiration: a quiet database
otherwise archives only when a 16MB segment fills.
Validated end to end against a real S3 repository: 2000 rows written, a
recovery target marked, the table truncated, then recovered to that
target in a clean volume with all 2000 rows intact while the live
database stayed empty.
Four defects found only by running it, each of which would have shipped:
- PGBACKREST_REPO_PASSWORD is parsed by pgBackRest as a `repo-password`
option that does not exist. The credential entry is now
OB_REPOSITORY_PASSPHRASE, outside that namespace.
- pgBackRest connects as the OS user, which is `postgres`; the driver
creates `onebox`. pg1-user and pg1-database are now written.
- Generated files are replaced atomically by rename, which gives the
path a new inode — and a Docker bind-mount of a *file* is bound to
the inode, so the configuration vanished from inside the running
container. The configuration directory is mounted instead.
- `tls: insecure` was rendered as unverified TLS over a plaintext
endpoint. pgBackRest has no plaintext mode for S3, so a plaintext
endpoint is now refused at render with what to do instead.
Credentials never enter the project, the rendered document, or its
digest: the document carries only the *names* of the credential entries,
and the image's entrypoint reads the values from the mode-0600 file on
the host.
Restore, scheduling, and drills are not in this commit. `ob backup
enable`, `create`, and `status` are.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit made point-in-time recovery executable using
pgBackRest, which forced a derived PostgreSQL image Onebox would have had
to publish and maintain forever. Measuring the two settled it:
- pgBackRest links against 41 shared libraries, so it cannot be dropped
into the official image. Bind-mounting it fails on libssh2.
- wal-g links against libc and nothing else, and runs unmodified in
stock postgres:18 from a bind mount.
Everything hard about the pgBackRest integration came from it being
configured by a file rather than by the environment: a config replaced
atomically vanished from the running container because a Docker file
bind-mount follows the inode; the credential entry name collided with its
PGBACKREST_<OPTION> namespace; and it writes restore_command as the
absolute resolved path of its own binary, so no wrapper — by name, by
PATH, or by argv[0] — is reachable from recovery. wal-g has none of these
because it has no configuration file.
So there is no image. Onebox stages the wal-g binary on the host,
verified against a checksum pinned in this repository, and mounts it into
the official image beside a generated wrapper that maps the project's
declared credential entry names onto the fixed names wal-g reads. The
binary is fetched by the machine running `ob` and uploaded, which keeps
verification on this side of the trust boundary and lets a host with no
outbound internet be protected.
That checksum is also the first honest provenance record in the lifecycle
catalogue: every other digest there is a placeholder, and this one is the
value the download is actually verified against.
PostgreSQL 18 is now qualified alongside 17. The driver has always run
any version and monk runs 18 in production, so the previous ^17 pin was
already behind the fleet it was meant to describe.
`ob backup prune` and `ob backup verify` are exposed rather than left as
engine code with no caller — the sin this whole line of work started by
finding. `verify` runs wal-verify over the archive, which is the check
that a base backup exiting zero does not imply: a gapped WAL stream
recovers to the backup and no further, and nothing notices until someone
needs it.
Validated end to end against a real S3 repository, with no local-registry
workaround because the image is stock: 2000 rows written, a recovery
target marked, the table truncated, then recovered to that target in a
clean volume with all 2000 rows intact while the live database stayed
empty. Prune retained exactly the declared minimum_generations.
Two further defects found by running it:
- Enablement wrote the lifecycle state before staging the runtime, so a
single failure left a service recorded as protected with no binary to
archive with, and every retry was refused. Staging now happens first,
and re-running enable on an already-enabled service re-converges.
- PGDATABASE was unset, so libpq defaulted the database to the user
name and every command failed with `database "onebox" does not
exist`, which reads like a broken cluster rather than a missing
variable.
Restore, scheduling, and drills are still not here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…it is protected Two defects introduced by making protection executable, both of the same kind: something claimed to be true that was not. `ob backup enable` refused to start unless a credential file already existed on the target, and told the operator to "stage it through the trusted secret flow". No such flow reaches protection credentials — `ob secrets push` handles application env_files and external connection projections, neither of which is this. The instruction pointed at nothing, and the only way to satisfy it was to write the file by hand. Enablement now installs it. The target already names its encrypted file, Onebox already decrypts SOPS material for every other secret it places, and the engine already has a mode-0600 installer that runs under the service lock — so the pieces existed and were simply not connected. The credentials are decrypted and checked on the machine running `ob`, before any of it crosses to the target: a missing entry found after the server has restarted with archive_mode on is a database whose WAL cannot drain, which is a far worse place to learn it. Every missing entry is reported at once rather than one decrypt-edit-encrypt cycle at a time, and the repository key is checked for being 64 hex characters, because wal-g reads it as hex and a passphrase-shaped value is not a weak key but one it refuses outright. Wiring that up surfaced a second gap with nothing to do with credentials: enablement took no locks at all. It restarts a database, so it now takes the application lock and fence like every other mutation, and the per-service protection lock the credential installer requires. Two enablements, or an enablement and a deploy, could previously interleave on the same service. minio declared policy_qualified: true while having no executable renderer, so `ob validate` accepted a minio protection policy that `ob backup enable` then refused. That cost nothing while protection rendered nothing at all; it is a lie now. It is false until something runs. Validated end to end with the real trusted-secret flow — an age-encrypted SOPS credential file, no hand-staged secrets, no local registry: enable installs the credentials, a malformed repository key is refused before anything restarts, and 2000 rows truncated from the live database are recovered to a marked point in time in a clean volume. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ob doctor` warned "managed service data lives only on this host; Onebox takes no backups yet" for every declared service, unconditionally. That was true when nothing rendered protection. It is now said over a database archiving continuously to an off-host repository, and a doctor that reports a healthy thing as broken is a doctor people stop reading. A service with no policy still gets the warning, now naming the key that would fix it. A service that declares one is reported as declaring it. What the check still cannot say is whether protection was ever enabled — that is durable state on the target and this check is local — so it says what it knows and points at `ob backup status`, which reads the repository itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ng anything
Backups existed and nothing could restore them, which makes them a
hypothesis rather than a protection. `ob backup restore` and `ob backup
drill` are the same operation with different endings.
The recovered cluster is always built beside the live one, never over it:
the base backup is fetched into a fresh volume, WAL is replayed to the
requested point, and the result must start, promote, and answer a query
before anything touches the running database. A repository that cannot
recover therefore fails while the database it would have replaced is
still serving.
A drill stops there and discards the volume. That it runs the same code
as a real restore is the entire point — a drill exercising its own path
would prove the drill works, which is not the claim anyone needs.
Promotion copies the live volume aside under a dated name first and never
deletes it. A restore is run on a day that is already going badly, and it
must not be the step that makes the day unrecoverable.
Retention now honours both bounds it was given. minimum_generations and
recovery_window are different promises — how many recoverable bases to
keep, and how far back recovery must reach — and only the first was
applied, so on a quiet database where N backups spanned an afternoon the
declared window was silently shortened. Both are computed and the more
conservative wins, including keeping the one base backup older than the
cutoff, because recovering to a point in time replays forward from the
backup before it.
Three defects found by running it:
- `pg_ctl -w` returns when the server accepts connections, which happens
while it is still read-only and still replaying, so the first probe
reported every recovery as unfinished. Promotion is now waited for.
- PostgreSQL refuses RFC 3339 for recovery_target_time — the `T` and `Z`
make the whole configuration file fail to parse. RFC 3339 stays the
input format because it is the unambiguous one; it is converted to a
timestamptz literal on the way in.
- The backup label was parsed out of wal-g's log with the surrounding
sentence's punctuation still attached.
Validated end to end: 2000 rows written, a point in time marked, the table
truncated, a drill passed without touching anything, then a restore to
that point put all 2000 rows back into the live service — healthy, still
archiving, with the replaced data kept in a dated volume.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A backup policy is a promise about time. Declaring `schedule: {cron: "0 2
* * *"}` and then only ever backing up when somebody types the command is
not a weaker version of that promise — it is a different thing wearing
its clothes, and it fails silently on exactly the night nobody watched.
The cron was validated and then ignored.
Two systemd timers per protected service, taken from the policy rather
than invented. The backup schedule pushes a base backup and then applies
retention, in that order, so the repository is never briefly below the
generations the policy promises. The restore-drill schedule verifies the
archived WAL forms an unbroken chain.
The units run wal-g directly rather than through `ob`, because there is no
`ob` on the target: Onebox is agentless, and the verified binary these
units invoke is the only thing it has already placed there.
That same agentlessness is why the drill timer verifies rather than
restores. A real drill recovers into a throwaway volume and proves the
cluster answers, and that orchestration lives in `ob`. Reimplementing it
in a unit file would produce a drill that exercises a different path from
a real restore, which proves the drill works rather than the backups. So
the unattended half is the check that can be made honestly on the target,
and `ob backup drill` stays the whole proof, to be run from CI or a
workstation on the cadence the policy already declares.
A host with no systemd is refused rather than warned about. It can run a
protected database perfectly well and would never take a scheduled
backup, and a warning at the foot of an otherwise green apply is how that
goes unnoticed until it matters. A host with systemd but no flock is
refused too: the units serialise on it, and without it a backup and a
retention pass would run over the same repository at once.
Interactive wal-g invocations take the same flock, so an operator running
a backup by hand and a timer firing cannot talk to the repository at
once. Where flock is absent the command runs unwrapped, which is not a
silent degradation: flock ships with util-linux on every host that can
run systemd, and a host that cannot run systemd has no timers to
serialise against.
Two regressions the restore surfaced, both fixed here:
- the destructive-mount exemption still named protection/config, which
the wal-g rework had renamed to protection/runtime, so every apply of
a protected service demanded --allow-destructive-mounts to detach
files Onebox wrote itself;
- anonymous volumes were reported as data about to detach. The official
PostgreSQL image declares VOLUME /var/lib/postgresql while Onebox
mounts the data one level below, so every container gets a scratch
volume with a 64-character hex id nothing declares.
The generated units were checked with `systemd-analyze verify` on a real
systemd, the calendar expression normalises and resolves its next elapse,
and every ExecStart line was run verbatim against a live protected
service. The timers themselves have not been observed firing on a real
host — that needs a systemd target, which this validation did not have.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Protection could be turned on and never cleanly off. Stopping it meant
deleting durable state by hand, which is the kind of instruction that
ends with somebody removing the wrong thing.
`ob backup disable` stops archiving, removes the timers, restarts the
service as an ordinary unprotected one, and removes the destination
credentials from the host. It does not touch the repository: the backups
already taken are the reason anyone turned protection on, and someone
disabling it today may still need to recover from last week.
The existing multi-phase disablement — request, plan, authorize, advance —
is a larger apparatus than the decision needs, and every phase of it is a
place to leave a service half-disabled. This transitions straight to
disabled and keeps LastEffective, so the record still says what the
service was protected by when it stopped.
Two defects fixed, one of them written minutes earlier in this same
change:
- The disable message promised that `ob backup status` and `ob backup
restore` still worked afterwards. They do not: both run wal-g inside
the service container, and an unprotected service mounts neither the
binary nor the credentials. The backups are untouched, which is the
part that matters, and the way back to them is to enable protection
again. Caught by testing the claim rather than trusting it.
- A backup command on a service that declared protection but never
established it reached into the container anyway and produced an OCI
runtime error about a missing path. It now says the service is not
protected and which command fixes that.
Enablement checks the host can run the schedules before anything durable
happens. It previously discovered that at the schedule sync, which is
after the service had already been recorded as protected and restarted
archiving — an enablement left half-applied with a failed command as its
only symptom.
Validated as a whole lifecycle against a real Linux target over SSH — a
privileged systemd container running its own Docker daemon, which is the
single box this product is built for. Bootstrap, enable (aarch64 wal-g
selected from the target's own uname, credentials decrypted from SOPS and
installed, timers installed), an unattended backup taken by the timer
firing on its own, drill, verify, prune, a point-in-time restore putting
2000 truncated rows back into the live service, and finally disable —
after which the timers and units are gone, archiving is off, the data is
intact, and every backup is still in the repository.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SyncSchedules owns the "ob-<app>-*" systemd namespace and removes the
units a project no longer declares, which is what keeps a deleted job
from running forever. Protection timers were named inside that namespace,
so every `ob deploy` reclaimed them as stale: all scheduled backups
stopped, and the only trace was a line in the deploy output saying the
schedule was "no longer declared".
A database would have gone on serving, apparently protected, with an
archive still running and no base backup taken since the last deploy —
until someone needed to recover and found the window ended weeks ago.
Protection timers now live under "ob-protection-", a namespace the job
scheduler cannot claim, and the job scheduler skips that prefix as well.
Belt and braces, because of what the failure costs and how quietly it
happens. A regression test pins both halves.
Found by deploying a real application. It did not appear in any unit
test, in the local Docker validation, or in the systemd-container run,
because none of them ran a deploy after enabling protection.
Also in this change, closing the two items left open:
- Destructive protection operations take a typed confirmation —
`--confirm <service>` rather than a bare `--yes`. Onebox's approval
flow binds a recorded confirmation to an exact plan, and a recovery
or a disablement has no plan to bind to, so the guard is the name of
the thing being replaced: it cannot be given by accident, or by a
shell history entry meant for another service.
- The protection artifact subsystem is retired. It generated twelve
JSON descriptors of what protection should look like, plus a digest
comparison to detect drift, and never had a caller. Every descriptor
is now either derived live — schedules are systemd units built from
the policy, retention is applied by prune from the same policy — or
superseded by the wal-g checksum pinned in this binary and verified
before the binary is placed on a host. Drift is asked of the target
instead: `ob backup status` reports when the staged binary is not the
pinned build or the credential wrapper is not what the project
renders. A second description of the truth is only somewhere for the
two to disagree.
Validated against Gitea 1.24.6 on an Ubuntu 24.04 VM: deployed with an
Onebox-managed protected PostgreSQL, installed through its own web
installer (112 tables), three repositories created through its API, one
deleted through its API, then recovered by point-in-time restore — after
which Gitea itself lists the deleted repository again and answers for it.
The timers survived the redeploy that would previously have removed them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scheduled retention applied only minimum_generations. Both bounds are minimums — at least this many recoverable bases, and at least this much continuous history — so keeping the count alone silently shortened the window. A service backing up every five minutes under a seven-day window retained ten minutes of history, and nothing said so. wal-g offers no working time bound to express the other half. Its documented `retain N --after <time>` flag is accepted and ignored, and `delete before FIND_FULL <timestamp>` deletes nothing even when the timestamp is now; both were measured against 3.0.8 rather than taken from the help text. `retain FULL n` does work. So the window becomes a count, computed from the policy alone: firing days in the window times runs per firing day, plus one — the oldest backup kept has to be older than the window, because recovering to its earliest moment replays forward from the base taken before it. The larger of that and minimum_generations wins. That needed arithmetic maximumCronGap does not do. It answers a different question — the longest gap in *days*, which is what a drill cadence is checked against — and ignores the minute and hour fields entirely, so it reads "*/5 * * * *" as daily. Counting the sub-daily firings is what turns a 5-minute schedule under a 7-day window into the 2017 generations it actually implies, rather than the 2 it was keeping. The unattended path and `ob backup prune` now compute the same number, so a schedule cannot expire what an operator's prune would have kept. Confirmed on the VM: the unit renders `delete retain FULL 2017` for that policy, a timer run grows the repository instead of trimming it to two, and Gitea still answers for all three repositories. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nine renames and two regroupings across the project schema, from a review
of every key and value against the vocabularies our users already speak —
Docker Compose, Kubernetes, GitHub Actions, restic, CloudNativePG.
Names:
services.<name>.protection -> backup (CNPG `spec.backup`,
Velero `Backup`; the
command is `ob backup`)
backup.retention.minimum_generations -> keep (restic and borg both
say keep; "generation"
is rsnapshot jargon)
backup.retention.recovery_window -> window (unambiguous inside
retention)
backup.restore_drill -> drill (matches `ob backup
drill`)
backup.allow_backup_interruption -> allow_downtime (names what the operator
is agreeing to)
verifications -> checks (shorter, and regrouped;
see below)
Values:
encryption.* client-side | archive-password | server-side-sse
-> client-side | server-side
archive-password was pgBackRest's repository cipher and became
unreachable when protection moved to wal-g. server-side-sse stutters:
SSE is server-side encryption.
backup_targets.<name>.tls required | insecure -> verify | skip-verify
"insecure" reads as plaintext; it means an unverified certificate, and
the transport is TLS either way.
Structure — two regroupings, both replacing a repeated prefix or an
untagged union with a group that carries the fact once:
`checks` groups by kind. The flat list was an untagged union of four
shapes, and status_codes, required_headers, contains and json_assertions
were legal only alongside `url` with nothing saying so until validation.
Each group now has one shape the schema can type, so an editor completes
it and a wrong field is a wrong field rather than a runtime refusal. The
regrouping renames three fields, because once the group names the kind
the field no longer has to: http -> path, exec -> run, and
migration_revisions becomes the group `migrations`.
`policy.migrations` groups the four keys that each repeated the word:
require_migration_backup, migration_backup_maximum_age,
require_migration_restore_test, migration_backup_key_material.
What this deliberately does NOT change, having checked d13cf93 — the
commit that locked these conventions before onebox.run/v1 went public:
- `published_ports` stays. It was renamed *from* `ports` on purpose, to
keep it distinct from `port`, the route and health port on the same
struct. Compose saying `ports` does not outweigh two keys one letter
apart meaning different things.
- `maximum_`/`minimum_` stay spelled out. d13cf93 renamed *.max_age to
*.maximum_age across probe, drill, migration policy and the plan
artifact. An abbreviation pass would reverse a deliberate decision for
four characters.
- `services` stays, despite colliding with Compose's `services` where
application containers live. The collision costs one error at validate
naming the exact path — not silence, and not wrong behaviour. Paying
for that with a longer word on every project is the wrong trade.
- `env`, `needs`, `health` stay: Kubernetes says `env`, GitHub Actions
says `needs`, and `health` is shorter than Compose's `healthcheck`.
- `health.within` stays. It is a rollout deadline, not Compose's
per-probe `timeout` — and Onebox already emits `timeout` into the
healthcheck it generates, so that name is taken by a different thing
in the same output.
The on-target state directory stays `<app>/protection/`. Renaming it would
turn a config rename into a data migration, orphaning credentials,
lifecycle state and the staged runtime on every deployed host.
No compatibility shims: the loader refuses every old spelling by name and
the guard test in load_test.go pins that, as it did for the previous
rename pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rename left the documentation describing a product that no longer
matches the one in the repository. Four pages still called the protection
layer "proposed", `ob doctor` warned that Onebox takes no backups over a
database archiving to an off-host repository, and the field reference
linked to a page the rename had deleted.
Corrected:
- `ob doctor` reports a service declaring `backup` as declaring it, and
keeps the warning for a service that declares nothing. A doctor that
reports a healthy thing as broken is a doctor people stop reading. It
still cannot say whether protection was ever *enabled* — that is
durable state on the target and the check is local — so it points at
`ob backup status`, which reads the repository itself.
- The landing page, capabilities page, drivers page and database guide
no longer say Onebox takes no backups. They now draw the line where it
actually falls: a managed service declaring `backup` is protected, a
workload's own volume is not.
- `backup_targets` is no longer marked schema-only in the field
reference, and the lifecycle failure codes no longer belong to a
"proposed" layer.
- `verifications` in the field index became `checks`, which is where the
dangling link came from.
Adds the guide the database page was already linking to. Its example was
extracted and run through `ob validate` rather than written by hand, and
it documents the three things that surprise people: declaring a policy is
not enabling it, the repository key is 64 hex characters and not a
passphrase, and retention is derived from the schedule and the window
together — a five-minute schedule under a seven-day window keeps 2017
generations, not two.
It also states plainly that the scheduled drill verifies archive
continuity rather than restoring, because a real drill needs `ob` and
Onebox puts no agent on the host. `ob backup drill` belongs in CI.
Site builds, 40 pages, no dangling internal links.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…replaced Two designs for running scheduled backups existed side by side. One staged a runner binary onto every host and drove it with sealed envelopes, publication proofs and a canonical operation graph. The other installs systemd timers that invoke the wal-g binary already staged there. The second shipped, and it is the one that matches the product: Onebox is agentless, and the runner design needed a per-host binary plus a publication-proof supply chain nothing builds. Keeping both meant ~2,000 lines describing an architecture the code does not use, which reads as capability rather than as abandoned work. Removed: cmd/ob-scheduled-runner, engine/scheduled_protection.go, onebox/scheduled_envelope.go, scheduled_install.go, scheduled_runner.go, protection_resources.go, the ScheduledLifecycleExecutor seam that had no backend, and the two Names helpers that addressed the runner and its envelopes on disk. Also removes the multi-phase disablement — request, plan, authorize, advance, roll back — which no path reached. Stopping a backup is not a data migration, and every phase was another place to leave a service half disabled. Disablement keeps two states, and the pending one now earns its place. It previously wrote "disabled" before doing any of the work, so a failure while restarting the service or removing credentials left a record claiming the service had stopped archiving while it still was. It now records the intent first and the completion second, which is what a retried run needs to read — the same ordering bug already fixed in enablement. Requesting disablement also stops the restore drills immediately. A drill materialises a whole recovered cluster, and running one for a service somebody has just asked to stop protecting is work nobody wants. Backups keep running until the work completes, so the recovery window has no hole in it while the disablement is in flight. `protection_disablement_not_authorized` moves to the reserved set: the authorization step that raised it is gone, and the reserved list's own comment no longer claims backup and disable are unwired, because they are not. Net 2,438 lines removed. Full suite green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follows the scheduling removal. Everything here was declared, tested
against itself, and reachable from nothing an operator can run.
Gone:
- lifecycle_operation_graph.go — the per-operation risk, reversibility
and approval declarations, and the graph builder that turned them into
steps. Nothing called it. The approval it declared was never enforced;
the destructive backup commands are guarded by a typed confirmation
instead, which is what actually runs.
- lifecycle_records.go — a second evidence-record format beside the
journal the engine writes.
- protected_identity.go — a sealed identity record for a protected
service, superseded by the lifecycle state the renderer reads.
- s3_target.go — a target adapter with its own probe and credential
binding, superseded by wal-g reaching the repository directly.
- active_volume.go and active_volume_seed.go — a durable record of which
volume a service's data is in. Recovery names the volume from the
service's own declaration.
- protection_credentials.go — resolved the credential slots a plan would
carry. Enablement reads the entries from the target and installs them
itself.
Kept, moved to operation_types.go: the lifecycle metadata grammar and the
digest pattern, which four live files still bind evidence with.
Kept as tests: the two assertions that were worth more than the code they
exercised — that a sealed operation plan carries no credential value, and
that no public failure contains secret-shaped data.
Three failure codes move to the reserved set, because the only paths that
raised them are gone: backup_target_unauthorized, backup_target_unreachable
and protected_service_identity_changed. `ob backup enable` reaches the
repository through wal-g and reports its own failure.
Every schema name from the rename pass is unchanged; verified after the
deletions rather than assumed.
Net 1,878 lines removed across this commit. Suite green, vet clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six keys lose four characters each: policy.minimum_onebox_version -> min_onebox_version policy.minimum_plan_schema -> min_plan_schema policy.migrations.backup_maximum_age -> backup_max_age backup.maximum_data_loss -> max_data_loss backup.drill.maximum_age -> max_age external_services.<n>.probe.maximum_age -> max_age Only the prefix is abbreviated, so nothing loses meaning: max_data_loss still says data loss, min_onebox_version still says which runner. This reverses d13cf93, which spelled them out to make the four sites consistent with each other. They stay consistent — the whole set moves together — so the change is purely brevity, which is why it was not recommended. It is made because the shorter form is wanted, and the argument against it was that four characters do not justify a breaking change rather than that the longer form is better. The guard test now refuses the spelled-out spellings, replacing the two cases that guarded the reverse rename and are obsolete for the same reason: `max_age` used to be the old name and is the new one again. Frozen contract verdicts re-recorded, schema republished, docs regenerated. `just check` passes, which also caught three files this session had never run gofmt over. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`just lint` found seven unused functions left behind when the unreachable protection model went: a path sorter whose only caller took the grouped form, a string helper restored alongside the metadata grammar but needed by nothing, and five remnants of the multi-phase disablement — the phase successor, the schedule deactivator, the active-kind reader, a remote deletion assertion, and a test helper. `just check` does not run the linter; `just ci` does. Running only the former after each commit is how these survived four passes. CI gate green: fmt, vet, tests, docs, site build, lint, govulncheck, actionlint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds executable PostgreSQL PITR using WAL-G, while replacing the former protection schema and removing superseded lifecycle infrastructure.
Changes:
- Adds backup enable/create/verify/prune/status/drill/restore operations and systemd scheduling.
- Renames backup, check, migration-policy, and minimum/maximum schema fields.
- Removes the unused scheduled-runner and protection-planning model.
Reviewed changes
Copilot reviewed 135 out of 135 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
README.md |
Updates schema examples. |
cmd/ob-docgen/main.go |
Updates generated references. |
cmd/ob-scheduled-runner/main.go |
Removes scheduled runner. |
cmd/ob-scheduled-runner/main_test.go |
Removes runner tests. |
cmd/ob/backup.go |
Adds backup CLI. |
cmd/ob/commands.go |
Updates plan output. |
cmd/ob/doctor.go |
Reports backup declarations. |
cmd/ob/main.go |
Registers backup commands. |
cmd/ob/output.go |
Defines backup output contracts. |
cmd/ob/output_test.go |
Tests output contracts. |
docs/onebox.run-v1.schema.json |
Regenerates project schema. |
e2e/testdata/app/ob.yml |
Migrates checks syntax. |
e2e/testdata/worker/ob-broken.yml |
Migrates checks syntax. |
e2e/testdata/worker/ob.yml |
Migrates checks syntax. |
internal/app/canonical_test.go |
Updates canonical facts. |
internal/app/constraints.go |
Renames enum values. |
internal/app/defaults.go |
Updates renamed defaults. |
internal/app/errors.go |
Adds backup errors. |
internal/app/external_schema.go |
Renames probe age. |
internal/app/generate.go |
Renders protected services. |
internal/app/jsonschema.go |
Updates schema constraints. |
internal/app/jsonschema_test.go |
Updates schema tests. |
internal/app/load.go |
Validates backup persistence. |
internal/app/load_test.go |
Tests breaking schema changes. |
internal/app/names.go |
Adds runtime and timer names. |
internal/app/protection_artifacts.go |
Removes old artifacts model. |
internal/app/protection_artifacts_test.go |
Removes artifact tests. |
internal/app/protection_names_test.go |
Updates backup naming tests. |
internal/app/protection_schema.go |
Validates backup policies. |
internal/app/protection_schema_test.go |
Tests renamed policy schema. |
internal/app/protection_walg.go |
Implements WAL-G projection. |
internal/app/protection_walg_test.go |
Tests retention derivation. |
internal/app/resolve.go |
Resolves backup overrides. |
internal/app/service_image_state_test.go |
Tests protected image state. |
internal/app/service_lifecycle.go |
Exposes lifecycle helpers. |
internal/app/service_lifecycle_records.go |
Qualifies PostgreSQL WAL-G. |
internal/app/services.go |
Renders archive configuration. |
internal/app/services_test.go |
Updates service tests. |
internal/app/testdata/contract-verdicts.json |
Updates contract verdicts. |
internal/app/testdata/corpus/ext-immich-sourced.compose.yaml |
Updates Compose fixture. |
internal/app/testdata/corpus/fanout.yml |
Migrates checks syntax. |
internal/app/testdata/corpus/goal.yml |
Migrates checks syntax. |
internal/app/testdata/corpus/monk.yml |
Migrates checks syntax. |
internal/app/testdata/corpus/pursue.yml |
Migrates checks syntax. |
internal/app/testdata/corpus/recast.yml |
Migrates renamed fields. |
internal/app/types.go |
Defines renamed schema types. |
internal/app/validate.go |
Validates grouped checks. |
internal/buildinfo/calver.go |
Updates field terminology. |
internal/engine/engine.go |
Caches flock capability. |
internal/engine/fixtures_test.go |
Migrates check fixtures. |
internal/engine/hooks_test.go |
Migrates URL checks. |
internal/engine/migration_backup.go |
Uses grouped migration policy. |
internal/engine/migration_backup_test.go |
Updates migration tests. |
internal/engine/protection_postgres.go |
Stages WAL-G runtime. |
internal/engine/protection_postgres_ops.go |
Implements repository operations. |
internal/engine/protection_restore.go |
Implements drill and restore. |
internal/engine/protection_schedule.go |
Installs backup timers. |
internal/engine/resume_test.go |
Migrates check state. |
internal/engine/schedule.go |
Separates timer namespaces. |
internal/engine/schedule_test.go |
Tests timer isolation. |
internal/engine/scheduled_protection.go |
Removes old scheduler. |
internal/engine/scheduled_protection_test.go |
Removes scheduler tests. |
internal/engine/service_apply.go |
Handles protection mounts. |
internal/engine/services.go |
Stages runtime and schedules. |
internal/engine/verify.go |
Executes grouped checks. |
internal/engine/verify_contract_test.go |
Updates verification tests. |
internal/engine/verify_injection_test.go |
Updates injection fixture. |
internal/onebox/active_volume.go |
Removes active-volume model. |
internal/onebox/active_volume_seed.go |
Removes volume seeding. |
internal/onebox/active_volume_seed_test.go |
Removes seeding tests. |
internal/onebox/active_volume_test.go |
Removes volume tests. |
internal/onebox/backup_evidence.go |
Renames migration fields. |
internal/onebox/backup_evidence_test.go |
Updates evidence tests. |
internal/onebox/backup_gate_test.go |
Updates policy terminology. |
internal/onebox/binding.go |
Adds backup runtime binding. |
internal/onebox/duration_contract_test.go |
Updates duration paths. |
internal/onebox/exec_test.go |
Updates minimum-version syntax. |
internal/onebox/execute.go |
Dispatches backup operations. |
internal/onebox/execution_types.go |
Adds recovery inputs. |
internal/onebox/job_plan_test.go |
Updates migration policy. |
internal/onebox/lifecycle_errors.go |
Updates failure registry. |
internal/onebox/lifecycle_errors_test.go |
Updates failure tests. |
internal/onebox/lifecycle_operation_graph.go |
Removes operation graph. |
internal/onebox/lifecycle_operation_graph_test.go |
Removes graph tests. |
internal/onebox/lifecycle_records.go |
Removes legacy records. |
internal/onebox/lifecycle_records_test.go |
Removes record tests. |
internal/onebox/operation_types.go |
Retains digest validation. |
internal/onebox/protected_identity.go |
Removes identity model. |
internal/onebox/protected_identity_test.go |
Removes identity tests. |
internal/onebox/protection_artifacts.go |
Removes artifact bindings. |
internal/onebox/protection_artifacts_test.go |
Removes binding tests. |
internal/onebox/protection_credentials.go |
Removes slot resolver. |
internal/onebox/protection_credentials_test.go |
Removes resolver tests. |
internal/onebox/protection_disable.go |
Implements disablement. |
internal/onebox/protection_enable.go |
Implements enablement. |
internal/onebox/protection_read.go |
Implements backup status. |
internal/onebox/protection_redaction_test.go |
Removes legacy redaction test. |
internal/onebox/protection_resources.go |
Removes resource model. |
internal/onebox/protection_resources_test.go |
Removes resource tests. |
internal/onebox/protection_restore.go |
Coordinates recovery locking. |
internal/onebox/protection_state.go |
Simplifies lifecycle state. |
internal/onebox/protection_state_test.go |
Tests state transitions. |
internal/onebox/s3_target.go |
Removes S3 adapter. |
internal/onebox/s3_target_test.go |
Removes adapter tests. |
internal/onebox/scheduled_envelope.go |
Removes sealed envelopes. |
internal/onebox/scheduled_envelope_test.go |
Removes envelope tests. |
internal/onebox/scheduled_install.go |
Removes runner installation. |
internal/onebox/scheduled_install_test.go |
Removes installation tests. |
internal/onebox/scheduled_runner.go |
Removes scheduled execution. |
internal/onebox/service.go |
Removes scheduler seam. |
internal/onebox/service_test.go |
Migrates check fixtures. |
internal/onebox/state_probe_test.go |
Removes volume probe tests. |
internal/onebox/types.go |
Renames proposal checks. |
scripts/validate_release_tag_test.go |
Updates field terminology. |
site/astro.config.mjs |
Adds backup guide. |
site/public/onebox.run-v1.schema.json |
Publishes regenerated schema. |
site/src/content/docs/explanation/evidence-not-declaration.mdx |
Documents executable backup evidence. |
site/src/content/docs/guides/add-a-database.mdx |
Links backup workflow. |
site/src/content/docs/guides/back-up-a-database.mdx |
Adds PostgreSQL backup guide. |
site/src/content/docs/guides/run-migrations.mdx |
Migrates checks example. |
site/src/content/docs/index.mdx |
Updates backup capability. |
site/src/content/docs/reference/cli.mdx |
Documents backup CLI. |
site/src/content/docs/reference/drivers.mdx |
Documents PostgreSQL support. |
site/src/content/docs/reference/errors.mdx |
Updates lifecycle errors. |
site/src/content/docs/reference/fields/backup_targets.mdx |
Marks targets shipped. |
site/src/content/docs/reference/fields/checks.mdx |
Adds grouped-check reference. |
site/src/content/docs/reference/fields/environments.mdx |
Documents policy regrouping. |
site/src/content/docs/reference/fields/external_services.mdx |
Renames probe age. |
site/src/content/docs/reference/fields/services.mdx |
Documents backup policy. |
site/src/content/docs/reference/fields/verifications.mdx |
Removes old reference. |
site/src/content/docs/reference/naming.mdx |
Updates naming rules. |
site/src/content/docs/reference/policies.mdx |
Documents renamed policy. |
site/src/content/docs/reference/project-file.mdx |
Updates field index. |
site/src/content/docs/start/install.mdx |
Updates minimum fields. |
site/src/content/docs/status/capabilities.mdx |
Updates capability status. |
Suppressed comments (1)
internal/app/validate.go:359
- The public field was deliberately retained as
published_ports, but this error now tells users to remove the unrecognizedportskey. Preserve the actual path and remediation spelling.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| volumes: | ||
| - ./library:/usr/src/app/upload | ||
| ports: | ||
| published_ports: |
| } | ||
| for i, port := range w.PublishedPorts { | ||
| pp := indexed(path+".published_ports", i) | ||
| pp := indexed(path+".ports", i) |
Comment on lines
+105
to
+109
| would read as approval. Managed services are different: one declaring | ||
| `protection` is backed up continuously and recoverable to a point in time — see | ||
| the backup reference. Only the `postgres` driver has an executable contract | ||
| today; every other driver declares `policy_qualified: false` and its protection | ||
| policy is refused rather than accepted and ignored. |
Comment on lines
+96
to
+100
| `postgres` is the one driver with an executable protection contract: declaring | ||
| `protection` on it gives a continuous off-host archive and point-in-time | ||
| recovery. Every other driver declares `policy_qualified: false`, so a protection | ||
| policy on it is refused at `ob validate` rather than accepted and left to do | ||
| nothing — and its data lives only on this host until you copy it off yourself. |
Comment on lines
+122
to
+125
| **Backups cover managed services, not workload volumes.** A service declaring | ||
| `protection` — PostgreSQL today — is backed up continuously to an off-host | ||
| repository and can be recovered to a point in time. A workload's own volume is | ||
| not, and `ob doctor` says so for every workload holding durable data, because |
Comment on lines
+180
to
+182
| case KindBackupCreate: | ||
| result.EvidenceID = operationID | ||
| err = e.BackupService(ctx, request.Service) |
Comment on lines
+89
to
+94
| "The repository is not touched. The backups already taken are the reason\n" + | ||
| "anyone turned protection on, and someone disabling it today may still need\n" + | ||
| "to recover from last week — `ob backup status` still reads them and\n" + | ||
| "`ob backup restore` can still recover from them.\n\n" + | ||
| "What does stop is the recovery window advancing: from here on there is no\n" + | ||
| "new WAL, so the newest recoverable point is the moment this ran.", |
Comment on lines
+278
to
+281
| if len(status.Generations) > 0 { | ||
| latest := status.Generations[len(status.Generations)-1] | ||
| status.LatestBackup = &latest | ||
| status.RecoverableTo = time.Unix(latest.StoppedAt, 0).UTC().Format(time.RFC3339) |
Comment on lines
+350
to
+356
| "docker volume create " + q(kept), | ||
| // Copied rather than renamed: Docker has no rename, and a copy leaves | ||
| // the original intact until the very last step. | ||
| "docker run --rm -v " + q(live+":/from") + " -v " + q(kept+":/to") + " alpine sh -c 'cp -a /from/. /to/'", | ||
| "docker volume rm -f " + q(live), | ||
| "docker volume create " + q(live), | ||
| "docker run --rm -v " + q(staging+":/from") + " -v " + q(live+":/to") + " alpine sh -c 'cp -a /from/. /to/'", |
| | `<name>.prefix` | string | — | Non-secret object prefix reserved for Onebox protection data. Expects a relative object prefix with no empty leading component or shell metacharacter. | | ||
| | `<name>.region` | string | — | S3-compatible region when the endpoint requires one. Expects a lower-case S3-compatible region of letters, digits and hyphens. | | ||
| | `<name>.tls` | `required` · `insecure` | `required` | TLS verification policy: required or insecure. | | ||
| | `<name>.tls` | `verify` · `skip-verify` | `required` | TLS verification policy: required or insecure. | |
Fourteen findings, all real. The three that could lose data or leak
credentials first.
**A failed promotion destroyed both copies.** The volume swap ran as one
&&-chain, and a failure after `docker volume rm -f <live>` returned before
setting Promoted — so the deferred cleanup then deleted the staging volume,
the only complete copy of the recovered data. Live empty, recovered gone,
and the error named neither volume.
The live volume is now copied aside while it is still intact, and only then
is anything destructive done. The copy is emptied and refilled in one
container rather than removed and recreated, so a failure cannot leave the
service with no volume at all. From the moment promotion starts, the staging
volume is retained on failure and both volumes are named in the error. On
success it is removed, because the live volume then holds that data.
**`tls: skip-verify` forced plaintext.** It mapped to AWS_S3_FORCE_HTTP, so
an https endpoint with a self-signed certificate would have had its
credentials and backup stream sent in the clear — and wal-g has no option to
skip certificate verification at all, only an endpoint protocol and a CA
file. The transport now comes from the endpoint's own scheme via the
documented S3_ENDPOINT_PROTOCOL, and https + skip-verify is refused with
what to do instead, rather than silently verifying anyway.
**Protection downgraded an authored `wal_level`.** The appended
`-c wal_level=replica` won over `settings: {wal_level: logical}`, because
the last -c wins — so enabling backups would have broken every logical
replication slot on the server. replica is the floor, so it is raised to
rather than set, and anything below it is refused at validate rather than
silently overridden.
Reporting, which matters more than usual for a product whose thesis is
evidence over declaration:
- `recovered_to` was wall-clock now, read after promotion, so a drill to a
point last month reported today. It is the replay target now, and the
drill line states it — evidence that does not say what it proved is not
evidence.
- Recovery, status and prune read the current project's target rather than
the recorded projection, contradicting the rule rendering follows. An
operator editing the target would have pointed a restore at an empty
repository while the server kept archiving to the old one.
- An unparseable wal-g timestamp became 1970-01-01 in a recoverability
report. It is refused. Generations are sorted by time rather than
trusting the order another tool printed.
- `backup disable --help` promised that status and restore still worked
afterwards. They do not, and the same PR said so in the command output.
- backup, prune and verify took no host-ownership check and no locks at
all. Prune deletes base backups; it could expire generations while a
restore was reading them. All three now take the same guards as enable,
disable and restore.
- `docker run --env-file` cannot parse `export NAME=` or quoted values,
which the validators explicitly accept and Compose's env_file handles —
so a credential file in that form enabled fine and then produced a
recovery container with no credentials. Installation normalises to one
form.
- Enablement refused to re-converge from disable-pending, trapping an
operator whose disable died between its two state writes.
- Help text still described pgBackRest stanzas; the tls struct tag still
published `default: "required"`, which the new enum rejects, and the
generated schema carried it; five symbols this PR added were never
called; and a comment claimed wal-g honours `--after`, which the code
beside it says it does not.
`ProtectionTimerForEnvironment` was dead because the schedule sync built the
unit name inline — two spellings that could drift. The helper is now the
single source, and the .service and .timer derive from it.
Re-validated on a fresh VM, including the paths that changed: an
export-and-quoted credential file installs normalised; the endpoint scheme
drives the transport; an authored `wal_level: logical` survives while
`minimal` is refused; a drill reports the point it recovered to; and a
restore to a marked point put 2000 truncated rows back with both the
replaced data and the staging copy accounted for.
Verifying that also caught a leak the review did not: the staging volume
survived a *successful* restore, because the retain flag was checked without
also checking whether promotion completed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight findings and four notes. The first one is the sharpest: the fix for the *previous* review's projection finding did not work. **The recorded projection did not actually win.** `EffectiveProtectionProjection` delegates to a helper that prefers LastEffective only while disable-pending, so recovery, status and prune still read the project's current intent — under a comment claiming the opposite, which is worse than the original bug because it reads as fixed. An operator editing `backup_targets` would have pointed a restore at a repository the history is not in, and at a credential file installed under the old target's name. The exported function now prefers the recorded projection whenever there is one, and rendering resolves through it, so there is one rule rather than four that can drift. **Recovery bypassed the repository lock.** `fetch` ran wal-g through a raw docker exec, so a backup timer firing mid-restore could run its retention pass and expire the generation the restore was streaming out. The per-service protection lock does not help — systemd units cannot take it, which is the whole reason the flock exists. **Re-enabling never rebound.** With the service already enabled the transition was skipped, discarding the freshly pinned image and the new projection, so `ob backup enable` after editing the target was a silent no-op and no command could move a service to a new repository. **Read-only status waited an hour.** `backup-list` took the exclusive flock, contradicting the documented promise that status answers while other work is in flight. Listing now takes a shared lock with a short timeout. **A quoted key was rejected with a false message.** Validation judged the raw value while installation strips quotes, so `OB_REPOSITORY_KEY="…"` was refused for being 66 characters with an error about hex. **`discardRecoveryStaging` could not fail.** The command ended in `true`, making its own exit-code guard unreachable; a staging volume that could not be removed reported success and recovery proceeded into it — exactly what the function exists to prevent. It now verifies the volume is gone. **https + skip-verify passed validate and failed at render**, after the durable state was already written, leaving every later apply failing with guidance pointing at `ob validate`, which passed. The check moved to validation. **The repository prefix was ambiguous.** It joined app and service with a hyphen while both may contain hyphens, so `a-b`/`c` and `a`/`b-c` shared a prefix and would interleave two clusters' backups — and it is unversioned, so it could not be corrected later. It uses the injective join the rest of the derived names use, which names.go mandates for exactly this reason. Notes: a comment naming a field that no longer exists, a collision example naming the wrong app, `disable` gaining the `--break-lock` every other mutation has, and a helper whose name claimed more than it checked. Three tests pin the behaviours worth keeping: the recorded projection winning over an edited intent, the prefix not colliding across hyphenated names, and a quoted credential value being accepted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ResolveProtectedImage` ran `docker pull` unconditionally, so every `ob backup enable` fetched the manifest again — including on a re-enable, where the reference is already a digest. A digest names exact bytes. Pulling it a second time cannot return anything different, so the request buys nothing and costs two things that matter: registry quota, and the ability to enable protection at all on a host that is offline or rate-limited while already holding precisely the image it needs. That last one contradicts the reason the wal-g binary is staged from the operator's machine rather than fetched by the target — so that a host with no outbound internet can still be protected. A digest already present is now accepted as-is. A tag still resolves through the registry, because a tag can move, which is why protection pins in the first place. Found by exhausting anonymous Docker Hub's pull limit during validation: retrying `ob backup enable` a dozen times issued a dozen manifest requests for bytes the host had after the first one, and the resulting 429 then blocked the rest of the run. One predicate decides what "pinned" means now, shared by the pull-skipping guard and the pinning check so the two cannot drift, with a test covering the tag that merely looks like a digest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v1 keys were renamed (protection -> backup, verifications -> checks,
minimum_generations -> keep, recovery_window -> window, restore_drill ->
drill, allow_backup_interruption -> allow_downtime, maximum_/minimum_ ->
max_/min_), but only the YAML tags moved. The identifiers, filenames,
error codes, on-disk paths, unit names, and prose kept the old spelling,
so every reader had to hold two vocabularies at once and the docs printed
CLI output the binary no longer emits.
This is that rename carried through:
- 24 files renamed (protection_*.go -> backup_*.go) and the identifiers
with them, including Verification -> RunnableCheck, which could not
take the obvious name because app.Check already exists in preflight.
- Error codes protection_* -> backup_* and protected_service_* ->
service_*, with the frozen contract verdicts re-recorded.
- On-disk state <app>/protection/ -> <app>/backup/, systemd units
ob-protection-* -> ob-backup-*, journal steps protection-step:* ->
backup-step:*. The SyncSchedules guard that keeps a deploy from
deleting backup timers follows the new prefix.
- ob doctor's report section protections -> backups.
- Docs: the stale strings only. "Protection from an adversarial
infrastructure provider" is English, not a key, and stays.
The guide transcript claimed output ("protection runtime wal-g",
"ob-protection-shop-...") that no longer matches any string in the
binary; both now match backup_postgres.go and backup_schedule.go
exactly. The guide's YAML block was re-validated against a build of
this commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Running the full cycle against a real host — enable, back up, drill, point-in-time restore, disable, re-enable — turned up three failures the test suite did not have a shape for. Each is fixed with a test that would have caught it. 1. A restore poisoned every later recovery. The promoted cluster kept the recovery settings that produced it in postgresql.conf, so every base backup taken afterwards carried that restore's `recovery_target_time`. A later recovery asking for the newest point inherited a target in the past and the cluster died with "recovery ended before configured recovery target was reached". The drill that proved the repository on Wednesday failed on Thursday because Wednesday's restore succeeded. Recovery now clears every target kind before stating its own, and promotion removes onebox's block from the cluster going into service. 2. `ob backup enable` on an already-enabled service always failed with "backup lifecycle state digest mismatch". The transition to a disabled source mutated two sealed fields and passed the record on without resealing. Re-running enable is the documented way to move a service to an edited policy or target, so the documented path was the broken one, and the error blamed the host's record, which was intact. 3. Re-enabling needed a registry it had no reason to need. A service already bound to a repository has its pinned digest on the host, but enable re-resolved the tag every time: a rate-limited Docker Hub failed a command with nothing to fetch, and a tag that had moved would have swapped the bytes running over a live data directory — the thing pinning exists to prevent. The lifecycle record now also holds the authored reference that produced the pin (the runtime one alternates between tag and digest with the protected state, so it cannot serve as an identity), and the pin is kept whenever the project still declares that reference and the host still holds those bytes. Verified on Ubuntu 24.04 + Docker CE: a table dropped after a base backup came back from a point-in-time restore (801 rows, correct point), the promoted cluster carried no recovery settings, a backup taken after that restore drilled clean, and enable succeeded from both enabled and disabled with no reachable registry at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…oing it A review of what the backup surface actually executes found four kinds of declared-but-inert code. Each was documented to users, validated, and read by nothing. Lifecycle capability catalogue (-450 lines). It carried delivery classes, repository ownership, artifact and SBOM provenance, patch transitions with compatibility and continuity probes, consistency/topology preconditions, achievable RPO, protected resource names, native retention mappings, per-kind encryption modes, operation identifiers, and a graduation evidence contract — for one executable driver. Nothing read any of it except the function that validated it against itself, and the "provenance" digests were placeholders: sixty-four repetitions of one hex character, in the same shape as the one real checksum in the repository. What is left is what gates behaviour: qualification, recovery kinds, supported versions, credential slots. Nineteen reserved failure codes. Enumerated, documented in the public error reference with a "reserved" column, raised by nothing. The reference page now says every code is raised by a path in the binary, and a test checks that in both directions rather than maintaining a list of the ones that are not. Never-emitted public types (internal/onebox/types.go, deleted). Deployment proposals, observations, proposal preconditions, host state, image pins, comparison statuses — a thirty-field JSON contract that no code path constructs. With them go the staged status classifier and digest that only their own tests called. Declared-only schema keys. `observability.logs/metrics/alerts` (a user could declare 30-day log retention and nothing collected a log), `backup.drill.staging_filesystem` (validated as an absolute path, then ignored), `build.platform`, `image.platform`, and `image.registry`, whose own description said it selects no login. Also honest now: the lifecycle record claimed a restore drill ran on the declared cadence. No such timer is ever installed — onebox is agentless, a drill is orchestrated by `ob`, and what the host can honestly run unattended is the archive verification. `ob backup status` listed a schedule nothing performed, which is the one thing this product says it does not do. It now lists what the host runs, verified against a live system: backup-create, backup-prune and backup-verify against the installed units, replay-archive against the server's own archive_timeout. The README claimed onebox does not take backups. It does, for PostgreSQL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`just dead-exports` is not part of `just ci`, which is why the previous commit passed the gate with seven exported identifiers left behind: their last callers went out with the never-emitted proposal types and the s3 target adapter. ActiveVolumeFile, BackupTargetEncryptionMode, ValidateBackupTarget, Load/SaveBackupLifecycleState and two methods on the lifecycle record are gone; the branch is back to zero unreferenced exports, the number main has. One of them was not dead weight but a dead guard. AllowOperation refused a deploy, service apply or restore against a service whose disablement was requested and never finished — and no production path called it. The failure code it named counted as reachable only because an uncalled function mentioned it, which is how the codes-are-honest test was satisfied. Rather than delete a documented refusal, it is wired where it has a clear meaning: executeRecovery reads the record under the backup lock and refuses to drill or restore into a service somebody has just asked to stop protecting. Verified on a VM that had never seen onebox: bootstrap, first deploy, backup enable, base backup, a table dropped after that backup recovered by point-in-time restore (700 rows, `1 tables`), no recovery settings left in the promoted cluster, a backup taken after the restore drilling clean, verify, disable, re-enable, and a second deploy that left both timers installed, archiving on, and the data intact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on a live host, for the same deploy: two registry calls became zero, and `ob backup status` went from one to none. Two causes, both the same shape — a question asked whose answer could not change the outcome. The load path asked the registry first. Every command that loads a project with a protected service ran `docker manifest inspect` against the registry for the pinned service image, then checked the local cache. The single consumer of those two observations accepts either one, so a host holding the exact digest already had its answer. Now the cache is checked first and the registry only when it misses — which also means validate, plan, status and every backup command stop touching the registry at all on a host that is up to date. `image.pull` was a schema key nothing read. The loader defaulted it, validation checked the enum against `always|missing|never`, the reference documented it — and both release paths ran `docker compose pull` for every workload on every deploy regardless, including for an image pinned by digest and already present. A project declaring `pull: never` was pulled from anyway. The policy now decides: never does not fetch, always fetches, and missing (the default) fetches only what the host does not hold, which for a digest-pinned image is an exact answer rather than a guess. The plan preview follows the same rule, because a plan listing a command that will not run is a plan nobody can check against. This is what a rate-limited Docker Hub was failing on: commands that needed nothing from it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… onebox Gating onebox's own `docker compose pull` step was half the job. Compose fetches a missing image during `up` on its own, so a workload declaring `pull: never` was still fetched: on a live host, a release with `pull: never` started an image the host did not have and pulled it without anything in the plan or the output saying so. The generated runtime now carries `pull_policy`, so the declaration reaches the process that acts on it. Verified against a real registry and a real host, all five cases: never + held no fetch, release succeeds never + absent no fetch, release fails "No such image" — the point missing + absent fetches missing + held no fetch always + held fetches anyway The frozen conformance verdicts move with the rendered runtime, which now states the policy it always had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pending record was written and then immediately overwritten with `disabled`, before the server was restarted without archive_mode and before the timers were removed. So the state that exists to say "the decision was made and the work is not finished" lasted between two consecutive writes and covered nothing, while the window that can really be interrupted ran under a record already claiming the service had stopped archiving. Killing a real disable 1.5 seconds in produced exactly the lie the state exists to prevent: `disabled` on disk, archive_mode still on, both timers still installed. The record now stays pending until the restart, the schedule removal and the credential removal have happened. With the window real, the refusal wired in the previous commit was exercised against it for the first time: `ob backup drill` and `ob backup restore` both refuse with backup_disable_pending rather than recovering into a service somebody has just asked to stop protecting. `ob backup status` was answering that same state with wal-g's own message — "stat /opt/onebox/backup/ob-wal-g: no such file or directory" — because the runtime is removed partway through disabling. It reports the state instead, which is the thing the operator can act on. Both ways out verified on a live host: finishing the disable removes the timers and stops archiving, and re-enabling abandons it and comes back to a service that drills clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…st one Point-in-time recovery fetched wal-g's LATEST base backup whatever point it had been asked for. Replay only moves forward, so a base backup that finished after the requested point can never reach it: PostgreSQL replays the WAL it has, runs out before the target, and dies with "recovery ended before configured recovery target was reached". Every point older than the newest base backup was therefore unrecoverable — with a daily backup and a seven-day window, six of those days — while `ob backup status` reported the whole window as recoverable and `ob backup drill` passed, because a drill with no target asks for the newest point and that is the one case this got right. The base is now chosen by the requested point: the newest backup that finished at or before it, which is both the only kind that can reach the point and the one with the least WAL to replay. A point older than every backup in the repository is refused with the time the oldest one finished, because an operator asking for a window needs to know where it really starts. Verified on a live host, against the case that used to fail: two base backups either side of a recovery point, and a restore to the point between them landed on base_...021 rather than base_...023 and produced exactly the right data — 711 rows, the 11 written before the point, none of the 11 written after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two defects found by reading the remaining backup code and testing what it claims against a live repository. wal-verify was judged by its exit code. wal-g prints the integrity table, says "integrity check status: WARNING", lists the ranges it could not find — and exits 0. Two WAL segments deleted out of the middle of a live repository produced exactly that, and `ob backup verify` printed a green check over a chain with holes in it. The scheduled unit had the same blind spot: systemd saw exit 0 and nothing was ever raised. This is the check the comment above it calls "the check worth running, the one a green backup does not imply", and it could not fail. The report is now the verdict. A missing range is a hole, except MISSING_UPLOADING: wal-g classifies a segment that way while the server could still be uploading it and reclassifies it as MISSING_LOST once that window passes — the segments deleted here were UPLOADING at first and LOST minutes later. Position in the table decides nothing: on a database taking backups in a loop, three segments were in flight while later ones had already arrived. Verified both ways on a live host — the damaged repository fails and names the ranges, and five verifies run immediately after five backups on a healthy repository all pass. `ob backup enable` could not move a service to a different repository. Enable resolved its projection through EffectiveBackupProjection, which prefers the recorded one — correctly, so that an edited target cannot redirect a restore at a repository the history is not in. But enable is the one command where the project's intent is supposed to take effect, and resolving it the same way meant editing a bucket and re-running enable reported success and rebound the service to the repository it was already using. Measured: bucket edited, enable green, every later backup still going to the old bucket. Enable now reads the declared projection, and says so when the repository changes — because the new one starts at the backup it is about to take, so the declared window begins there. Also fixed on the way: folding stderr into runWalg broke the JSON reads that parse backup-list. The report-reading command has its own path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…promise beside the facts The two things the review left open, and one defect found while closing them. minio was marked backup-qualified. `ob backup enable` refuses every driver except postgres, so a project could declare a backup policy on a minio service, pass `ob validate`, be reviewed and committed, and only fail when somebody tried to establish it. The flag now means what it says — the contract runs — and minio gets the same refusal at validate time as every other unqualified driver. The guide already told readers every other driver declares `policy_qualified: false`; that is now true. `ob backup status` reported only what the repository holds and left the operator to work out whether it satisfied the policy they wrote, which is the question they came with. It now states the declared window beside the history actually held, and says plainly when the repository does not reach that far back yet — a young repository is not a fault, but it is not the promise either, and only the report can say which. It also asks the server whether it is still archiving. The staged binary and wrapper being correct says the tooling is in place; it says nothing about `archive_mode` being on, the `archive_command` still being the one onebox installed, or `archive_timeout` still being inside the declared maximum data loss. Onebox passes all three as command-line flags, which outrank ALTER SYSTEM, so the way this actually goes wrong is a container recreated by hand — and that is what was tested: a replacement container with archiving off and a four-hour timeout produced three drift lines, and a healthy server produces none. Getting there took three attempts at one psql invocation, each of which failed in a way the exit-code branch read as "the server is down" and skipped. The check passed its unit tests while never having run. Only a live server showed it. Fixed on the way: retiring the old credential file on a target move deleted the file the same run had just installed whenever the bucket changed inside a target that kept its name — the next command needing the repository failed with "--env-file: no such file or directory". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit made `ob backup verify` judge wal-g's report instead of its exit code, and said the scheduled unit had the same blind spot. It still did: the unit invoked wal-g directly, so systemd marked it successful over a repository with holes in it — for the entire span between an operator running the command by hand, which is the span the scheduled check exists to cover. The verify unit now runs a rendered POSIX-sh script that reaches the same verdict as the command: wal-g's own failure still fails the unit, MISSING_LOST and MISSING_DELAYED are holes, and MISSING_UPLOADING is a segment the server may still be uploading and resolves itself. Verified by firing both units on a live host rather than waiting for 2am: the backup unit takes a base backup and expires down to the derived retention (9 generations to 8), and the verify unit exits 0 on a healthy repository and exit-code 1 on the damaged one, with "onebox: the archived WAL has gaps" in the journal. Also confirmed on that host: OnCalendar carries the declared timezone correctly — 0 2 * * * UTC schedules as 19:00 PDT. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
internal/journal/backup.go carried a parallel record model for backup operations — step ids derived from operation identity, attempt counters, retry classifications, incomplete-resource inventories, helper provenance, terminal-result records, and their own append and lookup path. 184 lines, plus six fields on the shared record and four types. No path that runs writes any of it. Backup operations journal through the same records every other operation uses, which is how the pre-release job and migration-backup evidence read on a live host. The rest described a retry and resumption model the backup commands do not have. `just dead-exports` does not catch this: the tests referenced every symbol, so the apparatus kept itself alive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes PostgreSQL backups real, then renames the surface they sit on.
Protection was fully described and entirely inert before this: the schema
accepted a policy, the catalogue marked postgres qualified, and the artifact,
lock, fence, journal and scheduled-runner machinery was all built and
unit-tested — with zero production callers.
ob validateaccepted a15-minute-RPO PITR policy while
ob previewrendered a plainpostgres:17with no archive command. A database could be declared protected and be backed
up by nothing.
What works now
Continuous WAL archiving to an off-host S3 repository, systemd timers from the
declared schedule, retention honouring both floors, and point-in-time recovery.
Validated end to end against a real application, not a fixture: Gitea 1.24.6
on an Ubuntu 24.04 VM with an Onebox-managed protected PostgreSQL, installed
through its own web installer (112 tables). Three repositories created through
its API, one deleted through its API, then recovered by point-in-time
restore — after which Gitea lists it again and answers for it. A timer was
observed firing unattended and taking a backup on its own.
wal-g, not pgBackRest
pgBackRest was implemented first and replaced after measuring both:
postgres:18libssh2.so.1: cannot open shared object filepgBackRest cannot be dropped into the official image, so it forces a derived
image Onebox would have to publish and rebuild for every PostgreSQL CVE. wal-g
is one glibc-linked binary configured entirely by environment, so there is no
image: a checksum-pinned binary is staged on the host and mounted in.
That checksum is also the first honest provenance record in the lifecycle
catalogue — every other digest there is a placeholder, and this one is the value
the download is verified against before it is ever placed on a host.
Spec changes (breaking)
Two
spec!:commits, from a review of all 219 key paths and 24 value enumsagainst Docker Compose, Kubernetes, GitHub Actions, restic and CloudNativePG.
Renames:
protection→backup·verifications→checks·minimum_generations→keep·recovery_window→window·restore_drill→drill·allow_backup_interruption→allow_downtime·maximum_/minimum_prefixes →max_/min_.Values:
archive-passworddropped (unreachable since wal-g) andserver-side-sse→server-side; backuptls: required|insecure→verify|skip-verify, because "insecure" reads as plaintext when it means anunverified certificate.
Regroupings:
checksby kind, which removes an untagged union of four shapes;and
policy.migrations, which drops the prefix four keys each repeated.Deliberately not renamed, with reasons recorded in the commits:
published_ports(distinct fromporton the same struct),services(theCompose collision costs one precise error at validate, not silence), and
env,needs,health— already standard in Kubernetes and GitHub Actions, andshorter.
No compatibility shims: the loader refuses every old spelling by name, and a
guard test pins that.
Deletions
~5,000 lines of unreachable protection model: an agent-on-target scheduling
design (runner binary, sealed envelopes, publication proofs) that the systemd
approach superseded, the operation graph and its unenforced approval
declarations, a second evidence-record format beside the journal, the
protected-identity record, the S3 target adapter, the active-volume records,
and the credential-slot resolver.
Bugs found by running it
Each was found by execution or by an adjacent check, not by writing the code:
ob deploysilently deleted every scheduled backup. The job schedulerowns
ob-<app>-*and reclaims units no longer declared; protection timerswere inside that namespace. Found only by deploying a real application.
minimum_generationswasapplied, so a five-minute schedule under a seven-day window kept ten minutes
of history.
PGBACKREST_REPO_PASSWORDparsed as a nonexistent option; pgBackRestconnecting as
postgreswhere the driver createsonebox; an atomicallyreplaced config vanishing from a running container because a Docker file
bind-mount follows the inode;
PGDATABASEunset; enablement writing statebefore staging the runtime; and disable claiming completion before doing the
work.
Not in this PR
Only
postgreshas an executable contract — every other driver refuses a policyrather than accepting one it cannot honour. The scheduled drill verifies archive
continuity rather than running a full restore: that needs
ob, and Onebox putsno agent on the host, so
ob backup drillbelongs in CI on the declared cadence.Checks
just cigreen — fmt, vet, 1386 tests, docs-generate-check, site-build,golangci-lint (0 issues), govulncheck, actionlint. gitleaks clean. Site builds
40 pages with no dangling internal links.
This has had no review but mine. The branch mixes a new subsystem with a
breaking 86-file rename, and the mechanical pass corrupted files twice during
development — Compose
ports:written into Compose documents, and a canonicalfact name partially replaced — both caught by a failing test rather than by
design. That class of damage is worth looking for.
🤖 Generated with Claude Code