diff --git a/README.md b/README.md index a07fd81c..74a01d6a 100644 --- a/README.md +++ b/README.md @@ -70,21 +70,17 @@ clickhouse, nats. Anything else is refused rather than guessed at, because inventing an image from a name produces a container that starts and stores nothing durable. -Onebox does **not** take backups, and `ob doctor` says so for every workload and -service holding durable data. It also refuses a major version change a driver -cannot perform in place, rather than replacing the container and leaving the -data intact and unreachable. - -The schema can already declare desired log, metric, and alert capabilities. -The local engine does **not** manage those continuous services yet, and reports -them as declared rather than managed. The planned -dashboard/control plane will add authenticated team approvals, continuous -evidence, shared policy, and recovery assurance without becoming a generic -Docker UI. - -Versioned driver contracts and continuous observability management are not -shipped. Plan/status drift observation and plan-bound migration backup reports -are shipped; Onebox still does not create or store the backup itself. +Onebox takes PostgreSQL backups: continuous WAL archiving to a repository you +own, point-in-time restore, and a drill that proves recovery without touching +the live service. Every other driver **refuses** a backup policy rather than +accepting one it cannot honour, and `ob doctor` says which is which for every +workload and service holding durable data. It also refuses a major version +change a driver cannot perform in place, rather than replacing the container +and leaving the data intact and unreachable. + +The planned dashboard/control plane will add authenticated team approvals, +continuous evidence, shared policy, and recovery assurance without becoming a +generic Docker UI. ## Start using it @@ -172,8 +168,8 @@ Executable plans use `onebox.run/executable-deploy-plan/v1alpha2` and include the planner's version, source revision, build time, dirty state, and supported schemas. Schema-less and unsupported plans are rejected. Environment policy can set -`minimum_onebox_version` using the exact CalVer release form and can set -`minimum_plan_schema`; `ob doctor` reports whether the runner selected by +`min_onebox_version` using the exact CalVer release form and can set +`min_plan_schema`; `ob doctor` reports whether the runner selected by `PATH` is compatible. When a minimum version is configured, commit-derived and dirty checkout builds fail closed because they are not released runners. @@ -289,16 +285,17 @@ headers, and scalar JSON values. Migration verification can bind the expected provider and applied revisions to the captured job-result evidence: ```yaml -verifications: - - url: https://app.example.com/healthz - status_codes: [200] - required_headers: - X-App-Ready: "yes" - json_assertions: - - path: service.ready - equals: true - - migration_revisions: - job: migrate +checks: + url: + - url: https://app.example.com/healthz + status_codes: [200] + required_headers: + X-App-Ready: "yes" + json_assertions: + - path: service.ready + equals: true + migrations: + - job: migrate provider: atlas applied_revisions: ["202607130001"] ``` diff --git a/cmd/ob-docgen/main.go b/cmd/ob-docgen/main.go index a80a77f4..b5b82dc5 100644 --- a/cmd/ob-docgen/main.go +++ b/cmd/ob-docgen/main.go @@ -293,8 +293,8 @@ var blocks = []block{ {Key: "deployment", Title: "deployment", Order: 50, Status: statusShipped, Summary: "Release ordering, how many releases are retained for rollback, and the migration policy.", ReadWhen: []string{"Changing release order, retention or migration gating"}}, - {Key: "verifications", Title: "verifications", Order: 60, Status: statusShipped, - Summary: "What must be true before a release becomes current: external URLs, in-workload checks, or migration revision evidence.", + {Key: "checks", Title: "checks", Order: 60, Status: statusShipped, + Summary: "What must be true before a release becomes current, grouped by kind: external URLs, in-workload HTTP or exec probes, or migration revision evidence.", ReadWhen: []string{"Gating release activation on a health endpoint or a smoke test"}}, {Key: "proxy", Title: "proxy", Order: 70, Status: statusShipped, Summary: "Who owns the ingress proxy, which image runs it, and how TLS is resolved.", @@ -308,14 +308,11 @@ var blocks = []block{ {Key: "notifications", Title: "notifications", Order: 100, Status: statusShipped, Summary: "Named webhooks that receive selected operation outcomes.", ReadWhen: []string{"Sending deploy outcomes to Slack, Discord or an incident tool"}}, - {Key: "observability", Title: "observability", Order: 110, Status: statusIntentOnly, - Summary: "Declared logging, metric and alerting intent. Validated and planned, but the local engine runs nothing continuous for it.", - ReadWhen: []string{"Recording observability intent that another system will act on"}}, - {Key: "backup_targets", Title: "backup_targets", Order: 200, Status: statusSchemaOnly, - Summary: "User-owned off-host S3-compatible repositories available to service protection policies. Accepted by the loader; not yet executable.", - ReadWhen: []string{"Evaluating the proposed protection layer", "Understanding why Onebox refuses a backup target that shares the protected host"}}, + {Key: "backup_targets", Title: "backup_targets", Order: 200, Status: statusShipped, + Summary: "User-owned off-host S3-compatible repositories a protected service writes its backups to. Executable for the postgres driver; every other driver refuses a policy rather than accepting one it cannot honour.", + ReadWhen: []string{"Declaring where a database's backups go", "Understanding why Onebox refuses a backup target that shares the protected host"}}, {Key: "external_services", Title: "external_services", Order: 210, Status: statusSchemaOnly, - Summary: "Typed dependencies operated outside Onebox, whose lifecycle and protection stay external. Accepted by the loader; not yet executable.", + Summary: "Typed dependencies operated outside Onebox, whose lifecycle and backups stay external. Accepted by the loader; not yet executable.", ReadWhen: []string{"Modelling an RDS, Neon, Supabase or Upstash dependency"}}, } @@ -888,20 +885,16 @@ func renderErrorPage() string { fmt.Fprintln(&buf, "## Lifecycle failure codes") fmt.Fprintln(&buf) - fmt.Fprintln(&buf, ":::caution[Belongs to the proposed protection layer]") - fmt.Fprintln(&buf, "These codes are defined and drift-tested in the binary, but the operations that") - fmt.Fprintln(&buf, "raise most of them are not yet executable. A row marked **reserved** is one no") - fmt.Fprintln(&buf, "path raises today: the code is fixed so it stays stable when the capability") - fmt.Fprintln(&buf, "lands, but you cannot cause it. The set is computed from the source, not") - fmt.Fprintln(&buf, "maintained by hand.") - fmt.Fprintln(&buf, ":::") + fmt.Fprintln(&buf, "Every code here is raised by a path in the shipped binary, checked against the") + fmt.Fprintln(&buf, "source by a test in both directions. The table is computed, not maintained by") + fmt.Fprintln(&buf, "hand.") fmt.Fprintln(&buf) fmt.Fprintln(&buf, "The failure contract shared by plans, event streams, terminal results, status and") fmt.Fprintln(&buf, "doctor. Each carries a stable code and one safe command in its semantic role; diagnostic") fmt.Fprintln(&buf, "detail stays in restricted local evidence, never in the public record.") fmt.Fprintln(&buf) - fmt.Fprintln(&buf, "| Code | Reachable | Means | Guidance role | Command |") - fmt.Fprintln(&buf, "| --- | --- | --- | --- | --- |") + fmt.Fprintln(&buf, "| Code | Means | Guidance role | Command |") + fmt.Fprintln(&buf, "| --- | --- | --- | --- |") for _, code := range onebox.LifecycleFailureCodes() { // A code that will not resolve is a defect in the contract, not a row to // drop: a shorter table is one nobody can tell is incomplete. @@ -909,11 +902,7 @@ func renderErrorPage() string { if err != nil { panic(fmt.Sprintf("lifecycle code %q does not resolve: %v", code, err)) } - reach := "yes" - if onebox.LifecycleFailureReserved(code) { - reach = "reserved" - } - fmt.Fprintf(&buf, "| `%s` | %s | %s | %s | `%s` |\n", code, reach, escapeCell(failure.Message), failure.GuidanceRole(), failure.GuidanceCommand()) + fmt.Fprintf(&buf, "| `%s` | %s | %s | `%s` |\n", code, escapeCell(failure.Message), failure.GuidanceRole(), failure.GuidanceCommand()) } return buf.String() diff --git a/cmd/ob-scheduled-runner/main.go b/cmd/ob-scheduled-runner/main.go deleted file mode 100644 index 5bb9d352..00000000 --- a/cmd/ob-scheduled-runner/main.go +++ /dev/null @@ -1,81 +0,0 @@ -package main - -import ( - "context" - "crypto/rand" - "errors" - "fmt" - "os" - "os/signal" - "syscall" - - "github.com/spf13/cobra" - - "github.com/labstack/onebox/internal/buildinfo" - "github.com/labstack/onebox/internal/onebox" -) - -type unavailableScheduledExecutor struct{} - -func (unavailableScheduledExecutor) ExecuteScheduledLifecycle(context.Context, onebox.ScheduledLifecycleExecution) error { - return errors.New("scheduled lifecycle backend is not available for this operation in the current build") -} - -func executeEnvelope(ctx context.Context, path string) error { - envelope, err := onebox.LoadScheduledOperationEnvelope(path) - if err != nil { - return fmt.Errorf("load scheduled envelope: %w", err) - } - service := onebox.New(onebox.Options{ScheduledLifecycleExecutor: unavailableScheduledExecutor{}}) - runner := onebox.ScheduledRunner{Executor: service} - return runner.ExecuteRecurring(ctx, envelope, rand.Reader) -} - -func newRootCmd(runEnvelope func(context.Context, string) error) *cobra.Command { - if runEnvelope == nil { - runEnvelope = executeEnvelope - } - root := &cobra.Command{ - Use: "ob-scheduled-runner", - Short: "short-lived Onebox scheduled lifecycle runner", - SilenceUsage: true, - SilenceErrors: true, - Args: cobra.NoArgs, - } - run := &cobra.Command{ - Use: "run ", - Short: "execute one sealed scheduled operation and exit", - Args: cobra.ExactArgs(1), - RunE: func(cmd *cobra.Command, args []string) error { - return runEnvelope(cmd.Context(), args[0]) - }, - } - version := &cobra.Command{ - Use: "version", - Short: "print runner and protocol versions", - Args: cobra.NoArgs, - RunE: func(cmd *cobra.Command, _ []string) error { - info := buildinfo.Read() - compatibility := onebox.CurrentScheduledRunnerCompatibility() - _, err := fmt.Fprintf(cmd.OutOrStdout(), "%s runner_protocol=%d envelope_protocols=%d-%d cli_protocols=%d-%d\n", - info.Version, compatibility.RunnerProtocol, - compatibility.EnvelopeProtocols.Minimum, compatibility.EnvelopeProtocols.Maximum, - compatibility.CLIProtocols.Minimum, compatibility.CLIProtocols.Maximum) - return err - }, - } - root.AddCommand(run, version) - return root -} - -func main() { - ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) - defer stop() - if err := newRootCmd(nil).ExecuteContext(ctx); err != nil { - fmt.Fprintln(os.Stderr, "ob-scheduled-runner:", err) - if errors.Is(err, context.Canceled) || errors.Is(ctx.Err(), context.Canceled) { - os.Exit(130) - } - os.Exit(1) - } -} diff --git a/cmd/ob-scheduled-runner/main_test.go b/cmd/ob-scheduled-runner/main_test.go deleted file mode 100644 index 5aa643cf..00000000 --- a/cmd/ob-scheduled-runner/main_test.go +++ /dev/null @@ -1,47 +0,0 @@ -package main - -import ( - "context" - "reflect" - "sort" - "testing" -) - -func TestScheduledRunnerCommandSurfaceIsRestricted(t *testing.T) { - root := newRootCmd(func(context.Context, string) error { return nil }) - var names []string - for _, command := range root.Commands() { - names = append(names, command.Name()) - } - sort.Strings(names) - if !reflect.DeepEqual(names, []string{"run", "version"}) { - t.Fatalf("runner commands = %#v", names) - } - for _, forbidden := range []string{"plan", "deploy", "approve", "bootstrap", "destroy", "serve", "listen"} { - for _, name := range names { - if name == forbidden { - t.Fatalf("scheduled runner exposes forbidden command %q", forbidden) - } - } - } -} - -func TestScheduledRunnerRunsExactlyOneEnvelope(t *testing.T) { - var calls []string - root := newRootCmd(func(_ context.Context, path string) error { - calls = append(calls, path) - return nil - }) - root.SetArgs([]string{"run", "/var/lib/onebox/example/protection/envelope.json"}) - if err := root.ExecuteContext(context.Background()); err != nil { - t.Fatal(err) - } - if !reflect.DeepEqual(calls, []string{"/var/lib/onebox/example/protection/envelope.json"}) { - t.Fatalf("runner calls = %#v", calls) - } - root = newRootCmd(func(context.Context, string) error { return nil }) - root.SetArgs([]string{"run", "one", "two"}) - if err := root.ExecuteContext(context.Background()); err == nil { - t.Fatal("runner accepted more than one envelope") - } -} diff --git a/cmd/ob/backup.go b/cmd/ob/backup.go new file mode 100644 index 00000000..944b5822 --- /dev/null +++ b/cmd/ob/backup.go @@ -0,0 +1,266 @@ +package main + +import ( + "encoding/json" + "fmt" + "text/tabwriter" + "time" + + "github.com/spf13/cobra" + + "github.com/labstack/onebox/internal/onebox" +) + +// `ob backup` is the operator's whole view of backup. +// +// Three verbs, and the split between them is the point. `enable` is the one +// that changes what the server is — it restarts the database under the +// protected image and does not return until a recoverable backup exists. +// `create` takes another one. `status` asks the repository, not the project, +// what can actually be recovered; every figure comes from the repository itself. +// +// There is deliberately no verb that reports backup as established from the +// project alone. A policy in `ob.yml` is a request, and until `enable` has +// succeeded the service renders as an ordinary unprotected server. +func addBackupCommands(root *cobra.Command, g *globalFlags) { + backupCmd := &cobra.Command{ + Use: "backup", + Short: "protect a data service and inspect what can be recovered", + Long: "Backup and recovery for the data services this project declares.\n\n" + + "Backup is physical: a base backup plus continuous WAL archiving to the\n" + + "off-host repository the project's backup_targets name, which is what makes\n" + + "recovery to a point in time possible rather than recovery to last night.\n\n" + + "Declaring a policy does not establish it. `ob backup enable` restarts the\n" + + "service with archiving on, stages the verified backup tooling, and takes\n" + + "the first base backup; only then does the service render as protected.", + Args: cobra.NoArgs, RunE: showCommandHelp, + } + + var enableBreakLock bool + enableCmd := &cobra.Command{ + Use: "enable ", + Short: "establish backup — restarts the service archiving and takes the first backup", + Long: "Make a declared backup policy real.\n\n" + + "The order is forced: the credentials are checked, the image is pinned by\n" + + "registry digest, the verified wal-g binary is staged on the host, and only\n" + + "then does the server restart with archiving on.\n\n" + + "The restart is a real restart of the database. It is not complete until the\n" + + "first base backup exists, because WAL archiving with nothing to replay onto\n" + + "can recover nothing.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + return runMutation(cmd, g, onebox.ExecuteRequest{ + Kind: onebox.KindBackupEnable, Service: args[0], BreakLock: enableBreakLock, + }, "backup enable") + }, + } + enableCmd.Flags().BoolVar(&enableBreakLock, "break-lock", false, "break a stale operation lock after inspecting its holder") + backupCmd.AddCommand(enableCmd) + + var createBreakLock bool + createCmd := &cobra.Command{ + Use: "create ", + Short: "take a base backup now", + Long: "Take a base backup of a protected service.\n\n" + + "Every base backup is complete: the space between them is covered by the WAL\n" + + "stream rather than by differential backups, so there is no type to choose.\n\n" + + "WAL archiving runs continuously and is not this command. Between backups the\n" + + "recoverable point keeps advancing on its own; a base backup bounds how much\n" + + "WAL a recovery has to replay, and how far back the window reaches.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + return runMutation(cmd, g, onebox.ExecuteRequest{ + Kind: onebox.KindBackupCreate, Service: args[0], BreakLock: createBreakLock, + }, "backup create") + }, + } + createCmd.Flags().BoolVar(&createBreakLock, "break-lock", false, "break a stale operation lock after inspecting its holder") + backupCmd.AddCommand(createCmd) + + var disableConfirm string + var disableBreakLock bool + disableCmd := &cobra.Command{ + Use: "disable ", + Short: "stop archiving; keep every backup already taken", + Long: "Take a service out of backup.\n\n" + + "Archiving stops, the schedules are removed, the service restarts as an\n" + + "ordinary unprotected one, and its destination credentials are removed from\n" + + "the host.\n\n" + + "The repository is not touched: every backup already taken stays where it\n" + + "is. Reading or recovering from them needs backup enabled again,\n" + + "because the binary and credentials that reach the repository live in the\n" + + "protected service.\n\n" + + "What does stop is the recovery window advancing: from here on there is no\n" + + "new WAL, so the newest recoverable point is the moment this ran.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + if disableConfirm != args[0] { + return fmt.Errorf( + "disabling backup for %s stops archiving, so the recovery window stops advancing from now.\n"+ + "Re-run with --confirm %s once you mean it", + args[0], args[0]) + } + return runMutation(cmd, g, onebox.ExecuteRequest{ + Kind: onebox.KindBackupDisable, Service: args[0], BreakLock: disableBreakLock, + }, "backup disable") + }, + } + disableCmd.Flags().StringVar(&disableConfirm, "confirm", "", "name of the service whose archiving may stop") + // Without this a stale lock leaves no way to stop archiving, which is the + // one operation an operator reaches for when something has already gone + // wrong. + disableCmd.Flags().BoolVar(&disableBreakLock, "break-lock", false, "break a stale operation lock after inspecting its holder") + backupCmd.AddCommand(disableCmd) + + var pruneBreakLock bool + pruneCmd := &cobra.Command{ + Use: "prune ", + Short: "expire backups outside the declared retention", + Long: "Expire everything the policy no longer promises to keep.\n\n" + + "Retention comes from services..backup.retention.keep,\n" + + "so this never removes more than the project says it may keep fewer of.\n\n" + + "WAL older than the oldest retained backup goes with it: WAL that cannot be\n" + + "replayed onto any surviving base backup recovers nothing and only costs storage.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + return runMutation(cmd, g, onebox.ExecuteRequest{ + Kind: onebox.KindBackupPrune, Service: args[0], BreakLock: pruneBreakLock, + }, "backup prune") + }, + } + pruneCmd.Flags().BoolVar(&pruneBreakLock, "break-lock", false, "break a stale operation lock after inspecting its holder") + backupCmd.AddCommand(pruneCmd) + + verifyCmd := &cobra.Command{ + Use: "verify ", + Short: "prove the archived WAL forms an unbroken chain", + Long: "Check that the WAL in the repository is continuous.\n\n" + + "This is the check worth running on a schedule, and it is not implied by a\n" + + "backup that exited zero. A base backup with a gapped WAL stream recovers to\n" + + "the backup and no further — a nightly snapshot wearing the label of\n" + + "point-in-time recovery — and nothing else notices until someone needs it.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + return runMutation(cmd, g, onebox.ExecuteRequest{ + Kind: onebox.KindAssuranceCheck, Service: args[0], + }, "backup verify") + }, + } + backupCmd.AddCommand(verifyCmd) + + var restoreTo, restoreConfirm string + var restoreBreakLock bool + restoreCmd := &cobra.Command{ + Use: "restore ", + Short: "recover to a point in time and put it in service", + Long: "Recover a protected service from its repository.\n\n" + + "The recovered cluster is always built beside the live one, never over it:\n" + + "the base backup is fetched into a fresh volume, WAL is replayed to the\n" + + "requested point, and the result has to start and answer a query before\n" + + "anything touches the running database. A repository that cannot recover\n" + + "fails while the database it would have replaced is still serving.\n\n" + + "The data being replaced is copied aside first, under a dated volume name,\n" + + "and never deleted. A restore is run on a day that is already going badly;\n" + + "it must not be the step that makes it unrecoverable.\n\n" + + "Without --to, recovery goes to the newest recoverable point.\n\n" + + "The service name has to be typed back with --confirm. Onebox's approval flow\n" + + "binds a recorded confirmation to an exact plan, and a recovery has no plan to\n" + + "bind to — so the guard is the name of the thing being replaced, which cannot\n" + + "be given by accident or by a shell history entry meant for another service.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + if restoreConfirm != args[0] { + return fmt.Errorf( + "restoring replaces the live data of service %s.\n"+ + "Re-run with --confirm %s once you mean it, or use `ob backup drill %s` to prove the repository recovers without touching anything", + args[0], args[0], args[0]) + } + return runMutation(cmd, g, onebox.ExecuteRequest{ + Kind: onebox.KindRestoreCutover, Service: args[0], + RecoveryTarget: restoreTo, BreakLock: restoreBreakLock, + }, "backup restore") + }, + } + restoreCmd.Flags().StringVar(&restoreTo, "to", "", "RFC 3339 point in time to recover to (default: the newest recoverable point)") + restoreCmd.Flags().StringVar(&restoreConfirm, "confirm", "", "name of the service whose live data may be replaced") + restoreCmd.Flags().BoolVar(&restoreBreakLock, "break-lock", false, "break a stale operation lock after inspecting its holder") + backupCmd.AddCommand(restoreCmd) + + var drillTo string + drillCmd := &cobra.Command{ + Use: "drill ", + Short: "prove the repository recovers, without touching anything", + Long: "Recover into a throwaway volume, prove the cluster opens and answers, then\n" + + "discard it. The live service is never touched.\n\n" + + "This runs the same code as `ob backup restore` and stops before the last\n" + + "step, which is the point: a drill that exercised a different path would\n" + + "prove the drill works rather than that the backups do.\n\n" + + "A backup nobody has restored is a hypothesis.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + return runMutation(cmd, g, onebox.ExecuteRequest{ + Kind: onebox.KindRestoreTest, Service: args[0], RecoveryTarget: drillTo, + }, "backup drill") + }, + } + drillCmd.Flags().StringVar(&drillTo, "to", "", "RFC 3339 point in time to prove recoverable (default: the newest recoverable point)") + backupCmd.AddCommand(drillCmd) + + statusCmd := &cobra.Command{ + Use: "status ", + Short: "what the repository can recover, read from the repository", + Long: "Report what is actually recoverable.\n\n" + + "Every figure comes from the repository rather than from the project: the\n" + + "policy states what should be true, and this states what is. A service whose\n" + + "policy is declared but never enabled has no repository to ask, and says so.", + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + status, err := operationsService(cmd, g).BackupStatus(cmd.Context(), args[0]) + if err != nil { + return err + } + if isStructuredOutput(g) { + encoder := json.NewEncoder(cmd.OutOrStdout()) + encoder.SetIndent("", " ") + return encoder.Encode(status) + } + out := cmd.OutOrStdout() + fmt.Fprintf(out, "service %s\nrepository %s\n", status.Service, status.Repository) + for _, issue := range status.RuntimeIssues { + fmt.Fprintf(out, "drift %s\n", issue) + } + if len(status.Generations) == 0 { + fmt.Fprintln(out, "\nno recoverable base backup yet") + return nil + } + fmt.Fprintf(out, "recoverable to %s or later, as far as the archived WAL reaches\n", status.RecoverableTo) + // The declared window beside what the repository actually holds. A + // report that states only the second leaves the reader to work out + // whether their policy is being kept, which is the question they + // came with. + if status.DeclaredWindow != "" && status.OldestRecoverable != "" { + reach := "covers the declared window" + if !status.WindowCovered { + reach = "shorter than the declared window — this repository does not reach that far back yet" + } + fmt.Fprintf(out, "history %s onwards; declared window %s: %s\n", + status.OldestRecoverable, status.DeclaredWindow, reach) + } + if status.DeclaredMaxDataLoss != "" { + fmt.Fprintf(out, "data loss at most %s declared; the write-ahead log is archived continuously and every drift in that is listed above\n", + status.DeclaredMaxDataLoss) + } + fmt.Fprintln(out) + w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0) + fmt.Fprintln(w, "BACKUP\tCOMPLETED\tFROM WAL") + for _, generation := range status.Generations { + fmt.Fprintf(w, "%s\t%s\t%s\n", generation.Label, + time.Unix(generation.StoppedAt, 0).UTC().Format(time.RFC3339), generation.WALStart) + } + return w.Flush() + }, + } + backupCmd.AddCommand(statusCmd) + + root.AddCommand(backupCmd) +} diff --git a/cmd/ob/commands.go b/cmd/ob/commands.go index e2915958..a0eb02a9 100644 --- a/cmd/ob/commands.go +++ b/cmd/ob/commands.go @@ -116,7 +116,7 @@ func addCommands(root *cobra.Command, g *globalFlags) { }, } planCmd.Flags().StringVarP(&planOut, "out", "o", "ob-plan.json", "plan artifact path") - planCmd.Flags().StringVar(&planBackupReportOut, "backup-report-out", "", "write a plan-bound backup report template when migration protection is required") + planCmd.Flags().StringVar(&planBackupReportOut, "backup-report-out", "", "write a plan-bound backup report template when migration backup is required") planCmd.Flags().StringArrayVar(&imageArgs, "image", nil, "resolved image as workload=reference, for build-sourced workloads (repeatable)") root.AddCommand(planCmd) @@ -358,8 +358,8 @@ func renderDeployPlan(cmd *cobra.Command, u *ui.UI, plan onebox.DeployPlan) { u.Println(" " + step.ID + detail) } if plan.MigrationBackup != nil { - u.Println(fmt.Sprintf(" migration_backup=maximum_age:%s restore_test:%t resources:%d keys:%d", - plan.MigrationBackup.MaximumAge, plan.MigrationBackup.RequireRestoreTest, + u.Println(fmt.Sprintf(" migration_backup=max_age:%s restore_test:%t resources:%d keys:%d", + plan.MigrationBackup.MaxAge, plan.MigrationBackup.RequireRestoreTest, len(plan.MigrationBackup.Resources), len(plan.MigrationBackup.RequiredKeyMaterial))) } fmt.Fprintln(out) diff --git a/cmd/ob/doctor.go b/cmd/ob/doctor.go index a86dbbba..12d910c7 100644 --- a/cmd/ob/doctor.go +++ b/cmd/ob/doctor.go @@ -33,12 +33,12 @@ const ( ) type doctorReport struct { - Status doctorStatus `json:"status"` - Binary doctorBinaryReport `json:"binary"` - SSHAgent doctorSSHAgentReport `json:"ssh_agent"` - Project doctorProjectReport `json:"project"` - Approval doctorApprovalReport `json:"approval"` - Protections doctorProtectionsReport `json:"protections"` + Status doctorStatus `json:"status"` + Binary doctorBinaryReport `json:"binary"` + SSHAgent doctorSSHAgentReport `json:"ssh_agent"` + Project doctorProjectReport `json:"project"` + Approval doctorApprovalReport `json:"approval"` + Backups doctorBackupsReport `json:"backups"` } type doctorBinaryReport struct { @@ -78,17 +78,17 @@ type doctorSSHAgentReport struct { } type doctorProjectReport struct { - Status doctorStatus `json:"status"` - Message string `json:"message"` - Path string `json:"path"` - Environment string `json:"environment"` - Found bool `json:"found"` - Valid bool `json:"valid"` - Compatible bool `json:"compatible"` - APIVersion string `json:"api_version"` - Application string `json:"application"` - MinimumOneboxVersion string `json:"minimum_onebox_version"` - MinimumPlanSchema string `json:"minimum_plan_schema"` + Status doctorStatus `json:"status"` + Message string `json:"message"` + Path string `json:"path"` + Environment string `json:"environment"` + Found bool `json:"found"` + Valid bool `json:"valid"` + Compatible bool `json:"compatible"` + APIVersion string `json:"api_version"` + Application string `json:"application"` + MinOneboxVersion string `json:"min_onebox_version"` + MinPlanSchema string `json:"min_plan_schema"` } type doctorApprovalReport struct { @@ -101,13 +101,13 @@ type doctorApprovalReport struct { ConfirmationSchemaVersion string `json:"confirmation_schema_version"` } -type doctorProtectionsReport struct { - Status doctorStatus `json:"status"` - Message string `json:"message"` - Checks []doctorProtectionCheck `json:"checks"` +type doctorBackupsReport struct { + Status doctorStatus `json:"status"` + Message string `json:"message"` + Checks []doctorBackupCheck `json:"checks"` } -type doctorProtectionCheck struct { +type doctorBackupCheck struct { Status doctorStatus `json:"status"` Workload string `json:"workload"` Mechanism string `json:"mechanism"` @@ -147,7 +147,7 @@ func addDoctorCommand(root *cobra.Command, g *globalFlags) { cmd := &cobra.Command{ Use: "doctor", Short: "check local runner provenance and deployment safety capabilities", - Long: "Check this runner and the safety capabilities of the environment it targets.\n\nReports the runner's provenance and whether it satisfies the environment's\nminimum version and plan schema, and names every workload and service holding\ndurable data that has no backup — Onebox does not take backups, and silence\nthere would read as approval. In structured output, automation should gate on\nthe report status: data.status for pass or warn, and error.details.status for\na failing diagnosis.", + Long: "Check this runner and the safety capabilities of the environment it targets.\n\nReports the runner's provenance and whether it satisfies the environment's\nminimum version and plan schema, and names every workload and service holding\ndurable data that nothing is copying off the box, because silence there would\nread as approval. A service declaring `backup` is reported as declaring it;\nwhat the repository can actually recover is `ob backup status`. In structured output, automation should gate on\nthe report status: data.status for pass or warn, and error.details.status for\na failing diagnosis.", Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { report := buildDoctorReport(cmd.Context(), g, newDoctorDependencies()) @@ -178,16 +178,16 @@ func buildDoctorReport(ctx context.Context, g *globalFlags, deps doctorDependenc sshAgent := inspectDoctorSSHAgent(ctx, deps) project, cfg, environment := inspectDoctorProject(g, deps) approval := inspectDoctorApproval(environment) - protections := inspectDoctorProtections(cfg, project.Path, deps) + backups := inspectDoctorBackups(cfg, project.Path, deps) report := doctorReport{ - Status: doctorPass, - Binary: binary, - SSHAgent: sshAgent, - Project: project, - Approval: approval, - Protections: protections, - } - for _, status := range []doctorStatus{binary.Status, sshAgent.Status, project.Status, approval.Status, protections.Status} { + Status: doctorPass, + Binary: binary, + SSHAgent: sshAgent, + Project: project, + Approval: approval, + Backups: backups, + } + for _, status := range []doctorStatus{binary.Status, sshAgent.Status, project.Status, approval.Status, backups.Status} { report.Status = worseDoctorStatus(report.Status, status) } return report @@ -438,8 +438,8 @@ func inspectDoctorProject(g *globalFlags, deps doctorDependencies) (doctorProjec report.Message = err.Error() return report, cfg, nil } - report.MinimumOneboxVersion = environment.Policy.MinimumOneboxVersion - report.MinimumPlanSchema = environment.Policy.MinimumPlanSchema + report.MinOneboxVersion = environment.Policy.MinOneboxVersion + report.MinPlanSchema = environment.Policy.MinPlanSchema if err := onebox.CheckRunnerCompatibility(environment.Policy, deps.runner); err != nil { report.Status = doctorFail report.Message = err.Error() @@ -472,16 +472,17 @@ func inspectDoctorApproval(environment *app.Environment) doctorApprovalReport { return report } -func inspectDoctorProtections(cfg *app.Spec, configPath string, deps doctorDependencies) doctorProtectionsReport { - report := doctorProtectionsReport{Status: doctorPass, Checks: []doctorProtectionCheck{}} +func inspectDoctorBackups(cfg *app.Spec, configPath string, deps doctorDependencies) doctorBackupsReport { + report := doctorBackupsReport{Status: doctorPass, Checks: []doctorBackupCheck{}} if cfg == nil { report.Status = doctorWarning - report.Message = "declared protection mechanisms were not evaluated because project config is unavailable" + report.Message = "declared backup mechanisms were not evaluated because project config is unavailable" return report } // Durable data with nothing copying it off the box is worth saying out - // loud. Onebox does not take backups, and a workload whose data only + // loud. Onebox backs up protected *services*; a workload's own volume is + // not something it copies anywhere, and a workload whose data only // exists on one machine is one disk away from gone — silence here would // read as approval. componentNames := make([]string, 0, len(cfg.Workloads)) @@ -492,7 +493,7 @@ func inspectDoctorProtections(cfg *app.Spec, configPath string, deps doctorDepen for _, name := range componentNames { w := cfg.Workloads[name] if w.HoldsDurableData() { - message := "holds durable data and Onebox takes no backups; copy it off this host yourself" + message := "holds durable data that Onebox does not copy off this host; back it up yourself, or hold the state in a managed service that declares backup" switch { case w.Replicas > 1: // Do not suggest declaring durable here: the loader refuses a @@ -504,22 +505,43 @@ func inspectDoctorProtections(cfg *app.Spec, configPath string, deps doctorDepen case w.Persistence == nil: message += ". Declare persistence: {mode: durable} to state this, or mode: ephemeral if the volume is not state" } - report.Checks = append(report.Checks, doctorProtectionCheck{ + report.Checks = append(report.Checks, doctorBackupCheck{ Status: doctorWarning, Workload: name, Mechanism: "backup", Available: false, Message: message, }) } if w.HasBindMounts() { - report.Checks = append(report.Checks, doctorProtectionCheck{ + report.Checks = append(report.Checks, doctorBackupCheck{ Status: doctorPass, Workload: name, Mechanism: "bind_mount", Available: true, Message: "mounts a host path Onebox does not own; its contents are yours to back up", }) } } + // A service that declares backup is not the same as one that does not, + // and this said otherwise for both — it warned that "Onebox takes no + // backups yet" over a database archiving to an off-host repository. A + // doctor that reports a healthy thing as broken is a doctor people stop + // reading. + // + // What it still cannot say is whether backup was ever *enabled*: that + // is durable state on the target and this check is local. So a declared + // policy is reported as declared, and the operator is pointed at the one + // command that reads the repository itself. for _, name := range cfg.ServiceNames() { - report.Checks = append(report.Checks, doctorProtectionCheck{ - Status: doctorWarning, Workload: name, Mechanism: "backup", Available: false, - Message: "managed service data lives only on this host; Onebox takes no backups yet", + service := cfg.Services[name] + if service.Backup == nil { + report.Checks = append(report.Checks, doctorBackupCheck{ + Status: doctorWarning, Workload: name, Mechanism: "backup", Available: false, + Message: "managed service data lives only on this host; declare services." + name + + ".backup to copy it off, or accept that one disk is all there is", + }) + continue + } + report.Checks = append(report.Checks, doctorBackupCheck{ + Status: doctorPass, Workload: name, Mechanism: "backup", Available: true, + Message: "declares " + service.Backup.RecoveryKind + " backup to target " + + service.Backup.Target + "; run `ob backup status " + name + + "` to see what the repository can actually recover", }) } @@ -529,7 +551,7 @@ func inspectDoctorProtections(cfg *app.Spec, configPath string, deps doctorDepen } _, sourceErr := deps.stat(source) sopsPath, sopsErr := deps.lookPath("sops") - check := doctorProtectionCheck{Workload: "", Mechanism: "sops_secrets"} + check := doctorBackupCheck{Workload: "", Mechanism: "sops_secrets"} switch { case sourceErr != nil: check.Status = doctorFail @@ -546,7 +568,7 @@ func inspectDoctorProtections(cfg *app.Spec, configPath string, deps doctorDepen } if cfg.Runtime != nil && len(cfg.Runtime.EnvChecks) > 0 { - check := doctorProtectionCheck{Mechanism: "runtime_env_checks"} + check := doctorBackupCheck{Mechanism: "runtime_env_checks"} if err := cfg.RunPreflight(filepath.Dir(configPath)); err != nil { check.Status = doctorFail check.Message = err.Error() @@ -564,11 +586,11 @@ func inspectDoctorProtections(cfg *app.Spec, configPath string, deps doctorDepen if len(report.Checks) == 0 { report.Message = "nothing on this host holds durable data" } else if report.Status == doctorFail { - report.Message = "one or more declared protection mechanisms are unavailable" + report.Message = "one or more declared backup mechanisms are unavailable" } else if report.Status == doctorWarning { report.Message = "durable data is present and Onebox does not back it up" } else { - report.Message = "declared local protection mechanisms are available" + report.Message = "declared local backup mechanisms are available" } return report } @@ -618,8 +640,8 @@ func formatDoctorReport(report doctorReport) string { fmt.Fprintf(&out, "%s project: %s\n", doctorStatusLabel(report.Project.Status), report.Project.Message) fmt.Fprintf(&out, " config: %s environment: %s\n", report.Project.Path, report.Project.Environment) fmt.Fprintf(&out, "%s approval: %s\n", doctorStatusLabel(report.Approval.Status), report.Approval.Message) - fmt.Fprintf(&out, "%s protections: %s\n", doctorStatusLabel(report.Protections.Status), report.Protections.Message) - for _, check := range report.Protections.Checks { + fmt.Fprintf(&out, "%s backups: %s\n", doctorStatusLabel(report.Backups.Status), report.Backups.Message) + for _, check := range report.Backups.Checks { name := check.Mechanism if check.Workload != "" { name = check.Workload + "/" + name diff --git a/cmd/ob/doctor_test.go b/cmd/ob/doctor_test.go index 6fb851eb..124eb52a 100644 --- a/cmd/ob/doctor_test.go +++ b/cmd/ob/doctor_test.go @@ -36,9 +36,9 @@ func doctorTestDependencies(t *testing.T) doctorDependencies { Environments: map[string]app.Environment{ "production": { Policy: app.Policy{ - RequireApproval: true, - MinimumOneboxVersion: "v2026.7.1", - MinimumPlanSchema: "onebox.run/executable-deploy-plan/v1alpha1", + RequireApproval: true, + MinOneboxVersion: "v2026.7.1", + MinPlanSchema: "onebox.run/executable-deploy-plan/v1alpha1", }, }, }, @@ -79,7 +79,7 @@ func doctorTestDependencies(t *testing.T) doctorDependencies { } } -func TestBuildDoctorReportFindsShadowingPolicyAndProtectionGaps(t *testing.T) { +func TestBuildDoctorReportFindsShadowingPolicyAndBackupGaps(t *testing.T) { deps := doctorTestDependencies(t) report := buildDoctorReport(context.Background(), &globalFlags{ConfigPath: "/project/ob.yml", Env: "production"}, deps) @@ -109,16 +109,16 @@ func TestBuildDoctorReportFindsShadowingPolicyAndProtectionGaps(t *testing.T) { } // Durable data with nothing copying it off the box must be said out loud; // silence would read as approval. - if report.Protections.Status != doctorWarning || len(report.Protections.Checks) != 1 { - t.Fatalf("protection report = %+v", report.Protections) + if report.Backups.Status != doctorWarning || len(report.Backups.Checks) != 1 { + t.Fatalf("backup report = %+v", report.Backups) } for _, mechanism := range []string{"backup"} { found := false - for _, check := range report.Protections.Checks { + for _, check := range report.Backups.Checks { found = found || check.Mechanism == mechanism && !check.Available } if !found { - t.Fatalf("missing unavailable %s check: %+v", mechanism, report.Protections.Checks) + t.Fatalf("missing unavailable %s check: %+v", mechanism, report.Backups.Checks) } } } @@ -170,7 +170,7 @@ func TestDoctorHumanOutputNamesEveryDiagnosticArea(t *testing.T) { "ssh-agent:", "project:", "approval:", - "protections:", + "backups:", "database/backup", } { if !strings.Contains(out, want) { @@ -186,8 +186,8 @@ func TestDoctorReportsMissingProjectWithoutAborting(t *testing.T) { if report.Project.Status != doctorWarning || report.Project.Found { t.Fatalf("project report = %+v", report.Project) } - if report.Protections.Status != doctorWarning { - t.Fatalf("protection report = %+v", report.Protections) + if report.Backups.Status != doctorWarning { + t.Fatalf("backup report = %+v", report.Backups) } } @@ -197,7 +197,7 @@ func TestDoctorReportsIncompatibleProjectPolicy(t *testing.T) { return &app.Spec{ APIVersion: "onebox.run/v1", Environments: map[string]app.Environment{ - "production": {Policy: app.Policy{MinimumOneboxVersion: "v2027.1.0"}}, + "production": {Policy: app.Policy{MinOneboxVersion: "v2027.1.0"}}, }, Workloads: map[string]app.Workload{}, }, nil diff --git a/cmd/ob/job.go b/cmd/ob/job.go index 2ce4b8c0..00484d9a 100644 --- a/cmd/ob/job.go +++ b/cmd/ob/job.go @@ -31,7 +31,7 @@ func addJobCommand(root *cobra.Command, g *globalFlags) { }, } plan.Flags().StringVarP(&planOut, "out", "o", "ob-job-plan.json", "job plan artifact path") - plan.Flags().StringVar(&backupReportOut, "backup-report-out", "", "write a plan-bound backup report template when migration protection is required") + plan.Flags().StringVar(&backupReportOut, "backup-report-out", "", "write a plan-bound backup report template when migration backup is required") var planPath, approvalPath, backupReportPath, overrideReason string var breakLock bool diff --git a/cmd/ob/main.go b/cmd/ob/main.go index f1e4a8cd..d1bea48d 100644 --- a/cmd/ob/main.go +++ b/cmd/ob/main.go @@ -62,6 +62,7 @@ func newRootCmd() *cobra.Command { addJobCommand(root, g) addInitCommand(root, g) addOpsCommands(root, g) + addBackupCommands(root, g) addPreviewCommand(root, g) addSchemaCommand(root, g) addPreflightCommand(root, g) diff --git a/cmd/ob/output.go b/cmd/ob/output.go index 37ebb0f6..ee10b1a8 100644 --- a/cmd/ob/output.go +++ b/cmd/ob/output.go @@ -76,34 +76,42 @@ const ( // Cobra's native human output. Keeping this list closed makes adding a command // an explicit CLI-contract decision instead of silently inheriting behavior. var cliOutputMatrix = map[string]cliOutputClass{ - "ob abort": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob approve": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob audit": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob bootstrap": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob canonical": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob deploy": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob destroy": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob doctor": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob eject": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob exec": {Class: cliClassOperatorPassthrough, NDJSON: true}, - "ob init": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob job plan": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob job run": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob logs": {Class: cliClassOperatorPassthrough, JSON: true, NDJSON: true}, - "ob plan": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob preflight": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob preview": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob proxy apply": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob resume": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob rollback": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob schema": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob secrets edit": {Class: cliClassTrustedEditor, JSON: true}, - "ob secrets list": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob secrets push": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob service apply": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, - "ob status": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob validate": {Class: cliClassFiniteEnvelope, JSON: true}, - "ob version": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob abort": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob approve": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob audit": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob bootstrap": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob canonical": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob deploy": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob destroy": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob doctor": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob eject": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob exec": {Class: cliClassOperatorPassthrough, NDJSON: true}, + "ob init": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob backup create": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob backup enable": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob backup disable": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob backup drill": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob backup restore": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob backup prune": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob backup verify": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob backup status": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob job plan": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob job run": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob logs": {Class: cliClassOperatorPassthrough, JSON: true, NDJSON: true}, + "ob plan": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob preflight": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob preview": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob proxy apply": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob resume": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob rollback": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob schema": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob secrets edit": {Class: cliClassTrustedEditor, JSON: true}, + "ob secrets list": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob secrets push": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob service apply": {Class: cliClassFiniteStream, JSON: true, NDJSON: true}, + "ob status": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob validate": {Class: cliClassFiniteEnvelope, JSON: true}, + "ob version": {Class: cliClassFiniteEnvelope, JSON: true}, } type cliExitError struct { diff --git a/cmd/ob/output_test.go b/cmd/ob/output_test.go index 2c1ef4fe..bf428112 100644 --- a/cmd/ob/output_test.go +++ b/cmd/ob/output_test.go @@ -490,34 +490,42 @@ workloads: func TestLeafOutputMatrixIsClosedAndHasNoAliases(t *testing.T) { wantMatrix := map[string]cliOutputClass{ - "ob abort": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob approve": {Class: "finite_envelope", JSON: true}, - "ob audit": {Class: "finite_envelope", JSON: true}, - "ob bootstrap": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob canonical": {Class: "finite_envelope", JSON: true}, - "ob deploy": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob destroy": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob doctor": {Class: "finite_envelope", JSON: true}, - "ob eject": {Class: "finite_envelope", JSON: true}, - "ob exec": {Class: "operator_passthrough", NDJSON: true}, - "ob init": {Class: "finite_envelope", JSON: true}, - "ob job plan": {Class: "finite_envelope", JSON: true}, - "ob job run": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob logs": {Class: "operator_passthrough", JSON: true, NDJSON: true}, - "ob plan": {Class: "finite_envelope", JSON: true}, - "ob preflight": {Class: "finite_envelope", JSON: true}, - "ob preview": {Class: "finite_envelope", JSON: true}, - "ob proxy apply": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob resume": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob rollback": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob schema": {Class: "finite_envelope", JSON: true}, - "ob secrets edit": {Class: "trusted_editor", JSON: true}, - "ob secrets list": {Class: "finite_envelope", JSON: true}, - "ob secrets push": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob service apply": {Class: "finite_stream", JSON: true, NDJSON: true}, - "ob status": {Class: "finite_envelope", JSON: true}, - "ob validate": {Class: "finite_envelope", JSON: true}, - "ob version": {Class: "finite_envelope", JSON: true}, + "ob abort": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob approve": {Class: "finite_envelope", JSON: true}, + "ob audit": {Class: "finite_envelope", JSON: true}, + "ob bootstrap": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob canonical": {Class: "finite_envelope", JSON: true}, + "ob deploy": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob destroy": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob doctor": {Class: "finite_envelope", JSON: true}, + "ob eject": {Class: "finite_envelope", JSON: true}, + "ob exec": {Class: "operator_passthrough", NDJSON: true}, + "ob init": {Class: "finite_envelope", JSON: true}, + "ob job plan": {Class: "finite_envelope", JSON: true}, + "ob backup create": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob backup enable": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob backup disable": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob backup drill": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob backup restore": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob backup prune": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob backup verify": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob backup status": {Class: "finite_envelope", JSON: true}, + "ob job run": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob logs": {Class: "operator_passthrough", JSON: true, NDJSON: true}, + "ob plan": {Class: "finite_envelope", JSON: true}, + "ob preflight": {Class: "finite_envelope", JSON: true}, + "ob preview": {Class: "finite_envelope", JSON: true}, + "ob proxy apply": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob resume": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob rollback": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob schema": {Class: "finite_envelope", JSON: true}, + "ob secrets edit": {Class: "trusted_editor", JSON: true}, + "ob secrets list": {Class: "finite_envelope", JSON: true}, + "ob secrets push": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob service apply": {Class: "finite_stream", JSON: true, NDJSON: true}, + "ob status": {Class: "finite_envelope", JSON: true}, + "ob validate": {Class: "finite_envelope", JSON: true}, + "ob version": {Class: "finite_envelope", JSON: true}, } if !reflect.DeepEqual(cliOutputMatrix, wantMatrix) { t.Fatalf("CLI output matrix changed:\ngot %#v\nwant %#v", cliOutputMatrix, wantMatrix) diff --git a/cmd/ob/remedy_test.go b/cmd/ob/remedy_test.go index a5059894..4d9259c8 100644 --- a/cmd/ob/remedy_test.go +++ b/cmd/ob/remedy_test.go @@ -15,7 +15,7 @@ import ( // lifecycle validation checks guidance is shell-safe and starts with `ob `, // which is a claim about its form, not its truth. Eighteen of thirty-five // lifecycle codes passed that guard while naming verbs the CLI has never had — -// `ob backup inspect`, `ob protection enable`, `ob assurance status`. A code is +// `ob backup inspect`, `ob backup enable`, `ob assurance status`. A code is // read at the moment something is broken, so a remedy that exits with // `unknown command` costs the operator a round trip and their trust in the // rest of the page. diff --git a/docs/onebox.run-v1.schema.json b/docs/onebox.run-v1.schema.json index 23c6644d..e4887f01 100644 --- a/docs/onebox.run-v1.schema.json +++ b/docs/onebox.run-v1.schema.json @@ -191,29 +191,26 @@ }, "properties": { "cold": { - "description": "Encryption mode required for cold recovery: client-side, archive-password, or server-side-sse.", + "description": "Encryption mode required for cold recovery: client-side or server-side.", "enum": [ "client-side", - "archive-password", - "server-side-sse" + "server-side" ], "type": "string" }, "pitr": { - "description": "Encryption mode required for point-in-time recovery: client-side, archive-password, or server-side-sse.", + "description": "Encryption mode required for point-in-time recovery: client-side or server-side.", "enum": [ "client-side", - "archive-password", - "server-side-sse" + "server-side" ], "type": "string" }, "snapshot": { - "description": "Encryption mode required for snapshot recovery: client-side, archive-password, or server-side-sse.", + "description": "Encryption mode required for snapshot recovery: client-side or server-side.", "enum": [ "client-side", - "archive-password", - "server-side-sse" + "server-side" ], "type": "string" } @@ -265,7 +262,7 @@ "type": "string" }, "prefix": { - "description": "Non-secret object prefix reserved for Onebox protection data. Expects a relative object prefix with no empty leading component or shell metacharacter.", + "description": "Non-secret object prefix reserved for Onebox backup data. Expects a relative object prefix with no empty leading component or shell metacharacter.", "examples": [ "production/shop" ], @@ -281,18 +278,18 @@ "type": "string" }, "tls": { - "default": "required", - "description": "TLS verification policy: required or insecure.", + "default": "verify", + "description": "Transport policy: verify, or skip-verify to accept a plaintext http endpoint.", "enum": [ - "required", - "insecure" + "verify", + "skip-verify" ], "type": "string" } }, "type": "object" }, - "description": "User-owned off-host repositories available to service protection policies.", + "description": "User-owned off-host repositories available to service backup policies.", "type": "object" }, "base_path": { @@ -337,13 +334,6 @@ "pattern": "^[^/\\x00-\\x1f'\"$`\\\\][^\\x00-\\x1f'\"$`\\\\]*$", "type": "string" }, - "platform": { - "description": "Target image platform for the external build.", - "examples": [ - "linux/amd64" - ], - "type": "string" - }, "target": { "description": "Named Dockerfile stage to build.", "type": "string" @@ -354,6 +344,198 @@ ], "description": "Build metadata for development. Production requires a resolved image supplied with --image. Also accepts a build context path." }, + "checks": { + "additionalProperties": false, + "description": "Assertions that must pass before a release becomes current unless marked advisory.", + "patternProperties": { + "^x-": {} + }, + "properties": { + "exec": { + "description": "Commands run inside a named workload.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "advisory": { + "default": false, + "description": "Report a failure without blocking release activation.", + "type": "boolean" + }, + "run": { + "description": "Shell command verified inside the workload.", + "examples": [ + "test -f /srv/ready" + ], + "type": "string" + }, + "workload": { + "description": "Workload the command runs inside.", + "examples": [ + "web" + ], + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "http": { + "description": "HTTP paths probed inside a named workload.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "advisory": { + "default": false, + "description": "Report a failure without blocking release activation.", + "type": "boolean" + }, + "path": { + "description": "HTTP path verified inside the workload. Expects a path beginning with /.", + "examples": [ + "/healthz" + ], + "pattern": "^/[^\\x00-\\x1f'\"$` \\\\]*$", + "type": "string" + }, + "port": { + "description": "Container port to probe.", + "examples": [ + 3000 + ], + "maximum": 65535, + "minimum": 1, + "type": "integer" + }, + "workload": { + "description": "Workload the path is probed inside.", + "examples": [ + "web" + ], + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "migrations": { + "description": "Migration revisions checked against captured job evidence.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "advisory": { + "default": false, + "description": "Report a failure without blocking release activation.", + "type": "boolean" + }, + "applied_revisions": { + "description": "Revisions the job must report as applied.", + "items": { + "type": "string" + }, + "type": "array" + }, + "job": { + "description": "Job workload whose captured evidence is checked.", + "examples": [ + "migrate" + ], + "type": "string" + }, + "provider": { + "description": "Migration tool that produced the revisions.", + "examples": [ + "alembic" + ], + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "url": { + "description": "External URLs probed from the operator side.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "advisory": { + "default": false, + "description": "Report a failure without blocking release activation.", + "type": "boolean" + }, + "contains": { + "description": "Text the response body must contain.", + "type": "string" + }, + "json_assertions": { + "description": "Scalar JSON response values that must match exactly.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "equals": { + "description": "Exact scalar value required at path." + }, + "path": { + "description": "Dot-separated path to a scalar value in the JSON response.", + "examples": [ + "service.ready" + ], + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "required_headers": { + "additionalProperties": { + "type": "string" + }, + "description": "Exact response headers required for success.", + "type": "object" + }, + "status_codes": { + "description": "Allowed response status codes. A successful 2xx response is expected when omitted.", + "items": { + "maximum": 599, + "minimum": 100, + "type": "integer" + }, + "type": "array" + }, + "url": { + "description": "External HTTP or HTTPS URL verified from the operator side. Expects an http or https URL.", + "examples": [ + "https://shop.example.com/healthz" + ], + "pattern": "^https?://", + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + } + }, + "type": "object" + }, "compose": { "description": "Existing Compose service to adopt, as repository path#service. Expects a reference of the form path/to/compose.yaml#service.", "examples": [ @@ -497,23 +679,46 @@ "description": "Declared permission for agent-authored proposals. The current CLI does not distinguish agent identity; execution remains approval-gated.", "type": "boolean" }, - "migration_backup_key_material": { - "description": "Names of key material whose usability must be covered by the migration backup report.", - "items": { - "type": "string" + "migrations": { + "additionalProperties": false, + "description": "What this environment requires of a release carrying migration risk.", + "patternProperties": { + "^x-": {} }, - "type": "array" - }, - "migration_backup_maximum_age": { - "default": "24h", - "description": "Maximum age of a backup report accepted for a migration. Expects a duration such as 30s, 5m, 1h30m or 14d.", - "examples": [ - "24h" - ], - "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", - "type": "string" + "properties": { + "backup_key_material": { + "description": "Key-material identities the backup report must name.", + "examples": [ + "BACKUP_ACCESS_KEY_ID" + ], + "items": { + "type": "string" + }, + "type": "array" + }, + "backup_max_age": { + "default": "24h", + "description": "Maximum age of a backup report accepted for a migration. Expects a duration such as 30s, 5m, 1h30m or 14d.", + "examples": [ + "24h" + ], + "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", + "type": "string" + }, + "require_backup": { + "default": false, + "description": "Require a plan-bound backup report before a release with migration risk.", + "type": "boolean" + }, + "require_restore_test": { + "default": false, + "description": "Require the backup report to state that a restore test succeeded.", + "type": "boolean" + } + }, + "type": "object" }, - "minimum_onebox_version": { + "min_onebox_version": { "description": "Oldest released Onebox runner allowed to operate this environment. Expects a CalVer release such as v2026.8.0.", "examples": [ "v2026.8.0" @@ -521,7 +726,7 @@ "pattern": "^v([1-9][0-9]{3})\\.([1-9]|1[0-2])\\.(0|[1-9][0-9]{0,18})$", "type": "string" }, - "minimum_plan_schema": { + "min_plan_schema": { "description": "Oldest executable plan schema accepted by this environment. Expects a plan schema identity such as onebox.run/executable-deploy-plan/v1alpha2.", "examples": [ "onebox.run/executable-deploy-plan/v1alpha2" @@ -533,16 +738,6 @@ "default": true, "description": "Require a plan-bound local confirmation before mutating this environment.", "type": "boolean" - }, - "require_migration_backup": { - "default": false, - "description": "Require a plan-bound backup report before a release with migration risk.", - "type": "boolean" - }, - "require_migration_restore_test": { - "default": false, - "description": "Require the backup report to state that a restore test succeeded.", - "type": "boolean" } }, "type": "object" @@ -603,6 +798,14 @@ "^x-": {} }, "properties": { + "backup_owner": { + "description": "Operator or provider responsible for backup, restore, upgrades, credentials, and durability. Expects a stable operator or provider identity of letters, digits, dots, @, colons, slashes, underscores and hyphens.", + "examples": [ + "platform-team/rds" + ], + "pattern": "^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,127}$", + "type": "string" + }, "connection": { "additionalProperties": false, "description": "Trusted connection source and driver-shaped entry mapping.", @@ -683,7 +886,7 @@ ], "type": "string" }, - "maximum_age": { + "max_age": { "default": "5m", "description": "Maximum age of a probe observation bound into a plan. Expects a duration such as 30s, 5m, 1h30m or 14d.", "examples": [ @@ -703,19 +906,11 @@ } }, "type": "object" - }, - "protection_owner": { - "description": "Operator or provider responsible for backup, restore, upgrades, credentials, and durability. Expects a stable operator or provider identity of letters, digits, dots, @, colons, slashes, underscores and hyphens.", - "examples": [ - "platform-team/rds" - ], - "pattern": "^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,127}$", - "type": "string" } }, "type": "object" }, - "description": "Typed dependencies operated outside Onebox. Their connection projection is trusted, but their lifecycle and protection remain external.", + "description": "Typed dependencies operated outside Onebox. Their connection projection is trusted, but their lifecycle and backup remain external.", "type": "object" }, "health": { @@ -842,16 +1037,9 @@ "^x-": {} }, "properties": { - "platform": { - "description": "Platform selected when the image is multi-platform.", - "examples": [ - "linux/amd64" - ], - "type": "string" - }, "pull": { "default": "missing", - "description": "Image pull policy: missing, always, or never.", + "description": "When to fetch the image from the registry: missing fetches only what the host does not already hold, always fetches every release, never fetches at all and fails on a missing image.", "enum": [ "always", "missing", @@ -866,10 +1054,6 @@ ], "pattern": "^((?:(?:(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])(?:\\.(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]))*|\\[(?:[a-fA-F0-9:]+)\\])(?::[0-9]+)?/)?[a-z0-9]+(?:(?:[._]|__|[-]+)[a-z0-9]+)*(?:/[a-z0-9]+(?:(?:[._]|__|[-]+)[a-z0-9]+)*)*)(?::([\\w][\\w.-]{0,127}))?(?:@([A-Za-z][A-Za-z0-9]*(?:[-_+.][A-Za-z][A-Za-z0-9]*)*[:][[:xdigit:]]{32,}))?$", "type": "string" - }, - "registry": { - "description": "Optional registry label retained in canonical configuration. Current authentication uses every top-level registries entry; this field does not select a login.", - "type": "string" } }, "type": "object" @@ -918,72 +1102,6 @@ "description": "Named webhooks that receive selected operation outcomes.", "type": "object" }, - "observability": { - "additionalProperties": false, - "description": "Declared logging, metrics, and alerting intent. Continuous management is not currently provided.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "alerts": { - "additionalProperties": false, - "description": "Declared alerting intent. Continuous management is not currently provided.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "unhealthy_after": { - "description": "Desired duration of unhealthy state before alerting. Expects a duration such as 30s, 5m, 1h30m or 14d.", - "examples": [ - "5m" - ], - "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", - "type": "string" - } - }, - "type": "object" - }, - "logs": { - "additionalProperties": false, - "description": "Declared log-retention intent. Continuous management is not currently provided.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "enabled": { - "default": false, - "description": "Declare that log collection is desired.", - "type": "boolean" - }, - "retention": { - "description": "Desired log-retention period. Expects a duration such as 30s, 5m, 1h30m or 14d.", - "examples": [ - "30d" - ], - "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", - "type": "string" - } - }, - "type": "object" - }, - "metrics": { - "additionalProperties": false, - "description": "Declared metric-collection intent. Continuous management is not currently provided.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "enabled": { - "default": false, - "description": "Declare that metric collection is desired.", - "type": "boolean" - } - }, - "type": "object" - } - }, - "type": "object" - }, "port": { "description": "Container port used with domain shorthand and as the default HTTP health port.", "examples": [ @@ -1250,74 +1368,26 @@ "^x-": {} }, "properties": { - "driver": { - "description": "Built-in service driver. Defaults to the service map key. Expects lower-case letters, digits and hyphens, starting with a letter, at most 40 characters.", - "examples": [ - "postgres" - ], - "pattern": "^[a-z]([a-z0-9-]{0,38}[a-z0-9])?$", - "type": "string" - }, - "persistence": { + "backup": { "additionalProperties": false, - "description": "Data-lifetime declaration for this supporting service.", + "description": "Recovery intent for this service. Onebox selects the qualified native implementation; declaring intent alone does not establish backup.", "patternProperties": { "^x-": {} }, "properties": { - "mode": { - "default": "durable", - "description": "Data lifetime: durable, ephemeral, or external.", - "enum": [ - "durable", - "ephemeral", - "external" - ], - "type": "string" - } - }, - "type": "object" - }, - "protection": { - "additionalProperties": false, - "description": "Recovery intent for this service. Onebox selects the qualified native implementation; declaring intent alone does not establish protection.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "allow_backup_interruption": { + "allow_downtime": { "default": false, "description": "Whether recurring backup operations may use the driver-declared stopped-service window.", "type": "boolean" }, - "maximum_data_loss": { - "description": "Maximum tolerable interval between the latest recoverable point and failure. Expects a duration such as 30s, 5m, 1h30m or 14d.", - "examples": [ - "15m" - ], - "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", - "type": "string" - }, - "recovery_kind": { - "description": "Required recovery envelope: snapshot, pitr, or cold.", - "enum": [ - "snapshot", - "pitr", - "cold" - ], - "examples": [ - "pitr" - ], - "type": "string" - }, - "restore_drill": { + "drill": { "additionalProperties": false, "description": "Exact isolated restore-test schedule, proof age, and optional staging filesystem.", "patternProperties": { "^x-": {} }, "properties": { - "proof_maximum_age": { + "max_age": { "default": "7d", "description": "Maximum age of the latest passing restore proof. Expects a duration such as 30s, 5m, 1h30m or 14d.", "examples": [ @@ -1352,18 +1422,30 @@ } }, "type": "object" - }, - "staging_filesystem": { - "description": "Absolute filesystem path used for isolated restore materialization instead of the host default. Expects an absolute path with no control character or shell metacharacter.", - "examples": [ - "/srv/onebox-restore" - ], - "pattern": "^/[^\\x00-\\x1f'\"$`\\\\]*$", - "type": "string" } }, "type": "object" }, + "max_data_loss": { + "description": "Maximum tolerable interval between the latest recoverable point and failure. Expects a duration such as 30s, 5m, 1h30m or 14d.", + "examples": [ + "15m" + ], + "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", + "type": "string" + }, + "recovery_kind": { + "description": "Required recovery envelope: snapshot, pitr, or cold.", + "enum": [ + "snapshot", + "pitr", + "cold" + ], + "examples": [ + "pitr" + ], + "type": "string" + }, "retention": { "additionalProperties": false, "description": "Portable minimum recovery history that the selected native driver must be able to preserve.", @@ -1371,7 +1453,7 @@ "^x-": {} }, "properties": { - "minimum_generations": { + "keep": { "default": 7, "description": "Minimum number of independently recoverable base generations to retain.", "examples": [ @@ -1380,7 +1462,7 @@ "minimum": 1, "type": "integer" }, - "recovery_window": { + "window": { "default": "7d", "description": "Minimum continuous recovery history the native retention mapping must preserve. Expects a duration such as 30s, 5m, 1h30m or 14d.", "examples": [ @@ -1430,6 +1512,34 @@ }, "type": "object" }, + "driver": { + "description": "Built-in service driver. Defaults to the service map key. Expects lower-case letters, digits and hyphens, starting with a letter, at most 40 characters.", + "examples": [ + "postgres" + ], + "pattern": "^[a-z]([a-z0-9-]{0,38}[a-z0-9])?$", + "type": "string" + }, + "persistence": { + "additionalProperties": false, + "description": "Data-lifetime declaration for this supporting service.", + "patternProperties": { + "^x-": {} + }, + "properties": { + "mode": { + "default": "durable", + "description": "Data lifetime: durable, ephemeral, or external.", + "enum": [ + "durable", + "ephemeral", + "external" + ], + "type": "string" + } + }, + "type": "object" + }, "resources": { "additionalProperties": false, "description": "Memory and CPU limits for this supporting service.", @@ -1488,131 +1598,6 @@ "description": "Supporting services managed outside application releases, such as databases and caches.", "type": "object" }, - "verifications": { - "description": "Checks that must pass before a release becomes current unless marked advisory.", - "items": { - "additionalProperties": false, - "patternProperties": { - "^x-": {} - }, - "properties": { - "advisory": { - "default": false, - "description": "Report a failed check without blocking release activation.", - "type": "boolean" - }, - "contains": { - "description": "Text that the HTTP response body must contain.", - "type": "string" - }, - "exec": { - "description": "Shell command verified inside the named workload.", - "type": "string" - }, - "http": { - "description": "HTTP path verified inside the named workload. Expects a path beginning with /.", - "examples": [ - "/healthz" - ], - "pattern": "^/[^\\x00-\\x1f'\"$` \\\\]*$", - "type": "string" - }, - "json_assertions": { - "description": "Scalar JSON response values that must match exactly.", - "items": { - "additionalProperties": false, - "patternProperties": { - "^x-": {} - }, - "properties": { - "equals": { - "description": "Exact scalar value required at path." - }, - "path": { - "description": "Dot-separated path to a scalar value in the JSON response.", - "examples": [ - "service.ready" - ], - "type": "string" - } - }, - "type": "object" - }, - "type": "array" - }, - "migration_revisions": { - "additionalProperties": false, - "description": "Expected migration provider and applied revisions, checked against captured job evidence.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "applied_revisions": { - "description": "Ordered migration revisions expected to be applied.", - "items": { - "type": "string" - }, - "type": "array" - }, - "job": { - "description": "Migration job whose result evidence is checked.", - "examples": [ - "migrate" - ], - "type": "string" - }, - "provider": { - "description": "Migration provider expected in the job result.", - "examples": [ - "atlas" - ], - "type": "string" - } - }, - "type": "object" - }, - "port": { - "description": "Container port used by an internal HTTP verification.", - "examples": [ - 3000 - ], - "maximum": 65535, - "minimum": 1, - "type": "integer" - }, - "required_headers": { - "additionalProperties": { - "type": "string" - }, - "description": "Exact HTTP response headers required for success.", - "type": "object" - }, - "status_codes": { - "description": "Allowed HTTP response status codes. A successful 2xx response is expected when omitted.", - "items": { - "maximum": 599, - "minimum": 100, - "type": "integer" - }, - "type": "array" - }, - "url": { - "description": "External HTTP or HTTPS URL verified from the operator side. Expects an http or https URL.", - "examples": [ - "https://shop.example.com/healthz" - ], - "pattern": "^https?://", - "type": "string" - }, - "workload": { - "description": "Workload in which an internal HTTP or exec verification runs.", - "type": "string" - } - }, - "type": "object" - }, - "type": "array" - }, "workloads": { "additionalProperties": { "additionalProperties": false, @@ -1860,13 +1845,6 @@ "pattern": "^[^/\\x00-\\x1f'\"$`\\\\][^\\x00-\\x1f'\"$`\\\\]*$", "type": "string" }, - "platform": { - "description": "Target image platform for the external build.", - "examples": [ - "linux/amd64" - ], - "type": "string" - }, "target": { "description": "Named Dockerfile stage to build.", "type": "string" @@ -2131,16 +2109,9 @@ "^x-": {} }, "properties": { - "platform": { - "description": "Platform selected when the image is multi-platform.", - "examples": [ - "linux/amd64" - ], - "type": "string" - }, "pull": { "default": "missing", - "description": "Image pull policy: missing, always, or never.", + "description": "When to fetch the image from the registry: missing fetches only what the host does not already hold, always fetches every release, never fetches at all and fails on a missing image.", "enum": [ "always", "missing", @@ -2155,10 +2126,6 @@ ], "pattern": "^((?:(?:(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])(?:\\.(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]))*|\\[(?:[a-fA-F0-9:]+)\\])(?::[0-9]+)?/)?[a-z0-9]+(?:(?:[._]|__|[-]+)[a-z0-9]+)*(?:/[a-z0-9]+(?:(?:[._]|__|[-]+)[a-z0-9]+)*)*)(?::([\\w][\\w.-]{0,127}))?(?:@([A-Za-z][A-Za-z0-9]*(?:[-_+.][A-Za-z][A-Za-z0-9]*)*[:][[:xdigit:]]{32,}))?$", "type": "string" - }, - "registry": { - "description": "Optional registry label retained in canonical configuration. Current authentication uses every top-level registries entry; this field does not select a login.", - "type": "string" } }, "type": "object" diff --git a/e2e/testdata/app/ob.yml b/e2e/testdata/app/ob.yml index e58230d1..a1d17b80 100644 --- a/e2e/testdata/app/ob.yml +++ b/e2e/testdata/app/ob.yml @@ -16,7 +16,8 @@ proxy: # Traefik is started by the test itself, from the same Compose file, so # Onebox must not try to manage one. managed: false -verifications: +checks: # exec (not http): on macOS the container bridge network is unreachable # from the host, so probe from inside the container. - - { exec: "wget -qO- http://localhost:8080/", workload: web } + exec: + - {workload: web, run: "wget -qO- http://localhost:8080/"} diff --git a/e2e/testdata/worker/ob-broken.yml b/e2e/testdata/worker/ob-broken.yml index ada562f3..370aa171 100644 --- a/e2e/testdata/worker/ob-broken.yml +++ b/e2e/testdata/worker/ob-broken.yml @@ -16,5 +16,6 @@ deployment: order: [worker, web] proxy: managed: false -verifications: - - { exec: "wget -qO- http://localhost:8080/", workload: web } +checks: + exec: + - {workload: web, run: "wget -qO- http://localhost:8080/"} diff --git a/e2e/testdata/worker/ob.yml b/e2e/testdata/worker/ob.yml index 2a32f72a..3b8df8c4 100644 --- a/e2e/testdata/worker/ob.yml +++ b/e2e/testdata/worker/ob.yml @@ -16,5 +16,6 @@ deployment: order: [worker, web] proxy: managed: false -verifications: - - { exec: "wget -qO- http://localhost:8080/", workload: web } +checks: + exec: + - {workload: web, run: "wget -qO- http://localhost:8080/"} diff --git a/internal/app/backup_artifacts.go b/internal/app/backup_artifacts.go new file mode 100644 index 00000000..01e65d71 --- /dev/null +++ b/internal/app/backup_artifacts.go @@ -0,0 +1,93 @@ +package app + +// BackupEffectiveProjection is the policy and target a service is protected +// by. It is recorded at enablement and carried in the durable lifecycle state, +// so a service keeps archiving to the repository it was bound to even if the +// project's intent is later edited. +type BackupEffectiveProjection struct { + Policy BackupPolicy `json:"policy"` + Target BackupTarget `json:"target"` +} + +// The projection a protected service is actually running under. +// +// This file used to also generate a set of twelve JSON descriptors — schedules, +// retention, provenance, restore templates — describing what backup *should* +// look like on the target, together with a digest comparison to detect drift. +// None of it ever had a caller, and the design it described no longer exists: +// the schedules are systemd units derived from the policy, retention is applied +// by the prune command from the same policy, and the provenance that matters is +// the wal-g checksum pinned in this binary and verified before the binary is +// ever placed on a host. +// +// Drift is now asked of the target directly rather than of a descriptor written +// beside it — see VerifyBackupRuntime. A second description of the truth is +// only somewhere for the two to disagree. + +func (r *Resolved) effectiveBackupProjection(serviceName string, service Service) (BackupEffectiveProjection, string, error) { + if state, ok := r.serviceRuntime[serviceName]; ok && state.BackupState == "disable-pending" { + if state.LastEffective == nil { + return BackupEffectiveProjection{}, "", errf("backup_image_revert_unsafe", "services."+serviceName, "ob backup disable --output ndjson", "disable-pending state has no durable last-effective backup projection") + } + return *state.LastEffective, "last-effective", nil + } + if service.Backup == nil { + return BackupEffectiveProjection{}, "", errf("project_invalid", "services."+serviceName+".backup", "ob validate", "service has no backup intent or retained projection") + } + target, ok := r.BackupTargets[service.Backup.Target] + if !ok { + return BackupEffectiveProjection{}, "", errf("backup_target_unknown", "services."+serviceName+".backup.target", "ob validate", "backup target is not declared") + } + return BackupEffectiveProjection{Policy: *service.Backup, Target: target}, "project-intent", nil +} + +// EffectiveBackupProjection is the repository a service is actually +// archiving to, which is not always the one the project currently names. +// +// The recorded projection wins whenever there is one. Enablement writes down +// exactly what it bound, and the server has been archiving there ever since, so +// editing `backup_targets` or `services..backup.target` afterwards must not +// silently redirect a restore at a repository the history is not in — nor at a +// credential file installed under the old target's name. +// +// Rendering, recovery, status and retention all resolve through here, so there +// is one rule rather than four that can drift. The project's intent takes +// effect at the next `ob backup enable`, which is where the change is made +// deliberately. +// DeclaredBackupProjection is what the project currently asks for, with no +// regard for what the service was last bound to. +// +// Exactly one caller wants this: `ob backup enable`, which is where the +// project's intent is supposed to take effect. Resolving enable through +// EffectiveBackupProjection instead meant the recorded projection won there +// too, so editing a target and re-running enable — the documented way to move a +// service to a different repository — reported success, rebound the service to +// the repository it was already using, and left the operator believing their +// backups had moved. Measured against a live host: bucket edited, enable green, +// every subsequent backup still going to the old bucket. +func (r *Resolved) DeclaredBackupProjection(serviceName string) (BackupEffectiveProjection, error) { + service, ok := r.Services[serviceName] + if !ok { + return BackupEffectiveProjection{}, errf("project_invalid", "services."+serviceName, "ob validate", "service is not declared") + } + if service.Backup == nil { + return BackupEffectiveProjection{}, errf("project_invalid", "services."+serviceName+".backup", "ob validate", "service declares no backup policy") + } + target, ok := r.BackupTargets[service.Backup.Target] + if !ok { + return BackupEffectiveProjection{}, errf("backup_target_unknown", "services."+serviceName+".backup.target", "ob validate", "backup target is not declared") + } + return BackupEffectiveProjection{Policy: *service.Backup, Target: target}, nil +} + +func (r *Resolved) EffectiveBackupProjection(serviceName string) (BackupEffectiveProjection, error) { + service, ok := r.Services[serviceName] + if !ok { + return BackupEffectiveProjection{}, errf("project_invalid", "services."+serviceName, "ob validate", "service is not declared") + } + if state, observed := r.serviceRuntime[serviceName]; observed && state.LastEffective != nil { + return *state.LastEffective, nil + } + projection, _, err := r.effectiveBackupProjection(serviceName, service) + return projection, err +} diff --git a/internal/app/protection_names_test.go b/internal/app/backup_names_test.go similarity index 95% rename from internal/app/protection_names_test.go rename to internal/app/backup_names_test.go index 6b5352b9..4029c1ac 100644 --- a/internal/app/protection_names_test.go +++ b/internal/app/backup_names_test.go @@ -9,7 +9,7 @@ func TestProtectedServiceReservesRestoreRuntimeNames(t *testing.T) { spec := &Spec{ Name: "example", Services: map[string]Service{ - "database": {Driver: "postgres", Version: 17, Volumes: []string{"data"}, Protection: &ProtectionPolicy{Target: "offsite"}}, + "database": {Driver: "postgres", Version: 17, Volumes: []string{"data"}, Backup: &BackupPolicy{Target: "offsite"}}, }, } all := spec.All("production") diff --git a/internal/app/protection_schema.go b/internal/app/backup_schema.go similarity index 66% rename from internal/app/protection_schema.go rename to internal/app/backup_schema.go index cfb0f7c6..26c83edf 100644 --- a/internal/app/protection_schema.go +++ b/internal/app/backup_schema.go @@ -18,6 +18,15 @@ func validateBackupTarget(target BackupTarget, path string) error { if err := checkEnum(path+".tls", target.TLS, eBackupTLS); err != nil { return err } + // Refused here rather than at render. Enablement writes the durable state + // before applying, so a render-time refusal first appeared after the service + // was already recorded as protected — and every later apply and deploy then + // failed with guidance pointing at `ob validate`, which passed. + if strings.HasPrefix(target.Endpoint, "https://") && target.TLS == "skip-verify" { + return errf("recovery_objective_unsupported", path+".tls", "ob validate", + "skip-verify cannot be honoured on an https endpoint: the backup tool has no option to skip certificate verification. "+ + "Install the certificate authority on the host so the certificate verifies, or use an http endpoint if the destination is on a trusted network") + } if err := gFailureDomain.check(path+".failure_domain.identity", target.FailureDomain.Identity); err != nil { return err } @@ -48,24 +57,6 @@ func validateBackupTarget(target BackupTarget, path string) error { return nil } -// ValidateBackupTarget keeps lifecycle adapters on the same closed target -// contract as project loading. Adapters receive already-resolved values, but -// revalidate at their trust boundary rather than assuming every caller loaded -// a complete project first. -func ValidateBackupTarget(name string, target BackupTarget) error { - if err := gIdent.check("backup_targets."+name, name); err != nil { - return err - } - return validateBackupTarget(target, "backup_targets."+name) -} - -// BackupTargetEncryptionMode returns the authored mode for one recovery kind. -// An empty result is deliberately not a default: the lifecycle adapter must -// refuse protection whose encryption evidence cannot be established. -func BackupTargetEncryptionMode(target BackupTarget, recoveryKind string) string { - return encryptionFor(target.Encryption, recoveryKind) -} - func validateBackupEndpoint(endpoint, tls, path string) error { u, err := url.Parse(endpoint) if err != nil || u.Host == "" || (u.Scheme != "https" && u.Scheme != "http") { @@ -74,8 +65,8 @@ func validateBackupEndpoint(endpoint, tls, path string) error { if u.User != nil || u.RawQuery != "" || u.Fragment != "" { return errf("project_invalid", path+".endpoint", "ob validate", "a backup endpoint may not contain userinfo, query credentials, or a fragment") } - if u.Scheme != "https" && tls != "insecure" { - return errf("project_invalid", path+".endpoint", "ob validate", "an http backup endpoint requires tls: insecure; use https for verified transport") + if u.Scheme != "https" && tls != "skip-verify" { + return errf("project_invalid", path+".endpoint", "ob validate", "an http backup endpoint requires tls: skip-verify; use https for verified transport") } return nil } @@ -111,70 +102,78 @@ func validateTargetEncryption(encryption TargetEncryption, path string) error { return nil } -func validateProtectionPolicy(policy ProtectionPolicy, path string) error { +func validateBackupPolicy(policy BackupPolicy, path string) error { if err := gIdent.check(path+".target", policy.Target); err != nil { return err } if err := checkEnum(path+".recovery_kind", policy.RecoveryKind, eRecoveryKind); err != nil { return err } - maximumDataLoss, err := PositiveDuration(policy.MaximumDataLoss) + maximumDataLoss, err := PositiveDuration(policy.MaxDataLoss) if err != nil { - return errf("project_invalid", path+".maximum_data_loss", "ob validate", "maximum_data_loss must be a positive duration: %v", err) + return errf("project_invalid", path+".max_data_loss", "ob validate", "max_data_loss must be a positive duration: %v", err) } if maximumDataLoss < time.Minute { - return errf("recovery_objective_unsupported", path+".maximum_data_loss", "ob validate", "maximum_data_loss must be at least one minute because the host scheduler has one-minute resolution") + return errf("recovery_objective_unsupported", path+".max_data_loss", "ob validate", "max_data_loss must be at least one minute because the host scheduler has one-minute resolution") } if err := validateSchedule(&policy.Schedule, path+".schedule"); err != nil { return err } - if policy.Retention.MinimumGenerations <= 0 { - return errf("backup_retention_unsupported", path+".retention.minimum_generations", "ob validate", "minimum_generations must be a positive whole number") + if policy.Retention.Keep <= 0 { + return errf("backup_retention_unsupported", path+".retention.keep", "ob validate", "keep must be a positive whole number") } - if _, err := PositiveDuration(policy.Retention.RecoveryWindow); err != nil { - return errf("backup_retention_unsupported", path+".retention.recovery_window", "ob validate", "recovery_window must be a positive duration: %v", err) + if _, err := PositiveDuration(policy.Retention.Window); err != nil { + return errf("backup_retention_unsupported", path+".retention.window", "ob validate", "window must be a positive duration: %v", err) } - if err := validateSchedule(&policy.RestoreDrill.Schedule, path+".restore_drill.schedule"); err != nil { + if err := validateSchedule(&policy.Drill.Schedule, path+".drill.schedule"); err != nil { return err } - proofAge, err := PositiveDuration(policy.RestoreDrill.ProofMaximumAge) + proofAge, err := PositiveDuration(policy.Drill.MaxAge) if err != nil { - return errf("project_invalid", path+".restore_drill.proof_maximum_age", "ob validate", "proof_maximum_age must be a positive duration: %v", err) + return errf("project_invalid", path+".drill.max_age", "ob validate", "max_age must be a positive duration: %v", err) } - gap, exact := maximumCronGap(policy.RestoreDrill.Schedule.Cron) + gap, exact := maximumCronGap(policy.Drill.Schedule.Cron) if !exact { - return errf("restore_drill_schedule_too_sparse", path+".restore_drill.schedule.cron", "ob validate", "restore drill cadence cannot be proven against proof_maximum_age; use a daily or weekday-based schedule") + return errf("drill_schedule_too_sparse", path+".drill.schedule.cron", "ob validate", "restore drill cadence cannot be proven against max_age; use a daily or weekday-based schedule") } if gap >= proofAge { - return errf("restore_drill_schedule_too_sparse", path+".restore_drill.schedule.cron", "ob validate", "restore drill maximum interval %s reaches or exceeds proof_maximum_age %s; use a more frequent schedule", gap, proofAge) - } - if err := gAbsPath.checkOptional(path+".restore_drill.staging_filesystem", policy.RestoreDrill.StagingFilesystem); err != nil { - return err + return errf("drill_schedule_too_sparse", path+".drill.schedule.cron", "ob validate", "restore drill maximum interval %s reaches or exceeds max_age %s; use a more frequent schedule", gap, proofAge) } return nil } -func validateProtectionSelection(p *Spec, serviceName, driverName string, service Service) error { - if service.Protection == nil { +func validateBackupSelection(p *Spec, serviceName, driverName string, service Service) error { + if service.Backup == nil { return nil } - path := "services." + serviceName + ".protection" - policy := service.Protection + path := "services." + serviceName + ".backup" + policy := service.Backup target, ok := p.BackupTargets[policy.Target] if !ok { - return errf("backup_target_unknown", path+".target", "ob validate", "protection target %q is not declared in backup_targets", policy.Target) + return errf("backup_target_unknown", path+".target", "ob validate", "backup target %q is not declared in backup_targets", policy.Target) + } + + // Archiving needs at least `replica`. A project asking for `minimal` has + // asked for something backup cannot deliver, so it is refused rather + // than silently raised — the same reason an authored `logical` is left + // alone rather than forced down. + if level, ok := service.Settings["wal_level"]; ok { + if spelled := fmt.Sprint(level); spelled != "replica" && spelled != "logical" { + return errf("project_invalid", "services."+serviceName+".settings.wal_level", "ob validate", + "wal_level %q cannot carry the write-ahead log a backup replays; backup needs replica or logical", spelled) + } } capability, exists := lifecycleCapabilityFor(driverName) version := versionString(service.Version) - if !exists || !capability.ProtectionQualified(version) { - return errf("backup_driver_unsupported", path, "ob validate", "driver %q version %q is runnable but has no qualified executable protection contract; remove the policy or choose a qualified driver version", driverName, version) + if !exists || !capability.BackupQualified(version) { + return errf("backup_driver_unsupported", path, "ob validate", "driver %q version %q is runnable but has no qualified executable backup contract; remove the policy or choose a qualified driver version", driverName, version) } if !capability.SupportsRecoveryKind(version, policy.RecoveryKind) { return errf("recovery_objective_unsupported", path+".recovery_kind", "ob validate", "driver %q version %q does not support recovery kind %q in its qualified contract", driverName, version, policy.RecoveryKind) } - if policy.RecoveryKind == "cold" && !policy.AllowBackupInterruption { - return errf("backup_interruption_not_authorized", path+".allow_backup_interruption", "ob validate", "driver %q requires an explicitly permitted recurring stopped-service backup window", driverName) + if policy.RecoveryKind == "cold" && !policy.AllowDowntime { + return errf("backup_interruption_not_authorized", path+".allow_downtime", "ob validate", "driver %q requires an explicitly permitted recurring stopped-service backup window", driverName) } if target.Kind != "s3-compatible" { return errf("recovery_objective_unsupported", path+".target", "ob validate", "%s recovery requires an s3-compatible repository target", policy.RecoveryKind) @@ -225,7 +224,7 @@ func PositiveDuration(value string) (time.Duration, error) { } // maximumCronGap recognizes the exact daily/weekly/monthly shapes accepted by -// the protection contract. Unknown advanced expressions remain executable but +// the backup contract. Unknown advanced expressions remain executable but // are not used to prove a sparse schedule safe until the scheduler owns a full // next-occurrence calculation. func maximumCronGap(expression string) (time.Duration, bool) { @@ -346,72 +345,72 @@ func sameEndpointHost(host, endpoint string) bool { return err == nil && sameHost(host, u.Hostname()) } -// prepareServiceOverride permits only operational tuning beneath protection. +// prepareServiceOverride permits only operational tuning beneath backup. // Target, recovery kind, interruption authority, proof age, and staging // identity remain project-level intent and cannot change by environment. func prepareServiceOverride(path string, service Service, patch map[string]any) (map[string]any, error) { - protectionValue, present := patch["protection"] + backupValue, present := patch["backup"] if !present { return patch, nil } - if service.Protection == nil { - return nil, errf("override_not_permitted", path+".protection", "ob validate", "an environment cannot enable protection for a service that has no project-level policy") + if service.Backup == nil { + return nil, errf("override_not_permitted", path+".backup", "ob validate", "an environment cannot enable backup for a service that has no project-level policy") } - protectionPatch, ok := protectionValue.(map[string]any) + backupPatch, ok := backupValue.(map[string]any) if !ok { - return nil, errf("override_invalid", path+".protection", "ob validate", "a protection override must be a mapping") + return nil, errf("override_invalid", path+".backup", "ob validate", "a backup override must be a mapping") } - allowed := map[string]bool{"schedule": true, "retention": true, "restore_drill": true} - for _, key := range sortedKeys(protectionPatch) { + allowed := map[string]bool{"schedule": true, "retention": true, "drill": true} + for _, key := range sortedKeys(backupPatch) { if !allowed[key] { - return nil, errf("override_not_permitted", path+".protection."+key, "ob validate", "%q may not be overridden per environment; protection overrides may tune only schedules and retention", key) + return nil, errf("override_not_permitted", path+".backup."+key, "ob validate", "%q may not be overridden per environment; backup overrides may tune only schedules and retention", key) } } - base, err := toGeneric(*service.Protection) + base, err := toGeneric(*service.Backup) if err != nil { return nil, err } - if value, ok := protectionPatch["schedule"]; ok { - merged, err := mergeClosedOverride(path+".protection.schedule", base["schedule"], value, map[string]bool{"cron": true, "timezone": true}) + if value, ok := backupPatch["schedule"]; ok { + merged, err := mergeClosedOverride(path+".backup.schedule", base["schedule"], value, map[string]bool{"cron": true, "timezone": true}) if err != nil { return nil, err } base["schedule"] = merged } - if value, ok := protectionPatch["retention"]; ok { - merged, err := mergeClosedOverride(path+".protection.retention", base["retention"], value, map[string]bool{"minimum_generations": true, "recovery_window": true}) + if value, ok := backupPatch["retention"]; ok { + merged, err := mergeClosedOverride(path+".backup.retention", base["retention"], value, map[string]bool{"keep": true, "window": true}) if err != nil { return nil, err } base["retention"] = merged } - if value, ok := protectionPatch["restore_drill"]; ok { + if value, ok := backupPatch["drill"]; ok { drillPatch, ok := value.(map[string]any) if !ok { - return nil, errf("override_invalid", path+".protection.restore_drill", "ob validate", "a restore_drill override must be a mapping") + return nil, errf("override_invalid", path+".backup.drill", "ob validate", "a drill override must be a mapping") } for _, key := range sortedKeys(drillPatch) { if key != "schedule" { - return nil, errf("override_not_permitted", path+".protection.restore_drill."+key, "ob validate", "%q may not be overridden per environment; only the drill schedule may vary", key) + return nil, errf("override_not_permitted", path+".backup.drill."+key, "ob validate", "%q may not be overridden per environment; only the drill schedule may vary", key) } } - drillBase, _ := base["restore_drill"].(map[string]any) + drillBase, _ := base["drill"].(map[string]any) if schedule, ok := drillPatch["schedule"]; ok { - merged, err := mergeClosedOverride(path+".protection.restore_drill.schedule", drillBase["schedule"], schedule, map[string]bool{"cron": true, "timezone": true}) + merged, err := mergeClosedOverride(path+".backup.drill.schedule", drillBase["schedule"], schedule, map[string]bool{"cron": true, "timezone": true}) if err != nil { return nil, err } drillBase["schedule"] = merged } - base["restore_drill"] = drillBase + base["drill"] = drillBase } out := make(map[string]any, len(patch)) for key, value := range patch { out[key] = value } - out["protection"] = base + out["backup"] = base return out, nil } diff --git a/internal/app/protection_schema_test.go b/internal/app/backup_schema_test.go similarity index 58% rename from internal/app/protection_schema_test.go rename to internal/app/backup_schema_test.go index e6332ead..806eaf82 100644 --- a/internal/app/protection_schema_test.go +++ b/internal/app/backup_schema_test.go @@ -6,7 +6,7 @@ import ( "testing" ) -const validProtectionProject = `api_version: onebox.run/v1 +const validBackupProject = `api_version: onebox.run/v1 app: shop environments: production: @@ -32,36 +32,40 @@ backup_targets: services: postgres: version: 17 - protection: + backup: target: offsite recovery_kind: pitr - maximum_data_loss: 15m + max_data_loss: 15m ` -func TestProtectionIntentLoadsAndDefaultsToExactSchedules(t *testing.T) { - p, err := LoadBytes([]byte(validProtectionProject), "ob.yml") +func TestBackupIntentLoadsAndDefaultsToExactSchedules(t *testing.T) { + p, err := LoadBytes([]byte(validBackupProject), "ob.yml") if err != nil { t.Fatal(err) } - policy := p.Services["postgres"].Protection + policy := p.Services["postgres"].Backup if policy == nil { - t.Fatal("protection policy was not decoded") + t.Fatal("backup policy was not decoded") } if policy.Schedule.Cron != "0 2 * * *" || policy.Schedule.Timezone != "UTC" { t.Fatalf("backup schedule = %#v, want exact daily UTC default", policy.Schedule) } - if policy.Retention.MinimumGenerations != 7 || policy.Retention.RecoveryWindow != "7d" { + if policy.Retention.Keep != 7 || policy.Retention.Window != "7d" { t.Fatalf("retention = %#v, want seven generations and seven days", policy.Retention) } - if policy.RestoreDrill.Schedule.Cron != "0 3 * * 0,3" || policy.RestoreDrill.ProofMaximumAge != "7d" { - t.Fatalf("restore drill = %#v, want exact twice-weekly schedule and seven-day proof age", policy.RestoreDrill) + if policy.Drill.Schedule.Cron != "0 3 * * 0,3" || policy.Drill.MaxAge != "7d" { + t.Fatalf("restore drill = %#v, want exact twice-weekly schedule and seven-day proof age", policy.Drill) } - if got := p.BackupTargets["offsite"]; got.TLS != "required" || got.Credentials.Provider != "sops" { + if got := p.BackupTargets["offsite"]; got.TLS != "verify" || got.Credentials.Provider != "sops" { t.Fatalf("target defaults = %#v", got) } } -func TestSingleNodeMinIOColdIntentLoads(t *testing.T) { +// minio was accepted with a backup policy and then refused at +// `ob backup enable`, because postgres is the only driver whose contract runs. +// The refusal belongs at the point the policy is written, so this is now the +// same rejection every other unqualified driver gets. +func TestMinIOBackupIntentIsRefusedUntilItsContractRuns(t *testing.T) { project := `api_version: onebox.run/v1 app: shop environments: {production: {server: deploy@app.example.net}} @@ -77,21 +81,25 @@ backup_targets: services: minio: version: RELEASE.2026-07-31T00-00-00Z - protection: {target: offsite, recovery_kind: cold, maximum_data_loss: 24h, allow_backup_interruption: true} + backup: {target: offsite, recovery_kind: cold, max_data_loss: 24h, allow_downtime: true} ` - if _, err := LoadBytes([]byte(project), "ob.yml"); err != nil { - t.Fatal(err) + _, err := LoadBytes([]byte(project), "ob.yml") + if err == nil { + t.Fatal("a minio backup policy was accepted, but no driver except postgres can establish one") + } + if !strings.Contains(err.Error(), "backup_driver_unsupported") { + t.Fatalf("refusal is not the unqualified-driver one: %v", err) } } func TestReplicationIntentIsRejected(t *testing.T) { - project := strings.ReplaceAll(validProtectionProject, "kind: s3-compatible", "kind: minio-replication") + project := strings.ReplaceAll(validBackupProject, "kind: s3-compatible", "kind: minio-replication") if _, err := LoadBytes([]byte(project), "ob.yml"); err == nil { t.Fatal("removed replication target was accepted") } } -func TestRunnableUnqualifiedDriverRejectsProtectionWithoutFallback(t *testing.T) { +func TestRunnableUnqualifiedDriverRejectsBackupWithoutFallback(t *testing.T) { if _, err := LoadBytes([]byte(`api_version: onebox.run/v1 app: shop environments: {production: {server: deploy@app.example.net}} @@ -101,7 +109,7 @@ services: {redis: 7} t.Fatalf("unqualified driver must remain runnable: %v", err) } - protected := strings.ReplaceAll(validProtectionProject, "postgres", "redis") + protected := strings.ReplaceAll(validBackupProject, "postgres", "redis") protected = strings.ReplaceAll(protected, "pitr", "snapshot") _, err := LoadBytes([]byte(protected), "ob.yml") assertAppErrorCode(t, err, "backup_driver_unsupported") @@ -117,7 +125,7 @@ func TestEveryRuntimeDriverHasAnExplicitLifecycleRecordAndNoDefault(t *testing.T } for _, name := range DriverNames() { capability, ok := lifecycleCapabilityFor(name) - if !ok || capability.DriverName() != name { + if !ok || capability.driver != name { t.Errorf("driver %q has no matching lifecycle capability record", name) } } @@ -126,7 +134,7 @@ func TestEveryRuntimeDriverHasAnExplicitLifecycleRecordAndNoDefault(t *testing.T } } -func TestProtectionIntentRefusals(t *testing.T) { +func TestBackupIntentRefusals(t *testing.T) { cases := []struct { name string yaml string @@ -134,47 +142,47 @@ func TestProtectionIntentRefusals(t *testing.T) { }{ { name: "inline storage secret", - yaml: strings.Replace(validProtectionProject, " secret_key_entry: BACKUP_SECRET_ACCESS_KEY\n", " secret_key_entry: BACKUP_SECRET_ACCESS_KEY\n secret_key: plaintext-must-not-enter-the-model\n", 1), + yaml: strings.Replace(validBackupProject, " secret_key_entry: BACKUP_SECRET_ACCESS_KEY\n", " secret_key_entry: BACKUP_SECRET_ACCESS_KEY\n secret_key: plaintext-must-not-enter-the-model\n", 1), code: "unknown_field", }, { name: "target shares protected host", - yaml: strings.Replace(validProtectionProject, " host: objects.example.net", " host: app.example.net", 1), + yaml: strings.Replace(validBackupProject, " host: objects.example.net", " host: app.example.net", 1), code: "backup_target_not_independent", }, { name: "author selects backup tool", - yaml: strings.Replace(validProtectionProject, " target: offsite\n", " target: offsite\n tool: pgbackrest\n", 1), + yaml: strings.Replace(validBackupProject, " target: offsite\n", " target: offsite\n tool: some-backup-tool\n", 1), code: "unknown_field", }, { name: "recurring policy tries to authorize enablement restart", - yaml: strings.Replace(validProtectionProject, " maximum_data_loss: 15m\n", " maximum_data_loss: 15m\n allow_enablement_restart: true\n", 1), + yaml: strings.Replace(validBackupProject, " max_data_loss: 15m\n", " max_data_loss: 15m\n allow_enablement_restart: true\n", 1), code: "unknown_field", }, { name: "restore drill too sparse", - yaml: strings.Replace(validProtectionProject, " maximum_data_loss: 15m\n", " maximum_data_loss: 15m\n restore_drill:\n schedule: {cron: '0 3 1 * *', timezone: UTC}\n proof_maximum_age: 7d\n", 1), - code: "restore_drill_schedule_too_sparse", + yaml: strings.Replace(validBackupProject, " max_data_loss: 15m\n", " max_data_loss: 15m\n drill:\n schedule: {cron: '0 3 1 * *', timezone: UTC}\n max_age: 7d\n", 1), + code: "drill_schedule_too_sparse", }, { name: "stepped weekday drill too sparse", - yaml: strings.Replace(validProtectionProject, " maximum_data_loss: 15m\n", " maximum_data_loss: 15m\n restore_drill:\n schedule: {cron: '0 3 * * */2', timezone: UTC}\n proof_maximum_age: 36h\n", 1), - code: "restore_drill_schedule_too_sparse", + yaml: strings.Replace(validBackupProject, " max_data_loss: 15m\n", " max_data_loss: 15m\n drill:\n schedule: {cron: '0 3 * * */2', timezone: UTC}\n max_age: 36h\n", 1), + code: "drill_schedule_too_sparse", }, { name: "sub-minute replay objective", - yaml: strings.Replace(validProtectionProject, "maximum_data_loss: 15m", "maximum_data_loss: 30s", 1), + yaml: strings.Replace(validBackupProject, "max_data_loss: 15m", "max_data_loss: 30s", 1), code: "recovery_objective_unsupported", }, { name: "unsupported retention", - yaml: strings.Replace(validProtectionProject, " maximum_data_loss: 15m\n", " maximum_data_loss: 15m\n retention: {minimum_generations: 0, recovery_window: 7d}\n", 1), + yaml: strings.Replace(validBackupProject, " max_data_loss: 15m\n", " max_data_loss: 15m\n retention: {keep: 0, window: 7d}\n", 1), code: "backup_retention_unsupported", }, { name: "unsupported objective", - yaml: strings.Replace(validProtectionProject, "recovery_kind: pitr", "recovery_kind: snapshot", 1), + yaml: strings.Replace(validBackupProject, "recovery_kind: pitr", "recovery_kind: snapshot", 1), code: "recovery_objective_unsupported", }, } @@ -189,15 +197,15 @@ func TestProtectionIntentRefusals(t *testing.T) { } } -func TestProtectionEnvironmentOverridesTuneOnlySchedulesAndRetention(t *testing.T) { - valid := strings.Replace(validProtectionProject, " server: deploy@app.example.net\n", ` server: deploy@app.example.net +func TestBackupEnvironmentOverridesTuneOnlySchedulesAndRetention(t *testing.T) { + valid := strings.Replace(validBackupProject, " server: deploy@app.example.net\n", ` server: deploy@app.example.net overrides: services: postgres: - protection: + backup: schedule: {cron: '0 4 * * *'} - retention: {minimum_generations: 10} - restore_drill: {schedule: {cron: '0 5 * * 1,4'}} + retention: {keep: 10} + drill: {schedule: {cron: '0 5 * * 1,4'}} `, 1) p, err := LoadBytes([]byte(valid), "ob.yml") if err != nil { @@ -207,16 +215,16 @@ func TestProtectionEnvironmentOverridesTuneOnlySchedulesAndRetention(t *testing. if err != nil { t.Fatal(err) } - policy := resolved.Services["postgres"].Protection - if policy.Target != "offsite" || policy.Schedule.Cron != "0 4 * * *" || policy.Retention.MinimumGenerations != 10 || policy.Retention.RecoveryWindow != "7d" || policy.RestoreDrill.Schedule.Cron != "0 5 * * 1,4" { + policy := resolved.Services["postgres"].Backup + if policy.Target != "offsite" || policy.Schedule.Cron != "0 4 * * *" || policy.Retention.Keep != 10 || policy.Retention.Window != "7d" || policy.Drill.Schedule.Cron != "0 5 * * 1,4" { t.Fatalf("resolved safe override = %#v", policy) } - unsafe := strings.Replace(validProtectionProject, " server: deploy@app.example.net\n", ` server: deploy@app.example.net + unsafe := strings.Replace(validBackupProject, " server: deploy@app.example.net\n", ` server: deploy@app.example.net overrides: services: postgres: - protection: {target: another-repository} + backup: {target: another-repository} `, 1) p, err = LoadBytes([]byte(unsafe), "ob.yml") if err != nil { diff --git a/internal/app/backup_walg.go b/internal/app/backup_walg.go new file mode 100644 index 00000000..ca0243f0 --- /dev/null +++ b/internal/app/backup_walg.go @@ -0,0 +1,546 @@ +package app + +import ( + "fmt" + "math" + "net/url" + "sort" + "strconv" + "strings" +) + +// wal-g is the backup engine for the postgres driver: a physical base +// backup plus continuous WAL archiving, which is the only combination that can +// honour the point-in-time recovery the policy declares. A logical dump cannot, +// and a copy of a running data directory is the generic live-volume archive the +// contract refuses outright. +// +// It runs from a verified binary staged on the host and mounted into the stock +// PostgreSQL image, rather than from a PostgreSQL image Onebox builds and +// publishes. That is the whole reason this file is short. wal-g links against +// libc and nothing else, and takes its entire configuration from the +// environment — so there is no image to maintain, no configuration file to +// place, and no second copy of anything to keep in step with the project. +// +// pgBackRest was implemented first and replaced. It is a fine tool, but it +// needs 41 shared libraries, so it cannot be dropped into the official image +// and forces a derived one; and it is configured by a file, which brought the +// file's own problems — an atomically replaced config vanishing from a running +// container, credential names colliding with its option namespace, and a +// restore_command it writes as the absolute path of its own binary. + +// PgDataPath is the data directory the postgres driver runs with. Every wal-g +// command that touches the cluster needs it exactly: the driver sets PGDATA to +// this subdirectory so the volume can hold a lost+found without confusing +// initdb, and pointing a backup at the volume root captures the wrong tree. +const PgDataPath = "/var/lib/postgresql/data/pgdata" + +// WalgMountPath is where the staged binary and its wrapper are mounted inside +// the container, read-only. Outside /usr/local/bin deliberately: the mount must +// not shadow anything the official image ships. +const WalgMountPath = "/opt/onebox/backup" + +// WalgBinary is the wrapper Onebox stages beside wal-g, and what every caller +// invokes. It exists because wal-g reads its credentials from fixed AWS_* names +// while a backup target names its own entries, so something has to bridge the +// two — and doing it here keeps the project's vocabulary out of wal-g's and +// wal-g's out of the operator's encrypted file. +const WalgBinary = WalgMountPath + "/ob-wal-g" + +// WalgRepositoryKeyEntry is the credential entry holding the repository +// encryption key. Unlike the destination keys it has a fixed name: the key is +// Onebox's own requirement rather than a property of the destination, so there +// is no backup_targets field to indirect through. +const WalgRepositoryKeyEntry = "OB_REPOSITORY_KEY" + +// WalgPrefix is the repository location for one protected service. +// +// The application and service are joined with the injective rule the rest of +// the derived names use, not a hyphen. Hyphens are legal in both, so `a-b`/`c` +// and `a`/`b-c` would land on the same prefix and interleave two clusters' base +// backups and WAL — and this string is unversioned by design, so it could not +// be corrected later without orphaning every backup taken before the fix. +func WalgPrefix(target BackupTarget, app, service string) string { + segments := []string{strings.Trim(target.Prefix, "/"), Join(app, service)} + if segments[0] == "" { + segments = segments[1:] + } + return "s3://" + target.Bucket + "/" + strings.Join(segments, "/") +} + +// WalgArchiveCommand is what the server runs for each completed WAL segment. +// A failure here must fail the archive: postgres then retries, and WAL +// accumulates on the data volume until it succeeds. That is what keeps the +// recovery window continuous instead of quietly gapped, and it is why archiving +// is worth alerting on — an unreachable repository eventually fills the volume +// rather than losing history silently. +func WalgArchiveCommand() string { return WalgBinary + " wal-push %p" } + +// WalgEnvironment is the non-secret configuration a protected service runs +// with. Every value here is derived from the project and safe to read: the +// destination's location, not its keys. +func WalgEnvironment(target BackupTarget, app, service string) (map[string]any, error) { + if target.Kind != "s3-compatible" { + return nil, fmt.Errorf("backup for %q: unsupported target kind %q", service, target.Kind) + } + endpoint, err := url.Parse(target.Endpoint) + if err != nil || endpoint.Host == "" { + return nil, fmt.Errorf("backup for %q: endpoint %q is not a URL naming a host", service, target.Endpoint) + } + if endpoint.Scheme == "https" && target.TLS == "skip-verify" { + // wal-g offers no way to skip certificate verification: its only + // transport controls are the endpoint protocol and a CA file. Accepting + // this would mean verifying anyway while the project says otherwise. + return nil, errf("recovery_objective_unsupported", "backup_targets.tls", "ob validate", + "backup for %q cannot skip certificate verification on an https endpoint: wal-g has no such option. "+ + "Install the certificate authority on the host so the certificate verifies, or use an http endpoint if the destination is on a trusted network", + service) + } + env := map[string]any{ + "WALG_S3_PREFIX": WalgPrefix(target, app, service), + "AWS_ENDPOINT": target.Endpoint, + // Path-style addressing, because an S3-compatible endpoint is usually + // not the one provider whose virtual-host naming works everywhere. A + // bucket name is then a path segment rather than a subdomain that has + // to resolve. + "AWS_S3_FORCE_PATH_STYLE": "true", + // The repository is encrypted whatever the destination does, because + // the schema refuses a policy whose recovery kind has no declared + // encryption mode. The key is hex so it can be a printable line in the + // operator's encrypted file rather than 32 raw bytes. + "WALG_LIBSODIUM_KEY_TRANSFORM": "hex", + // A completed segment is archived when it fills or when the server + // switches; refusing to overwrite one already in the repository turns a + // duplicated segment into a loud failure instead of a rewritten history. + "WALG_PREVENT_WAL_OVERWRITE": "true", + // backup-push connects to the local server over its Unix socket, so it + // needs no password and opens no TCP port to do it. + // + // PGDATABASE is not optional: left unset libpq defaults the database to + // the user name, and the driver's user is `onebox` while its database + // is the application's — so every command fails with `database "onebox" + // does not exist`, which reads like a broken cluster rather than a + // missing variable. + "PGHOST": "/var/run/postgresql", + "PGUSER": PgSuperuser, + "PGDATABASE": app, + "PGDATA": PgDataPath, + } + if target.Region != "" { + env["AWS_REGION"] = target.Region + } else { + // wal-g requires a region even where the destination ignores it. + // Written down rather than left implicit. + env["AWS_REGION"] = "us-east-1" + } + // The transport comes from the endpoint's own scheme, which is the only + // unambiguous source: `tls` says what the operator will accept, not what + // the endpoint speaks. Mapping skip-verify to plaintext — as this first did + // — sent credentials in the clear to an https endpoint that had merely + // presented a self-signed certificate. + env["S3_ENDPOINT_PROTOCOL"] = endpoint.Scheme + return env, nil +} + +// PgSuperuser is the role the postgres driver creates. It must agree with the +// driver's `user` field, and the contract test holds the two together. wal-g +// connects as it rather than as the operating-system user, which is `postgres` +// — a role the driver never creates, because Onebox owns the identity and makes +// it the application's so two projects on one host cannot silently share one. +const PgSuperuser = "onebox" + +// WalgCredentialEntries are the entry names the target-side credential file +// must define: the two or three the project named for the destination, plus the +// fixed repository key. Names only — no value is produced, read, or returned. +func WalgCredentialEntries(target BackupTarget) []string { + entries := []string{ + target.Credentials.AccessKeyEntry, + target.Credentials.SecretKeyEntry, + WalgRepositoryKeyEntry, + } + if target.Credentials.SessionTokenEntry != "" { + entries = append(entries, target.Credentials.SessionTokenEntry) + } + sort.Strings(entries) + return entries +} + +// RenderWalgWrapper generates the wrapper staged beside the binary. +// +// It maps the entry names the project declared onto the fixed AWS_* names wal-g +// reads, and it is generated rather than shipped because those names come from +// the project. It holds no value: the wrapper reads them from the environment, +// which the container gets from the mode-0600 credential file on the host. +func RenderWalgWrapper(target BackupTarget) []byte { + var b strings.Builder + b.WriteString("#!/bin/sh\n") + b.WriteString("# Generated by Onebox. Maps the credential entry names this project\n") + b.WriteString("# declared onto the fixed names wal-g reads, then runs it.\n") + b.WriteString("#\n") + b.WriteString("# No credential is in this file. The values arrive in the environment\n") + b.WriteString("# from the mode-0600 credential file on the host; only the names are\n") + b.WriteString("# here, and the names are not secret.\n") + b.WriteString("set -eu\n") + assign := func(walgName, entry string) { + if entry == "" { + return + } + b.WriteString("if [ -n \"${" + entry + "-}\" ]; then\n") + b.WriteString(" " + walgName + "=\"$" + entry + "\"\n") + b.WriteString(" export " + walgName + "\n") + b.WriteString("fi\n") + } + assign("AWS_ACCESS_KEY_ID", target.Credentials.AccessKeyEntry) + assign("AWS_SECRET_ACCESS_KEY", target.Credentials.SecretKeyEntry) + assign("AWS_SESSION_TOKEN", target.Credentials.SessionTokenEntry) + assign("WALG_LIBSODIUM_KEY", WalgRepositoryKeyEntry) + b.WriteString("exec " + WalgMountPath + "/wal-g \"$@\"\n") + return []byte(b.String()) +} + +// WalgVersion is the wal-g release Onebox stages, pinned in the binary rather +// than resolved at run time, together with the checksum of each architecture's +// asset. The checksums are the provenance: the binary is verified against these +// before it is ever placed on a host, so a compromised release page cannot +// substitute one. They were taken from the release and confirmed against the +// binary this was validated with. +const WalgVersion = "v3.0.8" + +// walgChecksums maps the target's `uname -m` to the published asset and its +// SHA-256. A host reporting anything else is refused rather than guessed at. +var walgChecksums = map[string]struct{ Asset, SHA256 string }{ + "x86_64": { + Asset: "wal-g-pg-22.04-amd64", + SHA256: "f30544c5ce93cf83b87578e3c4a2e9c0e0ffc3d160ef89ecddaf75f397d98deb", + }, + "aarch64": { + Asset: "wal-g-pg-22.04-aarch64", + SHA256: "794d1a81f0c27825a1603bd39c0f2cf5dd8bed7cc36b598ca05d8d963c3d5fcf", + }, +} + +// WalgAssetFor returns the download name and expected checksum for a target's +// machine architecture. +// +// The assets are built against Ubuntu 22.04 and link against glibc, which is +// what the official Debian-based PostgreSQL images provide. An Alpine variant +// would not run them, which is why the driver's image is not a matter of taste. +func WalgAssetFor(machine string) (asset, sha256 string, err error) { + entry, ok := walgChecksums[normalizeMachine(machine)] + if !ok { + return "", "", fmt.Errorf( + "no verified wal-g build for machine architecture %q; backup supports x86_64 and aarch64", machine) + } + return entry.Asset, entry.SHA256, nil +} + +// WalgDownloadURL is where the pinned asset comes from. Check is by the +// checksum above, not by trusting this location. +func WalgDownloadURL(asset string) string { + return "https://github.com/wal-g/wal-g/releases/download/" + WalgVersion + "/" + asset +} + +// serviceBackup is everything renderService needs to run a service under +// backup. It is derived from observed durable lifecycle state rather than +// from the presence of a policy, for the same reason image selection is: a +// declared intent is not an established backup, and rendering a server with +// an archive_command pointing at a repository that was never initialised would +// take the database down at its next WAL switch. +type serviceBackup struct { + RuntimeHostDir string + CredentialFile string + ArchiveCommand string + ArchiveTimeout string + Environment map[string]any +} + +// backupForRender returns the rendering inputs for a protected service, or +// nil when the service is not running under established backup. +// +// Both live states qualify: `enabled` is the ordinary case, and +// `disable-pending` still archives, because disablement has not completed and +// stopping the archive first would put a gap in the recovery window that the +// retained history claims is continuous. +func (r *Resolved) backupForRender(n Names, serviceName string) (*serviceBackup, error) { + state, observed := r.serviceRuntime[serviceName] + if !observed || (state.BackupState != "enabled" && state.BackupState != "disable-pending") { + return nil, nil + } + service, ok := r.Services[serviceName] + if !ok { + return nil, errf("project_invalid", "services."+serviceName, "ob validate", "service is not declared") + } + driverName := service.Driver + if driverName == "" { + driverName = serviceName + } + if driverName != "postgres" { + // Every other driver's backup is declared in the lifecycle + // catalogue but has no executable renderer yet. Refusing here is the + // difference between "not implemented" and a service that runs as if + // the policy had never been written. + return nil, errf("backup_driver_unsupported", "services."+serviceName+".backup", "ob validate", + "driver %q has no executable backup renderer; only postgres is executable today", driverName) + } + projection, err := r.renderBackupProjection(serviceName, service, state) + if err != nil { + return nil, err + } + // The RPO is the reason archive_timeout exists. Without it a quiet database + // archives only when a 16MB segment fills, so the newest recoverable point + // can trail by hours whatever the policy says. Forcing a segment switch at + // the declared interval is what makes max_data_loss a bound rather than + // an aspiration. + maximumDataLoss, err := PositiveDuration(projection.Policy.MaxDataLoss) + if err != nil { + return nil, err + } + environment, err := WalgEnvironment(projection.Target, r.Spec.Name, serviceName) + if err != nil { + return nil, err + } + environment["OB_S3_KEY_ENTRY"] = projection.Target.Credentials.AccessKeyEntry + environment["OB_S3_SECRET_ENTRY"] = projection.Target.Credentials.SecretKeyEntry + return &serviceBackup{ + RuntimeHostDir: n.BackupRuntimeDir(serviceName), + CredentialFile: n.BackupCredentialFile(serviceName, projection.Policy.Target), + ArchiveCommand: WalgArchiveCommand(), + ArchiveTimeout: fmt.Sprintf("%ds", int(maximumDataLoss.Seconds())), + Environment: environment, + }, nil +} + +// RenderServiceBackupWrappers generates the credential wrapper for every +// service running under established backup, keyed by the host path it must +// be written to. It is separate from RenderServices because the two are placed +// separately: the wrapper and the binary beside it must be in place before the +// container that mounts them starts. +func (r *Resolved) RenderServiceBackupWrappers(env string) (map[string][]byte, error) { + n := r.Spec.NamesFor(env) + out := map[string][]byte{} + for _, name := range sortedKeys(r.Spec.Services) { + backup, err := r.backupForRender(n, name) + if err != nil { + return nil, err + } + if backup == nil { + continue + } + projection, err := r.renderBackupProjection(name, r.Services[name], r.serviceRuntime[name]) + if err != nil { + return nil, err + } + out[n.BackupWrapperFile(name)] = RenderWalgWrapper(projection.Target) + } + return out, nil +} + +// renderBackupProjection is the policy and target a *running* protected +// service is archiving under. +// +// The recorded projection wins over the project's current intent, and that +// ordering is the point. Enablement writes down exactly what it bound, and the +// server has been archiving to that repository ever since. If someone edits the +// policy — or deletes it — the archive does not retroactively move, so +// rendering from the edited project would point a live server at a repository +// its own history is not in. The project's intent takes effect at the next +// enablement, which is where the change can be made deliberately. +func (r *Resolved) renderBackupProjection(serviceName string, service Service, state ServiceRuntimeState) (BackupEffectiveProjection, error) { + _ = service + _ = state + projection, err := r.EffectiveBackupProjection(serviceName) + if err != nil { + return BackupEffectiveProjection{}, errf("backup_state_incomplete", "services."+serviceName+".backup", "ob backup status "+serviceName, + "service %s is protected but neither its durable state nor the project says what it is protected by; restore the policy or disable backup", serviceName) + } + return projection, nil +} + +// ServiceIsProtected reports whether a service runs under established +// backup, from durable state rather than from the project's intent. +func (r *Resolved) ServiceIsProtected(serviceName string) bool { + state, observed := r.serviceRuntime[serviceName] + return observed && (state.BackupState == "enabled" || state.BackupState == "disable-pending") +} + +// normalizeMachine folds the spellings of one architecture onto a single name. +// +// `uname -m` is not standardised: Linux says aarch64 where Darwin says arm64, +// and amd64 appears for x86_64. The architecture that matters is the one the +// *container* runs, which is Linux — so a Darwin host reporting arm64 still +// needs the Linux aarch64 build, and folding the names is exactly right rather +// than merely convenient. +func normalizeMachine(machine string) string { + switch strings.TrimSpace(strings.ToLower(machine)) { + case "aarch64", "arm64", "armv8l": + return "aarch64" + case "x86_64", "amd64", "x64": + return "x86_64" + default: + return strings.TrimSpace(machine) + } +} + +// ValidateWalgCredentials checks decrypted credential material against what the +// repository needs, before any of it reaches the target. +// +// Every problem is reported at once. An operator fixing a SOPS file one error +// message at a time is an operator doing four decrypt-edit-encrypt cycles to +// learn what could have been said in one. +func ValidateWalgCredentials(plaintext []byte, target BackupTarget) error { + values := map[string]string{} + for index, line := range strings.Split(string(plaintext), "\n") { + trimmed := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(line), "export ")) + if trimmed == "" || strings.HasPrefix(trimmed, "#") { + continue + } + name, value, ok := strings.Cut(trimmed, "=") + if !ok { + return errf("backup_credentials_invalid", "backup_targets."+target.Credentials.File, "ob backup enable", + "the decrypted credential file has no name=value on line %d", index+1) + } + // Quotes are stripped here for the same reason the installer strips + // them: a shell-sourced dotenv commonly carries them, and judging the + // quoted form would reject a perfectly good 64-character key for being + // 66 characters — with a message about hex that says nothing true. + values[strings.TrimSpace(name)] = unquoteCredentialValue(value) + } + + var missing []string + for _, entry := range WalgCredentialEntries(target) { + if values[entry] == "" { + missing = append(missing, entry) + } + } + if len(missing) > 0 { + return errf("backup_credentials_invalid", "backup_targets."+target.Credentials.File, "ob backup enable", + "the credential file does not define %s", strings.Join(missing, ", ")) + } + + // The repository key is checked here rather than discovered by wal-g at the + // first backup. It is read as hex, so a passphrase-shaped value is not a + // weak key — it is a key wal-g refuses outright, and finding that out from + // a failed backup is finding it out too late. + key := values[WalgRepositoryKeyEntry] + if len(key) != 64 || strings.TrimLeft(strings.ToLower(key), "0123456789abcdef") != "" { + return errf("backup_credentials_invalid", "backup_targets."+target.Credentials.File, "ob backup enable", + "%s must be exactly 64 hexadecimal characters — it is a 32-byte key, not a passphrase. Generate one with `openssl rand -hex 32`", + WalgRepositoryKeyEntry) + } + return nil +} + +// DataVolumeFor is the durable volume a service's data lives in, as the +// generated runtime names it. Recovery needs it because putting recovered data +// in service means replacing exactly that volume and nothing else. +func DataVolumeFor(s Service) string { return dataVolume(s) } + +// WalgRetentionTimeLayout is how wal-g's `delete retain --after` reads a +// timestamp, and WalgRetentionDateFormat is the same shape for `date` on the +// target. They must agree: the scheduled unit computes the cutoff with one and +// wal-g parses it with the other. +const ( + WalgRetentionTimeLayout = "2006-01-02T15:04:05" + WalgRetentionDateFormat = `%Y-%m-%dT%H:%M:%S` +) + +// WalgRetainCount is how many base backups must be kept to honour both +// retention bounds at once. +// +// keep and window are both *minimums*: at least this +// many recoverable bases, and at least this much continuous history. Retention +// must therefore satisfy whichever is larger, and on a frequent schedule that is +// the window — a service backing up every five minutes under a seven-day window +// needs far more than the handful of generations the count alone would keep. +// +// It is computed here rather than at run time because wal-g offers no working +// time bound. Its `--after` flag is documented and ignored, and `delete before +// FIND_FULL ` deletes nothing; both were measured against 3.0.8 +// rather than taken from the help text. `retain FULL n` does work, and both +// bounds are known from the policy alone, so the arithmetic the policy already +// implies is done up front and expressed as a count. +func WalgRetainCount(policy BackupPolicy) (int, error) { + window, err := PositiveDuration(policy.Retention.Window) + if err != nil { + return 0, err + } + dayGap, exact := maximumCronGap(policy.Schedule.Cron) + perDay, counted := cronRunsPerFiringDay(policy.Schedule.Cron) + if !exact || dayGap <= 0 || !counted || perDay <= 0 { + // A schedule whose spacing cannot be bounded cannot say how many backups + // a window holds. The generation floor is what remains, and it is the + // operator's declared minimum rather than a guess. + return policy.Retention.Keep, nil + } + // Backups in the window = firing days in the window × runs per firing day. + // Plus one, because the oldest backup kept must be *older* than the window: + // recovering to the window's earliest moment replays forward from the base + // taken before it. + firingDays := float64(window) / float64(dayGap) + needed := int(math.Ceil(firingDays*float64(perDay))) + 1 + if needed < policy.Retention.Keep { + return policy.Retention.Keep, nil + } + return needed, nil +} + +// cronRunsPerFiringDay counts how many times a schedule fires on a day it fires +// at all, from its minute and hour fields. +// +// maximumCronGap deliberately ignores those fields — it answers a different +// question, the longest gap in *days*, which is what a drill cadence is checked +// against. Using it alone as a backup interval reads "*/5 * * * *" as daily and +// retains two generations for a service that takes 288 a day. +func cronRunsPerFiringDay(expression string) (int, bool) { + fields := strings.Fields(expression) + if len(fields) != 5 { + return 0, false + } + minutes, ok := cronFieldCount(fields[0], 60) + if !ok { + return 0, false + } + hours, ok := cronFieldCount(fields[1], 24) + if !ok { + return 0, false + } + return minutes * hours, true +} + +// cronFieldCount counts the values one cron field selects, over the forms a +// schedule realistically uses: every value, a step, a list, or one value. +func cronFieldCount(field string, size int) (int, bool) { + if field == "*" { + return size, true + } + if step, ok := strings.CutPrefix(field, "*/"); ok { + every, err := strconv.Atoi(step) + if err != nil || every <= 0 || every > size { + return 0, false + } + return (size + every - 1) / every, true + } + count := 0 + for _, part := range strings.Split(field, ",") { + value, err := strconv.Atoi(strings.TrimSpace(part)) + if err != nil || value < 0 || value >= size { + return 0, false + } + count++ + } + if count == 0 { + return 0, false + } + return count, true +} + +// unquoteCredentialValue removes one matched pair of surrounding quotes, which +// is what a shell would do when sourcing the file and what the installer does +// when normalising it. +func unquoteCredentialValue(value string) string { + value = strings.TrimSpace(value) + if len(value) >= 2 && (value[0] == '"' || value[0] == '\'') && value[len(value)-1] == value[0] { + return value[1 : len(value)-1] + } + return value +} diff --git a/internal/app/backup_walg_test.go b/internal/app/backup_walg_test.go new file mode 100644 index 00000000..cdb26f7a --- /dev/null +++ b/internal/app/backup_walg_test.go @@ -0,0 +1,103 @@ +package app + +import ( + "strings" + "testing" +) + +// Retention has two floors and the larger one wins. A frequent schedule under a +// long window needs far more generations than the declared minimum, and keeping +// only the minimum silently shortens the window the policy promised. +func TestWalgRetainCountSatisfiesBothRetentionFloors(t *testing.T) { + for _, tc := range []struct { + name string + cron string + window string + minimum int + wantAtLeast int + }{ + {"frequent schedule is bound by the window", "*/5 * * * *", "24h", 2, 288}, + {"daily schedule over a week", "0 2 * * *", "168h", 2, 7}, + {"sparse schedule falls back to the declared minimum", "0 2 * * *", "1h", 5, 5}, + } { + t.Run(tc.name, func(t *testing.T) { + got, err := WalgRetainCount(BackupPolicy{ + Schedule: Schedule{Cron: tc.cron}, + Retention: BackupRetention{Keep: tc.minimum, Window: tc.window}, + }) + if err != nil { + t.Fatal(err) + } + if got < tc.wantAtLeast { + t.Fatalf("retain count = %d, want at least %d to honour both floors", got, tc.wantAtLeast) + } + if got < tc.minimum { + t.Fatalf("retain count %d is below the declared minimum %d", got, tc.minimum) + } + }) + } +} + +// The recorded projection wins over the project's current intent. +// +// Enablement writes down which repository it bound, and the server has archived +// there ever since. An operator editing backup_targets afterwards must not +// silently redirect a restore at a repository the history is not in — nor at a +// credential file installed under the old target's name. +func TestRecordedProjectionWinsOverEditedIntent(t *testing.T) { + recorded := BackupEffectiveProjection{ + Policy: BackupPolicy{Target: "original", RecoveryKind: "pitr", MaxDataLoss: "15m"}, + Target: BackupTarget{Kind: "s3-compatible", Bucket: "recorded-bucket", Endpoint: "https://a.example.net"}, + } + edited := &Resolved{ + Spec: &Spec{ + Name: "shop", BasePath: "/var/lib/ob", + Services: map[string]Service{"db": {Driver: "postgres", Version: 18, Backup: &BackupPolicy{ + Target: "moved", RecoveryKind: "pitr", MaxDataLoss: "15m", + }}}, + BackupTargets: map[string]BackupTarget{"moved": {Kind: "s3-compatible", Bucket: "new-bucket", Endpoint: "https://b.example.net"}}, + }, + Env: "production", + } + bound, err := edited.WithServiceRuntimeStates(map[string]ServiceRuntimeState{ + "db": {BackupState: "enabled", ServiceImage: "postgres@sha256:" + strings.Repeat("a", 64), + PublicationVerified: true, DigestAvailable: true, LastEffective: &recorded}, + }) + if err != nil { + t.Fatal(err) + } + got, err := bound.EffectiveBackupProjection("db") + if err != nil { + t.Fatal(err) + } + if got.Target.Bucket != "recorded-bucket" || got.Policy.Target != "original" { + t.Fatalf("projection = %#v, want the recorded one — an edited target must not redirect a restore", got) + } +} + +// The repository prefix must be injective. Hyphens are legal in both an app and +// a service name, so a hyphen join would land app `a-b`/service `c` and app +// `a`/service `b-c` on one prefix and interleave two clusters' backups. The +// prefix is unversioned, so this cannot be corrected later. +func TestWalgPrefixCannotCollideAcrossHyphenatedNames(t *testing.T) { + target := BackupTarget{Bucket: "backups", Prefix: "production"} + first := WalgPrefix(target, "a-b", "c") + second := WalgPrefix(target, "a", "b-c") + if first == second { + t.Fatalf("two distinct services share the repository prefix %q", first) + } +} + +// A quoted value is ordinary in a shell-sourced dotenv and is stripped when the +// file is installed, so validation must judge the same form. Judging the quoted +// text rejected a perfectly good key for being 66 characters, with a message +// about hex that said nothing true. +func TestQuotedCredentialValuesAreAccepted(t *testing.T) { + target := BackupTarget{Credentials: CredentialReference{ + AccessKeyEntry: "K", SecretKeyEntry: "S", File: "secrets/backup.env", + }} + plaintext := []byte("export K=\"key\"\nS='secret'\n" + WalgRepositoryKeyEntry + "=\"" + strings.Repeat("ab", 32) + "\"\n") + if err := ValidateWalgCredentials(plaintext, target); err != nil { + t.Fatalf("quoted credential file rejected: %v", err) + } +} diff --git a/internal/app/canonical_facts.go b/internal/app/canonical_facts.go index 0a33a26d..2e338a25 100644 --- a/internal/app/canonical_facts.go +++ b/internal/app/canonical_facts.go @@ -21,7 +21,7 @@ type CanonicalHygieneFacts struct { } type CanonicalServiceFacts struct { - ProtectionState CanonicalFact `json:"protection_state"` + BackupState CanonicalFact `json:"backup_state"` Tier CanonicalFact `json:"tier"` RecoveryKind CanonicalFact `json:"recovery_kind"` ServiceImageDigest CanonicalFact `json:"service_image_digest"` @@ -92,7 +92,7 @@ func validateCanonicalFacts(facts CanonicalFacts) error { fact CanonicalFact valid *regexp.Regexp }{ - {"protection_state", service.ProtectionState, enumPattern("undeclared", "declared", "enabled", "disable-pending", "disabled")}, + {"backup_state", service.BackupState, enumPattern("undeclared", "declared", "enabled", "disable-pending", "disabled")}, {"tier", service.Tier, enumPattern("Run", "Managed", "External")}, {"recovery_kind", service.RecoveryKind, enumPattern(eRecoveryKind...)}, {"service_image_digest", service.ServiceImageDigest, canonicalDigest}, diff --git a/internal/app/canonical_test.go b/internal/app/canonical_test.go index 381a068e..c7721616 100644 --- a/internal/app/canonical_test.go +++ b/internal/app/canonical_test.go @@ -98,13 +98,13 @@ func TestCanonicalAnnotatesOnlyWhatWasNotWritten(t *testing.T) { } } -func TestCanonicalProtectionFactsCoverEveryPublicOrigin(t *testing.T) { - project := strings.Replace(validProtectionProject, " server: deploy@app.example.net\n", ` server: deploy@app.example.net +func TestCanonicalBackupFactsCoverEveryPublicOrigin(t *testing.T) { + project := strings.Replace(validBackupProject, " server: deploy@app.example.net\n", ` server: deploy@app.example.net overrides: services: postgres: - protection: - retention: {minimum_generations: 10} + backup: + retention: {keep: 10} `, 1) spec, err := LoadBytes([]byte(project), "ob.yml") if err != nil { @@ -119,9 +119,9 @@ func TestCanonicalProtectionFactsCoverEveryPublicOrigin(t *testing.T) { origins[row[0]] = Origin(row[1]) } for path, want := range map[string]Origin{ - "services.postgres.protection.recovery_kind": OriginAuthored, - "services.postgres.protection.schedule.cron": OriginDefault, - "services.postgres.protection.retention.minimum_generations": OriginEnvironmentOverride, + "services.postgres.backup.recovery_kind": OriginAuthored, + "services.postgres.backup.schedule.cron": OriginDefault, + "services.postgres.backup.retention.keep": OriginEnvironmentOverride, } { if got := origins[path]; got != want { t.Errorf("%s origin = %q, want %q", path, got, want) @@ -137,7 +137,7 @@ func TestCanonicalProtectionFactsCoverEveryPublicOrigin(t *testing.T) { }, Services: map[string]CanonicalServiceFacts{ "postgres": { - ProtectionState: fact("enabled", OriginDerived), + BackupState: fact("enabled", OriginDerived), Tier: fact("Managed", OriginDerived), RecoveryKind: fact("pitr", OriginDerived), ServiceImageDigest: fact("sha256:"+strings.Repeat("a", 64), OriginObserved), @@ -157,7 +157,7 @@ func TestCanonicalProtectionFactsCoverEveryPublicOrigin(t *testing.T) { out := string(body) for _, golden := range []string{ "recovery_kind: pitr", - "minimum_generations: 10 # environment-override", + "keep: 10 # environment-override", "logging_max_size:", "value: 20MB", "origin: default", @@ -178,7 +178,7 @@ func TestCanonicalProtectionFactsCoverEveryPublicOrigin(t *testing.T) { } func TestCanonicalFactsRejectUnsafeObservedValuesWithoutReflectingThem(t *testing.T) { - spec, err := LoadBytes([]byte(validProtectionProject), "ob.yml") + spec, err := LoadBytes([]byte(validBackupProject), "ob.yml") if err != nil { t.Fatal(err) } @@ -194,7 +194,7 @@ func TestCanonicalFactsRejectUnsafeObservedValuesWithoutReflectingThem(t *testin }, Services: map[string]CanonicalServiceFacts{ "postgres": { - ProtectionState: fact("enabled", OriginDerived), Tier: fact("Managed", OriginDerived), RecoveryKind: fact("pitr", OriginDerived), + BackupState: fact("enabled", OriginDerived), Tier: fact("Managed", OriginDerived), RecoveryKind: fact("pitr", OriginDerived), ServiceImageDigest: fact(canary, OriginObserved), EncryptionMode: fact("client-side", OriginDerived), ObservedRPO: fact("4m", OriginObserved), ObservedRecoveryWindow: fact("7d", OriginObserved), ExpectedInterruption: fact("none", OriginDerived), DrillCapacityState: fact("available", OriginObserved), diff --git a/internal/app/constraints.go b/internal/app/constraints.go index 556b2b91..f0fb2ff5 100644 --- a/internal/app/constraints.go +++ b/internal/app/constraints.go @@ -137,7 +137,7 @@ var ( gS3Region = grammar{"S3 region", regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,62}$`), "a lower-case S3-compatible region of letters, digits and hyphens"} - gProtectionOwner = grammar{"protection owner", regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,127}$`), + gBackupOwner = grammar{"backup owner", regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,127}$`), "a stable operator or provider identity of letters, digits, dots, @, colons, slashes, underscores and hyphens"} ) @@ -172,9 +172,9 @@ var ( eRole = []string{RoleApplication, RoleWorker, RoleDaemon, RoleJob} eConnectionPart = []string{"url", "host", "port", "user", "password", "database"} eBackupTargetKind = []string{"s3-compatible"} - eBackupTLS = []string{"required", "insecure"} + eBackupTLS = []string{"verify", "skip-verify"} eRecoveryKind = []string{"snapshot", "pitr", "cold"} - eEncryptionMode = []string{"client-side", "archive-password", "server-side-sse"} + eEncryptionMode = []string{"client-side", "server-side"} eExternalProbeKind = []string{"driver-health"} ) diff --git a/internal/app/defaults.go b/internal/app/defaults.go index 98390ff8..1a89c7e6 100644 --- a/internal/app/defaults.go +++ b/internal/app/defaults.go @@ -58,9 +58,9 @@ func applyDefaults(p *Spec, raw map[string]any, derived map[string]Origin) { // absence produced an untyped complaint about an empty duration for a // field the author had never heard of. A default is the answer the // evolution rules already allow. - if e.Policy.RequireMigrationBackup && e.Policy.MigrationBackupMaximumAge == "" { - e.Policy.MigrationBackupMaximumAge = "24h" - mark(path + ".policy.migration_backup_maximum_age") + if e.Policy.Migrations.RequireBackup && e.Policy.Migrations.BackupMaxAge == "" { + e.Policy.Migrations.BackupMaxAge = "24h" + mark(path + ".policy.migrations.backup_max_age") } p.Environments[name] = e } @@ -163,34 +163,34 @@ func applyDefaults(p *Spec, raw map[string]any, derived map[string]Origin) { s.Persistence.Mode = "durable" mark(path + ".persistence.mode") } - if s.Protection != nil { - if s.Protection.Schedule.Cron == "" { - s.Protection.Schedule.Cron = "0 2 * * *" - mark(path + ".protection.schedule.cron") + if s.Backup != nil { + if s.Backup.Schedule.Cron == "" { + s.Backup.Schedule.Cron = "0 2 * * *" + mark(path + ".backup.schedule.cron") } - if s.Protection.Schedule.Timezone == "" { - s.Protection.Schedule.Timezone = "UTC" - mark(path + ".protection.schedule.timezone") + if s.Backup.Schedule.Timezone == "" { + s.Backup.Schedule.Timezone = "UTC" + mark(path + ".backup.schedule.timezone") } - if s.Protection.Retention.MinimumGenerations == 0 && !stated(raw, path+".protection.retention.minimum_generations") { - s.Protection.Retention.MinimumGenerations = 7 - mark(path + ".protection.retention.minimum_generations") + if s.Backup.Retention.Keep == 0 && !stated(raw, path+".backup.retention.keep") { + s.Backup.Retention.Keep = 7 + mark(path + ".backup.retention.keep") } - if s.Protection.Retention.RecoveryWindow == "" { - s.Protection.Retention.RecoveryWindow = "7d" - mark(path + ".protection.retention.recovery_window") + if s.Backup.Retention.Window == "" { + s.Backup.Retention.Window = "7d" + mark(path + ".backup.retention.window") } - if s.Protection.RestoreDrill.Schedule.Cron == "" { - s.Protection.RestoreDrill.Schedule.Cron = "0 3 * * 0,3" - mark(path + ".protection.restore_drill.schedule.cron") + if s.Backup.Drill.Schedule.Cron == "" { + s.Backup.Drill.Schedule.Cron = "0 3 * * 0,3" + mark(path + ".backup.drill.schedule.cron") } - if s.Protection.RestoreDrill.Schedule.Timezone == "" { - s.Protection.RestoreDrill.Schedule.Timezone = "UTC" - mark(path + ".protection.restore_drill.schedule.timezone") + if s.Backup.Drill.Schedule.Timezone == "" { + s.Backup.Drill.Schedule.Timezone = "UTC" + mark(path + ".backup.drill.schedule.timezone") } - if s.Protection.RestoreDrill.ProofMaximumAge == "" { - s.Protection.RestoreDrill.ProofMaximumAge = "7d" - mark(path + ".protection.restore_drill.proof_maximum_age") + if s.Backup.Drill.MaxAge == "" { + s.Backup.Drill.MaxAge = "7d" + mark(path + ".backup.drill.max_age") } } p.Services[name] = s @@ -200,7 +200,7 @@ func applyDefaults(p *Spec, raw map[string]any, derived map[string]Origin) { target := p.BackupTargets[name] path := "backup_targets." + name if target.TLS == "" { - target.TLS = "required" + target.TLS = "verify" mark(path + ".tls") } if target.Credentials.Provider == "" { @@ -226,9 +226,9 @@ func applyDefaults(p *Spec, raw map[string]any, derived map[string]Origin) { external.Probe.Timeout = "5s" mark(path + ".probe.timeout") } - if external.Probe.MaximumAge == "" { - external.Probe.MaximumAge = "5m" - mark(path + ".probe.maximum_age") + if external.Probe.MaxAge == "" { + external.Probe.MaxAge = "5m" + mark(path + ".probe.max_age") } } p.ExternalServices[name] = external diff --git a/internal/app/errors.go b/internal/app/errors.go index 2aaea291..ceeb7603 100644 --- a/internal/app/errors.go +++ b/internal/app/errors.go @@ -23,39 +23,41 @@ var errorCodes = map[string]string{ "project_invalid": "a value that does not satisfy the contract", // Rules across more than one field. - "app_required": "the shorthand form needs an application name to attach the workload to", - "no_environment": "a project must declare at least one environment", - "no_workload": "a project must declare at least one workload", - "workload_malformed": "a workload is not a mapping", - "workload_source": "a workload declares other than exactly one of build, image or compose", - "stateful_replicas": "a workload keeping durable state asks for more than one replica", - "strategy_ungated": "a rolling release is asked for by a workload with no health check to gate it", - "shorthand_and_workloads": "top-level shorthand cannot be combined with a workloads block", - "routing_exclusive": "the domain shorthand and the routes list say the same thing twice", - "routing_incomplete": "domain and port are declared together or not at all", - "route_collision": "two workloads claim the same address", - "route_without_proxy": "a route is declared with nothing to route it", - "identifier_collision": "a name is used by both a workload and a service", - "derived_name_too_long": "a name Onebox derives exceeds the runtime's limit", - "unknown_prerequisite": "a prerequisite names something the project does not declare", - "prerequisite_has_no_health": "a wait for health names something with no health check", - "unknown_service_driver": "a service names a driver Onebox has no implementation for", - "service_settings_unsupported": "a setting was declared for a driver with no way to apply it", - "schedule_untranslatable": "a cron expression whose meaning the host's scheduler cannot preserve", - "unknown_environment": "an environment the project does not declare", - "backup_driver_unsupported": "a runnable service driver has no qualified executable protection contract", - "backup_target_unknown": "a protection policy selects no declared backup target", - "backup_target_not_independent": "a backup target shares the protected failure domain", - "backup_encryption_unverified": "the selected target cannot prove the encryption mode required by the recovery kind", - "backup_retention_unsupported": "the declared recovery history cannot map to supported retention semantics", - "backup_interruption_not_authorized": "the selected recovery contract needs a recurring stopped-service window the author did not permit", - "recovery_objective_unsupported": "the service driver, target, or version cannot execute the declared recovery kind", - "restore_drill_schedule_too_sparse": "the restore-drill cadence cannot keep restore proof current", - "protected_service_patch_unsupported": "no exact qualified protected current-to-candidate image transition exists", - "protection_image_revert_unsafe": "tag rendering would strand an effective protection prerequisite", - "protection_service_image_unpublished": "the protected service image lacks verified publication provenance", - "service_image_digest_unavailable": "the immutable service image is unavailable from registry and exact cache", - "service_image_patch_disable_pending": "protected image refresh is refused while disablement is pending", + "app_required": "the shorthand form needs an application name to attach the workload to", + "no_environment": "a project must declare at least one environment", + "no_workload": "a project must declare at least one workload", + "workload_malformed": "a workload is not a mapping", + "workload_source": "a workload declares other than exactly one of build, image or compose", + "stateful_replicas": "a workload keeping durable state asks for more than one replica", + "strategy_ungated": "a rolling release is asked for by a workload with no health check to gate it", + "shorthand_and_workloads": "top-level shorthand cannot be combined with a workloads block", + "routing_exclusive": "the domain shorthand and the routes list say the same thing twice", + "routing_incomplete": "domain and port are declared together or not at all", + "route_collision": "two workloads claim the same address", + "route_without_proxy": "a route is declared with nothing to route it", + "identifier_collision": "a name is used by both a workload and a service", + "derived_name_too_long": "a name Onebox derives exceeds the runtime's limit", + "unknown_prerequisite": "a prerequisite names something the project does not declare", + "prerequisite_has_no_health": "a wait for health names something with no health check", + "unknown_service_driver": "a service names a driver Onebox has no implementation for", + "service_settings_unsupported": "a setting was declared for a driver with no way to apply it", + "schedule_untranslatable": "a cron expression whose meaning the host's scheduler cannot preserve", + "unknown_environment": "an environment the project does not declare", + "backup_driver_unsupported": "a runnable service driver has no qualified executable backup contract", + "backup_target_unknown": "a backup policy selects no declared backup target", + "backup_target_not_independent": "a backup target shares the protected failure domain", + "backup_encryption_unverified": "the selected target cannot prove the encryption mode required by the recovery kind", + "backup_retention_unsupported": "the declared recovery history cannot map to supported retention semantics", + "backup_interruption_not_authorized": "the selected recovery contract needs a recurring stopped-service window the author did not permit", + "recovery_objective_unsupported": "the service driver, target, or version cannot execute the declared recovery kind", + "drill_schedule_too_sparse": "the declared drill cadence is too sparse to keep restore proof within its maximum age", + "service_patch_unsupported": "no exact qualified protected current-to-candidate image transition exists", + "backup_image_revert_unsafe": "tag rendering would strand an effective backup prerequisite", + "backup_state_incomplete": "a protected service does not record what it is protected by", + "backup_credentials_invalid": "decrypted backup credentials are missing or malformed", + "backup_service_image_unpublished": "the protected service image lacks verified publication provenance", + "service_image_digest_unavailable": "the immutable service image is unavailable from registry and exact cache", + "service_image_patch_disable_pending": "protected image refresh is refused while disablement is pending", // Repository paths. "path_absolute": "a repository path may not be absolute", diff --git a/internal/app/external_schema.go b/internal/app/external_schema.go index bbabfce9..e863bb9b 100644 --- a/internal/app/external_schema.go +++ b/internal/app/external_schema.go @@ -33,7 +33,7 @@ func validateExternalService(external ExternalService, path string) error { } } } - if err := gProtectionOwner.check(path+".protection_owner", external.ProtectionOwner); err != nil { + if err := gBackupOwner.check(path+".backup_owner", external.BackupOwner); err != nil { return err } if external.Probe != nil { @@ -43,8 +43,8 @@ func validateExternalService(external ExternalService, path string) error { if _, err := PositiveDuration(external.Probe.Timeout); err != nil { return errf("project_invalid", path+".probe.timeout", "ob validate", "probe timeout must be a positive duration: %v", err) } - if _, err := PositiveDuration(external.Probe.MaximumAge); err != nil { - return errf("project_invalid", path+".probe.maximum_age", "ob validate", "probe maximum_age must be a positive duration: %v", err) + if _, err := PositiveDuration(external.Probe.MaxAge); err != nil { + return errf("project_invalid", path+".probe.max_age", "ob validate", "probe max_age must be a positive duration: %v", err) } } return nil diff --git a/internal/app/external_schema_test.go b/internal/app/external_schema_test.go index 2d8a1002..f5b51203 100644 --- a/internal/app/external_schema_test.go +++ b/internal/app/external_schema_test.go @@ -21,7 +21,7 @@ external_services: connection: source: {file: secrets/database.env, provider: sops} entries: {url: DATABASE_URL} - protection_owner: platform-team/rds + backup_owner: platform-team/rds probe: {} ` @@ -48,7 +48,7 @@ external_services: connection: source: {file: secrets/database.env, provider: sops} entries: {url: DATABASE_URL} - protection_owner: platform-team/rds + backup_owner: platform-team/rds `, code: "identifier_collision", }, @@ -65,7 +65,7 @@ external_services: connection: source: {file: secrets/database.env, provider: sops} entries: {url: DATABASE_URL} - protection_owner: platform-team/rds + backup_owner: platform-team/rds `, code: "unknown_field", }, @@ -82,13 +82,13 @@ external_services: t.Fatal(err) } external := project.ExternalServices["database"] - if external.Driver != "postgres" || external.ProtectionOwner != "platform-team/rds" { + if external.Driver != "postgres" || external.BackupOwner != "platform-team/rds" { t.Fatalf("external service = %#v", external) } if external.Connection.Source.Provider != "sops" || external.Connection.Entries["url"] != "DATABASE_URL" { t.Fatalf("trusted connection = %#v", external.Connection) } - if external.Probe == nil || external.Probe.Kind != "driver-health" || external.Probe.Timeout != "5s" || external.Probe.MaximumAge != "5m" { + if external.Probe == nil || external.Probe.Kind != "driver-health" || external.Probe.Timeout != "5s" || external.Probe.MaxAge != "5m" { t.Fatalf("read-only probe defaults = %#v", external.Probe) } }) diff --git a/internal/app/generate.go b/internal/app/generate.go index a3df1274..773bdc10 100644 --- a/internal/app/generate.go +++ b/internal/app/generate.go @@ -143,7 +143,11 @@ func (r *Resolved) render(env, releaseID string, images Images) (*Rendered, erro if err != nil { return nil, err } - doc, err := p.renderService(n, name, p.Services[name], selection.Image) + backup, err := r.backupForRender(n, name) + if err != nil { + return nil, err + } + doc, err := p.renderService(n, name, p.Services[name], selection.Image, backup) if err != nil { return nil, err } @@ -200,6 +204,14 @@ func (p *Spec) renderWorkload(n Names, name string, w Workload, releaseID string if ref := images[name]; ref != "" { svc["image"] = ref } + // The declared policy has to reach Compose, not just the explicit pull + // step onebox runs before a release. Compose fetches a missing image + // during `up` on its own, so a workload declaring `pull: never` was + // fetched anyway — proved on a live host, where a release with + // `pull: never` started an image the host did not have. + if policy := composePullPolicy(w.Image.Pull); policy != "" { + svc["pull_policy"] = policy + } case w.Build != nil: ref, ok := images[name] if !ok || ref == "" { @@ -852,7 +864,11 @@ func (r *Resolved) RenderServices(env string) (map[string][]byte, error) { if err != nil { return nil, err } - doc, err := p.renderService(n, name, service, selection.Image) + backup, err := r.backupForRender(n, name) + if err != nil { + return nil, err + } + doc, err := p.renderService(n, name, service, selection.Image, backup) if err != nil { return nil, err } @@ -920,3 +936,18 @@ func (p *Spec) connectionVars(name string, w Workload) map[string]string { } return out } + +// composePullPolicy maps the declared `image.pull` onto Compose's own spelling. +// They agree on all three names today; the mapping exists so the schema is free +// to keep its vocabulary if Compose changes its own. +func composePullPolicy(declared string) string { + switch declared { + case "always": + return "always" + case "never": + return "never" + case "missing": + return "missing" + } + return "" +} diff --git a/internal/app/jsonschema.go b/internal/app/jsonschema.go index 8bdad172..6fb13b3d 100644 --- a/internal/app/jsonschema.go +++ b/internal/app/jsonschema.go @@ -312,9 +312,9 @@ var schemaConstraints = []struct { {[]string{"base_path"}, pattern(gAbsPath)}, {[]string{"environments", "*", "base_path"}, pattern(gAbsPath)}, - {[]string{"environments", "*", "policy", "minimum_onebox_version"}, pattern(gCalVer)}, - {[]string{"environments", "*", "policy", "minimum_plan_schema"}, pattern(gPlanSchema)}, - {[]string{"environments", "*", "policy", "migration_backup_maximum_age"}, pattern(gDur)}, + {[]string{"environments", "*", "policy", "min_onebox_version"}, pattern(gCalVer)}, + {[]string{"environments", "*", "policy", "min_plan_schema"}, pattern(gPlanSchema)}, + {[]string{"environments", "*", "policy", "migrations", "backup_max_age"}, pattern(gDur)}, {[]string{"workloads", "*", "role"}, enum(eRole)}, {[]string{"workloads", "*", "replicas"}, map[string]any{"minimum": 1}}, @@ -373,17 +373,16 @@ var schemaConstraints = []struct { {[]string{"services", "*", "resources", "memory"}, pattern(gSize)}, {[]string{"services", "*", "resources", "cpus"}, pattern(gCpus)}, {[]string{"services", "*", "volumes", "items"}, pattern(gIdent)}, - {[]string{"services", "*", "protection", "target"}, pattern(gIdent)}, - {[]string{"services", "*", "protection", "recovery_kind"}, enum(eRecoveryKind)}, - {[]string{"services", "*", "protection", "maximum_data_loss"}, pattern(gDur)}, - {[]string{"services", "*", "protection", "schedule", "cron"}, pattern(gCron)}, - {[]string{"services", "*", "protection", "schedule", "timezone"}, pattern(gTZ)}, - {[]string{"services", "*", "protection", "retention", "minimum_generations"}, map[string]any{"minimum": 1}}, - {[]string{"services", "*", "protection", "retention", "recovery_window"}, pattern(gDur)}, - {[]string{"services", "*", "protection", "restore_drill", "schedule", "cron"}, pattern(gCron)}, - {[]string{"services", "*", "protection", "restore_drill", "schedule", "timezone"}, pattern(gTZ)}, - {[]string{"services", "*", "protection", "restore_drill", "proof_maximum_age"}, pattern(gDur)}, - {[]string{"services", "*", "protection", "restore_drill", "staging_filesystem"}, pattern(gAbsPath)}, + {[]string{"services", "*", "backup", "target"}, pattern(gIdent)}, + {[]string{"services", "*", "backup", "recovery_kind"}, enum(eRecoveryKind)}, + {[]string{"services", "*", "backup", "max_data_loss"}, pattern(gDur)}, + {[]string{"services", "*", "backup", "schedule", "cron"}, pattern(gCron)}, + {[]string{"services", "*", "backup", "schedule", "timezone"}, pattern(gTZ)}, + {[]string{"services", "*", "backup", "retention", "keep"}, map[string]any{"minimum": 1}}, + {[]string{"services", "*", "backup", "retention", "window"}, pattern(gDur)}, + {[]string{"services", "*", "backup", "drill", "schedule", "cron"}, pattern(gCron)}, + {[]string{"services", "*", "backup", "drill", "schedule", "timezone"}, pattern(gTZ)}, + {[]string{"services", "*", "backup", "drill", "max_age"}, pattern(gDur)}, {[]string{"backup_targets", "*", "kind"}, enum(eBackupTargetKind)}, {[]string{"backup_targets", "*", "endpoint"}, pattern(gHTTPURL)}, @@ -406,10 +405,10 @@ var schemaConstraints = []struct { {[]string{"external_services", "*", "connection", "source", "file"}, pattern(gRepoPath)}, {[]string{"external_services", "*", "connection", "source", "provider"}, enum(eSecretProvider)}, {[]string{"external_services", "*", "connection", "entries", "*"}, pattern(gEnvName)}, - {[]string{"external_services", "*", "protection_owner"}, pattern(gProtectionOwner)}, + {[]string{"external_services", "*", "backup_owner"}, pattern(gBackupOwner)}, {[]string{"external_services", "*", "probe", "kind"}, enum(eExternalProbeKind)}, {[]string{"external_services", "*", "probe", "timeout"}, pattern(gDur)}, - {[]string{"external_services", "*", "probe", "maximum_age"}, pattern(gDur)}, + {[]string{"external_services", "*", "probe", "max_age"}, pattern(gDur)}, {[]string{"proxy", "kind"}, enum(eProxyKind)}, {[]string{"proxy", "image"}, pattern(gImageRef)}, @@ -427,12 +426,10 @@ var schemaConstraints = []struct { {[]string{"environments", "*", "env_files", "items", "provider"}, enum(eSecretProvider)}, {[]string{"environments", "*", "env_files", "items"}, map[string]any{"required": []any{"file"}}}, {[]string{"runtime", "env_checks", "items", "file"}, pattern(gRepoPath)}, - {[]string{"verifications", "items", "http"}, pattern(gURLPath)}, - {[]string{"verifications", "items", "url"}, pattern(gHTTPURL)}, - {[]string{"verifications", "items", "port"}, portBounds()}, - {[]string{"verifications", "items", "status_codes", "items"}, map[string]any{"minimum": 100, "maximum": 599}}, - {[]string{"observability", "alerts", "unhealthy_after"}, pattern(gDur)}, - {[]string{"observability", "logs", "retention"}, pattern(gDur)}, + {[]string{"checks", "http", "items", "path"}, pattern(gURLPath)}, + {[]string{"checks", "url", "items", "url"}, pattern(gHTTPURL)}, + {[]string{"checks", "http", "items", "port"}, portBounds()}, + {[]string{"checks", "url", "items", "status_codes", "items"}, map[string]any{"minimum": 100, "maximum": 599}}, {[]string{"notifications", "*", "on", "items"}, enum(eNotifyEvent)}, {[]string{"services", "*", "settings"}, propertyNames(gSettingKey)}, {[]string{"workloads", "*", "logging", "driver"}, pattern(gLogDriver)}, diff --git a/internal/app/jsonschema_test.go b/internal/app/jsonschema_test.go index 5801f7a4..67249c19 100644 --- a/internal/app/jsonschema_test.go +++ b/internal/app/jsonschema_test.go @@ -70,20 +70,19 @@ var beyondJSONSchema = map[string]string{ "workload and service share a name": "an identifier is unique across workloads and services, which are separate objects", // Facts about values that only resolution knows. - "unknown prerequisite": "a prerequisite must name something the project declares", - "absolute env_file": "a path that resolves inside the repository after joining", - "absolute compose ref": "a path that resolves inside the repository after joining", - "protection self target": "a target and environment host are declared in separate objects", - "hook naming an unlisted seam": "a hook key is a seam or a declared job, and the job list is a separate object", - "hook naming neither": "a hook key is a seam or a declared job, and the job list is a separate object", - "protection unsupported objective": "a recovery kind is qualified by the selected service driver", - "protection sparse drill": "cron cadence must be compared with restore proof age", + "unknown prerequisite": "a prerequisite must name something the project declares", + "absolute env_file": "a path that resolves inside the repository after joining", + "absolute compose ref": "a path that resolves inside the repository after joining", + "backup self target": "a target and environment host are declared in separate objects", + "hook naming an unlisted seam": "a hook key is a seam or a declared job, and the job list is a separate object", + "hook naming neither": "a hook key is a seam or a declared job, and the job list is a separate object", + "backup unsupported objective": "a recovery kind is qualified by the selected service driver", + "backup sparse drill": "cron cadence must be compared with restore proof age", // Exclusivity within one object that a schema could express, and does not // here because the resulting document would be harder to read than the // rule it encodes. - "verifications url with exec": "a verification is exactly one kind", - "verifications workload without probe": "an http or exec check names the workload it runs in", + "http check without a path": "a grouped http check still needs its path", } func TestPublishedSchemaMatchesTheLoader(t *testing.T) { diff --git a/internal/app/load.go b/internal/app/load.go index bfaacdd3..b7ef3058 100644 --- a/internal/app/load.go +++ b/internal/app/load.go @@ -544,7 +544,7 @@ func crossFieldRules(p *Spec) error { } if external, clash := p.ExternalServices[name]; clash { return errf("identifier_collision", "external_services."+name, "", - "%q is declared as both a Onebox-run service and an external service owned by %q", name, external.ProtectionOwner) + "%q is declared as both a Onebox-run service and an external service owned by %q", name, external.BackupOwner) } svc := p.Services[name] key, d, known := driverOf(name, svc) @@ -554,7 +554,7 @@ func crossFieldRules(p *Spec) error { "To run something else, declare it as a daemon workload — you own the image and the settings then.", key, strings.Join(DriverNames(), ", ")) } - if err := validateProtectionSelection(p, name, key, svc); err != nil { + if err := validateBackupSelection(p, name, key, svc); err != nil { return err } // Materialise the durable volume in the project rather than only in the @@ -569,13 +569,13 @@ func crossFieldRules(p *Spec) error { "%q declares volumes and persistence.mode: ephemeral; an ephemeral service owns no durable volume, "+ "so declare durable persistence or remove the volumes", name) } - // Protection is a contract about recovering durable data. An ephemeral + // Backup is a contract about recovering durable data. An ephemeral // service has none: seeding the active volume fails at apply time on a // volume that was never created, and the sealed identity would name it. - if serviceIsEphemeral(svc) && svc.Protection != nil { - return errf("project_invalid", "services."+name+".protection", "", - "%q declares protection and persistence.mode: ephemeral; there is no durable data to protect, "+ - "so declare durable persistence or remove the protection policy", name) + if serviceIsEphemeral(svc) && svc.Backup != nil { + return errf("project_invalid", "services."+name+".backup", "", + "%q declares backup and persistence.mode: ephemeral; there is no durable data to protect, "+ + "so declare durable persistence or remove the backup policy", name) } if d.dataPath != "" && len(svc.Volumes) == 0 && !serviceIsEphemeral(svc) { svc.Volumes = []string{"data"} diff --git a/internal/app/load_test.go b/internal/app/load_test.go index 44c4094d..87afcc96 100644 --- a/internal/app/load_test.go +++ b/internal/app/load_test.go @@ -56,9 +56,9 @@ func conformanceCases() []conformanceCase { {"absolute env_file", min + "runtime: {env_files: [/etc/x.env]}\n", false}, {"relative env_file", min + "runtime: {env_files: [.env.production]}\n", true}, {"base_path absolute", min + "base_path: /mnt/data/ob\n", true}, - {"duration in days", "api_version: onebox.run/v1\napp: a\nimage: nginx\nenvironments: {p: {server: h, policy: {migration_backup_maximum_age: 14d}}}\n", true}, - {"non-calver minimum version", "api_version: onebox.run/v1\napp: a\nimage: nginx\nenvironments: {p: {server: h, policy: {minimum_onebox_version: 0.0.1-m0}}}\n", false}, - {"incomplete plan schema", "api_version: onebox.run/v1\napp: a\nimage: nginx\nenvironments: {p: {server: h, policy: {minimum_plan_schema: \"onebox.run/executable-deploy-plan/v1alpha\"}}}\n", false}, + {"duration in days", "api_version: onebox.run/v1\napp: a\nimage: nginx\nenvironments: {p: {server: h, policy: {migrations: {backup_max_age: 14d}}}}\n", true}, + {"non-calver minimum version", "api_version: onebox.run/v1\napp: a\nimage: nginx\nenvironments: {p: {server: h, policy: {min_onebox_version: 0.0.1-m0}}}\n", false}, + {"incomplete plan schema", "api_version: onebox.run/v1\napp: a\nimage: nginx\nenvironments: {p: {server: h, policy: {min_plan_schema: \"onebox.run/executable-deploy-plan/v1alpha\"}}}\n", false}, {"hook with local", min + "hooks: {pre_release: {run: scripts/build.sh, local: true}}\n", true}, // A hook key is a lifecycle seam OR a declared job name. Both halves need // a case: an unlisted seam loads and never fires, and refusing a job name @@ -66,11 +66,6 @@ func conformanceCases() []conformanceCase { {"hook naming an unlisted seam", min + "hooks: {pre_deploy: {run: scripts/backup.sh}}\n", false}, {"hook naming a declared job", "api_version: onebox.run/v1\napp: a\nenvironments: {p: {server: h}}\nhooks: {migrate: {run: ./bin/migrate}}\nworkloads:\n w: {role: application, image: nginx}\n migrate: {role: job, image: nginx, data_effect: migration}\n", true}, {"hook naming neither", min + "hooks: {typo_hook: {run: scripts/x.sh}}\n", false}, - // observability sub-blocks are independent; each must be checked without - // the other present. - {"log retention without alerts", min + "observability: {logs: {retention: bogus}}\n", false}, - {"alerts without logs", min + "observability: {alerts: {unhealthy_after: 5m}}\n", true}, - {"log retention as an integer", min + "observability: {logs: {retention: 30}}\n", false}, // A settings key is interpolated into a generated shell command without // quoting, so the grammar is the only thing between a project file and // a root shell on the server. @@ -89,14 +84,14 @@ func conformanceCases() []conformanceCase { // The contract publishes persistence.mode defaulting to durable. That // default was unreachable while the block was absent, so a workload with // a managed volume read as holding nothing — and doctor, the backup gate - // and the protection gate each guessed the same wrong way. + // and the backup gate each guessed the same wrong way. {"volumes without persistence still load", wl("w: {image: nginx, volumes: [{name: data, path: /data}]}"), true}, {"a bind mount is not durable", wl("w: {image: nginx, volumes: [{source: ./cfg, path: /etc/app}], replicas: 3}"), true}, // Inference must not tighten a refusal against a project that loads. {"inferred durability does not refuse replicas", wl("w: {image: nginx, volumes: [{name: data, path: /data}], replicas: 3}"), true}, {"declared durability still refuses replicas", wl("w: {image: nginx, volumes: [{name: data, path: /data}], persistence: {mode: durable}, replicas: 3}"), false}, {"persistence block with no mode still refuses replicas", wl("w: {image: nginx, volumes: [{name: data, path: /data}], persistence: {}, replicas: 3}"), false}, - {"protection is no longer a field", wl("w: {image: nginx, protection: {backup: {schedule: {cron: \"0 3 * * *\"}}}}"), false}, + {"backup is no longer a field", wl("w: {image: nginx, backup: {backup: {schedule: {cron: \"0 3 * * *\"}}}}"), false}, {"a near-miss field name", wl("w: {image: nginx, replicaz: 3}"), false}, // A closed value set is only closed if a value outside it is refused, // and the refusal has to name the set rather than the type. @@ -114,10 +109,10 @@ func conformanceCases() []conformanceCase { {"unknown env file provider", min + "runtime: {env_files: [{file: s.env, provider: vault}]}\n", false}, {"env file entry without a file", min + "runtime: {env_files: [{provider: sops}]}\n", false}, {"environment-scoped env files", "api_version: onebox.run/v1\napp: a\nimage: nginx\nenvironments: {p: {server: h, env_files: [.env.p]}}\n", true}, - {"verifications workload without probe", min + "verifications: [{workload: ledger}]\n", false}, - {"verifications url with exec", min + "verifications: [{url: \"https://x/\", exec: \"echo\"}]\n", false}, - {"verifications url contains advisory", min + "verifications: [{url: \"https://x/\", contains: \"-*" +// as its own and removes the ones no longer declared, so backup timers named +// that way were deleted by the next deploy — every scheduled backup silently +// stopped, and the only trace was a line in the deploy output saying the +// schedule was "no longer declared". +func (n Names) BackupTimerForEnvironment(environment, service, operation string) string { + return n.BackupUnitForEnvironment(environment, service, operation) + ".timer" +} + +// BackupUnitForEnvironment is the systemd unit name without its suffix, so +// the .service and .timer that pair together cannot be spelled differently. +func (n Names) BackupUnitForEnvironment(environment, service, operation string) string { + return BackupUnitPrefix + n.App + "-" + environment + "-" + service + "-" + operation } +// BackupUnitPrefix is the systemd namespace backup owns outright. +const BackupUnitPrefix = "ob-backup-" + // Container is a workload's stable runtime slot. Container names are // host-global, so every one carries the application, component, and a // one-based replica ordinal — including singleton workloads. @@ -263,12 +297,12 @@ func (p *Spec) All(env string) []string { for _, v := range p.Services[s].Volumes { out = append(out, n.ServiceVolume(s, v)) } - if p.Services[s].Protection != nil { + if p.Services[s].Backup != nil { out = append(out, - n.ProtectionRestoreProject(s), - n.ProtectionRestoreContainer(s), - n.ProtectionRestoreNetwork(s), - n.ProtectionRestoreVolume(s), + n.BackupRestoreProject(s), + n.BackupRestoreContainer(s), + n.BackupRestoreNetwork(s), + n.BackupRestoreVolume(s), ) } } @@ -295,6 +329,10 @@ func routesOf(w Workload) []Route { // expanded, so callers never handle two shapes. func (w Workload) NormalisedRoutes() []Route { return routesOf(w) } +// Join is the injective separator rule above, exported for derived identifiers +// that live outside this file — a backup repository prefix among them. +func Join(parts ...string) string { return join(parts...) } + func join(parts ...string) string { out := "" for i, p := range parts { @@ -320,3 +358,21 @@ func runtimeName(parts ...string) string { } return strings.Join(escaped, "-") } + +// BackupRunLock is the mutex over actual repository work for one service. +// +// It exists because Onebox's backup lock is a value written to a file and +// verified by comparing it — a protocol a systemd unit's shell cannot join. So +// scheduled units and the engine's own wal-g invocations both take this flock +// instead, which makes it the one thing serialising a timer against an operator +// running `ob backup` at the same moment. +func (n Names) BackupRunLock(service string) string { + return path.Join(n.AppDir(), "backup", "run-"+service+".lock") +} + +// BackupVerifyScript is the host-side archive check the scheduled verify unit +// runs. It lives beside the locks rather than in the runtime directory that is +// mounted into the container, because it drives docker from the host. +func (n Names) BackupVerifyScript(service string) string { + return path.Join(n.AppDir(), "backup", "verify-"+service+".sh") +} diff --git a/internal/app/names_test.go b/internal/app/names_test.go index 479e8dd0..f90d763e 100644 --- a/internal/app/names_test.go +++ b/internal/app/names_test.go @@ -138,7 +138,7 @@ func TestContainerNamesEscapeSegmentHyphens(t *testing.T) { if got := n.TransientContainer("web-api"); got != "help--desk-web--api-new" { t.Errorf("hyphenated transient = %q, want help--desk-web--api-new", got) } - if restore, workload := n.ProtectionRestoreContainer("database"), n.Container("database-restore", 1); restore == workload { + if restore, workload := n.BackupRestoreContainer("database"), n.Container("database-restore", 1); restore == workload { t.Fatalf("restore container collides with declared workload: %q", restore) } } @@ -160,7 +160,7 @@ func TestRuntimeContainerDerivationIsInjective(t *testing.T) { add(n.Container(component, replica), fmt.Sprintf("container %s/%s/%d", application, component, replica)) } add(n.TransientContainer(component), "transient "+application+"/"+component) - add(n.ProtectionRestoreContainer(component), "restore "+application+"/"+component) + add(n.BackupRestoreContainer(component), "restore "+application+"/"+component) } } } diff --git a/internal/app/protection_artifacts.go b/internal/app/protection_artifacts.go deleted file mode 100644 index 5cba3ddc..00000000 --- a/internal/app/protection_artifacts.go +++ /dev/null @@ -1,186 +0,0 @@ -package app - -import ( - "crypto/sha256" - "encoding/hex" - "encoding/json" - "errors" - "path" - "sort" -) - -type ProtectionEffectiveProjection struct { - Policy ProtectionPolicy `json:"policy"` - Target BackupTarget `json:"target"` -} - -type GeneratedProtectionArtifact struct { - Class string `json:"class"` - Path string `json:"path"` - Mode uint32 `json:"mode"` - Digest string `json:"digest"` - Content []byte `json:"-"` -} - -type ProtectionArtifactSet struct { - Service string `json:"service"` - Driver string `json:"driver"` - Source string `json:"source"` - Artifacts []GeneratedProtectionArtifact `json:"artifacts"` -} - -type ProtectionArtifactDrift struct { - Class string `json:"class"` - ExpectedDigest string `json:"expected_digest"` - ObservedDigest string `json:"observed_digest,omitempty"` -} - -func (r *Resolved) GenerateProtectionArtifacts(serviceName string) (ProtectionArtifactSet, error) { - if r == nil || r.Spec == nil { - return ProtectionArtifactSet{}, errors.New("resolved project is nil") - } - service, ok := r.Services[serviceName] - if !ok { - return ProtectionArtifactSet{}, errf("project_invalid", "services."+serviceName, "ob validate", "service is not declared") - } - driverName := service.Driver - if driverName == "" { - driverName = serviceName - } - capability, ok := lifecycleCapabilityFor(driverName) - if !ok || !capability.ProtectionQualified(r.DeclaredVersion(serviceName)) { - return ProtectionArtifactSet{}, errf("backup_driver_unsupported", "services."+serviceName+".protection", "ob validate", "service has no qualified protection artifact contract") - } - record := capability.Record() - projection, source, err := r.effectiveProtectionProjection(serviceName, service) - if err != nil { - return ProtectionArtifactSet{}, err - } - if err := validateProtectionPolicy(projection.Policy, "services."+serviceName+".protection"); err != nil { - return ProtectionArtifactSet{}, err - } - if err := validateBackupTarget(projection.Target, "backup_targets."+projection.Policy.Target); err != nil { - return ProtectionArtifactSet{}, err - } - if !capability.SupportsRecoveryKind(r.DeclaredVersion(serviceName), projection.Policy.RecoveryKind) { - return ProtectionArtifactSet{}, errf("backup_driver_unsupported", "services."+serviceName+".protection.recovery_kind", "ob validate", "driver does not support the retained recovery kind") - } - names := r.NamesFor(r.Env) - base := path.Join(names.AppDir(), "protection", "artifacts", serviceName) - credentialPath := names.ProtectionCredentialFile(serviceName, projection.Policy.Target) - credentialEntries := []string{projection.Target.Credentials.AccessKeyEntry, projection.Target.Credentials.SecretKeyEntry} - if projection.Target.Credentials.SessionTokenEntry != "" { - credentialEntries = append(credentialEntries, projection.Target.Credentials.SessionTokenEntry) - } - inputs := map[string]any{ - "service": serviceName, "driver": driverName, "recovery_kind": projection.Policy.RecoveryKind, - "maximum_data_loss": projection.Policy.MaximumDataLoss, "target": projection.Policy.Target, - "endpoint": projection.Target.Endpoint, "bucket": projection.Target.Bucket, "prefix": projection.Target.Prefix, - "region": projection.Target.Region, "tls": projection.Target.TLS, "failure_domain": projection.Target.FailureDomain, - "credential_file": credentialPath, - "credential_entries": credentialEntries, - } - artifacts := make([]GeneratedProtectionArtifact, 0, 12) - appendArtifact := func(class string, mode uint32, value any) error { - content, err := json.Marshal(value) - if err != nil { - return err - } - content = append(content, '\n') - sum := sha256.Sum256(content) - artifacts = append(artifacts, GeneratedProtectionArtifact{ - Class: class, Path: path.Join(base, class+".json"), Mode: mode, - Digest: "sha256:" + hex.EncodeToString(sum[:]), Content: content, - }) - return nil - } - if err := appendArtifact("inputs", 0o600, inputs); err != nil { - return ProtectionArtifactSet{}, err - } - if err := appendArtifact("backup-schedule", 0o644, projection.Policy.Schedule); err != nil { - return ProtectionArtifactSet{}, err - } - if err := appendArtifact("drill-schedule", 0o644, projection.Policy.RestoreDrill.Schedule); err != nil { - return ProtectionArtifactSet{}, err - } - if err := appendArtifact("enablement", 0o600, map[string]any{ - "operation": "protection_enable", "preconditions": record.preconditions, - }); err != nil { - return ProtectionArtifactSet{}, err - } - if err := appendArtifact("disablement", 0o600, map[string]any{"operation": "protection_disable", "service": serviceName}); err != nil { - return ProtectionArtifactSet{}, err - } - if requiresReplayArtifact(projection.Policy.RecoveryKind) { - if err := appendArtifact("archive-hook", 0o600, map[string]any{ - "operation": "replay_archive", "service": serviceName, "credential_file": credentialPath, - }); err != nil { - return ProtectionArtifactSet{}, err - } - } - if err := appendArtifact("service-image", 0o644, record.serviceArtifact); err != nil { - return ProtectionArtifactSet{}, err - } - if record.helperArtifact != nil { - if err := appendArtifact("helper", 0o644, record.helperArtifact); err != nil { - return ProtectionArtifactSet{}, err - } - } - if err := appendArtifact("lifecycle-state", 0o600, map[string]any{ - "path": names.ActiveVolumeFile(serviceName), "credential_file": credentialPath, - }); err != nil { - return ProtectionArtifactSet{}, err - } - if err := appendArtifact("retention", 0o644, map[string]any{ - "minimum_generations": projection.Policy.Retention.MinimumGenerations, - "recovery_window": projection.Policy.Retention.RecoveryWindow, "native_mapping": record.retentionMapping, - }); err != nil { - return ProtectionArtifactSet{}, err - } - if err := appendArtifact("restore-template", 0o600, map[string]any{ - "service": serviceName, "operation": record.operations.Restore, "verify": record.operations.Verify, - "staging_filesystem": projection.Policy.RestoreDrill.StagingFilesystem, - }); err != nil { - return ProtectionArtifactSet{}, err - } - if err := appendArtifact("provenance-sbom", 0o644, map[string]any{ - "service": record.serviceArtifact, "helper": record.helperArtifact, - }); err != nil { - return ProtectionArtifactSet{}, err - } - sort.Slice(artifacts, func(i, j int) bool { return artifacts[i].Class < artifacts[j].Class }) - return ProtectionArtifactSet{Service: serviceName, Driver: driverName, Source: source, Artifacts: artifacts}, nil -} - -func requiresReplayArtifact(recoveryKind string) bool { - return recoveryKind == "pitr" -} - -func (r *Resolved) effectiveProtectionProjection(serviceName string, service Service) (ProtectionEffectiveProjection, string, error) { - if state, ok := r.serviceRuntime[serviceName]; ok && state.ProtectionState == "disable-pending" { - if state.LastEffective == nil { - return ProtectionEffectiveProjection{}, "", errf("protection_image_revert_unsafe", "services."+serviceName, "ob protection disable --output ndjson", "disable-pending state has no durable last-effective protection projection") - } - return *state.LastEffective, "last-effective", nil - } - if service.Protection == nil { - return ProtectionEffectiveProjection{}, "", errf("project_invalid", "services."+serviceName+".protection", "ob validate", "service has no protection intent or retained projection") - } - target, ok := r.BackupTargets[service.Protection.Target] - if !ok { - return ProtectionEffectiveProjection{}, "", errf("backup_target_unknown", "services."+serviceName+".protection.target", "ob validate", "protection target is not declared") - } - return ProtectionEffectiveProjection{Policy: *service.Protection, Target: target}, "project-intent", nil -} - -func CompareProtectionArtifacts(desired ProtectionArtifactSet, observed map[string]string) []ProtectionArtifactDrift { - drift := make([]ProtectionArtifactDrift, 0) - for _, artifact := range desired.Artifacts { - if observed[artifact.Class] != artifact.Digest { - drift = append(drift, ProtectionArtifactDrift{ - Class: artifact.Class, ExpectedDigest: artifact.Digest, ObservedDigest: observed[artifact.Class], - }) - } - } - return drift -} diff --git a/internal/app/protection_artifacts_test.go b/internal/app/protection_artifacts_test.go deleted file mode 100644 index 6dc4e02b..00000000 --- a/internal/app/protection_artifacts_test.go +++ /dev/null @@ -1,159 +0,0 @@ -package app - -import ( - "bytes" - "strings" - "testing" -) - -func protectionArtifactFixture() (*Resolved, ProtectionPolicy, BackupTarget) { - policy := ProtectionPolicy{ - Target: "offsite", RecoveryKind: "pitr", MaximumDataLoss: "5m", - Schedule: Schedule{Cron: "17 */6 * * *", Timezone: "UTC"}, - Retention: ProtectionRetention{MinimumGenerations: 7, RecoveryWindow: "7d"}, - RestoreDrill: RestoreDrillPolicy{ - Schedule: Schedule{Cron: "23 4 * * 1,4", Timezone: "UTC"}, - ProofMaximumAge: "7d", StagingFilesystem: "/srv/onebox-restore", - }, - } - target := BackupTarget{ - Kind: "s3-compatible", Endpoint: "https://objects.example.test", Bucket: "onebox-backups", - Prefix: "production/example", Region: "us-east-1", TLS: "required", - FailureDomain: FailureDomain{Identity: "provider-a/us-east-1/account-42"}, - Credentials: CredentialReference{ - File: "secrets/backup.env", Provider: "sops", AccessKeyEntry: "BACKUP_ACCESS_KEY_ID", - SecretKeyEntry: "BACKUP_SECRET_ACCESS_KEY", SessionTokenEntry: "BACKUP_SESSION_TOKEN", - }, - Encryption: TargetEncryption{PITR: "archive-password"}, - } - resolved := &Resolved{ - Spec: &Spec{ - Name: "example", BasePath: "/var/lib/onebox", - Services: map[string]Service{"database": {Driver: "postgres", Version: 17, Protection: &policy}}, - BackupTargets: map[string]BackupTarget{"offsite": target}, - }, - Env: "production", - } - return resolved, policy, target -} - -func TestGenerateProtectionArtifactsIsDeterministicRedactedAndComplete(t *testing.T) { - resolved, _, _ := protectionArtifactFixture() - first, err := resolved.GenerateProtectionArtifacts("database") - if err != nil { - t.Fatal(err) - } - second, err := resolved.GenerateProtectionArtifacts("database") - if err != nil { - t.Fatal(err) - } - if len(first.Artifacts) != 11 || len(second.Artifacts) != len(first.Artifacts) { - t.Fatalf("artifact counts = %d and %d, want 11", len(first.Artifacts), len(second.Artifacts)) - } - wantClasses := []string{ - "archive-hook", "backup-schedule", "disablement", "drill-schedule", "enablement", "inputs", - "lifecycle-state", "provenance-sbom", "restore-template", "retention", "service-image", - } - seen := make(map[string]bool, len(first.Artifacts)) - for index, artifact := range first.Artifacts { - seen[artifact.Class] = true - if artifact.Digest != second.Artifacts[index].Digest || !bytes.Equal(artifact.Content, second.Artifacts[index].Content) { - t.Fatalf("artifact %q is not deterministic", artifact.Class) - } - if !strings.HasPrefix(artifact.Path, "/var/lib/onebox/") || !lifecycleDigest.MatchString(artifact.Digest) { - t.Fatalf("artifact metadata is not sealed: %#v", artifact) - } - for _, forbidden := range []string{"secrets/backup.env", "database-content-canary", "secret-value-canary"} { - if bytes.Contains(artifact.Content, []byte(forbidden)) { - t.Fatalf("artifact %q leaked %q: %s", artifact.Class, forbidden, artifact.Content) - } - } - } - for _, class := range wantClasses { - if !seen[class] { - t.Errorf("missing artifact class %q", class) - } - } - if seen["helper"] { - t.Fatal("derived-image PostgreSQL unexpectedly received an external helper artifact") - } - if got := string(artifactContent(t, first, "backup-schedule")); !strings.Contains(got, `"cron":"17 */6 * * *"`) { - t.Fatalf("backup schedule is not exact: %s", got) - } - if got := string(artifactContent(t, first, "drill-schedule")); !strings.Contains(got, `"cron":"23 4 * * 1,4"`) { - t.Fatalf("drill schedule is not exact: %s", got) - } -} - -func TestProtectionArtifactsRetainLastEffectiveProjectionAndDetectRealDrift(t *testing.T) { - resolved, policy, target := protectionArtifactFixture() - original, err := resolved.GenerateProtectionArtifacts("database") - if err != nil { - t.Fatal(err) - } - withoutIntent := *resolved - withoutIntent.Spec = &Spec{Name: "example", BasePath: "/var/lib/onebox", Services: map[string]Service{ - "database": {Driver: "postgres", Version: 17}, - }} - retained, err := withoutIntent.WithServiceRuntimeStates(map[string]ServiceRuntimeState{ - "database": { - ProtectionState: "disable-pending", ServiceImage: pinnedServiceImage("postgres", '1'), - PublicationVerified: true, DigestAvailable: true, - LastEffective: &ProtectionEffectiveProjection{Policy: policy, Target: target}, - }, - }) - if err != nil { - t.Fatal(err) - } - desired, err := retained.GenerateProtectionArtifacts("database") - if err != nil { - t.Fatal(err) - } - if desired.Source != "last-effective" { - t.Fatalf("artifact source = %q, want last-effective", desired.Source) - } - observed := make(map[string]string, len(original.Artifacts)) - for index, artifact := range original.Artifacts { - if desired.Artifacts[index].Class != artifact.Class || desired.Artifacts[index].Digest != artifact.Digest { - t.Fatalf("retained artifact %d differs: got %#v want %#v", index, desired.Artifacts[index], artifact) - } - observed[artifact.Class] = artifact.Digest - } - if drift := CompareProtectionArtifacts(desired, observed); len(drift) != 0 { - t.Fatalf("matching retained artifacts reported drift: %#v", drift) - } - for _, artifact := range desired.Artifacts { - changed := make(map[string]string, len(observed)) - for class, digest := range observed { - changed[class] = digest - } - changed[artifact.Class] = "sha256:" + strings.Repeat("f", 64) - if changed[artifact.Class] == artifact.Digest { - changed[artifact.Class] = "sha256:" + strings.Repeat("e", 64) - } - drift := CompareProtectionArtifacts(desired, changed) - if len(drift) != 1 || drift[0].Class != artifact.Class { - t.Fatalf("real drift for %q = %#v", artifact.Class, drift) - } - } -} - -func TestSnapshotOnlyDriversReceiveNoReplayArtifact(t *testing.T) { - for name, capability := range lifecycleCapabilities { - record := capability.Record() - if record.recoveryKinds["snapshot"] && requiresReplayArtifact("snapshot") { - t.Fatalf("snapshot-only driver %q would receive a replay artifact", name) - } - } -} - -func artifactContent(t *testing.T, set ProtectionArtifactSet, class string) []byte { - t.Helper() - for _, artifact := range set.Artifacts { - if artifact.Class == class { - return artifact.Content - } - } - t.Fatalf("artifact class %q not found", class) - return nil -} diff --git a/internal/app/resolve.go b/internal/app/resolve.go index 769cdab9..4ffb916e 100644 --- a/internal/app/resolve.go +++ b/internal/app/resolve.go @@ -31,7 +31,7 @@ var ( "env_files": true, } overridableService = map[string]bool{ - "resources": true, "settings": true, "protection": true, + "resources": true, "settings": true, "backup": true, } ) @@ -116,14 +116,14 @@ func (p *Spec) Resolve(env string) (*Resolved, error) { if err != nil { return nil, err } - if _, protected := authoredPatch["protection"]; protected { - prefix := "services." + name + ".protection" + if _, protected := authoredPatch["backup"]; protected { + prefix := "services." + name + ".backup" for originPath := range out.Origins { if originPath == prefix || strings.HasPrefix(originPath, prefix+".") { delete(out.Origins, originPath) } } - markOverrideValue(prefix, authoredPatch["protection"], out.Origins) + markOverrideValue(prefix, authoredPatch["backup"], out.Origins) } clone.Services[name] = merged } diff --git a/internal/app/runtime.go b/internal/app/runtime.go index 72f5400a..6b27dc5d 100644 --- a/internal/app/runtime.go +++ b/internal/app/runtime.go @@ -117,7 +117,7 @@ func (w Workload) IsJob() bool { return w.Role == RoleJob } // // The contract publishes `persistence.mode` defaulting to durable, but the // block is optional, so the default was unreachable unless it was written — -// and doctor, the migration-backup requirement and the protection gate each +// and doctor, the migration-backup requirement and the backup gate each // read an absent block as "not durable". A workload with a managed named // volume holds data that outlives the release whether or not it says so. // @@ -349,3 +349,42 @@ func (w Workload) ProbePort() int { } return 0 } + +// ParsePostgresDuration reads the form PostgreSQL uses when it reports a +// setting — "15min", "1h", "300s", or a bare number meaning seconds — which is +// not the form ParseDuration accepts. It exists so a declared policy can be +// compared with what the server says it is doing. +func ParsePostgresDuration(value string) (time.Duration, bool) { + trimmed := strings.TrimSpace(value) + if trimmed == "" { + return 0, false + } + digits := len(trimmed) + for index, char := range trimmed { + if char < '0' || char > '9' { + digits = index + break + } + } + if digits == 0 { + return 0, false + } + count, err := strconv.Atoi(trimmed[:digits]) + if err != nil || count < 0 { + return 0, false + } + unit := strings.ToLower(strings.TrimSpace(trimmed[digits:])) + switch unit { + case "", "s": + return time.Duration(count) * time.Second, true + case "ms": + return time.Duration(count) * time.Millisecond, true + case "min": + return time.Duration(count) * time.Minute, true + case "h": + return time.Duration(count) * time.Hour, true + case "d": + return time.Duration(count) * 24 * time.Hour, true + } + return 0, false +} diff --git a/internal/app/runtime_test.go b/internal/app/runtime_test.go index a255b74e..48ebd801 100644 --- a/internal/app/runtime_test.go +++ b/internal/app/runtime_test.go @@ -103,3 +103,28 @@ func TestTargetCarriesTheDeclaredPort(t *testing.T) { }) } } + +// PostgreSQL reports a duration in its own spelling — "15min", "4h", "300s", or +// a bare number meaning seconds — and a declared policy has to be comparable +// with it. Reading "15min" as fifteen somethings-else is how a check says the +// server is fine when it is not. +func TestPostgresDurationsParseInTheirOwnSpelling(t *testing.T) { + for spelling, want := range map[string]time.Duration{ + "15min": 15 * time.Minute, + "4h": 4 * time.Hour, + "300s": 300 * time.Second, + "900": 900 * time.Second, + "250ms": 250 * time.Millisecond, + "1d": 24 * time.Hour, + } { + got, ok := ParsePostgresDuration(spelling) + if !ok || got != want { + t.Errorf("ParsePostgresDuration(%q) = %v, %v; want %v", spelling, got, ok, want) + } + } + for _, refused := range []string{"", "soon", "min", "-5s", "15 minutes"} { + if _, ok := ParsePostgresDuration(refused); ok { + t.Errorf("ParsePostgresDuration(%q) was accepted", refused) + } + } +} diff --git a/internal/app/secrets_graph_test.go b/internal/app/secrets_graph_test.go index ff55c23f..d31b1f18 100644 --- a/internal/app/secrets_graph_test.go +++ b/internal/app/secrets_graph_test.go @@ -138,7 +138,7 @@ external_services: connection: source: {file: secrets/database.env, provider: sops} entries: {url: DATABASE_URL} - protection_owner: platform-team/rds + backup_owner: platform-team/rds probe: {} `) want := []SecretDeclaration{{ diff --git a/internal/app/service_image_state.go b/internal/app/service_image_state.go index 393e7126..b6f0c92b 100644 --- a/internal/app/service_image_state.go +++ b/internal/app/service_image_state.go @@ -2,6 +2,7 @@ package app import ( "errors" + "regexp" "sort" "strings" @@ -17,10 +18,10 @@ type ServiceImageCandidate struct { } // ServiceRuntimeState is observed durable lifecycle state, not authoring -// input. Rendering consumes it through WithServiceRuntimeStates so protection +// input. Rendering consumes it through WithServiceRuntimeStates so backup // image behavior cannot be inferred from policy presence alone. type ServiceRuntimeState struct { - ProtectionState string + BackupState string ServiceImage string PublicationVerified bool DigestAvailable bool @@ -28,7 +29,7 @@ type ServiceRuntimeState struct { ManifestRootImages []string TagObservedDigest string RefreshCandidate *ServiceImageCandidate - LastEffective *ProtectionEffectiveProjection + LastEffective *BackupEffectiveProjection } type ServiceImageSelection struct { @@ -65,8 +66,8 @@ func (r *Resolved) WithServiceRuntimeStates(states map[string]ServiceRuntimeStat } func (state ServiceRuntimeState) validate(service string) error { - if !contains([]string{"never-enabled", "enabled", "disable-pending", "disabled"}, state.ProtectionState) { - return errf("project_invalid", "services."+service, "ob status --output json", "service runtime state has an invalid protection lifecycle state") + if !contains([]string{"never-enabled", "enabled", "disable-pending", "disabled"}, state.BackupState) { + return errf("project_invalid", "services."+service, "ob status --output json", "service runtime state has an invalid backup lifecycle state") } if state.ServiceImage != "" { if err := validatePinnedServiceImage(state.ServiceImage); err != nil { @@ -93,14 +94,14 @@ func (state ServiceRuntimeState) validate(service string) error { func (r *Resolved) selectServiceImage(serviceName, tagImage string) (ServiceImageSelection, error) { state, observed := r.serviceRuntime[serviceName] - if !observed || state.ProtectionState == "never-enabled" || state.ProtectionState == "disabled" { + if !observed || state.BackupState == "never-enabled" || state.BackupState == "disabled" { return ServiceImageSelection{Image: tagImage, Origin: OriginAuthored}, nil } if state.ServiceImage == "" { - return ServiceImageSelection{}, errf("protection_image_revert_unsafe", "services."+serviceName+".version", "ob protection disable --output ndjson", "protected runtime state has no immutable service image; refusing tag reversion") + return ServiceImageSelection{}, errf("backup_image_revert_unsafe", "services."+serviceName+".version", "ob backup disable --output ndjson", "protected runtime state has no immutable service image; refusing tag reversion") } if !state.PublicationVerified { - return ServiceImageSelection{}, errf("protection_service_image_unpublished", "services."+serviceName+".version", "ob service status --output json", "protected service image has no verified publication provenance") + return ServiceImageSelection{}, errf("backup_service_image_unpublished", "services."+serviceName+".version", "ob service status --output json", "protected service image has no verified publication provenance") } if !state.DigestAvailable && !state.CacheVerified { return ServiceImageSelection{}, errf("service_image_digest_unavailable", "services."+serviceName+".version", "ob service status --output json", "protected service image is unavailable from the registry and exact local cache") @@ -108,17 +109,17 @@ func (r *Resolved) selectServiceImage(serviceName, tagImage string) (ServiceImag selected := state.ServiceImage retained := append([]string{state.ServiceImage}, state.ManifestRootImages...) if candidate := state.RefreshCandidate; candidate != nil { - if state.ProtectionState == "disable-pending" { - return ServiceImageSelection{}, errf("service_image_patch_disable_pending", "services."+serviceName+".version", "ob protection disable --output ndjson", "protected service image refresh is refused while disablement is pending") + if state.BackupState == "disable-pending" { + return ServiceImageSelection{}, errf("service_image_patch_disable_pending", "services."+serviceName+".version", "ob backup disable --output ndjson", "protected service image refresh is refused while disablement is pending") } if !candidate.PublicationVerified { - return ServiceImageSelection{}, errf("protection_service_image_unpublished", "services."+serviceName+".version", "ob service status --output json", "candidate protected service image has no verified publication provenance") + return ServiceImageSelection{}, errf("backup_service_image_unpublished", "services."+serviceName+".version", "ob service status --output json", "candidate protected service image has no verified publication provenance") } if !candidate.DigestAvailable && !candidate.CacheVerified { return ServiceImageSelection{}, errf("service_image_digest_unavailable", "services."+serviceName+".version", "ob service status --output json", "candidate protected service image is unavailable from the registry and exact local cache") } if !candidate.ExactTransition { - return ServiceImageSelection{}, errf("protected_service_patch_unsupported", "services."+serviceName+".version", "ob service status --output json", "candidate image has no exact qualified protected transition") + return ServiceImageSelection{}, errf("service_patch_unsupported", "services."+serviceName+".version", "ob service status --output json", "candidate image has no exact qualified protected transition") } selected = candidate.Image retained = append(retained, candidate.Image) @@ -128,6 +129,28 @@ func (r *Resolved) selectServiceImage(serviceName, tagImage string) (ServiceImag return ServiceImageSelection{Image: selected, RetainedImages: retained, Origin: OriginObserved}, nil } +// DeclaredServiceImage is the reference the project authored — the driver's +// repository at the declared version — with no regard for what the host is +// running. It is deliberately not ServiceImageForRuntime: that one answers with +// the pinned digest while a service is protected and with the tag once it is +// not, so it cannot be used to recognise "the same image the operator asked +// for" across an enable/disable cycle. This can. +func (r *Resolved) DeclaredServiceImage(serviceName string) (string, error) { + service, ok := r.Services[serviceName] + if !ok { + return "", errf("project_invalid", "services."+serviceName, "ob validate", "service is not declared") + } + driverName := service.Driver + if driverName == "" { + driverName = serviceName + } + driver, ok := drivers[driverName] + if !ok { + return "", errf("unknown_service_driver", "services."+serviceName, "ob validate", "no managed driver named %q", driverName) + } + return driver.image + ":" + versionString(service.Version), nil +} + // ServiceImageForRuntime exposes the same selection used by generation so // planners, cache checks, and pruning all retain identical immutable roots. func (r *Resolved) ServiceImageForRuntime(serviceName string) (ServiceImageSelection, error) { @@ -146,12 +169,16 @@ func (r *Resolved) ServiceImageForRuntime(serviceName string) (ServiceImageSelec return r.selectServiceImage(serviceName, driver.image+":"+versionString(service.Version)) } +// serviceImageDigest matches a full sha256 reference, which is what "pinned" +// means for a protected service image. +var serviceImageDigest = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) + func validatePinnedServiceImage(image string) error { if err := imageref.Validate(image); err != nil { return err } marker := strings.LastIndex(image, "@sha256:") - if marker < 1 || len(image)-marker != len("@sha256:")+64 || !lifecycleDigest.MatchString(image[marker+1:]) { + if marker < 1 || len(image)-marker != len("@sha256:")+64 || !serviceImageDigest.MatchString(image[marker+1:]) { return errors.New("service image must be pinned by lowercase sha256 digest") } return nil diff --git a/internal/app/service_image_state_test.go b/internal/app/service_image_state_test.go index a538eebf..b0eac1c2 100644 --- a/internal/app/service_image_state_test.go +++ b/internal/app/service_image_state_test.go @@ -9,14 +9,44 @@ import ( func serviceImageTestResolved(withPolicy bool) *Resolved { service := Service{Driver: "postgres", Version: 17} if withPolicy { - service.Protection = &ProtectionPolicy{Target: "offsite"} + service.Backup = backupTestPolicy() } return &Resolved{ - Spec: &Spec{Name: "example", BasePath: "/var/lib/ob", Services: map[string]Service{"database": service}}, - Env: "production", + Spec: &Spec{ + Name: "example", BasePath: "/var/lib/ob", + Services: map[string]Service{"database": service}, + BackupTargets: map[string]BackupTarget{"offsite": backupTestTarget()}, + }, + Env: "production", + } +} + +func backupTestPolicy() *BackupPolicy { + return &BackupPolicy{ + Target: "offsite", RecoveryKind: "pitr", MaxDataLoss: "15m", + Retention: BackupRetention{Keep: 7, Window: "7d"}, } } +func backupTestTarget() BackupTarget { + return BackupTarget{ + Kind: "s3-compatible", Endpoint: "https://objects.example.net", + Bucket: "onebox-backups", Prefix: "production/example", Region: "us-east-1", TLS: "verify", + Credentials: CredentialReference{ + File: "secrets/backup.env", Provider: "sops", + AccessKeyEntry: "BACKUP_ACCESS_KEY_ID", SecretKeyEntry: "BACKUP_SECRET_ACCESS_KEY", + }, + Encryption: TargetEncryption{PITR: "client-side"}, + } +} + +// backupTestProjection is what enablement records. A runtime state that +// says "enabled" without one describes a service that is archiving somewhere +// nothing can name, which is not a state enablement can produce. +func backupTestProjection() *BackupEffectiveProjection { + return &BackupEffectiveProjection{Policy: *backupTestPolicy(), Target: backupTestTarget()} +} + func pinnedServiceImage(repository string, digit byte) string { return repository + "@sha256:" + strings.Repeat(string(digit), 64) } @@ -43,7 +73,7 @@ func TestUnprotectedServiceKeepsTagRenderingOffline(t *testing.T) { t.Fatalf("unprotected service image = %q", got) } withState, err := resolved.WithServiceRuntimeStates(map[string]ServiceRuntimeState{ - "database": {ProtectionState: "never-enabled"}, + "database": {BackupState: "never-enabled"}, }) if err != nil { t.Fatal(err) @@ -54,9 +84,9 @@ func TestUnprotectedServiceKeepsTagRenderingOffline(t *testing.T) { } func TestProtectedServiceRenderingUsesDurableDigestNotPolicyOrMovedTag(t *testing.T) { - image := pinnedServiceImage("ghcr.io/labstack/onebox-postgres-pgbackrest", 'a') + image := pinnedServiceImage("postgres", 'a') state := ServiceRuntimeState{ - ProtectionState: "enabled", ServiceImage: image, + BackupState: "enabled", ServiceImage: image, LastEffective: backupTestProjection(), PublicationVerified: true, DigestAvailable: true, TagObservedDigest: "sha256:" + strings.Repeat("b", 64), } @@ -78,9 +108,9 @@ func TestProtectedServiceImageFailureCodes(t *testing.T) { state ServiceRuntimeState code string }{ - {"unsafe-revert", ServiceRuntimeState{ProtectionState: "enabled", PublicationVerified: true, DigestAvailable: true}, "protection_image_revert_unsafe"}, - {"unpublished", ServiceRuntimeState{ProtectionState: "enabled", ServiceImage: image, DigestAvailable: true}, "protection_service_image_unpublished"}, - {"unavailable", ServiceRuntimeState{ProtectionState: "enabled", ServiceImage: image, PublicationVerified: true}, "service_image_digest_unavailable"}, + {"unsafe-revert", ServiceRuntimeState{BackupState: "enabled", PublicationVerified: true, DigestAvailable: true}, "backup_image_revert_unsafe"}, + {"unpublished", ServiceRuntimeState{BackupState: "enabled", ServiceImage: image, DigestAvailable: true}, "backup_service_image_unpublished"}, + {"unavailable", ServiceRuntimeState{BackupState: "enabled", ServiceImage: image, PublicationVerified: true}, "service_image_digest_unavailable"}, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { @@ -100,7 +130,7 @@ func TestProtectedServiceImageFailureCodes(t *testing.T) { func TestProtectedServicePermitsExactVerifiedCacheDuringRegistryOutage(t *testing.T) { image := pinnedServiceImage("postgres", 'a') resolved, err := serviceImageTestResolved(true).WithServiceRuntimeStates(map[string]ServiceRuntimeState{ - "database": {ProtectionState: "enabled", ServiceImage: image, PublicationVerified: true, CacheVerified: true}, + "database": {BackupState: "enabled", ServiceImage: image, PublicationVerified: true, CacheVerified: true}, }) if err != nil { t.Fatal(err) @@ -116,7 +146,7 @@ func TestProtectedRefreshRetainsCurrentAndManifestImageRoots(t *testing.T) { candidate := pinnedServiceImage("postgres", 'c') resolved, err := serviceImageTestResolved(true).WithServiceRuntimeStates(map[string]ServiceRuntimeState{ "database": { - ProtectionState: "enabled", ServiceImage: current, PublicationVerified: true, DigestAvailable: true, + BackupState: "enabled", ServiceImage: current, PublicationVerified: true, DigestAvailable: true, ManifestRootImages: []string{manifest}, RefreshCandidate: &ServiceImageCandidate{Image: candidate, PublicationVerified: true, DigestAvailable: true, ExactTransition: true}, }, @@ -145,12 +175,12 @@ func TestProtectedRefreshRefusesDisablePendingAndUnqualifiedTransition(t *testin code string }{ {"disable-pending", true, "service_image_patch_disable_pending"}, - {"enabled", false, "protected_service_patch_unsupported"}, + {"enabled", false, "service_patch_unsupported"}, } { copy := *candidate copy.ExactTransition = test.exact resolved, err := serviceImageTestResolved(true).WithServiceRuntimeStates(map[string]ServiceRuntimeState{ - "database": {ProtectionState: test.state, ServiceImage: image, PublicationVerified: true, DigestAvailable: true, RefreshCandidate: ©}, + "database": {BackupState: test.state, ServiceImage: image, PublicationVerified: true, DigestAvailable: true, RefreshCandidate: ©}, }) if err != nil { t.Fatal(err) @@ -166,7 +196,7 @@ func TestProtectedRefreshRefusesDisablePendingAndUnqualifiedTransition(t *testin func TestProtectedServiceGoldenUsesImmutableImage(t *testing.T) { image := pinnedServiceImage("postgres", 'd') resolved, err := serviceImageTestResolved(true).WithServiceRuntimeStates(map[string]ServiceRuntimeState{ - "database": {ProtectionState: "enabled", ServiceImage: image, PublicationVerified: true, DigestAvailable: true}, + "database": {BackupState: "enabled", ServiceImage: image, PublicationVerified: true, DigestAvailable: true}, }) if err != nil { t.Fatal(err) diff --git a/internal/app/service_lifecycle.go b/internal/app/service_lifecycle.go index 63a25d36..a8e4a06d 100644 --- a/internal/app/service_lifecycle.go +++ b/internal/app/service_lifecycle.go @@ -6,106 +6,45 @@ import ( "sort" ) -type serviceDeliveryClass string - -const ( - deliveryUpstreamDigest serviceDeliveryClass = "upstream-digest" - deliveryDerivedImage serviceDeliveryClass = "derived-image" - deliveryExternalHelper serviceDeliveryClass = "external-helper" -) - -type repositoryOwnership string - -const ( - repositoryNativeDirect repositoryOwnership = "native-direct" - repositoryArtifact repositoryOwnership = "artifact" -) - -// lifecycleCapability is deliberately separate from the runtime driver. A -// service can be runnable without an executable backup contract, and an absent -// record never inherits volume-copy or generic-helper behavior. -type lifecycleCapability interface { - DriverName() string - ProtectionQualified(version string) bool - SupportsRecoveryKind(version, recoveryKind string) bool - Record() lifecycleCapabilityRecord -} - -type lifecycleArtifactProvenance struct { - Repository string `json:"repository"` - Digest string `json:"digest"` - UpstreamDigest string `json:"upstream_digest"` - SBOMDigest string `json:"sbom_digest"` - ProvenanceID string `json:"provenance_id"` -} - -type lifecycleVersionRange struct { - Pattern string -} - -type protectedPatchTransition struct { - CurrentServiceDigest string - CandidateServiceDigest string - CurrentHelperDigest string - CandidateHelperDigest string - MaintenanceRange string - CompatibilityProbes []string - ContinuityProbes []string - RollbackLimit string -} - -type lifecyclePrecondition struct { - Code string `json:"code"` - Consistency string `json:"consistency"` - Topology string `json:"topology"` - RestartRequired bool `json:"restart_required"` -} - -type lifecycleOperations struct { - Backup string - Restore string - Verify string -} - -// lifecycleCapabilityRecord models every generic seam. policyQualified means -// the project schema may accept the contract; graduated is separately gated by -// live backup/restore evidence and is never inferred from this record. -type lifecycleCapabilityRecord struct { - driver string - policyQualified bool - graduated bool - recoveryKinds map[string]bool - delivery serviceDeliveryClass - serviceArtifact lifecycleArtifactProvenance - helperArtifact *lifecycleArtifactProvenance - supportedVersions []lifecycleVersionRange - patchTransitions []protectedPatchTransition - repository repositoryOwnership - encryptionByKind map[string]string - retentionMapping string - preconditions []lifecyclePrecondition - achievableRPO string - credentialSlots []string - protectedResources []string - operations lifecycleOperations - graduationEvidence []string -} - -func (record lifecycleCapabilityRecord) DriverName() string { return record.driver } - -func (record lifecycleCapabilityRecord) Record() lifecycleCapabilityRecord { return record } - -func (record lifecycleCapabilityRecord) ProtectionQualified(version string) bool { - return record.policyQualified && record.supportsVersion(version) -} - -func (record lifecycleCapabilityRecord) SupportsRecoveryKind(version, recoveryKind string) bool { - return record.ProtectionQualified(version) && record.recoveryKinds[recoveryKind] -} - -func (record lifecycleCapabilityRecord) supportsVersion(version string) bool { - for _, supported := range record.supportedVersions { - matched, err := regexp.MatchString(supported.Pattern, version) +// lifecycleCapability records what a driver's backup contract can actually do. +// +// It is deliberately separate from the runtime driver: a service can be +// runnable without an executable backup contract, and a driver with no record +// never inherits one. +// +// Only what gates behaviour lives here. An earlier version of this catalogue +// also carried delivery classes, repository ownership, artifact and SBOM +// provenance, patch transitions with continuity probes, consistency and +// topology preconditions, achievable RPO, protected resource names, native +// retention mappings, per-kind encryption modes, operation identifiers, and a +// graduation-evidence contract — around 450 lines describing a system that did +// not exist. Nothing read any of it except the function that validated it +// against itself, and the provenance digests were placeholders (sixty-four +// repetitions of one hex character) presented in the same shape as the one real +// checksum in the repository. Metadata that describes nothing and proves +// nothing is worse than absent: it reads as a guarantee. +type lifecycleCapability struct { + driver string + policyQualified bool + recoveryKinds map[string]bool + // Version patterns the contract is qualified for, as regular expressions. + supportedVersions []string + // Names the driver expects in its target-side credential file. Values never + // cross this catalogue. + credentialSlots []string +} + +func (capability lifecycleCapability) BackupQualified(version string) bool { + return capability.policyQualified && capability.supportsVersion(version) +} + +func (capability lifecycleCapability) SupportsRecoveryKind(version, recoveryKind string) bool { + return capability.BackupQualified(version) && capability.recoveryKinds[recoveryKind] +} + +func (capability lifecycleCapability) supportsVersion(version string) bool { + for _, pattern := range capability.supportedVersions { + matched, err := regexp.MatchString(pattern, version) if err == nil && matched { return true } @@ -113,136 +52,37 @@ func (record lifecycleCapabilityRecord) supportsVersion(version string) bool { return false } -func (record lifecycleCapabilityRecord) validate() error { - // `_test_lifecycle` is the one name exempt from being a real runtime driver. - // It belongs to the fixture in service_lifecycle_test.go, which exercises - // every generic seam without entering the runtime catalogue — so the seams - // stay tested even while no shipped driver uses all of them. - // - // The exemption is not a hole a project can reach. A record is only ever - // constructed in this repository — by buildLifecycleCapabilities and by that - // fixture — and this method has exactly two callers: the fixture, and - // validateLifecycleCatalogue over the fixed catalogue. An authored `driver` - // is a key looked up in lifecycleCapabilities by lifecycleCapabilityFor; it - // never builds a record, so it never arrives here whatever it is named. - if _, ok := drivers[record.driver]; !ok && record.driver != "_test_lifecycle" { - return fmt.Errorf("lifecycle driver %q is not a runtime driver", record.driver) - } - if len(record.recoveryKinds) == 0 { - return fmt.Errorf("lifecycle driver %q has no recovery kinds", record.driver) - } - if record.delivery != deliveryUpstreamDigest && record.delivery != deliveryDerivedImage && record.delivery != deliveryExternalHelper { - return fmt.Errorf("lifecycle driver %q has invalid service delivery class", record.driver) - } - if err := record.serviceArtifact.validate("service artifact"); err != nil { - return fmt.Errorf("lifecycle driver %q: %w", record.driver, err) - } - if record.delivery == deliveryExternalHelper && record.helperArtifact == nil { - return fmt.Errorf("lifecycle driver %q external-helper delivery has no helper provenance", record.driver) +func (capability lifecycleCapability) validate() error { + if _, ok := drivers[capability.driver]; !ok { + return fmt.Errorf("lifecycle driver %q is not a runtime driver", capability.driver) } - if record.helperArtifact != nil { - if err := record.helperArtifact.validate("helper artifact"); err != nil { - return fmt.Errorf("lifecycle driver %q: %w", record.driver, err) - } - } - if len(record.supportedVersions) == 0 { - return fmt.Errorf("lifecycle driver %q has no supported version range", record.driver) - } - for _, supported := range record.supportedVersions { - if _, err := regexp.Compile(supported.Pattern); err != nil { - return fmt.Errorf("lifecycle driver %q has invalid version range", record.driver) - } + if len(capability.recoveryKinds) == 0 { + return fmt.Errorf("lifecycle driver %q has no recovery kinds", capability.driver) } - if record.repository != repositoryNativeDirect && record.repository != repositoryArtifact { - return fmt.Errorf("lifecycle driver %q has invalid repository ownership", record.driver) - } - for kind := range record.recoveryKinds { + for kind := range capability.recoveryKinds { if !contains(eRecoveryKind, kind) { - return fmt.Errorf("lifecycle driver %q has invalid recovery kind %q", record.driver, kind) - } - if !contains(eEncryptionMode, record.encryptionByKind[kind]) { - return fmt.Errorf("lifecycle driver %q has no valid encryption mode for %q", record.driver, kind) + return fmt.Errorf("lifecycle driver %q has invalid recovery kind %q", capability.driver, kind) } } - if !contains([]string{"pgbackrest", "pbm", "clickhouse-chain", "artifact", "snapshot"}, record.retentionMapping) { - return fmt.Errorf("lifecycle driver %q has invalid native retention mapping", record.driver) - } - if _, err := PositiveDuration(record.achievableRPO); err != nil { - return fmt.Errorf("lifecycle driver %q has invalid achievable RPO", record.driver) - } - if len(record.preconditions) == 0 { - return fmt.Errorf("lifecycle driver %q has no consistency/topology preconditions", record.driver) + if len(capability.supportedVersions) == 0 { + return fmt.Errorf("lifecycle driver %q has no supported version range", capability.driver) } - for _, precondition := range record.preconditions { - if !gIdent.pattern.MatchString(precondition.Code) || precondition.Consistency == "" || precondition.Topology == "" { - return fmt.Errorf("lifecycle driver %q has incomplete precondition metadata", record.driver) + for _, pattern := range capability.supportedVersions { + if _, err := regexp.Compile(pattern); err != nil { + return fmt.Errorf("lifecycle driver %q has invalid version range", capability.driver) } } - if len(record.credentialSlots) == 0 || len(record.protectedResources) == 0 { - return fmt.Errorf("lifecycle driver %q has no credentials or protected resources", record.driver) + if len(capability.credentialSlots) == 0 { + return fmt.Errorf("lifecycle driver %q has no credential slots", capability.driver) } - for _, slot := range record.credentialSlots { + for _, slot := range capability.credentialSlots { if !gEnvName.pattern.MatchString(slot) { - return fmt.Errorf("lifecycle driver %q has unsafe credential slot metadata", record.driver) - } - } - for _, resource := range record.protectedResources { - if !gFailureDomain.pattern.MatchString(resource) { - return fmt.Errorf("lifecycle driver %q has unsafe protected resource metadata", record.driver) - } - } - for name, operation := range map[string]string{"backup": record.operations.Backup, "restore": record.operations.Restore, "verify": record.operations.Verify} { - if !gIdent.pattern.MatchString(operation) { - return fmt.Errorf("lifecycle driver %q has invalid %s operation", record.driver, name) - } - } - if len(record.graduationEvidence) == 0 { - return fmt.Errorf("lifecycle driver %q has no graduation evidence contract", record.driver) - } - for _, evidence := range record.graduationEvidence { - if !gIdent.pattern.MatchString(evidence) { - return fmt.Errorf("lifecycle driver %q has unsafe graduation evidence metadata", record.driver) + return fmt.Errorf("lifecycle driver %q has unsafe credential slot metadata", capability.driver) } } - if record.graduated && !record.policyQualified { - return fmt.Errorf("lifecycle driver %q cannot graduate without a qualified policy", record.driver) - } - for i, transition := range record.patchTransitions { - if err := transition.validate(record.delivery, record.helperArtifact != nil); err != nil { - return fmt.Errorf("lifecycle driver %q transition %d: %w", record.driver, i, err) - } - } - return nil -} - -func (artifact lifecycleArtifactProvenance) validate(name string) error { - if artifact.Repository == "" || !lifecycleDigest.MatchString(artifact.Digest) || - !lifecycleDigest.MatchString(artifact.UpstreamDigest) || !lifecycleDigest.MatchString(artifact.SBOMDigest) || - !gFailureDomain.pattern.MatchString(artifact.ProvenanceID) { - return fmt.Errorf("%s provenance is incomplete or unpinned", name) - } return nil } -func (transition protectedPatchTransition) validate(delivery serviceDeliveryClass, helper bool) error { - if !lifecycleDigest.MatchString(transition.CurrentServiceDigest) || !lifecycleDigest.MatchString(transition.CandidateServiceDigest) || - transition.CurrentServiceDigest == transition.CandidateServiceDigest { - return fmt.Errorf("service digests do not identify an exact transition") - } - if helper && (!lifecycleDigest.MatchString(transition.CurrentHelperDigest) || !lifecycleDigest.MatchString(transition.CandidateHelperDigest)) { - return fmt.Errorf("helper digests do not identify an exact transition") - } - if !helper && (transition.CurrentHelperDigest != "" || transition.CandidateHelperDigest != "") { - return fmt.Errorf("transition invents helper digests for %s delivery", delivery) - } - if transition.MaintenanceRange == "" || len(transition.CompatibilityProbes) == 0 || len(transition.ContinuityProbes) == 0 || transition.RollbackLimit == "" { - return fmt.Errorf("transition lacks range, probes, or rollback limit") - } - return nil -} - -var lifecycleDigest = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) - func contains(values []string, want string) bool { for _, value := range values { if value == want { @@ -258,13 +98,13 @@ func lifecycleCapabilityFor(driverName string) (lifecycleCapability, bool) { } // LifecycleCredentialSlots returns the names a qualified driver expects in its -// trusted target-side credential file. Values never cross this catalogue API. +// trusted target-side credential file. func LifecycleCredentialSlots(driverName, version string) ([]string, bool) { capability, ok := lifecycleCapabilityFor(driverName) - if !ok || !capability.ProtectionQualified(version) { + if !ok || !capability.BackupQualified(version) { return nil, false } - slots := append([]string(nil), capability.Record().credentialSlots...) + slots := append([]string(nil), capability.credentialSlots...) sort.Strings(slots) return slots, true } @@ -275,10 +115,10 @@ func validateLifecycleCatalogue() error { } for _, driverName := range sortedKeys(lifecycleCapabilities) { capability := lifecycleCapabilities[driverName] - if capability.DriverName() != driverName { - return fmt.Errorf("lifecycle record %q identifies driver %q", driverName, capability.DriverName()) + if capability.driver != driverName { + return fmt.Errorf("lifecycle record %q identifies driver %q", driverName, capability.driver) } - if err := capability.Record().validate(); err != nil { + if err := capability.validate(); err != nil { return err } } diff --git a/internal/app/service_lifecycle_records.go b/internal/app/service_lifecycle_records.go index 88e12c8f..ae7c26be 100644 --- a/internal/app/service_lifecycle_records.go +++ b/internal/app/service_lifecycle_records.go @@ -1,135 +1,79 @@ package app -import "strings" - -var lifecycleCapabilities = buildLifecycleCapabilities() - -func buildLifecycleCapabilities() map[string]lifecycleCapability { - return map[string]lifecycleCapability{ - "postgres": lifecycleRecord( - "postgres", true, "pitr", deliveryDerivedImage, repositoryNativeDirect, "archive-password", "pgbackrest", "5m", - "^17([.][0-9]+)*$", artifact("ghcr.io/labstack/onebox-postgres-pgbackrest", "postgres", '1'), nil, - []lifecyclePrecondition{{Code: "archive-mode", Consistency: "physical-base-wal", Topology: "single-primary", RestartRequired: true}}, - []string{"POSTGRES_PASSWORD", "PGBACKREST_REPO_PASSWORD"}, []string{"data-volume", "wal-stream"}, - lifecycleOperations{Backup: "pgbackrest-backup", Restore: "pgbackrest-restore", Verify: "pgbackrest-check"}), - "mysql": lifecycleRecord( - "mysql", false, "pitr", deliveryExternalHelper, repositoryArtifact, "client-side", "artifact", "5m", - "^8[.](0|4)([.][0-9]+)*$", artifact("mysql", "mysql", '2'), helperArtifact("percona/percona-xtrabackup", "xtrabackup", '3'), - []lifecyclePrecondition{{Code: "binary-log", Consistency: "physical-base-binlog", Topology: "single-primary"}}, - []string{"MYSQL_PASSWORD", "MYSQL_ROOT_PASSWORD", "RESTIC_PASSWORD"}, []string{"data-volume", "binary-log"}, - lifecycleOperations{Backup: "xtrabackup-create", Restore: "xtrabackup-restore", Verify: "mysql-verify"}), - "mariadb": lifecycleRecord( - "mariadb", false, "pitr", deliveryExternalHelper, repositoryArtifact, "client-side", "artifact", "5m", - "^11[.][0-9]+([.][0-9]+)*$", artifact("mariadb", "mariadb", '4'), helperArtifact("mariadb", "mariadb-backup", '5'), - []lifecyclePrecondition{{Code: "binary-log", Consistency: "physical-base-binlog", Topology: "single-primary", RestartRequired: true}}, - []string{"MARIADB_PASSWORD", "MARIADB_ROOT_PASSWORD", "RESTIC_PASSWORD"}, []string{"data-volume", "binary-log"}, - lifecycleOperations{Backup: "mariadb-backup", Restore: "mariadb-restore", Verify: "mariadb-verify"}), - "mongodb": lifecycleRecord( - "mongodb", false, "pitr", deliveryExternalHelper, repositoryNativeDirect, "server-side-sse", "pbm", "5m", - "^8[.]0([.][0-9]+)*$", artifact("mongo", "mongodb", '6'), helperArtifact("percona/percona-backup-mongodb", "pbm", '7'), - []lifecyclePrecondition{{Code: "replica-set", Consistency: "pbm-oplog", Topology: "single-node-replica-set"}}, - []string{"MONGO_INITDB_ROOT_PASSWORD", "PBM_STORAGE_CREDENTIAL"}, []string{"data-volume", "oplog", "replica-set-identity"}, - lifecycleOperations{Backup: "pbm-backup", Restore: "pbm-restore", Verify: "mongodb-verify"}), - "clickhouse": lifecycleRecord( - "clickhouse", false, "snapshot", deliveryUpstreamDigest, repositoryNativeDirect, "server-side-sse", "clickhouse-chain", "30m", - "^25([.][0-9]+)*$", artifact("clickhouse/clickhouse-server", "clickhouse", '8'), nil, - []lifecyclePrecondition{{Code: "named-collection", Consistency: "native-backup", Topology: "single-server", RestartRequired: true}}, - []string{"CLICKHOUSE_PASSWORD", "CLICKHOUSE_BACKUP_CREDENTIAL"}, []string{"data-volume", "named-collection"}, - lifecycleOperations{Backup: "clickhouse-backup", Restore: "clickhouse-restore", Verify: "clickhouse-verify"}), - "redis": lifecycleRecord( - "redis", false, "snapshot", deliveryExternalHelper, repositoryArtifact, "client-side", "snapshot", "1h", - "^8([.][0-9]+)*$", artifact("redis", "redis", '9'), helperArtifact("restic/restic", "restic", 'a'), - []lifecyclePrecondition{{Code: "persistence-mode", Consistency: "sealed-set-or-rdb", Topology: "single-server"}}, - []string{"REDIS_PASSWORD", "RESTIC_PASSWORD"}, []string{"data-volume", "rdb-or-sealed-set"}, - lifecycleOperations{Backup: "redis-snapshot", Restore: "redis-restore", Verify: "redis-verify"}), - "valkey": lifecycleRecord( - "valkey", false, "snapshot", deliveryExternalHelper, repositoryArtifact, "client-side", "snapshot", "1h", - "^8([.][0-9]+)*$", artifact("valkey/valkey", "valkey", 'b'), helperArtifact("restic/restic", "restic", 'c'), - []lifecyclePrecondition{{Code: "persistence-mode", Consistency: "immutable-rdb", Topology: "single-server"}}, - []string{"REDIS_PASSWORD", "RESTIC_PASSWORD"}, []string{"data-volume", "rdb"}, - lifecycleOperations{Backup: "valkey-snapshot", Restore: "valkey-restore", Verify: "valkey-verify"}), - "rabbitmq": lifecycleRecord( - "rabbitmq", false, "cold", deliveryExternalHelper, repositoryArtifact, "client-side", "artifact", "24h", - "^4([.][0-9]+)*$", artifact("rabbitmq", "rabbitmq", 'd'), helperArtifact("restic/restic", "restic", 'e'), - []lifecyclePrecondition{{Code: "stopped-node", Consistency: "cold-node-store", Topology: "single-node"}}, - []string{"RABBITMQ_DEFAULT_PASS", "RABBITMQ_ERLANG_COOKIE", "RESTIC_PASSWORD"}, []string{"data-volume", "node-name", "erlang-cookie"}, - lifecycleOperations{Backup: "rabbitmq-cold", Restore: "rabbitmq-restore", Verify: "rabbitmq-verify"}), - "minio": lifecycleRecord( - "minio", true, "cold", deliveryExternalHelper, repositoryArtifact, "client-side", "artifact", "24h", - "^RELEASE[.][0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9-]+Z$", artifact("minio/minio", "minio", 'f'), helperArtifact("restic/restic", "restic", '0'), - []lifecyclePrecondition{{Code: "stopped-service", Consistency: "cold-data-store", Topology: "single-node"}}, - []string{"MINIO_ROOT_PASSWORD", "RESTIC_PASSWORD"}, []string{"data-volume", "minio-configuration"}, - lifecycleOperations{Backup: "minio-cold", Restore: "minio-restore", Verify: "minio-verify"}), - "meilisearch": lifecycleRecord( - "meilisearch", false, "snapshot", deliveryExternalHelper, repositoryArtifact, "client-side", "snapshot", "24h", - "^1([.][0-9]+)*$", artifact("getmeili/meilisearch", "meilisearch", '0'), helperArtifact("restic/restic", "restic", '1'), - []lifecyclePrecondition{{Code: "snapshot-task", Consistency: "native-snapshot", Topology: "single-node"}}, - []string{"MEILI_MASTER_KEY", "RESTIC_PASSWORD"}, []string{"data-volume", "snapshot"}, - lifecycleOperations{Backup: "meilisearch-snapshot", Restore: "meilisearch-restore", Verify: "meilisearch-verify"}), - "nats": lifecycleRecord( - "nats", false, "snapshot", deliveryExternalHelper, repositoryArtifact, "client-side", "artifact", "1h", - "^2([.][0-9]+)*$", artifact("nats", "nats", '2'), helperArtifact("natsio/nats-box", "nats-cli", '3'), - []lifecyclePrecondition{{Code: "file-streams", Consistency: "account-snapshot", Topology: "single-server"}}, - []string{"NATS_ACCOUNT_CREDENTIAL", "RESTIC_PASSWORD"}, []string{"data-volume", "streams", "consumers"}, - lifecycleOperations{Backup: "nats-account-backup", Restore: "nats-account-restore", Verify: "nats-verify"}), - } -} - -func lifecycleRecord( - driver string, - policyQualified bool, - recoveryKind string, - delivery serviceDeliveryClass, - repository repositoryOwnership, - encryption string, - retention string, - rpo string, - versionPattern string, - service lifecycleArtifactProvenance, - helper *lifecycleArtifactProvenance, - preconditions []lifecyclePrecondition, - credentials []string, - resources []string, - operations lifecycleOperations, -) lifecycleCapabilityRecord { - return lifecycleCapabilityRecord{ - driver: driver, policyQualified: policyQualified, graduated: false, - recoveryKinds: map[string]bool{recoveryKind: true}, delivery: delivery, - serviceArtifact: service, helperArtifact: helper, - supportedVersions: []lifecycleVersionRange{{Pattern: versionPattern}}, - patchTransitions: []protectedPatchTransition{}, repository: repository, - encryptionByKind: map[string]string{recoveryKind: encryption}, retentionMapping: retention, - preconditions: preconditions, achievableRPO: rpo, - credentialSlots: credentials, protectedResources: resources, operations: operations, - graduationEvidence: []string{"runtime-health", "recoverable-point", "retention-current", "restore-proof"}, - } -} - -func artifact(repository, name string, seed byte) lifecycleArtifactProvenance { - digest := seededDigest(seed) - return lifecycleArtifactProvenance{ - Repository: repository, Digest: digest, UpstreamDigest: digest, - SBOMDigest: seededDigest(nextHex(seed)), ProvenanceID: "onebox/catalog/" + name + "/v1", - } -} - -func helperArtifact(repository, name string, seed byte) *lifecycleArtifactProvenance { - value := artifact(repository, name, seed) - return &value -} - -func seededDigest(seed byte) string { - if !strings.ContainsRune("0123456789abcdef", rune(seed)) { - seed = '0' - } - return "sha256:" + strings.Repeat(string(seed), 64) -} - -func nextHex(seed byte) byte { - const digits = "0123456789abcdef" - index := strings.IndexByte(digits, seed) - if index < 0 { - return '0' - } - return digits[(index+1)%len(digits)] +// The catalogue. policyQualified says the project schema may accept a backup +// policy for this driver at all, and it means one thing: `ob backup enable` can +// actually establish it. Today that is postgres alone. +// +// minio was marked qualified too, so a project could declare a backup policy on +// a minio service, pass `ob validate`, and only discover at `ob backup enable` +// that no driver but postgres is executable — a refusal arriving after the +// project had been written, reviewed and committed. A driver earns this flag +// when its contract runs, not when its contract is described. +var lifecycleCapabilities = map[string]lifecycleCapability{ + "postgres": { + driver: "postgres", policyQualified: true, + recoveryKinds: map[string]bool{"pitr": true}, + supportedVersions: []string{`^1[78]([.][0-9]+)*$`}, + credentialSlots: []string{"POSTGRES_PASSWORD", WalgRepositoryKeyEntry}, + }, + "mysql": { + driver: "mysql", + recoveryKinds: map[string]bool{"pitr": true}, + supportedVersions: []string{`^8[.](0|4)([.][0-9]+)*$`}, + credentialSlots: []string{"MYSQL_PASSWORD", "MYSQL_ROOT_PASSWORD", "RESTIC_PASSWORD"}, + }, + "mariadb": { + driver: "mariadb", + recoveryKinds: map[string]bool{"pitr": true}, + supportedVersions: []string{`^11[.][0-9]+([.][0-9]+)*$`}, + credentialSlots: []string{"MARIADB_PASSWORD", "MARIADB_ROOT_PASSWORD", "RESTIC_PASSWORD"}, + }, + "mongodb": { + driver: "mongodb", + recoveryKinds: map[string]bool{"pitr": true}, + supportedVersions: []string{`^8[.]0([.][0-9]+)*$`}, + credentialSlots: []string{"MONGO_INITDB_ROOT_PASSWORD", "PBM_STORAGE_CREDENTIAL"}, + }, + "clickhouse": { + driver: "clickhouse", + recoveryKinds: map[string]bool{"snapshot": true}, + supportedVersions: []string{`^25([.][0-9]+)*$`}, + credentialSlots: []string{"CLICKHOUSE_PASSWORD", "CLICKHOUSE_BACKUP_CREDENTIAL"}, + }, + "redis": { + driver: "redis", + recoveryKinds: map[string]bool{"snapshot": true}, + supportedVersions: []string{`^8([.][0-9]+)*$`}, + credentialSlots: []string{"REDIS_PASSWORD", "RESTIC_PASSWORD"}, + }, + "valkey": { + driver: "valkey", + recoveryKinds: map[string]bool{"snapshot": true}, + supportedVersions: []string{`^8([.][0-9]+)*$`}, + credentialSlots: []string{"REDIS_PASSWORD", "RESTIC_PASSWORD"}, + }, + "rabbitmq": { + driver: "rabbitmq", + recoveryKinds: map[string]bool{"cold": true}, + supportedVersions: []string{`^4([.][0-9]+)*$`}, + credentialSlots: []string{"RABBITMQ_DEFAULT_PASS", "RABBITMQ_ERLANG_COOKIE", "RESTIC_PASSWORD"}, + }, + "minio": { + driver: "minio", + recoveryKinds: map[string]bool{"cold": true}, + supportedVersions: []string{`^RELEASE[.][0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9-]+Z$`}, + credentialSlots: []string{"MINIO_ROOT_PASSWORD", "RESTIC_PASSWORD"}, + }, + "meilisearch": { + driver: "meilisearch", + recoveryKinds: map[string]bool{"snapshot": true}, + supportedVersions: []string{`^1([.][0-9]+)*$`}, + credentialSlots: []string{"MEILI_MASTER_KEY", "RESTIC_PASSWORD"}, + }, + "nats": { + driver: "nats", + recoveryKinds: map[string]bool{"snapshot": true}, + supportedVersions: []string{`^2([.][0-9]+)*$`}, + credentialSlots: []string{"NATS_ACCOUNT_CREDENTIAL", "RESTIC_PASSWORD"}, + }, } diff --git a/internal/app/service_lifecycle_test.go b/internal/app/service_lifecycle_test.go index ee9315bc..6cf9805d 100644 --- a/internal/app/service_lifecycle_test.go +++ b/internal/app/service_lifecycle_test.go @@ -2,107 +2,77 @@ package app import "testing" -func TestProductionLifecycleCatalogueIsComplete(t *testing.T) { +func TestEveryRuntimeDriverHasALifecycleRecord(t *testing.T) { if err := validateLifecycleCatalogue(); err != nil { t.Fatalf("validate lifecycle catalogue: %v", err) } if got, want := len(lifecycleCapabilities), 11; got != want { t.Fatalf("lifecycle capability count = %d, want %d", got, want) } - for driverName := range drivers { - capability, ok := lifecycleCapabilityFor(driverName) - if !ok { + if _, ok := lifecycleCapabilityFor(driverName); !ok { t.Errorf("runtime driver %q has no lifecycle record", driverName) - continue - } - record := capability.Record() - if record.graduated { - t.Errorf("driver %q graduated without runtime evidence", driverName) - } - if len(record.patchTransitions) != 0 { - t.Errorf("driver %q has a default protected patch transition", driverName) - } - if len(record.encryptionByKind) != len(record.recoveryKinds) { - t.Errorf("driver %q does not model encryption for every recovery kind", driverName) } } } -func TestInternalLifecycleDriverExercisesEverySeamWithoutGraduating(t *testing.T) { - record := nonGraduatingTestLifecycleCapability() - if err := record.validate(); err != nil { - t.Fatalf("validate internal lifecycle driver: %v", err) - } - if !record.policyQualified || record.graduated { - t.Fatalf("internal driver qualification/graduation = %v/%v, want true/false", record.policyQualified, record.graduated) - } - if !record.SupportsRecoveryKind("1", "pitr") { - t.Fatal("internal driver does not exercise versioned recovery-kind selection") +// Qualification is per version, not per driver: a project declaring a version +// outside the qualified range must be refused rather than accepted on the +// driver's name alone. +func TestQualificationIsScopedToTheSupportedVersions(t *testing.T) { + postgres, ok := lifecycleCapabilityFor("postgres") + if !ok { + t.Fatal("postgres has no lifecycle record") } - if record.delivery != deliveryExternalHelper || record.helperArtifact == nil { - t.Fatal("internal driver does not exercise helper delivery and provenance") - } - if len(record.patchTransitions) != 1 { - t.Fatalf("internal driver patch transitions = %d, want 1", len(record.patchTransitions)) - } - if len(record.preconditions) == 0 || !record.preconditions[0].RestartRequired { - t.Fatal("internal driver does not exercise restart-gated enablement") - } - if record.repository == "" || record.retentionMapping == "" || record.achievableRPO == "" || - len(record.credentialSlots) == 0 || len(record.protectedResources) == 0 || len(record.graduationEvidence) == 0 || - record.operations.Backup == "" || record.operations.Restore == "" || record.operations.Verify == "" { - t.Fatal("internal driver does not exercise every lifecycle seam") + for _, tc := range []struct { + version string + qualified bool + }{ + {"18", true}, {"17.4", true}, {"16", false}, {"19", false}, {"latest", false}, + } { + if got := postgres.BackupQualified(tc.version); got != tc.qualified { + t.Errorf("BackupQualified(%q) = %v, want %v", tc.version, got, tc.qualified) + } } - if _, ok := drivers[record.driver]; ok { - t.Fatal("internal driver leaked into the runtime/schema catalogue") + if postgres.SupportsRecoveryKind("18", "snapshot") { + t.Error("postgres accepted a recovery kind its contract does not execute") } - if _, ok := lifecycleCapabilities[record.driver]; ok { - t.Fatal("internal driver leaked into the lifecycle/status catalogue") + if !postgres.SupportsRecoveryKind("18", "pitr") { + t.Error("postgres refused the recovery kind its contract executes") } } -func TestProtectedPatchTransitionsAreExact(t *testing.T) { - record := nonGraduatingTestLifecycleCapability() - transition := record.patchTransitions[0] - transition.CandidateServiceDigest = transition.CurrentServiceDigest - if err := transition.validate(record.delivery, true); err == nil { - t.Fatal("same-digest protected patch transition was accepted") +// An unqualified driver hands out no credential slots, so nothing downstream can +// treat it as having a backup contract. +func TestAnUnqualifiedDriverExposesNoCredentialContract(t *testing.T) { + if _, ok := LifecycleCredentialSlots("redis", "8"); ok { + t.Fatal("an unqualified driver exposed credential slots") } - - transition = record.patchTransitions[0] - transition.ContinuityProbes = nil - if err := transition.validate(record.delivery, true); err == nil { - t.Fatal("protected patch transition without continuity probes was accepted") + slots, ok := LifecycleCredentialSlots("postgres", "18") + if !ok || len(slots) == 0 { + t.Fatalf("postgres credential slots = %v, ok = %v", slots, ok) } } -// nonGraduatingTestLifecycleCapability is the internal driver that exercises -// every generic seam — external-helper delivery, an explicit protected patch -// transition, restart-gated preconditions — without entering the runtime/schema -// catalogue or ever graduating to Managed. -// -// It lives here rather than beside the real records because it is a fixture, and -// a fake driver defined in production code is a fake driver compiled into the -// shipped binary. The `_test_lifecycle` name is still known to -// lifecycleCapabilityRecord.validate, which is what lets this record validate -// without being a real driver; that exemption is the seam this fixture uses, and -// removing it would mean the test could no longer check the thing it exists to -// check. -func nonGraduatingTestLifecycleCapability() lifecycleCapabilityRecord { - record := lifecycleRecord( - "_test_lifecycle", true, "pitr", deliveryExternalHelper, repositoryArtifact, "client-side", "artifact", "1m", - "^1$", artifact("example.invalid/test-service", "test-service", '4'), helperArtifact("example.invalid/test-helper", "test-helper", '5'), - []lifecyclePrecondition{{Code: "test-topology", Consistency: "test-consistency", Topology: "test-topology", RestartRequired: true}}, - []string{"TEST_DATABASE_PASSWORD", "TEST_REPOSITORY_PASSWORD"}, []string{"test-data", "test-replay"}, - lifecycleOperations{Backup: "test-backup", Restore: "test-restore", Verify: "test-verify"}, - ) - record.patchTransitions = []protectedPatchTransition{{ - CurrentServiceDigest: seededDigest('4'), CandidateServiceDigest: seededDigest('6'), - CurrentHelperDigest: seededDigest('5'), CandidateHelperDigest: seededDigest('7'), - MaintenanceRange: "1.x", CompatibilityProbes: []string{"format-check"}, - ContinuityProbes: []string{"replay-check"}, RollbackLimit: "before-write", - }} - record.graduated = false - return record +// The catalogue is fixed data, so its own consistency rules are worth checking +// against a record that breaks each one. +func TestCatalogueValidationRejectsIncompleteRecords(t *testing.T) { + base := lifecycleCapabilities["postgres"] + for name, mutate := range map[string]func(*lifecycleCapability){ + "unknown driver": func(c *lifecycleCapability) { c.driver = "not-a-driver" }, + "no recovery kind": func(c *lifecycleCapability) { c.recoveryKinds = nil }, + "bad recovery kind": func(c *lifecycleCapability) { c.recoveryKinds = map[string]bool{"eventually": true} }, + "no versions": func(c *lifecycleCapability) { c.supportedVersions = nil }, + "bad version": func(c *lifecycleCapability) { c.supportedVersions = []string{"^1[7"} }, + "no credentials": func(c *lifecycleCapability) { c.credentialSlots = nil }, + "unsafe credential": func(c *lifecycleCapability) { c.credentialSlots = []string{"NOT AN ENV NAME"} }, + } { + t.Run(name, func(t *testing.T) { + record := base + mutate(&record) + if err := record.validate(); err == nil { + t.Fatal("an incomplete lifecycle record validated") + } + }) + } } diff --git a/internal/app/services.go b/internal/app/services.go index a860f12a..b29f3f51 100644 --- a/internal/app/services.go +++ b/internal/app/services.go @@ -307,7 +307,7 @@ func (p *Spec) serviceHasHealth(name string) bool { // renderService generates one service's Compose document. It is its own // project, so it survives every release of the application beside it. -func (p *Spec) renderService(n Names, name string, s Service, selectedImage string) ([]byte, error) { +func (p *Spec) renderService(n Names, name string, s Service, selectedImage string, backup *serviceBackup) ([]byte, error) { key, d, ok := driverOf(name, s) if !ok { return nil, errf("unknown_service_driver", "services."+name, "", strings.Join([]string{ @@ -339,6 +339,13 @@ func (p *Spec) renderService(n Names, name string, s Service, selectedImage stri "env_file": []string{n.ServiceSecretFile(name)}, } + // The backup credential file comes second so its entries are present + // alongside the service credential, not instead of it: Compose applies each + // env_file over the ones before it, and the two name disjoint variables. + if backup != nil { + svc["env_file"] = []string{n.ServiceSecretFile(name), backup.CredentialFile} + } + env := map[string]any{} for k, v := range d.env { env[k] = v @@ -353,6 +360,38 @@ func (p *Spec) renderService(n Names, name string, s Service, selectedImage stri if err := applySettings(name, d, s, s.Settings, env, &command); err != nil { return nil, err } + + // Backup is applied over authored settings rather than under them. The + // archive configuration is not a default a project may prefer differently: + // a server whose archive_mode an author turned back off would keep running + // while its recovery window silently stopped advancing. + if backup != nil { + // Where the repository is, never how to open it. The credential entry + // *names* are here too; their values live in the mode-0600 file on the + // host and are read by the wrapper, so no secret enters this document + // or its digest. + for key, value := range backup.Environment { + env[key] = value + } + if len(command) == 0 { + // The official entrypoint dispatches on argv[0], so the server has + // to be named again once this document supplies a command at all. + command = []string{"postgres"} + } + command = append(command, + "-c", "archive_mode=on", + "-c", "archive_command="+backup.ArchiveCommand, + "-c", "archive_timeout="+backup.ArchiveTimeout, + ) + // wal_level=replica is the floor archiving needs, so it is raised to + // rather than set. Appending it unconditionally would override an + // authored `wal_level: logical` — the last -c wins — and every logical + // replication slot on the server would stop working the moment somebody + // enabled backups. + if !declaresAtLeastReplicaWAL(s.Settings) { + command = append(command, "-c", "wal_level=replica") + } + } if len(env) > 0 { svc["environment"] = env } @@ -370,7 +409,15 @@ func (p *Spec) renderService(n Names, name string, s Service, selectedImage stri if d.dataPath != "" && !serviceIsEphemeral(s) { vol := dataVolume(s) full := n.ServiceVolume(name, vol) - svc["volumes"] = []string{full + ":" + d.dataPath} + mounts := []string{full + ":" + d.dataPath} + if backup != nil { + // The directory, not the files — see BackupRuntimeDir for why + // an atomically replaced file vanishes from a running container. + // Read-only, because a container that could rewrite the binary it + // archives with could send the archive anywhere. + mounts = append(mounts, backup.RuntimeHostDir+":"+WalgMountPath+":ro") + } + svc["volumes"] = mounts volumes[full] = map[string]any{ "name": full, "labels": map[string]any{"ob.app": p.Name, "ob.service": name}, @@ -773,3 +820,17 @@ func atomicEnvFile(path string, body func(target string) string) string { func shellQuote(s string) string { return shellquote.Quote(s) } + +// declaresAtLeastReplicaWAL reports whether the project already asks for a WAL +// level that carries everything archiving needs. `logical` is a superset of +// `replica`; `minimal` is not, and is refused rather than silently raised +// because a project asking for minimal has asked for something backup +// cannot deliver. +func declaresAtLeastReplicaWAL(settings map[string]any) bool { + value, ok := settings["wal_level"] + if !ok { + return false + } + level := fmt.Sprint(value) + return level == "replica" || level == "logical" +} diff --git a/internal/app/services_test.go b/internal/app/services_test.go index 21995ff2..8f014944 100644 --- a/internal/app/services_test.go +++ b/internal/app/services_test.go @@ -525,16 +525,16 @@ services: } } -// Protection is a contract about recovering durable data. With no volume +// Backup is a contract about recovering durable data. With no volume // rendered, seeding the active volume fails at apply time against one that was // never created, and the sealed identity names it anyway. -func TestEphemeralServiceCannotDeclareProtection(t *testing.T) { - src := strings.Replace(validProtectionProject, - " postgres:\n version: 17\n protection:", - " postgres:\n version: 17\n persistence: {mode: ephemeral}\n protection:", 1) +func TestEphemeralServiceCannotDeclareBackup(t *testing.T) { + src := strings.Replace(validBackupProject, + " postgres:\n version: 17\n backup:", + " postgres:\n version: 17\n persistence: {mode: ephemeral}\n backup:", 1) _, err := LoadBytes([]byte(src), "ob.yml") if err == nil { - t.Fatal("an ephemeral service declaring protection was accepted") + t.Fatal("an ephemeral service declaring backup was accepted") } if !strings.Contains(err.Error(), "no durable data to protect") { t.Fatalf("refusal does not explain itself: %v", err) diff --git a/internal/app/testdata/contract-verdicts.json b/internal/app/testdata/contract-verdicts.json index 190001c8..f10b7fd7 100644 --- a/internal/app/testdata/contract-verdicts.json +++ b/internal/app/testdata/contract-verdicts.json @@ -2,7 +2,7 @@ { "case": "conformance/a bind mount is not durable", "loads": true, - "digest": "9153d2cdebfe7fa9fb99aca9c8e43ee29c0caa01c5c6b2f21e7669f322607693" + "digest": "ed4d60c87aec3dca217f0c3721377c7f5330ebfc7362b0a41608b57c1a44ea0f" }, { "case": "conformance/a near-miss field name", @@ -12,7 +12,7 @@ { "case": "conformance/a plugin log driver", "loads": true, - "digest": "288c7699e676ca18be4d0e354786bb0d54278f1d68e48a68ef33b3fc7e0e0b01" + "digest": "0f64ad0f34da5c6b72a781b5866a36009049ade8784229bdeeea15489ac6a766" }, { "case": "conformance/absolute compose ref", @@ -24,11 +24,6 @@ "loads": false, "code": "project_invalid" }, - { - "case": "conformance/alerts without logs", - "loads": true, - "digest": "2687af022cbd5bbc2daa11a9a9fe0aad37933378d5d5a4f759f5fec4331915bc" - }, { "case": "conformance/app starting ob-", "loads": false, @@ -44,6 +39,41 @@ "loads": false, "code": "project_invalid" }, + { + "case": "conformance/backup authored tool", + "loads": false, + "code": "unknown_field" + }, + { + "case": "conformance/backup inline secret", + "loads": false, + "code": "unknown_field" + }, + { + "case": "conformance/backup is no longer a field", + "loads": false, + "code": "unknown_field" + }, + { + "case": "conformance/backup self target", + "loads": false, + "code": "backup_target_not_independent" + }, + { + "case": "conformance/backup sparse drill", + "loads": false, + "code": "drill_schedule_too_sparse" + }, + { + "case": "conformance/backup unsupported objective", + "loads": false, + "code": "recovery_objective_unsupported" + }, + { + "case": "conformance/backup unsupported retention", + "loads": false, + "code": "backup_retention_unsupported" + }, { "case": "conformance/bad protocol", "loads": false, @@ -57,7 +87,7 @@ { "case": "conformance/bind mount volume", "loads": true, - "digest": "a64857a79c535d2e8c3e7ecb0c3b7a88d591f49fa764d4ab0e67fe898347fb51" + "digest": "37f4eb9c3ab03f69198a65f3f8c0e68d1a8515a99e6a6539390f8891d6de7965" }, { "case": "conformance/components is not a field", @@ -67,7 +97,7 @@ { "case": "conformance/daemon role", "loads": true, - "digest": "215a0ae8accd2f64f6ec5cd12b6a93ba5a63a6edeb2d2333bf5a79d7f1250680" + "digest": "0481a5635e8afaee4f26d84f4df9ef123c82f3c5f74516de0eb757fb499335c8" }, { "case": "conformance/declared durability still refuses replicas", @@ -87,7 +117,7 @@ { "case": "conformance/duration in days", "loads": true, - "digest": "5ac4d1f97c5a8fc93a3ebab678f6196fd68051d6f5eae0df66aaaa49d19687c6" + "digest": "7a431791a1c3b31ce1d06ec2cd56702bdc68208a3c286671e769baad96aa2892" }, { "case": "conformance/encrypted env file entry", @@ -102,17 +132,17 @@ { "case": "conformance/environment-scoped env files", "loads": true, - "digest": "5d4ab0e5117c9cbd57cf39e07f8cc8cab4e81cb73c1dc7dc8d331bd8b6ef39cc" + "digest": "1e92b0ede1f0e996df3f8dc9af17f80b16ae876d73a9758b5d486a38e1b7d58d" }, { "case": "conformance/explicit manual job remains a runtime service", "loads": true, - "digest": "e81a245edbe89dd02c6cf76ef2aed249272f4d13739311afde5c37cea88d3b7e" + "digest": "9a3e5ddfb89ac4b00767ad3a27be20eb69fb2ab07e8a2ce8cfbfebe0d9cce208" }, { "case": "conformance/explicit workloads block", "loads": true, - "digest": "9a25e5e99e9f1eb1cc71d17a0c8a13089938ddfadd052499097277b5dd7fcfe3" + "digest": "74af0694829a1c0c88c17ed313cbcdd8387e70c80fc7c0089d099cfcab4d53a4" }, { "case": "conformance/external lifecycle field", @@ -122,12 +152,12 @@ { "case": "conformance/external service connection", "loads": true, - "digest": "6a5f26a10eb914e99aa12a6ea86d0f7ccb780c50ba3e8710cb7a2dc59d0dbfc9" + "digest": "21ea4b4de122e9f7c3b02d200aa1fc6fcafb85adb8c2ac0e360736d4952c03e6" }, { "case": "conformance/hook naming a declared job", "loads": true, - "digest": "c2f3297325f024100b235a642cd05b17f154ff00c5da60c1ab545a149298681f" + "digest": "78f7b728401351cc89b2ea0419c17ff53bbe5626984e70a0b0bc3205340a7971" }, { "case": "conformance/hook naming an unlisted seam", @@ -149,10 +179,15 @@ "loads": false, "code": "project_invalid" }, + { + "case": "conformance/http check without a path", + "loads": false, + "code": "project_invalid" + }, { "case": "conformance/image reference with registry port", "loads": true, - "digest": "4f17d1c2310f306e61106858d29aaf3c80f276b09f8c36d6db1264955733fe7d" + "digest": "21b0f222f6b5bb6be9e820b5f78764624d1584c6cf5adea0454a749e312ead5d" }, { "case": "conformance/image reference with uppercase repository", @@ -167,12 +202,12 @@ { "case": "conformance/inferred durability does not refuse replicas", "loads": true, - "digest": "7772a6933822463c53f1e568b0d9c3287d6abbe53d11c166dc7f67588839957b" + "digest": "da7276b116782df45c1fe6ec8fc5eab89392adf7b433885c8733d93cb56e41a4" }, { "case": "conformance/job data_effect unknown", "loads": true, - "digest": "e81a245edbe89dd02c6cf76ef2aed249272f4d13739311afde5c37cea88d3b7e" + "digest": "9a3e5ddfb89ac4b00767ad3a27be20eb69fb2ab07e8a2ce8cfbfebe0d9cce208" }, { "case": "conformance/job requires data_effect", @@ -182,7 +217,7 @@ { "case": "conformance/job with data_effect", "loads": true, - "digest": "e81a245edbe89dd02c6cf76ef2aed249272f4d13739311afde5c37cea88d3b7e" + "digest": "9a3e5ddfb89ac4b00767ad3a27be20eb69fb2ab07e8a2ce8cfbfebe0d9cce208" }, { "case": "conformance/log driver with a space", @@ -194,16 +229,6 @@ "loads": false, "code": "project_invalid" }, - { - "case": "conformance/log retention as an integer", - "loads": false, - "code": "project_invalid" - }, - { - "case": "conformance/log retention without alerts", - "loads": false, - "code": "project_invalid" - }, { "case": "conformance/managed route middleware without proxy config", "loads": false, @@ -257,12 +282,12 @@ { "case": "conformance/one-char identifier", "loads": true, - "digest": "5ac4d1f97c5a8fc93a3ebab678f6196fd68051d6f5eae0df66aaaa49d19687c6" + "digest": "7a431791a1c3b31ce1d06ec2cd56702bdc68208a3c286671e769baad96aa2892" }, { "case": "conformance/operator proxy owns route middleware", "loads": true, - "digest": "6506674c2e724800df2920c0dad03d56bd53191127c44ee3a3cc3997c9c7389e" + "digest": "6513d9dbfe8373cd8c986f78d61346f49490b3d27264047ded4c2f5af62ee50b" }, { "case": "conformance/persistence block with no mode still refuses replicas", @@ -272,52 +297,17 @@ { "case": "conformance/persistence external", "loads": true, - "digest": "b9d10fff3e5a558e178a95969ef02e7bb9a88e667344d3771f2fd0dbcdec9029" + "digest": "c7d47947b4512f4b43a42f49fc3f0ad6f26bbaf68dca6418a40b3bd377712e25" }, { "case": "conformance/port out of range", "loads": false, "code": "project_invalid" }, - { - "case": "conformance/protection authored tool", - "loads": false, - "code": "unknown_field" - }, - { - "case": "conformance/protection inline secret", - "loads": false, - "code": "unknown_field" - }, - { - "case": "conformance/protection is no longer a field", - "loads": false, - "code": "unknown_field" - }, - { - "case": "conformance/protection self target", - "loads": false, - "code": "backup_target_not_independent" - }, - { - "case": "conformance/protection sparse drill", - "loads": false, - "code": "restore_drill_schedule_too_sparse" - }, - { - "case": "conformance/protection unsupported objective", - "loads": false, - "code": "recovery_objective_unsupported" - }, - { - "case": "conformance/protection unsupported retention", - "loads": false, - "code": "backup_retention_unsupported" - }, { "case": "conformance/provider-qualified route middlewares", "loads": true, - "digest": "d1be2e35c3638c2532e28f43043520b085bdc091815355023ee02a1d535e26cf" + "digest": "5b51f8b77ea5a83635d1dba0470729b0d4ba2925f7a029b3fe6b86203bab5582" }, { "case": "conformance/proxy kind none with a route", @@ -327,17 +317,17 @@ { "case": "conformance/proxy kind none without a route", "loads": true, - "digest": "9a25e5e99e9f1eb1cc71d17a0c8a13089938ddfadd052499097277b5dd7fcfe3" + "digest": "74af0694829a1c0c88c17ed313cbcdd8387e70c80fc7c0089d099cfcab4d53a4" }, { "case": "conformance/published udp port", "loads": true, - "digest": "b4bdfafcbb5b413b0a26a2347869340ad3a6fee5988c586620ed911e688bf0b3" + "digest": "0979f4d25c39158939bb41f04ca82839ae8021d56cff62ae0f41043d831bd695" }, { "case": "conformance/recreate workload with published host port", "loads": true, - "digest": "4085d5da5b79188a4136716a328fc7923f19d45d383e08502a55033846bc501b" + "digest": "08d756ffdd909355169e1edea244892a06354aed9720123717b5ad9e03d5b1dc" }, { "case": "conformance/relative compose ref", @@ -352,7 +342,7 @@ { "case": "conformance/repeated route middleware remains ordered", "loads": true, - "digest": "6cdb57177de03068697a4c5449d1fa21945581a1b68b27e1364e8818a81c9732" + "digest": "af54580b167899b337a4b7fb37e6f88b98904dac3906de38c5b17d21a865e2fe" }, { "case": "conformance/rolling workload with published host port", @@ -362,17 +352,17 @@ { "case": "conformance/routes list", "loads": true, - "digest": "e6d2c5b4594cf8f94a16f49b738fff5c446e61ddfc46d4f16648f3c991d2b6c7" + "digest": "ec9b3e1de67e1bf376ff19512c1435f4e7b999921895c7a03b25651754ae0e1b" }, { "case": "conformance/scheduled job", "loads": true, - "digest": "e81a245edbe89dd02c6cf76ef2aed249272f4d13739311afde5c37cea88d3b7e" + "digest": "9a3e5ddfb89ac4b00767ad3a27be20eb69fb2ab07e8a2ce8cfbfebe0d9cce208" }, { - "case": "conformance/service protection policy", + "case": "conformance/service backup policy", "loads": true, - "digest": "235dcc9445a139b939e7a13950f90e69cd5bf67621cc9f4588f04ec2a013af5c postgres=c1475eb63145a73b" + "digest": "1bce81b664f94d47dea544b3de676178615ed8a0a2c9efb8eda0a9d7f424b1e4 postgres=c1475eb63145a73b" }, { "case": "conformance/service scalar", @@ -465,24 +455,19 @@ "code": "project_invalid" }, { - "case": "conformance/verifications url contains advisory", - "loads": true, - "digest": "2687af022cbd5bbc2daa11a9a9fe0aad37933378d5d5a4f759f5fec4331915bc" - }, - { - "case": "conformance/verifications url with exec", + "case": "conformance/url check carrying an exec field", "loads": false, - "code": "project_invalid" + "code": "unknown_field" }, { - "case": "conformance/verifications workload without probe", - "loads": false, - "code": "project_invalid" + "case": "conformance/url check with contains and advisory", + "loads": true, + "digest": "2687af022cbd5bbc2daa11a9a9fe0aad37933378d5d5a4f759f5fec4331915bc" }, { "case": "conformance/volume scalar with a path", "loads": true, - "digest": "a42e5587db73041bb0ef4c563ae6ea4f8eaaf390afbfe79daefcc4cb06a2b135" + "digest": "d45e80f712c3333cfd93df0ea3c4354b21192fd0c8e54ed709bc093bd6865e38" }, { "case": "conformance/volume scalar without a path", @@ -492,7 +477,7 @@ { "case": "conformance/volumes without persistence still load", "loads": true, - "digest": "a42e5587db73041bb0ef4c563ae6ea4f8eaaf390afbfe79daefcc4cb06a2b135" + "digest": "d45e80f712c3333cfd93df0ea3c4354b21192fd0c8e54ed709bc093bd6865e38" }, { "case": "conformance/worker with schedule", @@ -517,17 +502,17 @@ { "case": "corpus/authentik.yml", "loads": true, - "digest": "6d3a2518ab077033b35018bb81aa9702a641a2ad7433468afacb6070083d60a8 postgres=dc4f8448b8b82b4a redis=d2660eeb4faa49fe" + "digest": "63f575430f70b00d2708dcbc93b8239e0f7c92f9889d156b1a4d6ca914e00aac postgres=dc4f8448b8b82b4a redis=d2660eeb4faa49fe" }, { "case": "corpus/ext-authentik-managed.yml", "loads": true, - "digest": "f6b0b547d0bc7d55b8309e57f4f928a57e2ce44dd4fecc773599c9785580083f postgres=dc4f8448b8b82b4a redis=d2660eeb4faa49fe" + "digest": "52a1eae2842164841682dcc468a20ffe3f700279f3e18224f5cf348e3eef3cea postgres=dc4f8448b8b82b4a redis=d2660eeb4faa49fe" }, { "case": "corpus/ext-authentik.yml", "loads": true, - "digest": "c3536ec0316f5d8ebb123c8cc7e924a108740029f1b7112b0a165742229fd46b" + "digest": "f25496dd235618927b97f8476fbdecc3ccce577cc52995559aa20d373a490f56" }, { "case": "corpus/ext-frigate.yml", @@ -537,12 +522,12 @@ { "case": "corpus/ext-gitea.yml", "loads": true, - "digest": "b190e64ca141e0af31a7a2e831cc218a43abdbcd554e1b0c7cf01cf9657daf42 postgres=e70cc45c347098f9" + "digest": "24544823fe1007300da3b22f82f06c2cd1a693adcec6ee47307f1f09e264d708 postgres=e70cc45c347098f9" }, { "case": "corpus/ext-immich-sourced.yml", "loads": true, - "digest": "b852204b8ab417c0b5ca7c162108f95189829c69f8b7284d58120fe1e17c47d9 postgres=76af15324857fa90 redis=c7c8bf8044d5342a" + "digest": "unparseable:compose parse: validating compose.yaml: services.immich-server additional properties 'published_ports' not allowed" }, { "case": "corpus/ext-immich.yml", @@ -552,22 +537,22 @@ { "case": "corpus/ext-n8n.yml", "loads": true, - "digest": "d9b22f801a91ee7c4f6e9d24811e9454374067466263cc08cbc18cb9aefc8241 postgres=809549d286e2dbdc redis=86933b446609e6d8" + "digest": "9b6fbbcdcfe32b45d1e095b90e0b96ff93123ebdc58d884eaf18d3014d6c93ac postgres=809549d286e2dbdc redis=86933b446609e6d8" }, { "case": "corpus/ext-paperless.yml", "loads": true, - "digest": "7399471473ceeaf2e4c4c12c1f7fb12510229ced25da1e7555fcb1e4ca53e79f" + "digest": "f11f765d9b72177c94d492020440c6245f62472bdce107289d6540ac0aa6d39f" }, { "case": "corpus/ext-plausible.yml", "loads": true, - "digest": "7c5101a2b9631b348ad45ec54b6e6a046fda23f52c0afbb088ac00573bb27251 events=1e798590e3a5dda4 postgres=b1fac70440c33545" + "digest": "be23ed70c8eec24e2b4746bd6839d46e6ec4fb2b45c1a234c8618caffa74e2e3 events=1e798590e3a5dda4 postgres=b1fac70440c33545" }, { "case": "corpus/ext-umami.yml", "loads": true, - "digest": "5ccbd8f60a2d58d1d0add204390f261084dcf76da9b5efb938740ec137198e19 postgres=36c6c38ba304b445" + "digest": "822b9460beb259b9b7772258465e79c621e0e0571afc765d0d9948aaba7939f4 postgres=36c6c38ba304b445" }, { "case": "corpus/fanout.yml", @@ -577,12 +562,12 @@ { "case": "corpus/ghost.yml", "loads": true, - "digest": "5839e5acd73a72aa8781b9431813de5c7882b78f0ce88054003f9a3390bcdc62 mysql=0f13a6374095d11b" + "digest": "88dd73e41c3566e107aec5fc23b1458daf406a29d7fb3a10c4fe64c5d0532479 mysql=0f13a6374095d11b" }, { "case": "corpus/gitea.yml", "loads": true, - "digest": "e55007fd6bd484f3af4ed13e764295ed6827be33335a191c2c6b39a9f03bef4f" + "digest": "cbe804b82e046201170393161481c3faef0649c52e64ee009c4612997ff95233" }, { "case": "corpus/goal.yml", @@ -592,7 +577,7 @@ { "case": "corpus/immich.yml", "loads": true, - "digest": "49d5a7c01f7967af67333602772e376a3c1fd355c23fc25429557787cb9d60b0" + "digest": "eccb7be79bef04153b3e998ce4f2f6cd5e35f072da74b5b9eac6b334c1118a8a" }, { "case": "corpus/monk.yml", @@ -602,17 +587,17 @@ { "case": "corpus/n8n.yml", "loads": true, - "digest": "2f919c34b74b3ac300d253189b9dc33031439ea70e3b052799597da444959eb3" + "digest": "404e41227962f8e1b0ed02980f4c2f245c2c5addda87feba7d4be314eb9dd4cf" }, { "case": "corpus/paperless.yml", "loads": true, - "digest": "bc40a993726a4bd48a043b9a2fc873000fda8d961288e5844b49f6442e644a2c" + "digest": "223eb7692be0b2d680f1e095bbf96ec5b4a03e7965ee0cef98c4d70b041d19f9" }, { "case": "corpus/penpot.yml", "loads": true, - "digest": "10d0d86d56452027d9d317f6d77d0a696ed95eb2595e9cf039db6d187ff338e6 postgres=fc584b1b50db23a6 redis=fcccb6a023ae5734" + "digest": "bdd52bee442c6f91c2dbe9c9dafa4cae1050116a58c7151d870f33e72cde5991 postgres=fc584b1b50db23a6 redis=fcccb6a023ae5734" }, { "case": "corpus/pursue.yml", @@ -627,21 +612,21 @@ { "case": "corpus/rocketchat.yml", "loads": true, - "digest": "99221f9fbd9730302616845b03ff75ade0dfa7e7857990f97904375b1e95ae1d mongodb=eaca06e5d1b88e4b" + "digest": "d406317867cf94a915ae85a90abf4be709e8a6b08aa7a834417be5f0f6f0d690 mongodb=eaca06e5d1b88e4b" }, { "case": "corpus/umami.yml", "loads": true, - "digest": "157ee9ab8c95702b93040e613c0636963dc1db1d5b0d46c4e30d95e27af11b5f" + "digest": "e969c3fa27ec71f300e8c65e42ae8c95e08ab56388692438b46640c4350b8c64" }, { "case": "corpus/uptime-kuma.yml", "loads": true, - "digest": "539c9c46853a6e942dcec9a2fee116f0492b223128c5f6d664a70c746a37b7b6" + "digest": "1ec690b55cbe4361efbcc5be90f7468d9e6b75b0133f79b7740129d1ef9d144c" }, { "case": "corpus/vaultwarden.yml", "loads": true, - "digest": "3489995628a73046fe58cb5eb7fd47fc86cd9c6eac16eff5f410ae4d6e658ca0" + "digest": "8d2b1e2990a723b085e09497ff135c3435ed96f55035d7470f0d5fddbdbbccc0" } ] diff --git a/internal/app/testdata/corpus/ext-immich-sourced.compose.yaml b/internal/app/testdata/corpus/ext-immich-sourced.compose.yaml index 3a4969c9..33a09fd4 100644 --- a/internal/app/testdata/corpus/ext-immich-sourced.compose.yaml +++ b/internal/app/testdata/corpus/ext-immich-sourced.compose.yaml @@ -13,7 +13,7 @@ services: IMMICH_WORKERS_INCLUDE: api volumes: - ./library:/usr/src/app/upload - ports: + published_ports: - 2283:2283 immich-machine-learning: diff --git a/internal/app/testdata/corpus/fanout.yml b/internal/app/testdata/corpus/fanout.yml index f77c2a54..40bfdd25 100644 --- a/internal/app/testdata/corpus/fanout.yml +++ b/internal/app/testdata/corpus/fanout.yml @@ -42,6 +42,7 @@ runtime: env_checks: - {file: fanout/.env, require: [FANOUT_SECRET]} - {file: traefik/.env, require: [CF_DNS_API_TOKEN]} -verifications: - - {url: "https://fanout.run/"} - - {url: "https://fanout.labstack.com/healthz"} +checks: + url: + - {url: "https://fanout.run/"} + - {url: "https://fanout.labstack.com/healthz"} diff --git a/internal/app/testdata/corpus/goal.yml b/internal/app/testdata/corpus/goal.yml index 0040700b..45eaa02c 100644 --- a/internal/app/testdata/corpus/goal.yml +++ b/internal/app/testdata/corpus/goal.yml @@ -33,6 +33,8 @@ runtime: proxy: {managed: true, config: traefik} hooks: bootstrap: {run: 'scripts/bootstrap.sh "$OB_SERVER"', local: true} -verifications: - - {workload: server, http: /readyz, port: 7510} - - {url: "https://goal.fit/healthz"} +checks: + http: + - {workload: server, path: /readyz, port: 7510} + url: + - {url: "https://goal.fit/healthz"} diff --git a/internal/app/testdata/corpus/monk.yml b/internal/app/testdata/corpus/monk.yml index a5603255..d65cb4f0 100644 --- a/internal/app/testdata/corpus/monk.yml +++ b/internal/app/testdata/corpus/monk.yml @@ -50,7 +50,9 @@ hooks: bootstrap: {run: 'scripts/bootstrap.sh "$OB_SERVER"', local: true} pre_release: {run: "cd web && bun install --frozen-lockfile && bun run build && rsync -az dist/ $OB_SERVER:/data/monk/web/", local: true} post_deploy: {run: "rsync -az web/dist/index.html $OB_SERVER:/data/monk/web/index.html", local: true} -verifications: - - {workload: server, http: /healthz, port: 7500} - - {url: "https://monk.trade/healthz", advisory: true} - - {url: "https://monk.trade/", contains: 'id="root"', advisory: true} +checks: + http: + - {workload: server, path: /healthz, port: 7500} + url: + - {url: "https://monk.trade/healthz", advisory: true} + - {url: "https://monk.trade/", contains: 'id="root"', advisory: true} diff --git a/internal/app/testdata/corpus/pursue.yml b/internal/app/testdata/corpus/pursue.yml index 1bf4cd40..fe2103bb 100644 --- a/internal/app/testdata/corpus/pursue.yml +++ b/internal/app/testdata/corpus/pursue.yml @@ -29,9 +29,11 @@ runtime: env_checks: - file: .env.production require: [DATABASE_URL, POSTGRES_PASSWORD, PUBLIC_BASE_URL, HTTP_ADDR, PURSUE_SECRET_KEY] -verifications: - - {workload: server, http: /healthz, port: 8080} - - {url: "https://pursue.run/healthz"} +checks: + http: + - {workload: server, path: /healthz, port: 8080} + url: + - {url: "https://pursue.run/healthz"} proxy: managed: true kind: traefik-docker diff --git a/internal/app/testdata/corpus/recast.yml b/internal/app/testdata/corpus/recast.yml index 5b1cf254..cd77a656 100644 --- a/internal/app/testdata/corpus/recast.yml +++ b/internal/app/testdata/corpus/recast.yml @@ -6,9 +6,9 @@ environments: policy: require_approval: true allow_agent_proposals: true - minimum_onebox_version: v2026.8.0 - minimum_plan_schema: onebox.run/executable-deploy-plan/v1alpha2 - require_migration_backup: false + min_onebox_version: v2026.8.0 + min_plan_schema: onebox.run/executable-deploy-plan/v1alpha2 + migrations: {require_backup: false} workloads: server: role: application @@ -37,9 +37,11 @@ runtime: env_checks: - file: .env.production require: [RECAST_ENV, RECAST_BASE_URL, DATABASE_URL, RECAST_AUTH_PEPPER, POSTGRES_PASSWORD] -verifications: - - {workload: server, http: /healthz, port: 8080} - - {url: "https://recast.report/healthz"} +checks: + http: + - {workload: server, path: /healthz, port: 8080} + url: + - {url: "https://recast.report/healthz"} proxy: managed: true kind: traefik-docker diff --git a/internal/app/types.go b/internal/app/types.go index 2eb0ce50..a46e5ecc 100644 --- a/internal/app/types.go +++ b/internal/app/types.go @@ -38,8 +38,8 @@ type Spec struct { Environments map[string]Environment `json:"environments" description:"Named environments, each naming the server it deploys to and the policy applied to it."` Workloads map[string]Workload `json:"workloads,omitempty" description:"Application containers, workers, daemons, and jobs managed as releases."` Services map[string]Service `json:"services,omitempty" description:"Supporting services managed outside application releases, such as databases and caches."` - ExternalServices map[string]ExternalService `json:"external_services,omitempty" description:"Typed dependencies operated outside Onebox. Their connection projection is trusted, but their lifecycle and protection remain external."` - BackupTargets map[string]BackupTarget `json:"backup_targets,omitempty" description:"User-owned off-host repositories available to service protection policies."` + ExternalServices map[string]ExternalService `json:"external_services,omitempty" description:"Typed dependencies operated outside Onebox. Their connection projection is trusted, but their lifecycle and backup remain external."` + BackupTargets map[string]BackupTarget `json:"backup_targets,omitempty" description:"User-owned off-host repositories available to service backup policies."` Deployment Deployment `json:"deployment" description:"Release ordering, retention, and migration behavior."` Runtime *Runtime `json:"runtime,omitempty" description:"Project-wide environment files and local environment-file requirements."` @@ -50,11 +50,10 @@ type Spec struct { // closedness check reads the contract's fields from these tags. envDefault []EnvFile Hooks map[string]Command `json:"hooks,omitempty" description:"Lifecycle commands keyed by seam: bootstrap, pre_release, post_release, or post_deploy."` - Verifications []Verification `json:"verifications,omitempty" description:"Checks that must pass before a release becomes current unless marked advisory."` + Checks Checks `json:"checks,omitzero" description:"Assertions that must pass before a release becomes current unless marked advisory."` Notifications map[string]Notification `json:"notifications,omitempty" description:"Named webhooks that receive selected operation outcomes."` Registries map[string]Registry `json:"registries,omitempty" description:"Named container registries and the environment variables holding their credentials."` Proxy Proxy `json:"proxy" description:"Ownership and configuration of the host ingress proxy."` - Observability *Observability `json:"observability,omitempty" description:"Declared logging, metrics, and alerting intent. Continuous management is not currently provided."` // rawExpanded is the authored input after shorthand expansion, kept so a // value's origin can be reported without threading a marker through every @@ -87,14 +86,21 @@ type Server struct { } type Policy struct { - RequireApproval bool `json:"require_approval" description:"Require a plan-bound local confirmation before mutating this environment." default:"true"` - AllowAgentProposals bool `json:"allow_agent_proposals" description:"Declared permission for agent-authored proposals. The current CLI does not distinguish agent identity; execution remains approval-gated." default:"true"` - MinimumOneboxVersion string `json:"minimum_onebox_version,omitempty" description:"Oldest released Onebox runner allowed to operate this environment." example:"v2026.8.0"` - MinimumPlanSchema string `json:"minimum_plan_schema,omitempty" description:"Oldest executable plan schema accepted by this environment." example:"onebox.run/executable-deploy-plan/v1alpha2"` - RequireMigrationBackup bool `json:"require_migration_backup" description:"Require a plan-bound backup report before a release with migration risk." default:"false"` - MigrationBackupMaximumAge string `json:"migration_backup_maximum_age,omitempty" description:"Maximum age of a backup report accepted for a migration." default:"24h" example:"24h"` - RequireMigrationRestoreTest bool `json:"require_migration_restore_test" description:"Require the backup report to state that a restore test succeeded." default:"false"` - MigrationBackupKeyMaterial []string `json:"migration_backup_key_material,omitempty" description:"Names of key material whose usability must be covered by the migration backup report."` + RequireApproval bool `json:"require_approval" description:"Require a plan-bound local confirmation before mutating this environment." default:"true"` + AllowAgentProposals bool `json:"allow_agent_proposals" description:"Declared permission for agent-authored proposals. The current CLI does not distinguish agent identity; execution remains approval-gated." default:"true"` + MinOneboxVersion string `json:"min_onebox_version,omitempty" description:"Oldest released Onebox runner allowed to operate this environment." example:"v2026.8.0"` + MinPlanSchema string `json:"min_plan_schema,omitempty" description:"Oldest executable plan schema accepted by this environment." example:"onebox.run/executable-deploy-plan/v1alpha2"` + // Migrations groups what this environment demands of a release that carries + // migration risk. Grouped rather than four flat keys each repeating the + // word: the prefix is the block's name now. + Migrations MigrationPolicy `json:"migrations,omitzero" description:"What this environment requires of a release carrying migration risk."` +} + +type MigrationPolicy struct { + RequireBackup bool `json:"require_backup" description:"Require a plan-bound backup report before a release with migration risk." default:"false"` + BackupMaxAge string `json:"backup_max_age,omitempty" description:"Maximum age of a backup report accepted for a migration." default:"24h" example:"24h"` + RequireRestoreTest bool `json:"require_restore_test" description:"Require the backup report to state that a restore test succeeded." default:"false"` + BackupKeyMaterial []string `json:"backup_key_material,omitempty" description:"Key-material identities the backup report must name." example:"BACKUP_ACCESS_KEY_ID"` } type Overrides struct { @@ -158,14 +164,11 @@ type Build struct { Dockerfile string `json:"dockerfile,omitempty" description:"Repository-relative Dockerfile path." example:"Dockerfile"` Target string `json:"target,omitempty" description:"Named Dockerfile stage to build."` Args map[string]any `json:"args,omitempty" description:"Build arguments supplied by the external build system."` - Platform string `json:"platform,omitempty" description:"Target image platform for the external build." example:"linux/amd64"` } type Image struct { Reference string `json:"reference" description:"Complete container image reference, optionally tagged or digest-pinned." example:"ghcr.io/acme/shop:1.4.0"` - Platform string `json:"platform,omitempty" description:"Platform selected when the image is multi-platform." example:"linux/amd64"` - Pull string `json:"pull" description:"Image pull policy: missing, always, or never." default:"missing"` - Registry string `json:"registry,omitempty" description:"Optional registry label retained in canonical configuration. Current authentication uses every top-level registries entry; this field does not select a login."` + Pull string `json:"pull" description:"When to fetch the image from the registry: missing fetches only what the host does not already hold, always fetches every release, never fetches at all and fails on a missing image." default:"missing"` } type Route struct { @@ -250,13 +253,13 @@ type Schedule struct { } type Service struct { - Driver string `json:"driver,omitempty" description:"Built-in service driver. Defaults to the service map key." example:"postgres"` - Version any `json:"version" description:"Driver version or image tag to run." example:"17"` - Volumes []string `json:"volumes,omitempty" description:"Additional driver-defined persistent volume names."` - Persistence *Persistence `json:"persistence,omitempty" description:"Data-lifetime declaration for this supporting service."` - Resources *Resources `json:"resources,omitempty" description:"Memory and CPU limits for this supporting service."` - Settings map[string]any `json:"settings,omitempty" description:"Driver-specific settings validated by the selected service driver."` - Protection *ProtectionPolicy `json:"protection,omitempty" description:"Recovery intent for this service. Onebox selects the qualified native implementation; declaring intent alone does not establish protection."` + Driver string `json:"driver,omitempty" description:"Built-in service driver. Defaults to the service map key." example:"postgres"` + Version any `json:"version" description:"Driver version or image tag to run." example:"17"` + Volumes []string `json:"volumes,omitempty" description:"Additional driver-defined persistent volume names."` + Persistence *Persistence `json:"persistence,omitempty" description:"Data-lifetime declaration for this supporting service."` + Resources *Resources `json:"resources,omitempty" description:"Memory and CPU limits for this supporting service."` + Settings map[string]any `json:"settings,omitempty" description:"Driver-specific settings validated by the selected service driver."` + Backup *BackupPolicy `json:"backup,omitempty" description:"Recovery intent for this service. Onebox selects the qualified native implementation; declaring intent alone does not establish backup."` } // BackupTarget is a closed S3-compatible destination declaration. It accepts @@ -265,9 +268,9 @@ type BackupTarget struct { Kind string `json:"kind" description:"Destination kind. Only s3-compatible is supported." example:"s3-compatible"` Endpoint string `json:"endpoint" description:"Destination API endpoint. HTTPS is required unless tls is explicitly insecure." example:"https://objects.example.com"` Bucket string `json:"bucket,omitempty" description:"Existing destination bucket used by this target." example:"onebox-backups"` - Prefix string `json:"prefix,omitempty" description:"Non-secret object prefix reserved for Onebox protection data." example:"production/shop"` + Prefix string `json:"prefix,omitempty" description:"Non-secret object prefix reserved for Onebox backup data." example:"production/shop"` Region string `json:"region,omitempty" description:"S3-compatible region when the endpoint requires one." example:"us-east-1"` - TLS string `json:"tls" description:"TLS verification policy: required or insecure." default:"required"` + TLS string `json:"tls" description:"Transport policy: verify, or skip-verify to accept a plaintext http endpoint." default:"verify"` FailureDomain FailureDomain `json:"failure_domain" description:"Operator-declared identity used to prove the destination does not share the protected host."` Credentials CredentialReference `json:"credentials" description:"Trusted encrypted-file entries containing destination credentials; values never appear in the project."` Encryption TargetEncryption `json:"encryption" description:"Required encryption mode for each recovery kind this target may store."` @@ -290,37 +293,36 @@ type CredentialReference struct { } type TargetEncryption struct { - Snapshot string `json:"snapshot,omitempty" description:"Encryption mode required for snapshot recovery: client-side, archive-password, or server-side-sse."` - PITR string `json:"pitr,omitempty" description:"Encryption mode required for point-in-time recovery: client-side, archive-password, or server-side-sse."` - Cold string `json:"cold,omitempty" description:"Encryption mode required for cold recovery: client-side, archive-password, or server-side-sse."` + Snapshot string `json:"snapshot,omitempty" description:"Encryption mode required for snapshot recovery: client-side or server-side."` + PITR string `json:"pitr,omitempty" description:"Encryption mode required for point-in-time recovery: client-side or server-side."` + Cold string `json:"cold,omitempty" description:"Encryption mode required for cold recovery: client-side or server-side."` } -type ProtectionPolicy struct { - Target string `json:"target" description:"Name of a project-level backup target." example:"offsite"` - RecoveryKind string `json:"recovery_kind" description:"Required recovery envelope: snapshot, pitr, or cold." example:"pitr"` - MaximumDataLoss string `json:"maximum_data_loss" description:"Maximum tolerable interval between the latest recoverable point and failure." example:"15m"` - AllowBackupInterruption bool `json:"allow_backup_interruption" description:"Whether recurring backup operations may use the driver-declared stopped-service window." default:"false"` - Schedule Schedule `json:"schedule" description:"Exact recurring base-backup schedule."` - Retention ProtectionRetention `json:"retention" description:"Portable minimum recovery history that the selected native driver must be able to preserve."` - RestoreDrill RestoreDrillPolicy `json:"restore_drill" description:"Exact isolated restore-test schedule, proof age, and optional staging filesystem."` +type BackupPolicy struct { + Target string `json:"target" description:"Name of a project-level backup target." example:"offsite"` + RecoveryKind string `json:"recovery_kind" description:"Required recovery envelope: snapshot, pitr, or cold." example:"pitr"` + MaxDataLoss string `json:"max_data_loss" description:"Maximum tolerable interval between the latest recoverable point and failure." example:"15m"` + AllowDowntime bool `json:"allow_downtime" description:"Whether recurring backup operations may use the driver-declared stopped-service window." default:"false"` + Schedule Schedule `json:"schedule" description:"Exact recurring base-backup schedule."` + Retention BackupRetention `json:"retention" description:"Portable minimum recovery history that the selected native driver must be able to preserve."` + Drill BackupDrill `json:"drill" description:"Exact isolated restore-test schedule, proof age, and optional staging filesystem."` } -type ProtectionRetention struct { - MinimumGenerations int `json:"minimum_generations" description:"Minimum number of independently recoverable base generations to retain." default:"7" example:"7"` - RecoveryWindow string `json:"recovery_window" description:"Minimum continuous recovery history the native retention mapping must preserve." default:"7d" example:"7d"` +type BackupRetention struct { + Keep int `json:"keep" description:"Minimum number of independently recoverable base generations to retain." default:"7" example:"7"` + Window string `json:"window" description:"Minimum continuous recovery history the native retention mapping must preserve." default:"7d" example:"7d"` } -type RestoreDrillPolicy struct { - Schedule Schedule `json:"schedule" description:"Exact recurring isolated restore-test schedule."` - ProofMaximumAge string `json:"proof_maximum_age" description:"Maximum age of the latest passing restore proof." default:"7d" example:"7d"` - StagingFilesystem string `json:"staging_filesystem,omitempty" description:"Absolute filesystem path used for isolated restore materialization instead of the host default." example:"/srv/onebox-restore"` +type BackupDrill struct { + Schedule Schedule `json:"schedule" description:"Exact recurring isolated restore-test schedule."` + MaxAge string `json:"max_age" description:"Maximum age of the latest passing restore proof." default:"7d" example:"7d"` } type ExternalService struct { - Driver string `json:"driver" description:"Built-in connection shape used to validate and project this dependency." example:"postgres"` - Connection ExternalConnection `json:"connection" description:"Trusted connection source and driver-shaped entry mapping."` - ProtectionOwner string `json:"protection_owner" description:"Operator or provider responsible for backup, restore, upgrades, credentials, and durability." example:"platform-team/rds"` - Probe *ExternalReadOnlyProbe `json:"probe,omitempty" description:"Optional bounded read-only health observation; it never creates or repairs provider resources."` + Driver string `json:"driver" description:"Built-in connection shape used to validate and project this dependency." example:"postgres"` + Connection ExternalConnection `json:"connection" description:"Trusted connection source and driver-shaped entry mapping."` + BackupOwner string `json:"backup_owner" description:"Operator or provider responsible for backup, restore, upgrades, credentials, and durability." example:"platform-team/rds"` + Probe *ExternalReadOnlyProbe `json:"probe,omitempty" description:"Optional bounded read-only health observation; it never creates or repairs provider resources."` } type ExternalConnection struct { @@ -334,9 +336,9 @@ type ExternalConnectionSource struct { } type ExternalReadOnlyProbe struct { - Kind string `json:"kind" description:"Read-only observation kind: driver-health." default:"driver-health"` - Timeout string `json:"timeout" description:"Maximum duration of one read-only probe." default:"5s" example:"5s"` - MaximumAge string `json:"maximum_age" description:"Maximum age of a probe observation bound into a plan." default:"5m" example:"5m"` + Kind string `json:"kind" description:"Read-only observation kind: driver-health." default:"driver-health"` + Timeout string `json:"timeout" description:"Maximum duration of one read-only probe." default:"5s" example:"5s"` + MaxAge string `json:"max_age" description:"Maximum age of a probe observation bound into a plan." default:"5m" example:"5m"` } type Deployment struct { @@ -361,18 +363,86 @@ type Command struct { Local bool `json:"local" description:"Run on the operator machine instead of the server." default:"false"` } -type Verification struct { - Workload string `json:"workload,omitempty" description:"Workload in which an internal HTTP or exec verification runs."` - HTTP string `json:"http,omitempty" description:"HTTP path verified inside the named workload." example:"/healthz"` - Exec string `json:"exec,omitempty" description:"Shell command verified inside the named workload."` - Port int `json:"port,omitempty" description:"Container port used by an internal HTTP verification." example:"3000"` - URL string `json:"url,omitempty" description:"External HTTP or HTTPS URL verified from the operator side." example:"https://shop.example.com/healthz"` - StatusCodes []int `json:"status_codes,omitempty" description:"Allowed HTTP response status codes. A successful 2xx response is expected when omitted."` - RequiredHeaders map[string]string `json:"required_headers,omitempty" description:"Exact HTTP response headers required for success."` - Contains string `json:"contains,omitempty" description:"Text that the HTTP response body must contain."` - JSONAssertions []JSONAssertion `json:"json_assertions,omitempty" description:"Scalar JSON response values that must match exactly."` - MigrationRevisions *MigrationRevs `json:"migration_revisions,omitempty" description:"Expected migration provider and applied revisions, checked against captured job evidence."` - Advisory bool `json:"advisory" description:"Report a failed check without blocking release activation." default:"false"` +// Checks are the post-release assertions, grouped by kind. +// +// Grouped rather than one list of four shapes. The flat form was an untagged +// union — `status_codes`, `required_headers`, `contains`, and `json_assertions` +// were legal only alongside `url`, and nothing said so until validation. A group +// per kind lets the schema type each one, so an editor can complete it and a +// wrong field is a wrong field rather than a runtime refusal. +type Checks struct { + HTTP []HTTPCheck `json:"http,omitempty" description:"HTTP paths probed inside a named workload."` + URL []URLCheck `json:"url,omitempty" description:"External URLs probed from the operator side."` + Exec []ExecCheck `json:"exec,omitempty" description:"Commands run inside a named workload."` + Migrations []MigrationCheck `json:"migrations,omitempty" description:"Migration revisions checked against captured job evidence."` +} + +type HTTPCheck struct { + Workload string `json:"workload" description:"Workload the path is probed inside." example:"web"` + Path string `json:"path" description:"HTTP path verified inside the workload." example:"/healthz"` + Port int `json:"port,omitempty" description:"Container port to probe." example:"3000"` + Advisory bool `json:"advisory,omitempty" description:"Report a failure without blocking release activation." default:"false"` +} + +type URLCheck struct { + URL string `json:"url" description:"External HTTP or HTTPS URL verified from the operator side." example:"https://shop.example.com/healthz"` + StatusCodes []int `json:"status_codes,omitempty" description:"Allowed response status codes. A successful 2xx response is expected when omitted."` + RequiredHeaders map[string]string `json:"required_headers,omitempty" description:"Exact response headers required for success."` + Contains string `json:"contains,omitempty" description:"Text the response body must contain."` + JSONAssertions []JSONAssertion `json:"json_assertions,omitempty" description:"Scalar JSON response values that must match exactly."` + Advisory bool `json:"advisory,omitempty" description:"Report a failure without blocking release activation." default:"false"` +} + +type ExecCheck struct { + Workload string `json:"workload" description:"Workload the command runs inside." example:"web"` + Run string `json:"run" description:"Shell command verified inside the workload." example:"test -f /srv/ready"` + Advisory bool `json:"advisory,omitempty" description:"Report a failure without blocking release activation." default:"false"` +} + +type MigrationCheck struct { + Job string `json:"job" description:"Job workload whose captured evidence is checked." example:"migrate"` + Provider string `json:"provider" description:"Migration tool that produced the revisions." example:"alembic"` + AppliedRevisions []string `json:"applied_revisions" description:"Revisions the job must report as applied."` + Advisory bool `json:"advisory,omitempty" description:"Report a failure without blocking release activation." default:"false"` +} + +// RunnableCheck is the flat form every check is executed as. Authors never +// write it; Checks.All produces it, so the runner keeps one shape to run. +type RunnableCheck struct { + Workload string + HTTP string + Exec string + Port int + URL string + StatusCodes []int + RequiredHeaders map[string]string + Contains string + JSONAssertions []JSONAssertion + MigrationRevisions *MigrationRevs + Advisory bool +} + +// All flattens the grouped checks into the order they are declared in: every +// HTTP check, then URL, then exec, then migrations. +func (c Checks) All() []RunnableCheck { + out := make([]RunnableCheck, 0, len(c.HTTP)+len(c.URL)+len(c.Exec)+len(c.Migrations)) + for _, check := range c.HTTP { + out = append(out, RunnableCheck{Workload: check.Workload, HTTP: check.Path, Port: check.Port, Advisory: check.Advisory}) + } + for _, check := range c.URL { + out = append(out, RunnableCheck{ + URL: check.URL, StatusCodes: check.StatusCodes, RequiredHeaders: check.RequiredHeaders, + Contains: check.Contains, JSONAssertions: check.JSONAssertions, Advisory: check.Advisory, + }) + } + for _, check := range c.Exec { + out = append(out, RunnableCheck{Workload: check.Workload, Exec: check.Run, Advisory: check.Advisory}) + } + for _, check := range c.Migrations { + revisions := MigrationRevs{Job: check.Job, Provider: check.Provider, AppliedRevisions: check.AppliedRevisions} + out = append(out, RunnableCheck{MigrationRevisions: &revisions, Advisory: check.Advisory}) + } + return out } type JSONAssertion struct { @@ -447,22 +517,3 @@ func (e EnvFile) StagedPath() string { escaped := strings.ReplaceAll(strings.ReplaceAll(e.File, "-", "--"), "/", "-") return ".ob-decrypted-" + e.Provider + "-" + escaped } - -type Observability struct { - Logs *LogSettings `json:"logs,omitempty" description:"Declared log-retention intent. Continuous management is not currently provided."` - Metrics *MetricSettings `json:"metrics,omitempty" description:"Declared metric-collection intent. Continuous management is not currently provided."` - Alerts *AlertSettings `json:"alerts,omitempty" description:"Declared alerting intent. Continuous management is not currently provided."` -} - -type LogSettings struct { - Enabled bool `json:"enabled" description:"Declare that log collection is desired." default:"false"` - Retention string `json:"retention,omitempty" description:"Desired log-retention period." example:"30d"` -} - -type MetricSettings struct { - Enabled bool `json:"enabled" description:"Declare that metric collection is desired." default:"false"` -} - -type AlertSettings struct { - UnhealthyAfter string `json:"unhealthy_after,omitempty" description:"Desired duration of unhealthy state before alerting." example:"5m"` -} diff --git a/internal/app/validate.go b/internal/app/validate.go index a0fd7b4a..2d794407 100644 --- a/internal/app/validate.go +++ b/internal/app/validate.go @@ -137,24 +137,8 @@ func validateTopLevel(p *Spec) error { } } } - for i, v := range p.Verifications { - if err := validateVerification(v, indexed("verifications", i)); err != nil { - return err - } - } - if p.Observability != nil { - // Each sub-block is optional and independent. Checking one behind the - // other's nil guard both skipped the check and dereferenced a nil. - if p.Observability.Logs != nil { - if err := gDur.checkOptional("observability.logs.retention", p.Observability.Logs.Retention); err != nil { - return err - } - } - if p.Observability.Alerts != nil { - if err := gDur.checkOptional("observability.alerts.unhealthy_after", p.Observability.Alerts.UnhealthyAfter); err != nil { - return err - } - } + if err := validateChecks(p.Checks); err != nil { + return err } return nil } @@ -192,13 +176,13 @@ func validateEnvironment(e Environment, path string) error { if err := validateEnvFiles(e.EnvFiles, path+".env_files"); err != nil { return err } - if err := gDur.checkOptional(path+".policy.migration_backup_maximum_age", e.Policy.MigrationBackupMaximumAge); err != nil { + if err := gDur.checkOptional(path+".policy.migrations.backup_max_age", e.Policy.Migrations.BackupMaxAge); err != nil { return err } - if err := gPlanSchema.checkOptional(path+".policy.minimum_plan_schema", e.Policy.MinimumPlanSchema); err != nil { + if err := gPlanSchema.checkOptional(path+".policy.min_plan_schema", e.Policy.MinPlanSchema); err != nil { return err } - return gCalVer.checkOptional(path+".policy.minimum_onebox_version", e.Policy.MinimumOneboxVersion) + return gCalVer.checkOptional(path+".policy.min_onebox_version", e.Policy.MinOneboxVersion) } func validateWorkload(w Workload, path string) error { @@ -345,7 +329,7 @@ func validateWorkload(w Workload, path string) error { } } for i, port := range w.PublishedPorts { - pp := indexed(path+".published_ports", i) + pp := indexed(path+".ports", i) if err := checkPort(pp+".host", port.Host); err != nil { return err } @@ -357,8 +341,8 @@ func validateWorkload(w Workload, path string) error { } } if len(w.PublishedPorts) > 0 && w.Mode() == "rolling" { - return errf("project_invalid", path+".published_ports", "", - "rolling workloads cannot publish fixed host ports because the serving and replacement replicas must coexist; remove published_ports or set strategy to recreate") + return errf("project_invalid", path+".ports", "", + "rolling workloads cannot publish fixed host ports because the serving and replacement replicas must coexist; remove ports or set strategy to recreate") } if w.Persistence != nil { if err := checkEnum(path+".persistence.mode", w.Persistence.Mode, ePersistence); err != nil { @@ -463,8 +447,8 @@ func validateService(s Service, path string) error { return err } } - if s.Protection != nil { - if err := validateProtectionPolicy(*s.Protection, path+".protection"); err != nil { + if s.Backup != nil { + if err := validateBackupPolicy(*s.Backup, path+".backup"); err != nil { return err } } @@ -513,62 +497,56 @@ func itoa(i int) string { // validateVerification enforces that a check is exactly one kind. The four // kinds probe different things and carry different fields; a check that named // two would have to be run as one of them, and nothing would say which. -func validateVerification(v Verification, path string) error { - kinds := 0 - for _, present := range []bool{v.HTTP != "", v.Exec != "", v.URL != "", v.MigrationRevisions != nil} { - if present { - kinds++ - } - } - if kinds != 1 { - return errf("project_invalid", path, "", - "a verification declares exactly one of http, exec, url or migration_revisions (found %d)", kinds) - } - - // A container probe runs inside a workload, so it has to name one. - if v.HTTP != "" || v.Exec != "" { - if v.Workload == "" { - return errf("project_invalid", path+".workload", "", - "an http or exec check runs inside a workload and must name it") +// validateChecks validates each group against its own shape. +// +// The union check this replaced — "exactly one of http, exec, url or +// migration_revisions" — is gone because the shape can no longer express more +// than one. So is the rule that response assertions belong to a url check: they +// are now fields of URLCheck and nowhere else, which the schema enforces before +// this function runs. +func validateChecks(c Checks) error { + for i, check := range c.HTTP { + path := indexed("checks.http", i) + if err := gIdent.check(path+".workload", check.Workload); err != nil { + return err } - if err := gIdent.check(path+".workload", v.Workload); err != nil { + if err := gURLPath.check(path+".path", check.Path); err != nil { return err } - } else if v.Workload != "" { - return errf("project_invalid", path+".workload", "", - "a url or migration check does not run inside a workload") - } - - if err := gURLPath.checkOptional(path+".http", v.HTTP); err != nil { - return err + if check.Port != 0 { + if err := checkPort(path+".port", check.Port); err != nil { + return err + } + } } - if v.URL != "" { - if err := gHTTPURL.check(path+".url", v.URL); err != nil { + for i, check := range c.URL { + path := indexed("checks.url", i) + if err := gHTTPURL.check(path+".url", check.URL); err != nil { return err } + for _, code := range check.StatusCodes { + if code < 100 || code > 599 { + return errf("project_invalid", path+".status_codes", "", "%d is not an HTTP status code", code) + } + } } - if v.Port != 0 { - if err := checkPort(path+".port", v.Port); err != nil { + for i, check := range c.Exec { + path := indexed("checks.exec", i) + if err := gIdent.check(path+".workload", check.Workload); err != nil { return err } - } - for _, code := range v.StatusCodes { - if code < 100 || code > 599 { - return errf("project_invalid", path+".status_codes", "", - "%d is not an HTTP status code", code) + if strings.TrimSpace(check.Run) == "" { + return errf("project_invalid", path+".run", "", "an exec check must name a command to run") } } - // Response-shape assertions describe a response, which only a url check - // receives; on any other kind they would be silently ignored. - if v.URL == "" && (len(v.StatusCodes) > 0 || len(v.RequiredHeaders) > 0 || - len(v.JSONAssertions) > 0 || v.Contains != "") { - return errf("project_invalid", path, "", - "status_codes, required_headers, json_assertions and contains describe a response, so they belong to a url check") - } - if v.MigrationRevisions != nil { - if err := gIdent.check(path+".migration_revisions.job", v.MigrationRevisions.Job); err != nil { + for i, check := range c.Migrations { + path := indexed("checks.migrations", i) + if err := gIdent.check(path+".job", check.Job); err != nil { return err } + if strings.TrimSpace(check.Provider) == "" { + return errf("project_invalid", path+".provider", "", "a migration check must name the provider that produced the revisions") + } } return nil } diff --git a/internal/buildinfo/calver.go b/internal/buildinfo/calver.go index 4bff32e7..3579563a 100644 --- a/internal/buildinfo/calver.go +++ b/internal/buildinfo/calver.go @@ -7,7 +7,7 @@ import ( ) // ReleaseVersionPattern is the one grammar for a release identity. It is -// exported so the project loader validates `minimum_onebox_version` against the +// exported so the project loader validates `min_version` against the // same expression this package parses: a loader that accepted a version no tag // can carry would fail closed on a release that is perfectly valid. var ReleaseVersionPattern = regexp.MustCompile(`^v([1-9][0-9]{3})\.([1-9]|1[0-2])\.(0|[1-9][0-9]{0,18})$`) diff --git a/internal/engine/backup_base_selection_test.go b/internal/engine/backup_base_selection_test.go new file mode 100644 index 00000000..c24362ed --- /dev/null +++ b/internal/engine/backup_base_selection_test.go @@ -0,0 +1,208 @@ +package engine + +import ( + "context" + "io" + "strings" + "testing" + "time" + + "github.com/labstack/onebox/internal/app" + "github.com/labstack/onebox/internal/transport" +) + +const walgListing = `[ + {"backup_name":"base_A","finish_time":"2026-08-20T10:00:00Z"}, + {"backup_name":"base_C","finish_time":"2026-08-20T14:00:00Z"}, + {"backup_name":"base_B","finish_time":"2026-08-20T12:00:00Z"} +]` + +func baseSelectionEngine(listing string) (*Engine, *transport.Fake) { + fake := &transport.Fake{Dynamic: func(cmd string) (transport.Result, bool) { + if strings.Contains(cmd, "backup-list") { + return transport.Result{Stdout: listing}, true + } + return transport.Result{}, false + }} + spec := &app.Spec{ + Name: "shop", + BasePath: "/var/lib/ob", + Services: map[string]app.Service{"database": {Driver: "postgres", Version: "18"}}, + } + e := New(&app.Resolved{Spec: spec, Env: "production"}, nil, fake, + Options{Out: io.Discard, Sleep: func(time.Duration) {}}) + return e, fake +} + +// Replay only moves forward, so the base backup has to be one that finished at +// or before the requested point. Fetching wal-g's LATEST unconditionally made +// every point older than the newest base backup unrecoverable: PostgreSQL ran +// out of WAL before the target and died with "recovery ended before configured +// recovery target was reached". With a daily backup and a seven-day window that +// is six days of a window `ob backup status` reported as recoverable. +func TestTheBaseBackupIsChosenByTheRequestedPoint(t *testing.T) { + for _, tc := range []struct { + name string + target string + want string + }{ + {"between two backups picks the earlier", "2026-08-20T13:00:00Z", "base_B"}, + {"after every backup picks the newest", "2026-08-20T20:00:00Z", "base_C"}, + {"exactly at a finish time picks it", "2026-08-20T12:00:00Z", "base_B"}, + {"just before a finish time picks the one before", "2026-08-20T11:59:59Z", "base_A"}, + } { + t.Run(tc.name, func(t *testing.T) { + e, _ := baseSelectionEngine(walgListing) + got, err := e.baseBackupFor(context.Background(), "c1", "database", tc.target) + if err != nil { + t.Fatalf("selecting a base backup: %v", err) + } + if got != tc.want { + t.Fatalf("chose %q, want %q", got, tc.want) + } + }) + } +} + +// A point older than everything in the repository is refused, and the refusal +// says what the repository can actually reach — the operator asking has a +// window in mind and needs to know where it really starts. +func TestAPointOlderThanTheRepositoryIsRefusedWithItsOldestBackup(t *testing.T) { + e, _ := baseSelectionEngine(walgListing) + _, err := e.baseBackupFor(context.Background(), "c1", "database", "2020-01-01T00:00:00Z") + if err == nil { + t.Fatal("a point older than every base backup was accepted") + } + if !strings.Contains(err.Error(), "2026-08-20T10:00:00Z") { + t.Fatalf("refusal does not name the oldest recoverable base: %v", err) + } +} + +// An entry whose completion time cannot be read is refused, not defaulted: a +// zero time sorts before every real one, so defaulting would make the +// unreadable entry the answer to "what can reach this point". +func TestAnUnreadableCompletionTimeIsRefused(t *testing.T) { + if _, err := parseWalgBackupList(`[{"backup_name":"base_A","finish_time":"whenever"}]`); err == nil { + t.Fatal("an unreadable completion time was accepted") + } + entries, err := parseWalgBackupList("null") + if err != nil || len(entries) != 0 { + t.Fatalf("empty listing = %v, %v", entries, err) + } +} + +// Without a requested point the recovery takes the newest backup, which is what +// "the newest recoverable point" means and needs no listing at all. +func TestNoRequestedPointStillUsesLatest(t *testing.T) { + e, fake := baseSelectionEngine(walgListing) + if _, err := e.fetchRecoveryBase(context.Background(), "c1", "database", ""); err != nil { + t.Fatalf("fetching the newest base: %v", err) + } + for _, cmd := range fake.Commands { + if strings.Contains(cmd, "backup-list") { + t.Fatalf("listed the repository for a recovery that asked for no point: %s", cmd) + } + } + if len(fake.Commands) == 0 || !strings.Contains(fake.Commands[len(fake.Commands)-1], "backup-fetch") { + t.Fatalf("no backup-fetch was issued: %v", fake.Commands) + } + if !strings.Contains(fake.Commands[len(fake.Commands)-1], "LATEST") { + t.Fatalf("did not fetch LATEST: %v", fake.Commands) + } +} + +// wal-g reports a broken WAL chain and exits 0. It prints the integrity table, +// says "integrity check status: WARNING", lists the ranges it could not find — +// and returns success, so a check that reads only the exit code reports green +// over a repository with holes in it. Two segments deleted from the middle of a +// live repository produced exactly that, and the scheduled unit had the same +// blind spot. +func TestABrokenWALChainIsNotAPassingVerification(t *testing.T) { + report := `[wal-verify] integrity check status: FAILURE +[wal-verify] integrity check details: ++-----+--------------------------+--------------------------+----------------+-------------------+ +| TLI | START | END | SEGMENTS COUNT | STATUS | ++-----+--------------------------+--------------------------+----------------+-------------------+ +| 3 | 000000030000000000000027 | 00000003000000000000002A | 4 | FOUND | +| 3 | 00000003000000000000002B | 00000003000000000000002B | 1 | MISSING_LOST | +| 3 | 00000003000000000000002C | 00000003000000000000002E | 3 | FOUND | ++-----+--------------------------+--------------------------+----------------+-------------------+ +[wal-verify] timeline check status: OK` + err := walVerifyResult(report) + if err == nil { + t.Fatal("a gap in the middle of the chain passed verification") + } + if !strings.Contains(err.Error(), "00000003000000000000002B") { + t.Fatalf("the failure does not name the missing range: %v", err) + } +} + +// A segment still uploading is not yet a hole, wherever it sits in the table. +// On a database taking backups in a loop, three segments were in flight while +// later ones had already arrived — so failing on in-flight ranges would fail +// every check that lands mid-upload. +func TestASegmentStillUploadingIsNotAGap(t *testing.T) { + report := `[wal-verify] integrity check status: WARNING +| TLI | START | END | SEGMENTS COUNT | STATUS | +| 3 | 000000030000000000000027 | 00000003000000000000002D | 7 | FOUND | +| 3 | 00000003000000000000002E | 000000030000000000000030 | 3 | MISSING_UPLOADING | +| 3 | 000000030000000000000031 | 000000030000000000000031 | 1 | FOUND | +[wal-verify] timeline check status: OK` + if err := walVerifyResult(report); err != nil { + t.Fatalf("segments in flight were treated as a gap: %v", err) + } +} + +// A lost segment is a hole wherever it sits. +func TestALostSegmentAtTheHeadIsAGap(t *testing.T) { + report := `[wal-verify] integrity check status: FAILURE +| TLI | START | END | SEGMENTS COUNT | STATUS | +| 3 | 000000030000000000000027 | 00000003000000000000002D | 7 | FOUND | +| 3 | 00000003000000000000002E | 00000003000000000000002E | 1 | MISSING_LOST | +[wal-verify] timeline check status: OK` + if err := walVerifyResult(report); err == nil { + t.Fatal("a lost segment at the head passed verification") + } +} + +// A healthy chain passes, and a report whose table this parser does not +// recognise still fails on the status line rather than reading as clean. +func TestAHealthyChainPassesAndAnUnreadableReportDoesNot(t *testing.T) { + healthy := `[wal-verify] integrity check status: OK +| TLI | START | END | SEGMENTS COUNT | STATUS | +| 3 | 000000030000000000000027 | 00000003000000000000002E | 8 | FOUND | +[wal-verify] timeline check status: OK` + if err := walVerifyResult(healthy); err != nil { + t.Fatalf("a healthy chain failed verification: %v", err) + } + if err := walVerifyResult("[wal-verify] timeline check status: FAILURE"); err == nil { + t.Fatal("a failing check with no table read as clean") + } +} + +// The scheduled check has to reach the same verdict as the interactive one. +// wal-g exits 0 over a broken chain, so a unit that invoked it directly was +// marked successful by systemd for as long as nobody looked — which is the +// whole span a scheduled check exists to cover. +func TestTheScheduledVerificationJudgesTheReportRatherThanTheExitCode(t *testing.T) { + script := backupVerifyScript("shop-database-1") + for _, required := range []string{ + "wal-verify", + "MISSING_(LOST|DELAYED)", + "exit 1", + } { + if !strings.Contains(script, required) { + t.Fatalf("the scheduled verification does not contain %q:\n%s", required, script) + } + } + // A segment still uploading must not fail the unit, for the same reason it + // does not fail the command: it resolves itself, and a check that flags it + // flaps on every busy database. + if strings.Contains(script, "MISSING_UPLOADING") { + t.Fatalf("the scheduled verification fails on segments still in flight:\n%s", script) + } + // wal-g's own failure must still fail the unit. + if !strings.Contains(script, `[ "$status" -eq 0 ] || exit "$status"`) { + t.Fatalf("the scheduled verification swallows wal-g's own exit code:\n%s", script) + } +} diff --git a/internal/engine/backup_credentials.go b/internal/engine/backup_credentials.go new file mode 100644 index 00000000..9841d47d --- /dev/null +++ b/internal/engine/backup_credentials.go @@ -0,0 +1,138 @@ +package engine + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "time" +) + +var backupCredentialEntry = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]{0,127}$`) + +// InstallBackupCredentialFile moves already-resolved credential material +// through a private upload into its target-side mode-0600 file. Secret bytes +// are never interpolated into a command, journal, result, or error. +func (e *Engine) InstallBackupCredentialFile(ctx context.Context, service, target string, requiredEntries []string, plaintext []byte) (string, error) { + if !backupIdentity.MatchString(service) || !backupIdentity.MatchString(target) { + return "", errors.New("backup credential service and target identities are invalid") + } + entries, err := backupCredentialEntries(plaintext) + if err != nil { + return "", err + } + // Normalised before it is written. The decrypted file may use `export NAME=` + // and quoted values — both ordinary in a shell-sourced dotenv, and both + // accepted by Compose's env_file parser. `docker run --env-file` accepts + // neither: it would create a variable literally named "export NAME" and keep + // the quotes as part of the value, so a recovery container would start with + // no credentials at all. Writing one form means every consumer reads the + // same thing. + plaintext = normalizeCredentialFile(plaintext) + requiredEntries = append([]string(nil), requiredEntries...) + sort.Strings(requiredEntries) + for _, entry := range requiredEntries { + if !backupCredentialEntry.MatchString(entry) { + return "", errors.New("backup credential contract contains an invalid slot") + } + if !entries[entry] { + return "", fmt.Errorf("backup credential file is missing required entry %s", entry) + } + } + + localStaging, err := os.MkdirTemp("", "ob-backup-credentials-") + if err != nil { + return "", errors.New("create private backup credential staging") + } + defer os.RemoveAll(localStaging) + const stagedName = "credentials.env" + if err := os.WriteFile(filepath.Join(localStaging, stagedName), plaintext, 0o600); err != nil { + return "", errors.New("write private backup credential staging") + } + + names := e.names() + destination := names.BackupCredentialFile(service, target) + tokenBytes := sha256.Sum256([]byte(e.backupFenceVals[service] + "\x00" + target)) + token := hex.EncodeToString(tokenBytes[:])[:16] + remoteStaging := names.AppDir() + "/backup/.credential-staging-" + service + "-" + token + if err := e.T.Upload(ctx, localStaging, remoteStaging); err != nil { + return "", errors.New("upload private backup credential staging") + } + // Cleanup cannot use BackupMutate: a failed/fenced install is exactly + // when that guard may refuse the command. The paths are deterministic, + // narrowly scoped staging artifacts, and best-effort removal must run even + // after cancellation so plaintext is not stranded on the host. + defer func() { + cleanupContext, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _, _ = e.T.Run(cleanupContext, "rm -rf "+q(remoteStaging)+"; rm -f "+q(destination+".tmp")) + }() + install := "mkdir -p " + q(names.BackupSecretDir()) + + " && chmod 700 " + q(names.BackupSecretDir()) + + " && cp " + q(remoteStaging+"/"+stagedName) + " " + q(destination+".tmp") + + " && chmod 600 " + q(destination+".tmp") + + " && mv -f " + q(destination+".tmp") + " " + q(destination) + + " && rm -rf " + q(remoteStaging) + result, err := e.BackupMutate(ctx, service, install) + if err != nil { + return "", errors.New("install target-side backup credential file") + } + if result.ExitCode != 0 { + return "", errors.New("install target-side backup credential file failed") + } + return destination, nil +} + +func backupCredentialEntries(plaintext []byte) (map[string]bool, error) { + entries := make(map[string]bool) + for index, line := range strings.Split(string(plaintext), "\n") { + trimmed := strings.TrimSpace(line) + if trimmed == "" || strings.HasPrefix(trimmed, "#") { + continue + } + trimmed = strings.TrimPrefix(trimmed, "export ") + entry, _, ok := strings.Cut(trimmed, "=") + entry = strings.TrimSpace(entry) + if !ok || !backupCredentialEntry.MatchString(entry) { + return nil, fmt.Errorf("backup credential file has an invalid entry at line %d", index+1) + } + if entries[entry] { + return nil, fmt.Errorf("backup credential file repeats entry %s", entry) + } + entries[entry] = true + } + if len(entries) == 0 { + return nil, errors.New("backup credential file has no entries") + } + return entries, nil +} + +// normalizeCredentialFile rewrites decrypted credential material into the one +// form every consumer parses: NAME=value, no export prefix, no surrounding +// quotes, comments and blank lines dropped. +func normalizeCredentialFile(plaintext []byte) []byte { + var out strings.Builder + for _, line := range strings.Split(string(plaintext), "\n") { + trimmed := strings.TrimSpace(line) + if trimmed == "" || strings.HasPrefix(trimmed, "#") { + continue + } + trimmed = strings.TrimPrefix(trimmed, "export ") + name, value, ok := strings.Cut(trimmed, "=") + if !ok { + continue + } + value = strings.TrimSpace(value) + if len(value) >= 2 && (value[0] == '"' || value[0] == '\'') && value[len(value)-1] == value[0] { + value = value[1 : len(value)-1] + } + out.WriteString(strings.TrimSpace(name) + "=" + value + "\n") + } + return []byte(out.String()) +} diff --git a/internal/engine/protection_credentials_test.go b/internal/engine/backup_credentials_test.go similarity index 69% rename from internal/engine/protection_credentials_test.go rename to internal/engine/backup_credentials_test.go index 0b61f70b..0154ae11 100644 --- a/internal/engine/protection_credentials_test.go +++ b/internal/engine/backup_credentials_test.go @@ -27,26 +27,26 @@ func (transport *credentialInspectTransport) Upload(_ context.Context, localDir, return err } -func TestInstallProtectionCredentialFileUsesPrivateTargetFileWithoutCommandLeak(t *testing.T) { +func TestInstallBackupCredentialFileUsesPrivateTargetFileWithoutCommandLeak(t *testing.T) { fake := &transport.Fake{} inspector := &credentialInspectTransport{Fake: fake} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) engine.T = inspector engine.fenceVal = "deploy-1 1" - engine.protectionLockVals = map[string]string{"database": "service-lock"} - engine.protectionFenceVals = map[string]string{"database": "backup-1 1"} + engine.backupLockVals = map[string]string{"database": "service-lock"} + engine.backupFenceVals = map[string]string{"database": "backup-1 1"} credentialCanary := "credential-canary-value" databaseCanary := "database-row-canary" plaintext := []byte("BACKUP_ACCESS_KEY_ID=access\nBACKUP_SECRET_ACCESS_KEY=" + credentialCanary + "\nDATABASE_CONTENT=" + databaseCanary + "\n") - path, err := engine.InstallProtectionCredentialFile( + path, err := engine.InstallBackupCredentialFile( context.Background(), "database", "offsite", []string{"BACKUP_ACCESS_KEY_ID", "BACKUP_SECRET_ACCESS_KEY"}, plaintext, ) if err != nil { - t.Fatalf("install protection credentials: %v", err) + t.Fatalf("install backup credentials: %v", err) } - if path != "/var/lib/ob/example/protection/secrets/database-offsite.env" { + if path != "/var/lib/ob/example/backup/secrets/database-offsite.env" { t.Fatalf("credential path = %q", path) } if inspector.mode != 0o600 { @@ -62,11 +62,11 @@ func TestInstallProtectionCredentialFileUsesPrivateTargetFileWithoutCommandLeak( } } -func TestProtectionCredentialErrorsDoNotEchoValues(t *testing.T) { +func TestBackupCredentialErrorsDoNotEchoValues(t *testing.T) { fake := &transport.Fake{} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) secret := "credential-canary-value" - _, err := engine.InstallProtectionCredentialFile( + _, err := engine.InstallBackupCredentialFile( context.Background(), "database", "offsite", []string{"REQUIRED_ENTRY"}, []byte("PRESENT_ENTRY="+secret+"\n"), ) @@ -78,31 +78,31 @@ func TestProtectionCredentialErrorsDoNotEchoValues(t *testing.T) { } } -func TestProtectionCredentialGrammarAcceptsLowercaseEnvironmentNames(t *testing.T) { +func TestBackupCredentialGrammarAcceptsLowercaseEnvironmentNames(t *testing.T) { fake := &transport.Fake{} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) engine.fenceVal = "deploy-1 1" - engine.protectionLockVals = map[string]string{"database": "service-lock"} - engine.protectionFenceVals = map[string]string{"database": "backup-1 1"} - if _, err := engine.InstallProtectionCredentialFile( + engine.backupLockVals = map[string]string{"database": "service-lock"} + engine.backupFenceVals = map[string]string{"database": "backup-1 1"} + if _, err := engine.InstallBackupCredentialFile( context.Background(), "database", "offsite", []string{"aws_access_key"}, []byte("aws_access_key=value\n"), ); err != nil { t.Fatalf("lowercase schema-valid credential entry was rejected: %v", err) } } -func TestProtectionCredentialInstallFailureCleansRemotePlaintextStaging(t *testing.T) { +func TestBackupCredentialInstallFailureCleansRemotePlaintextStaging(t *testing.T) { fake := &transport.Fake{Dynamic: func(command string) (transport.Result, bool) { if strings.Contains(command, "cp ") && strings.Contains(command, ".credential-staging-") { return transport.Result{ExitCode: 1}, true } return transport.Result{}, false }} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) engine.fenceVal = "deploy-1 1" - engine.protectionLockVals = map[string]string{"database": "service-lock"} - engine.protectionFenceVals = map[string]string{"database": "backup-1 1"} - if _, err := engine.InstallProtectionCredentialFile( + engine.backupLockVals = map[string]string{"database": "service-lock"} + engine.backupFenceVals = map[string]string{"database": "backup-1 1"} + if _, err := engine.InstallBackupCredentialFile( context.Background(), "database", "offsite", []string{"REQUIRED_ENTRY"}, []byte("REQUIRED_ENTRY=value\n"), ); err == nil { t.Fatal("failed target install was accepted") diff --git a/internal/engine/backup_image_test.go b/internal/engine/backup_image_test.go new file mode 100644 index 00000000..7f6b583b --- /dev/null +++ b/internal/engine/backup_image_test.go @@ -0,0 +1,179 @@ +package engine + +import ( + "context" + "io" + "strings" + "testing" + "time" + + ctypes "github.com/compose-spec/compose-go/v2/types" + "github.com/labstack/onebox/internal/app" + "github.com/labstack/onebox/internal/transport" +) + +// A tag must always resolve through the registry, because a tag can move. A +// digest must not: it is immutable, so a second pull cannot return different +// bytes — it only spends registry quota and makes a re-enable fail on a host +// that is offline or rate-limited while already holding exactly what it needs. +func TestOnlyDigestPinnedReferencesSkipTheRegistry(t *testing.T) { + for _, tc := range []struct { + name string + reference string + skippable bool + }{ + {"tag", "postgres:18", false}, + {"tag that looks pinned", "postgres:sha256-abc", false}, + {"digest", "postgres@sha256:" + "a" + "b" + "c", true}, + } { + t.Run(tc.name, func(t *testing.T) { + // The guard is the reference shape; the registry call is what it + // gates. Asserting the shape keeps the rule readable without a + // docker daemon. + got := containsDigest(tc.reference) + if got != tc.skippable { + t.Fatalf("containsDigest(%q) = %v, want %v", tc.reference, got, tc.skippable) + } + }) + } +} + +// A service that is already bound to a repository keeps the exact bytes it was +// bound with. Re-running enable — after a policy edit, or after a disable +// somebody changed their mind about — must not re-resolve the tag: the bytes +// would move under a live data directory the moment upstream published a patch, +// and the command would need a registry it has no reason to need. Both showed +// up the same way in a live run: `ob backup enable` on an already-enabled +// service failed on a Docker Hub 429 while the host held the pinned image. +func TestReEnableKeepsTheRecordedPinAndDoesNotReachTheRegistry(t *testing.T) { + const pin = "postgres@sha256:06cad38a5d9f5d24b4d83d86def30795d5e4b757fedbf5281172b576dedcd941" + fake := &transport.Fake{Dynamic: func(cmd string) (transport.Result, bool) { + if strings.Contains(cmd, "docker image inspect") && strings.Contains(cmd, pin) { + return transport.Result{Stdout: "present\n"}, true + } + return transport.Result{}, false + }} + e := protectedImageTestEngine(fake) + + // Recorded as the authored reference, which is what an enable after a + // disable still declares even though the runtime selection has reverted. + got, err := e.ResolveProtectedImage(context.Background(), "database", pin, "postgres:18") + if err != nil { + t.Fatalf("resolving an already-bound image: %v", err) + } + if got != pin { + t.Fatalf("resolved %q, want the recorded pin %q", got, pin) + } + for _, cmd := range fake.Commands { + if strings.Contains(cmd, "docker pull") { + t.Fatalf("pulled while holding the recorded pin: %s", cmd) + } + } +} + +// The pin is only reusable while the project still declares the reference that +// produced it. Changing the declared version is how an operator asks for +// different bytes, and that has to reach the registry. +func TestADeclaredVersionChangeStillResolvesThroughTheRegistry(t *testing.T) { + const stalePin = "postgres@sha256:06cad38a5d9f5d24b4d83d86def30795d5e4b757fedbf5281172b576dedcd941" + fake := &transport.Fake{Dynamic: func(cmd string) (transport.Result, bool) { + switch { + case strings.Contains(cmd, "docker pull"): + return transport.Result{}, true + case strings.Contains(cmd, "RepoDigests"): + return transport.Result{Stdout: "postgres@sha256:" + strings.Repeat("b", 64) + "\n"}, true + } + return transport.Result{Stdout: "absent\n"}, true + }} + e := protectedImageTestEngine(fake) + + // Recorded against postgres:17; the project now declares 18. + got, err := e.ResolveProtectedImage(context.Background(), "database", stalePin, "postgres:17") + if err != nil { + t.Fatalf("resolving after a declared version change: %v", err) + } + if got == stalePin { + t.Fatal("kept the pin recorded for a reference the project no longer declares") + } + pulled := false + for _, cmd := range fake.Commands { + if strings.Contains(cmd, "docker pull") { + pulled = true + } + } + if !pulled { + t.Fatal("a changed declared reference resolved without reaching the registry") + } +} + +func protectedImageTestEngine(fake *transport.Fake) *Engine { + spec := &app.Spec{ + Name: "shop", + BasePath: "/var/lib/ob", + Services: map[string]app.Service{"database": {Driver: "postgres", Version: "18"}}, + } + resolved := &app.Resolved{Spec: spec, Env: "production"} + return New(resolved, nil, fake, Options{Out: io.Discard, Sleep: func(time.Duration) {}}) +} + +// `image.pull` was defaulted, validated and documented, and read by nothing: +// every release pulled every workload from the registry, including an image +// already pinned by digest and already on the host. That is a request that +// cannot change the outcome, and on a rate-limited registry it failed a deploy +// with nothing to fetch. +func TestPullPolicyDecidesWhetherTheRegistryIsAskedAtAll(t *testing.T) { + const pinned = "nginx@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10" + for _, tc := range []struct { + name string + policy string + held bool + pull bool + }{ + {"never, held", "never", true, false}, + {"never, absent", "never", false, false}, + {"missing, held", "missing", true, false}, + {"missing, absent", "missing", false, true}, + {"always, held", "always", true, true}, + } { + t.Run(tc.name, func(t *testing.T) { + presence := "absent\n" + if tc.held { + presence = "present\n" + } + fake := &transport.Fake{Dynamic: func(cmd string) (transport.Result, bool) { + if strings.Contains(cmd, "docker image inspect") { + return transport.Result{Stdout: presence}, true + } + return transport.Result{}, false + }} + e := pullPolicyTestEngine(fake, tc.policy, pinned) + + if err := e.pullBeforeRelease(context.Background(), "web", "docker compose -p shop"); err != nil { + t.Fatalf("pull decision: %v", err) + } + pulled := false + for _, cmd := range fake.Commands { + if strings.Contains(cmd, "pull --quiet") { + pulled = true + } + } + if pulled != tc.pull { + t.Fatalf("pulled = %v, want %v (commands: %v)", pulled, tc.pull, fake.Commands) + } + }) + } +} + +func pullPolicyTestEngine(fake *transport.Fake, policy, image string) *Engine { + spec := &app.Spec{ + Name: "shop", + BasePath: "/var/lib/ob", + Workloads: map[string]app.Workload{ + "web": {Role: "application", Image: &app.Image{Reference: image, Pull: policy}}, + }, + } + e := New(&app.Resolved{Spec: spec, Env: "production"}, nil, fake, + Options{Out: io.Discard, Sleep: func(time.Duration) {}}) + e.Compose = &ctypes.Project{Services: ctypes.Services{"web": ctypes.ServiceConfig{Name: "web", Image: image}}} + return e +} diff --git a/internal/engine/backup_lock.go b/internal/engine/backup_lock.go new file mode 100644 index 00000000..30a60d31 --- /dev/null +++ b/internal/engine/backup_lock.go @@ -0,0 +1,261 @@ +package engine + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "regexp" + "strconv" + "strings" + "time" + + "github.com/labstack/onebox/internal/journal" + "github.com/labstack/onebox/internal/transport" +) + +var ( + ErrBackupConflict = errors.New("backup_conflict") + ErrBackupFenced = errors.New("backup operation fenced by a newer owner") + backupIdentity = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._:-]{0,127}$`) +) + +const backupLockPollInterval = 100 * time.Millisecond + +type backupLockMeta struct { + Owner string `json:"owner"` + OperationID string `json:"operation_id"` + Service string `json:"service"` + Epoch int `json:"epoch"` + TTLSeconds int `json:"ttl_s"` + AcquiredAt string `json:"acquired_at"` +} + +// BackupConflictError is safe to serialize as a lifecycle failure. The +// code is stable and the holder identity is operational metadata, never a +// command, credential, or database value. +type BackupConflictError struct { + Service string + OperationID string + AgeSeconds int +} + +func (err *BackupConflictError) Error() string { + return fmt.Sprintf("backup_conflict: service %s is held by operation %s (age %ds)", err.Service, err.OperationID, err.AgeSeconds) +} + +func (err *BackupConflictError) Unwrap() error { return ErrBackupConflict } +func (err *BackupConflictError) Code() string { return "backup_conflict" } +func (err *BackupConflictError) Retryable() bool { + return true +} + +func (e *Engine) backupLockDir() string { return e.base() + "/backup/locks" } +func (e *Engine) backupLockPath(service string) string { + return e.backupLockDir() + "/" + service + ".lock" +} +func (e *Engine) backupEpochPath(service string) string { + return e.backupLockDir() + "/" + service + ".epoch" +} +func (e *Engine) backupFencePath(service string) string { + return e.backupLockDir() + "/" + service + ".fence" +} + +// AcquireBackupLock acquires the per-service lock beneath the application +// lock. wait is a bounded contention budget; expiry returns backup_conflict. +// An expired lock or the same operation identity is reclaimed with a new epoch, +// fencing the former runner. +func (e *Engine) AcquireBackupLock(ctx context.Context, service, operationID string, wait time.Duration) (int, error) { + if e.lockVal == "" { + return 0, errors.New("backup lock requires the application lock") + } + if !backupIdentity.MatchString(service) || !backupIdentity.MatchString(operationID) { + return 0, errors.New("backup lock service and operation identity are invalid") + } + if wait < 0 { + return 0, errors.New("backup lock wait must not be negative") + } + if err := ctx.Err(); err != nil { + return 0, err + } + if res, err := e.T.Run(ctx, "mkdir -p "+q(e.backupLockDir())); err != nil { + return 0, err + } else if res.ExitCode != 0 { + return 0, fmt.Errorf("create backup lock directory: %s", strings.TrimSpace(res.Stderr)) + } + + maxAttempts := 1 + if wait > 0 { + maxAttempts += int((wait + backupLockPollInterval - 1) / backupLockPollInterval) + } + var conflict *BackupConflictError + staleReclaims := 0 + for attempt := 0; attempt < maxAttempts; attempt++ { + if err := ctx.Err(); err != nil { + return 0, err + } + epoch, err := e.nextBackupEpoch(ctx, service) + if err != nil { + return 0, err + } + meta := backupLockMeta{ + Owner: journal.DefaultOperator(), OperationID: operationID, Service: service, + Epoch: epoch, TTLSeconds: int(e.lockTTL().Seconds()), AcquiredAt: e.Opts.Now().UTC().Format(time.RFC3339), + } + encoded, _ := json.Marshal(meta) + lockValue := string(encoded) + create := "set -C; echo " + q(lockValue) + " > " + q(e.backupLockPath(service)) + " 2>/dev/null" + res, err := e.T.Run(ctx, create) + if err != nil { + return 0, err + } + if res.ExitCode == 0 { + if err := e.writeBackupFence(ctx, service, operationID, epoch, lockValue); err != nil { + return 0, err + } + return epoch, nil + } + + observedResult, err := e.T.Run(ctx, "cat "+q(e.backupLockPath(service))+" 2>/dev/null || true") + if err != nil { + return 0, err + } + observed := strings.TrimSpace(observedResult.Stdout) + if observed == "" { + continue + } + var holder backupLockMeta + _ = json.Unmarshal([]byte(observed), &holder) + ageResult, err := e.T.Run(ctx, lockAgeCmd(e.backupLockPath(service))) + if err != nil { + return 0, err + } + age, _ := strconv.Atoi(strings.TrimSpace(ageResult.Stdout)) + if age > int(e.lockTTL().Seconds()) || holder.OperationID == operationID { + if staleReclaims >= 4 { + return 0, errors.New("could not reclaim stale backup lock") + } + staleReclaims++ + removeObserved := `if [ "$(cat ` + q(e.backupLockPath(service)) + ` 2>/dev/null)" = ` + q(observed) + ` ]; then rm -f ` + q(e.backupLockPath(service)) + `; else exit 75; fi` + removed, err := e.T.Run(ctx, removeObserved) + if err != nil { + return 0, err + } + if removed.ExitCode == 0 || removed.ExitCode == 75 { + attempt-- // a stale-holder race does not consume contention budget + continue + } + return 0, fmt.Errorf("reclaim backup lock: %s", strings.TrimSpace(removed.Stderr)) + } + conflict = &BackupConflictError{Service: service, OperationID: safeBackupHolder(holder.OperationID), AgeSeconds: age} + if attempt+1 < maxAttempts { + e.Opts.Sleep(backupLockPollInterval) + } + } + if conflict == nil { + conflict = &BackupConflictError{Service: service, OperationID: "unknown", AgeSeconds: 0} + } + return 0, conflict +} + +func (e *Engine) nextBackupEpoch(ctx context.Context, service string) (int, error) { + result, err := e.T.Run(ctx, "cat "+q(e.backupEpochPath(service))+" 2>/dev/null || echo 0") + if err != nil { + return 0, err + } + previous, _ := strconv.Atoi(strings.TrimSpace(result.Stdout)) + return previous + 1, nil +} + +func (e *Engine) writeBackupFence(ctx context.Context, service, operationID string, epoch int, lockValue string) error { + fenceValue := operationID + " " + strconv.Itoa(epoch) + command := `if [ "$(cat ` + q(e.backupLockPath(service)) + ` 2>/dev/null)" = ` + q(lockValue) + ` ]; then echo ` + strconv.Itoa(epoch) + ` > ` + q(e.backupEpochPath(service)) + ` && echo ` + q(fenceValue) + ` > ` + q(e.backupFencePath(service)) + `; else echo ob-backup-lock-lost >&2; exit 96; fi` + result, err := e.T.Run(ctx, command) + if err != nil { + return err + } + if result.ExitCode == 96 && strings.Contains(result.Stderr, "ob-backup-lock-lost") { + return ErrBackupFenced + } + if result.ExitCode != 0 { + return fmt.Errorf("write backup fence: %s", strings.TrimSpace(result.Stderr)) + } + if e.backupLockVals == nil { + e.backupLockVals = make(map[string]string) + e.backupFenceVals = make(map[string]string) + } + e.backupLockVals[service] = lockValue + e.backupFenceVals[service] = fenceValue + return nil +} + +func (e *Engine) ReleaseBackupLock(service string) { + expected := e.backupLockVals[service] + if expected == "" { + return + } + cleanupContext, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + result, err := e.T.Run(cleanupContext, `if [ "$(cat `+q(e.backupLockPath(service))+` 2>/dev/null)" = `+q(expected)+` ]; then rm -f `+q(e.backupLockPath(service))+`; fi`) + if err != nil || result.ExitCode != 0 { + e.warnf("release backup lock failed: %v %s", err, strings.TrimSpace(result.Stderr)) + return + } + delete(e.backupLockVals, service) + delete(e.backupFenceVals, service) +} + +// StartBackupHeartbeat keeps a service lock fresh only while both its +// exact lock value and fence still belong to this runner. +func (e *Engine) StartBackupHeartbeat(ctx context.Context, service string) (func(), error) { + lockValue := e.backupLockVals[service] + fenceValue := e.backupFenceVals[service] + if lockValue == "" || fenceValue == "" { + return nil, errors.New("backup heartbeat requires service lock ownership") + } + heartbeatContext, cancel := context.WithCancel(ctx) + done := make(chan struct{}) + go func() { + defer close(done) + ticker := time.NewTicker(e.lockTTL() / 10) + defer ticker.Stop() + for { + select { + case <-heartbeatContext.Done(): + return + case <-ticker.C: + command := `if [ "$(cat ` + q(e.backupLockPath(service)) + ` 2>/dev/null)" = ` + q(lockValue) + ` ] && [ "$(cat ` + q(e.backupFencePath(service)) + ` 2>/dev/null)" = ` + q(fenceValue) + ` ]; then touch -c ` + q(e.backupLockPath(service)) + `; else exit 3; fi` + if result, err := e.T.Run(heartbeatContext, command); err == nil && result.ExitCode != 0 && result.ExitCode != 3 { + e.warnf("backup heartbeat for %s failed (exit %d): %s", service, result.ExitCode, strings.TrimSpace(result.Stderr)) + } + } + } + }() + return func() { cancel(); <-done }, nil +} + +// BackupMutate nests the exact service lock/fence guard inside the app +// fence guard. A runner that loses either authority cannot mutate service data. +func (e *Engine) BackupMutate(ctx context.Context, service, command string) (transport.Result, error) { + lockValue := e.backupLockVals[service] + fenceValue := e.backupFenceVals[service] + if e.lockVal == "" || e.fenceVal == "" || lockValue == "" || fenceValue == "" { + return transport.Result{}, errors.New("backup mutation requires application and service lock ownership") + } + guarded := `if [ "$(cat ` + q(e.backupLockPath(service)) + ` 2>/dev/null)" = ` + q(lockValue) + ` ] && [ "$(cat ` + q(e.backupFencePath(service)) + ` 2>/dev/null)" = ` + q(fenceValue) + ` ]; then ` + command + `; else echo ob-backup-fenced >&2; exit 98; fi` + result, err := e.mutate(ctx, guarded) + if err != nil { + return result, err + } + if result.ExitCode == 98 && strings.Contains(result.Stderr, "ob-backup-fenced") { + return result, ErrBackupFenced + } + return result, nil +} + +func safeBackupHolder(value string) string { + if backupIdentity.MatchString(value) { + return value + } + return "unknown" +} diff --git a/internal/engine/protection_lock_test.go b/internal/engine/backup_lock_test.go similarity index 62% rename from internal/engine/protection_lock_test.go rename to internal/engine/backup_lock_test.go index 161db550..532fc2b3 100644 --- a/internal/engine/protection_lock_test.go +++ b/internal/engine/backup_lock_test.go @@ -12,7 +12,7 @@ import ( "github.com/labstack/onebox/internal/transport" ) -func protectionLockTestEngine(fake *transport.Fake) *Engine { +func backupLockTestEngine(fake *transport.Fake) *Engine { engine := New( &app.Resolved{Spec: &app.Spec{Name: "example", BasePath: "/var/lib/ob"}, Env: "production"}, nil, @@ -25,19 +25,19 @@ func protectionLockTestEngine(fake *transport.Fake) *Engine { return engine } -func TestProtectionLockRequiresApplicationLock(t *testing.T) { +func TestBackupLockRequiresApplicationLock(t *testing.T) { fake := &transport.Fake{} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) engine.lockVal = "" - if _, err := engine.AcquireProtectionLock(context.Background(), "database", "backup-1", 0); err == nil { - t.Fatal("protection lock acquired without application lock") + if _, err := engine.AcquireBackupLock(context.Background(), "database", "backup-1", 0); err == nil { + t.Fatal("backup lock acquired without application lock") } if len(fake.Commands) != 0 { - t.Fatal("protection lock touched the host before checking lock order") + t.Fatal("backup lock touched the host before checking lock order") } } -func TestProtectionLockReturnsBoundedRetryableBackupConflict(t *testing.T) { +func TestBackupLockReturnsBoundedRetryableBackupConflict(t *testing.T) { holder := `{"owner":"operator","operation_id":"restore-7","service":"database","epoch":4,"ttl_s":10,"acquired_at":"2026-08-07T12:00:00Z"}` createAttempts := 0 fake := &transport.Fake{Dynamic: func(command string) (transport.Result, bool) { @@ -52,13 +52,13 @@ func TestProtectionLockReturnsBoundedRetryableBackupConflict(t *testing.T) { } return transport.Result{}, false }} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) - _, err := engine.AcquireProtectionLock(context.Background(), "database", "backup-8", 200*time.Millisecond) - if !errors.Is(err, ErrProtectionConflict) { - t.Fatalf("acquire error = %v, want protection conflict", err) + _, err := engine.AcquireBackupLock(context.Background(), "database", "backup-8", 200*time.Millisecond) + if !errors.Is(err, ErrBackupConflict) { + t.Fatalf("acquire error = %v, want backup conflict", err) } - var conflict *ProtectionConflictError + var conflict *BackupConflictError if !errors.As(err, &conflict) || conflict.Code() != "backup_conflict" || !conflict.Retryable() { t.Fatalf("conflict classification = %#v", err) } @@ -67,12 +67,12 @@ func TestProtectionLockReturnsBoundedRetryableBackupConflict(t *testing.T) { } } -func TestProtectionLockHonorsCancellation(t *testing.T) { +func TestBackupLockHonorsCancellation(t *testing.T) { fake := &transport.Fake{} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) ctx, cancel := context.WithCancel(context.Background()) cancel() - if _, err := engine.AcquireProtectionLock(ctx, "database", "backup-1", time.Second); !errors.Is(err, context.Canceled) { + if _, err := engine.AcquireBackupLock(ctx, "database", "backup-1", time.Second); !errors.Is(err, context.Canceled) { t.Fatalf("acquire error = %v, want context cancellation", err) } if len(fake.Commands) != 0 { @@ -80,7 +80,7 @@ func TestProtectionLockHonorsCancellation(t *testing.T) { } } -func TestProtectionLockReclaimsStaleHolderWithNewFence(t *testing.T) { +func TestBackupLockReclaimsStaleHolderWithNewFence(t *testing.T) { holder := `{"owner":"operator","operation_id":"backup-old","service":"database","epoch":4,"ttl_s":10,"acquired_at":"2026-08-07T11:00:00Z"}` createAttempts := 0 fake := &transport.Fake{Dynamic: func(command string) (transport.Result, bool) { @@ -100,33 +100,33 @@ func TestProtectionLockReclaimsStaleHolderWithNewFence(t *testing.T) { } return transport.Result{}, false }} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) - epoch, err := engine.AcquireProtectionLock(context.Background(), "database", "backup-new", 0) + epoch, err := engine.AcquireBackupLock(context.Background(), "database", "backup-new", 0) if err != nil { - t.Fatalf("reclaim stale protection lock: %v", err) + t.Fatalf("reclaim stale backup lock: %v", err) } if epoch != 5 || createAttempts != 2 { t.Fatalf("reclaimed epoch/attempts = %d/%d, want 5/2", epoch, createAttempts) } - if got := engine.protectionFenceVals["database"]; got != "backup-new 5" { - t.Fatalf("protection fence = %q", got) + if got := engine.backupFenceVals["database"]; got != "backup-new 5" { + t.Fatalf("backup fence = %q", got) } } -func TestProtectionMutationRejectsStaleFence(t *testing.T) { +func TestBackupMutationRejectsStaleFence(t *testing.T) { fake := &transport.Fake{Dynamic: func(command string) (transport.Result, bool) { if strings.Contains(command, "write-database-data") { - return transport.Result{ExitCode: 98, Stderr: "ob-protection-fenced\n"}, true + return transport.Result{ExitCode: 98, Stderr: "ob-backup-fenced\n"}, true } return transport.Result{}, false }} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) engine.fenceVal = "deploy-1 1" - engine.protectionLockVals = map[string]string{"database": "old-lock"} - engine.protectionFenceVals = map[string]string{"database": "backup-old 4"} + engine.backupLockVals = map[string]string{"database": "old-lock"} + engine.backupFenceVals = map[string]string{"database": "backup-old 4"} - if _, err := engine.ProtectionMutate(context.Background(), "database", "write-database-data"); !errors.Is(err, ErrProtectionFenced) { - t.Fatalf("protection mutation error = %v, want stale fence", err) + if _, err := engine.BackupMutate(context.Background(), "database", "write-database-data"); !errors.Is(err, ErrBackupFenced) { + t.Fatalf("backup mutation error = %v, want stale fence", err) } } diff --git a/internal/engine/backup_postgres.go b/internal/engine/backup_postgres.go new file mode 100644 index 00000000..84217f22 --- /dev/null +++ b/internal/engine/backup_postgres.go @@ -0,0 +1,509 @@ +package engine + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "net/http" + "os" + "path" + "path/filepath" + "strings" + "time" + + "github.com/labstack/onebox/internal/app" +) + +// Executable backup for the postgres driver. +// +// Everything above this file describes backup: the project declares intent, +// the catalogue declares what each driver could support, the lifecycle state +// records whether it was ever established, and the artifact set records what a +// protected service should look like. None of it runs anything. This file and +// its _ops sibling are the part that does, and they are deliberately narrow — +// one driver, one recovery kind, physical base plus WAL. +// +// Whether a service *is* protected is not decided here. It is durable state on +// the target, observed at project load and bound into the rendered project +// before anything renders, so a policy that was declared but never enabled +// produces an ordinary server rather than one archiving to a repository nobody +// initialised. + +// StageBackupRuntime places the verified wal-g binary and its generated +// wrapper on the target, then makes them readable by the service. +// +// The binary is fetched here — on the machine running `ob` — and uploaded, +// rather than downloaded by the target. That keeps the agentless model intact +// and, more importantly, keeps verification on this side of the trust boundary: +// the checksum is pinned in the Onebox binary, so a host with no outbound +// internet still gets backup, and a compromised release page cannot +// substitute a binary that a target-side `curl | sha256sum` would happily +// accept against a checksum from the same source. +func (e *Engine) StageBackupRuntime(ctx context.Context, service string, wrapper []byte) error { + machine, err := e.targetMachine(ctx) + if err != nil { + return err + } + asset, expected, err := app.WalgAssetFor(machine) + if err != nil { + return err + } + n := e.names() + destination := n.BackupBinaryFile(service) + + present, err := e.fileHasChecksum(ctx, destination, expected) + if err != nil { + return err + } + if !present { + st := e.ui.Step("backup runtime wal-g "+app.WalgVersion+" ("+machine+")", false) + staged, cleanup, err := fetchVerifiedBinary(ctx, app.WalgDownloadURL(asset), expected) + if err != nil { + st(err) + return err + } + defer cleanup() + if err := e.uploadBackupBinary(ctx, n.BackupRuntimeDir(service), staged, destination); err != nil { + st(err) + return err + } + st(nil) + } + + // The wrapper is passed in rather than rendered here, because enablement + // has to stage the runtime *before* it records that the service is + // protected — and until that record exists there is no bound state to + // render from. It is rewritten every time: it is derived from the declared + // credential entry names, which can change without the binary changing. + wrapperPath := n.BackupWrapperFile(service) + if err := e.writeServiceFile(ctx, wrapperPath, wrapper); err != nil { + return fmt.Errorf("cannot place backup wrapper %s: %w", wrapperPath, err) + } + // Readable and executable by the unprivileged server user inside the + // container, which is the whole point of it being there. Safe because it + // holds no credential — only the names of entries it reads from the + // environment. + if err := e.chmodPath(ctx, wrapperPath, "0755"); err != nil { + return err + } + return e.chmodPath(ctx, n.BackupRuntimeDir(service), "0755") +} + +func (e *Engine) targetMachine(ctx context.Context) (string, error) { + res, err := e.T.Run(ctx, "uname -m") + if err != nil { + return "", err + } + machine := strings.TrimSpace(res.Stdout) + if res.ExitCode != 0 || machine == "" { + return "", fmt.Errorf("cannot determine the target's machine architecture") + } + return machine, nil +} + +// fileHasChecksum reports whether the target already holds exactly the expected +// bytes. Re-uploading 60MB on every enable would be the kind of cost that makes +// people avoid running the command. +func (e *Engine) fileHasChecksum(ctx context.Context, remotePath, expected string) (bool, error) { + res, err := e.T.Run(ctx, "sha256sum "+q(remotePath)+" 2>/dev/null | cut -d' ' -f1") + if err != nil { + return false, err + } + return strings.TrimSpace(res.Stdout) == expected, nil +} + +// fetchVerifiedBinary downloads an asset and refuses it unless it hashes to the +// pinned value. The file is never made executable and never leaves the +// temporary directory until it has matched. +func fetchVerifiedBinary(ctx context.Context, url, expected string) (string, func(), error) { + dir, err := os.MkdirTemp("", "ob-backup-runtime-") + if err != nil { + return "", nil, fmt.Errorf("create staging directory: %w", err) + } + cleanup := func() { os.RemoveAll(dir) } + + request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + cleanup() + return "", nil, err + } + client := &http.Client{Timeout: 10 * time.Minute} + response, err := client.Do(request) + if err != nil { + cleanup() + return "", nil, fmt.Errorf("fetch %s: %w", url, err) + } + defer response.Body.Close() + if response.StatusCode != http.StatusOK { + cleanup() + return "", nil, fmt.Errorf("fetch %s: %s", url, response.Status) + } + + staged := filepath.Join(dir, "wal-g") + file, err := os.OpenFile(staged, os.O_CREATE|os.O_WRONLY|os.O_EXCL, 0o600) + if err != nil { + cleanup() + return "", nil, err + } + digest := sha256.New() + if _, err := io.Copy(io.MultiWriter(file, digest), response.Body); err != nil { + file.Close() + cleanup() + return "", nil, fmt.Errorf("download %s: %w", url, err) + } + if err := file.Close(); err != nil { + cleanup() + return "", nil, err + } + if observed := hex.EncodeToString(digest.Sum(nil)); observed != expected { + cleanup() + return "", nil, fmt.Errorf( + "the wal-g download does not match its pinned checksum (expected %s, got %s); refusing to place it on the host", + expected, observed) + } + return staged, cleanup, nil +} + +// uploadBackupBinary moves the verified binary into place. Upload writes a +// directory, so the staged file is placed alone in one and moved across. +func (e *Engine) uploadBackupBinary(ctx context.Context, runtimeDir, staged, destination string) error { + res, err := e.T.Run(ctx, "mkdir -p "+q(runtimeDir)) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot create the backup runtime directory: %s", strings.TrimSpace(res.Stderr)) + } + remoteStaging := runtimeDir + "/.staging" + if err := e.T.Upload(ctx, filepath.Dir(staged), remoteStaging); err != nil { + return fmt.Errorf("upload the wal-g binary: %w", err) + } + install := "mv -f " + q(remoteStaging+"/"+filepath.Base(staged)) + " " + q(destination) + + " && chmod 0755 " + q(destination) + + " && rm -rf " + q(remoteStaging) + res, err = e.T.Run(ctx, install) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot install the wal-g binary: %s", strings.TrimSpace(res.Stderr)) + } + return nil +} + +func (e *Engine) chmodPath(ctx context.Context, target, mode string) error { + res, err := e.T.Run(ctx, "chmod "+mode+" "+q(target)) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot set mode %s on %s", mode, target) + } + return nil +} + +// WriteBackupLifecycleState places the already-sealed lifecycle record that +// makes a service protected. The record's schema, transitions, and digest all +// belong to the layer above; the engine only puts the bytes on the target, +// under the same fence as every other generated file. +func (e *Engine) WriteBackupLifecycleState(ctx context.Context, service string, body []byte) error { + n := e.names() + res, err := e.T.Run(ctx, "mkdir -p "+q(path.Join(n.AppDir(), "backup", "state"))) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot create the backup state directory: %s", strings.TrimSpace(res.Stderr)) + } + return e.writeServiceFile(ctx, n.BackupLifecycleStateFile(service), body) +} + +// ReadBackupLifecycleState returns the raw lifecycle record for a service, +// or nil when none exists. Decoding belongs to the layer that owns the schema; +// the engine only fetches the bytes. +func (e *Engine) ReadBackupLifecycleState(ctx context.Context, service string) ([]byte, error) { + res, err := e.T.Run(ctx, "cat "+q(e.names().BackupLifecycleStateFile(service))+" 2>/dev/null || true") + if err != nil { + return nil, err + } + if len(res.Stdout) == 0 { + return nil, nil + } + return []byte(res.Stdout), nil +} + +// RebindServiceRuntimeStates re-derives the rendered project from lifecycle +// state the caller has just written. Enablement is the one flow that needs it: +// the project was loaded before the service was protected, so without this the +// same run would render the unprotected server it started from. +func (e *Engine) RebindServiceRuntimeStates(states map[string]app.ServiceRuntimeState) error { + bound, err := e.Spec.WithServiceRuntimeStates(states) + if err != nil { + return err + } + e.Spec = bound + return nil +} + +// ResolveProtectedImage pins the service image by the digest the host actually +// has, after pulling it. +// +// It is the stock PostgreSQL image — wal-g is mounted in beside it rather than +// baked into a derived one — but it is still pinned, because the reason +// protected image selection is durable state has nothing to do with which image +// it is: the bytes running over a live data directory must not change because a +// tag moved. +// recordedPin and recordedReference come from the service's lifecycle record: +// the digest it was last bound with, and the reference that produced it. When +// the project still declares that same reference and the host still holds those +// bytes, they are what the service keeps running. Re-resolving a tag here would +// mean a re-enable — after a policy edit, or after a disable somebody changed +// their mind about — could move the bytes running over a live data directory +// because the tag moved in the meantime, which is the exact thing pinning +// exists to prevent. It also made re-enable need the registry on a host that +// already had everything it needed, so a rate-limited Docker Hub failed a +// command that had nothing to fetch. +// +// Moving a protected service to a new image is a deliberate act with its own +// path; it is not a side effect of re-running enable. +func (e *Engine) ResolveProtectedImage(ctx context.Context, service, recordedPin, recordedReference string) (string, error) { + reference, err := e.Spec.ServiceImageForRuntime(service) + if err != nil { + return "", err + } + declared, err := e.Spec.DeclaredServiceImage(service) + if err != nil { + return "", err + } + st := e.ui.Step("protected image "+reference.Image, false) + if recordedPin != "" && recordedReference == declared && containsDigest(recordedPin) { + held, err := e.imagePresentByDigest(ctx, recordedPin) + if err != nil { + st(err) + return "", err + } + if held { + st(nil) + return recordedPin, nil + } + } + // A digest already on the host is not pulled again. The reference is + // immutable by construction, so a second pull cannot return different bytes + // — it only spends registry quota and turns a re-enable into a failure on a + // host that is offline or rate-limited while holding exactly what it needs. + // A tag still resolves through the registry, because a tag can move. + present, err := e.imagePresentByDigest(ctx, reference.Image) + if err != nil { + st(err) + return "", err + } + if present { + st(nil) + return reference.Image, nil + } + res, err := e.T.Run(ctx, "docker pull "+q(reference.Image)) + if err != nil { + st(err) + return "", err + } + if res.ExitCode != 0 { + err := fmt.Errorf("cannot pull %s: %s", reference.Image, lastLines(res.Stderr, 3)) + st(err) + return "", err + } + res, err = e.T.Run(ctx, "docker image inspect --format '{{index .RepoDigests 0}}' "+q(reference.Image)) + if err != nil { + st(err) + return "", err + } + pinned := strings.TrimSpace(res.Stdout) + if res.ExitCode != 0 || !containsDigest(pinned) { + err := fmt.Errorf( + "%s has no registry digest on this host; a protected service runs an image pinned by digest, so it must come from a registry rather than a local build", + reference.Image) + st(err) + return "", err + } + st(nil) + return pinned, nil +} + +// RemoveBackupCredentials deletes the target-side credential file for a +// service that is no longer protected. The repository it pointed at is left +// exactly as it is. +func (e *Engine) RemoveBackupCredentials(ctx context.Context, service string, last *app.BackupEffectiveProjection) error { + if last == nil { + return nil + } + path := e.names().BackupCredentialFile(service, last.Policy.Target) + res, err := e.T.Run(ctx, "rm -f "+q(path)) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot remove the backup credential file %s", path) + } + return nil +} + +// ReportDisabled says what disablement did and, more usefully, what it did not. +// +// The second line used to promise that `ob backup status` and `ob backup +// restore` still worked. They do not: both run wal-g inside the service +// container, and an unprotected service mounts neither the binary nor the +// credentials. The backups themselves are untouched, which is the part that +// matters, and the way back to them is to enable backup again. +func (e *Engine) ReportDisabled(service string) { + e.ui.Successf("%s is no longer archiving; its schedules are removed", service) + e.ui.Infof("every backup already taken is untouched in the repository. Reading or recovering from them "+ + "needs backup enabled again (`ob backup enable %s`), because the tooling and credentials that "+ + "reach the repository live in the protected service", service) +} + +// ReportTargetMoved says that this enablement bound the service to a different +// repository from the one it was archiving to, which is a fact with a +// consequence: the new repository starts at the backup this run is about to +// take, so the declared recovery window begins now. +func (e *Engine) ReportTargetMoved(service, from, to string) { + e.ui.Infof("%s now archives to backup target %q. What %q holds is untouched, but this repository "+ + "starts from the backup being taken now, so the declared recovery window begins here", + service, to, from) +} + +// VerifyBackupRuntime asks the target whether what is staged there is still +// what Onebox expects. +// +// This is the drift question that matters, and it is asked of the target +// directly: the binary that pushes the backups must be the one whose checksum +// is pinned in this release, and the wrapper that gives it credentials must be +// the one the project renders. A descriptor written beside them saying what they +// ought to be would only be somewhere for the two to disagree. +func (e *Engine) VerifyBackupRuntime(ctx context.Context, service string) ([]string, error) { + n := e.names() + var issues []string + + machine, err := e.targetMachine(ctx) + if err != nil { + return nil, err + } + _, expected, err := app.WalgAssetFor(machine) + if err != nil { + return nil, err + } + matches, err := e.fileHasChecksum(ctx, n.BackupBinaryFile(service), expected) + if err != nil { + return nil, err + } + if !matches { + issues = append(issues, fmt.Sprintf( + "the wal-g binary at %s is not the %s build this release pins; re-run `ob service apply` to replace it", + n.BackupBinaryFile(service), app.WalgVersion)) + } + + wrappers, err := e.Spec.RenderServiceBackupWrappers(e.Opts.Environment) + if err != nil { + return nil, err + } + wanted, ok := wrappers[n.BackupWrapperFile(service)] + if !ok { + return issues, nil + } + res, err := e.T.Run(ctx, "cat "+q(n.BackupWrapperFile(service))+" 2>/dev/null || true") + if err != nil { + return nil, err + } + if res.Stdout != string(wanted) { + issues = append(issues, fmt.Sprintf( + "the credential wrapper at %s is not what this project renders; re-run `ob service apply` to replace it", + n.BackupWrapperFile(service))) + } + archiving, err := e.archivingIssues(ctx, service) + if err != nil { + return nil, err + } + return append(issues, archiving...), nil +} + +// archivingIssues asks the running server whether it is still archiving, and +// whether it is doing so often enough for the loss the policy tolerates. +// +// The binary and the wrapper being correct says the tooling is in place. It +// says nothing about whether the server is using it: `archive_mode` can be +// turned off, `archive_command` can be replaced, and `archive_timeout` can be +// raised past the declared maximum data loss — each of which leaves every +// existing backup intact and quietly stops the recovery point advancing the way +// the policy promises. That is a state worth naming, because nothing else here +// notices it. +func (e *Engine) archivingIssues(ctx context.Context, service string) ([]string, error) { + projection, err := e.Spec.EffectiveBackupProjection(service) + if err != nil { + return nil, err + } + n := e.names() + // Connected as the role the driver creates, against the database that always + // exists. + // + // Two earlier spellings of this line both failed and were both read by the + // exit-code branch below as "the server is down", so the check silently did + // nothing on a perfectly healthy server: asking as the OS user reaches a + // role that does not exist, and asking as the right role without a database + // reaches one named after the role, which does not exist either. Only + // running it against a live server showed that — the unit tests were happy + // with a command that never worked. + read := "docker exec -u postgres " + q(n.ServiceContainer(service)) + + " psql -U " + q(app.PgSuperuser) + " -d postgres -Atc " + q("show archive_mode;") + + " -Atc " + q("show archive_command;") + + " -Atc " + q("show archive_timeout;") + res, err := e.T.Run(ctx, read) + if err != nil { + return nil, err + } + if res.ExitCode != 0 { + // Not an assertion that archiving is broken: the server may simply be + // down, which every other part of status already reports. + return nil, nil + } + fields := strings.Split(strings.TrimSpace(res.Stdout), "\n") + if len(fields) < 3 { + return nil, nil + } + mode, command, timeout := strings.TrimSpace(fields[0]), strings.TrimSpace(fields[1]), strings.TrimSpace(fields[2]) + var issues []string + if mode != "on" { + issues = append(issues, fmt.Sprintf( + "the server has archive_mode %q, so no write-ahead log is reaching the repository and the recovery point stopped advancing; re-run `ob backup enable %s`", mode, service)) + } + if !strings.Contains(command, app.WalgBinary) { + issues = append(issues, fmt.Sprintf( + "the server's archive_command is not the one Onebox installed, so where the write-ahead log goes is not what this project describes; re-run `ob backup enable %s`", service)) + } + if declared, ok := app.ParseDuration(projection.Policy.MaxDataLoss); ok { + if observed, parsed := app.ParsePostgresDuration(timeout); parsed && observed > declared { + issues = append(issues, fmt.Sprintf( + "the server closes a write-ahead log segment every %s, but the policy tolerates losing at most %s; an idle database can lose more than the policy permits", + timeout, projection.Policy.MaxDataLoss)) + } + } + return issues, nil +} + +// imagePresentByDigest reports whether a digest-pinned reference is already on +// the host. A tag always answers false: it may point somewhere else now, which +// is the reason backup pins in the first place. +func (e *Engine) imagePresentByDigest(ctx context.Context, reference string) (bool, error) { + if !containsDigest(reference) { + return false, nil + } + res, err := e.T.Run(ctx, "docker image inspect "+q(reference)+" >/dev/null 2>&1 && echo present || echo absent") + if err != nil { + return false, err + } + return strings.TrimSpace(res.Stdout) == "present", nil +} + +// containsDigest reports whether a reference names exact bytes rather than a +// tag. It is the one place that decides, so the pull-skipping guard and the +// pinning check cannot disagree about what "pinned" means. +func containsDigest(reference string) bool { return strings.Contains(reference, "@sha256:") } diff --git a/internal/engine/backup_postgres_ops.go b/internal/engine/backup_postgres_ops.go new file mode 100644 index 00000000..d969f3da --- /dev/null +++ b/internal/engine/backup_postgres_ops.go @@ -0,0 +1,449 @@ +package engine + +import ( + "context" + "encoding/json" + "fmt" + "regexp" + "sort" + "strings" + "time" + + "github.com/labstack/onebox/internal/app" +) + +// BackupGeneration is one recoverable base backup as the repository reports it. +type BackupGeneration struct { + Label string `json:"label"` + Type string `json:"type"` + StartedAt int64 `json:"started_at"` + StoppedAt int64 `json:"stopped_at"` + WALStart string `json:"wal_start,omitempty"` + SizeBytes int64 `json:"size_bytes,omitempty"` +} + +// BackupStatus is what the repository can actually recover, read from the +// repository rather than from the project. The distinction is the whole point: +// a policy says what should be true, and only the repository says what is. +type BackupStatus struct { + Service string `json:"service"` + Repository string `json:"repository"` + Generations []BackupGeneration `json:"generations"` + RuntimeIssues []string `json:"runtime_issues,omitempty"` + LatestBackup *BackupGeneration `json:"latest_backup,omitempty"` + RecoverableTo string `json:"recoverable_to,omitempty"` + // The declared promise, carried alongside the facts so the report can be + // read against it. Reporting only what the repository holds left the + // operator to work out whether it satisfies the policy they wrote — which + // is the one question they came with. + DeclaredWindow string `json:"declared_window,omitempty"` + DeclaredMaxDataLoss string `json:"declared_max_data_loss,omitempty"` + // OldestRecoverable is where the history starts: the completion of the + // oldest base backup still in the repository. Anything before it is gone. + OldestRecoverable string `json:"oldest_recoverable,omitempty"` + // WindowCovered is whether the history already reaches back as far as the + // declared window. A repository younger than its window is not a fault — + // but it is not the promise either, and only the report can say which. + WindowCovered bool `json:"window_covered"` +} + +// backedUpService resolves a service to its policy and driver, refusing every +// service this file cannot actually protect. It is the single gate: no caller +// below reaches wal-g without passing through it. +func (e *Engine) backedUpService(service string) (app.Service, *app.BackupPolicy, error) { + declared, ok := e.Spec.Services[service] + if !ok { + return app.Service{}, nil, fmt.Errorf("service %s is not declared in this project", service) + } + driver := declared.Driver + if driver == "" { + driver = service + } + if driver != "postgres" { + return app.Service{}, nil, fmt.Errorf( + "service %s runs the %s driver; executable backup exists for postgres only today", service, driver) + } + if declared.Backup == nil { + return app.Service{}, nil, fmt.Errorf( + "service %s declares no backup policy; add services.%s.backup to the project first", service, service) + } + // Declared is not established. Without this the command reaches into a + // container that mounts neither wal-g nor its credentials, and the operator + // gets an OCI runtime error about a missing path instead of being told the + // service is not protected. + if !e.Spec.ServiceIsProtected(service) { + return app.Service{}, nil, fmt.Errorf( + "service %s declares backup but it has never been established, or it was disabled; run `ob backup enable %s` first", + service, service) + } + return declared, declared.Backup, nil +} + +// runWalg executes one wal-g operation inside the service container, as the +// server's own user, through the wrapper that puts its credentials in scope. +// +// It deliberately does not go through mutate: the repository is off-host, these +// operations are the ones the fence and the service lock already guard at a +// higher level, and routing them through the application's release machinery +// would tie a backup to a deploy that has nothing to do with it. +func (e *Engine) runWalg(ctx context.Context, service string, args ...string) (string, error) { + return e.runWalgLocked(ctx, service, e.walgLockPrefix(ctx, service), args...) +} + +// runWalgRead is for commands that only read the repository. It waits briefly +// on a shared lock instead of an hour on an exclusive one, so status answers +// while a backup is running rather than blocking behind it. +func (e *Engine) runWalgRead(ctx context.Context, service string, args ...string) (string, error) { + return e.runWalgLocked(ctx, service, e.walgReadLockPrefix(ctx, service), args...) +} + +func (e *Engine) runWalgLocked(ctx context.Context, service, lockPrefix string, args ...string) (string, error) { + // Under the same flock the scheduled units take, so an operator running a + // backup by hand and a timer firing cannot talk to the repository at once. + // + // Absent flock the command runs unwrapped, and that is not a silent + // degradation: flock ships with util-linux on every host that can run + // systemd, and a host that cannot run systemd has no timers, so there is + // nothing for the lock to serialise against. SyncBackupSchedules + // refuses such a host outright rather than leaving backups unscheduled. + n := e.names() + var command []string + if lockPrefix != "" { + command = append(command, strings.TrimSpace(lockPrefix)) + } + command = append(command, "docker", "exec", "-u", "postgres", q(n.ServiceContainer(service)), app.WalgBinary) + for _, arg := range args { + command = append(command, q(arg)) + } + res, err := e.T.Run(ctx, strings.Join(command, " ")) + if err != nil { + return "", err + } + if res.ExitCode != 0 { + // wal-g's own message is the useful one and it carries no secret: the + // credentials reach it through the environment and it reports its + // configuration by name. + return "", fmt.Errorf("wal-g %s: %s", strings.Join(args, " "), lastLines(res.Stderr+res.Stdout, 6)) + } + return res.Stdout, nil +} + +// runWalgReport is for commands whose *report* is the answer rather than their +// exit code. wal-g writes its tables and status lines across both streams, so a +// caller that judges the report needs both; the JSON-producing commands must +// not have stderr folded in, which is a parse failure waiting to happen. +func (e *Engine) runWalgReport(ctx context.Context, service string, args ...string) (string, error) { + n := e.names() + command := []string{strings.TrimSpace(e.walgLockPrefix(ctx, service)), + "docker", "exec", "-u", "postgres", q(n.ServiceContainer(service)), app.WalgBinary} + for _, arg := range args { + command = append(command, q(arg)) + } + res, err := e.T.Run(ctx, strings.TrimSpace(strings.Join(command, " "))) + if err != nil { + return "", err + } + if res.ExitCode != 0 { + return "", fmt.Errorf("wal-g %s: %s", strings.Join(args, " "), lastLines(res.Stderr+res.Stdout, 6)) + } + return res.Stdout + res.Stderr, nil +} + +// BackupService takes one base backup. +// +// wal-g has no incremental base backup for PostgreSQL in the sense pgBackRest +// does: every base backup is complete, and the space between them is covered by +// the WAL stream rather than by differential backups. So there is no type to +// choose, and retention counts backups directly. +func (e *Engine) BackupService(ctx context.Context, service string) error { + if _, _, err := e.backedUpService(service); err != nil { + return err + } + st := e.ui.Step("backup "+service, false) + if _, err := e.runWalg(ctx, service, "backup-push", app.PgDataPath); err != nil { + st(err) + return err + } + st(nil) + return nil +} + +// PruneServiceBackups expires everything outside the declared retention. +// +// Retention has two bounds and they are not the same promise. +// keep says how many independently recoverable base backups to +// keep; window says how far back a point-in-time recovery must be able +// to reach. On a busy database the count is the binding one; on a quiet one the +// window is, because N backups might span an afternoon. Honouring only the +// count quietly shortens the window the policy promised, and nobody finds out +// until they try to recover to last Tuesday. +// +// wal-g has no working time bound — its `--after` flag is accepted and ignored +// — so the window is folded into the count before it gets here. See +// app.WalgRetainCount for that arithmetic and the measurements behind it. +// +// Separate from taking a backup because the order matters: expiring first would +// briefly hold one generation fewer than the policy promises. +func (e *Engine) PruneServiceBackups(ctx context.Context, service string) error { + if _, _, err := e.backedUpService(service); err != nil { + return err + } + projection, err := e.Spec.EffectiveBackupProjection(service) + if err != nil { + return err + } + policy := projection.Policy + retain, err := app.WalgRetainCount(policy) + if err != nil { + return fmt.Errorf("service %s: %w", service, err) + } + label := fmt.Sprintf("prune %s (keep %d generations — %d declared, %s of history)", + service, retain, policy.Retention.Keep, policy.Retention.Window) + st := e.ui.Step(label, false) + if _, err := e.runWalg(ctx, service, "delete", "retain", "FULL", + fmt.Sprint(retain), "--confirm"); err != nil { + st(err) + return err + } + st(nil) + return nil +} + +// VerifyServiceArchive checks that the WAL segments in the repository form an +// unbroken chain. +// +// This is the check worth running, and it has no equivalent in "did the backup +// command exit zero". A base backup plus a gapped WAL stream recovers to the +// backup and no further, which is a nightly snapshot wearing the label of +// point-in-time recovery — and nothing else notices until a restore. +func (e *Engine) VerifyServiceArchive(ctx context.Context, service string) error { + if _, _, err := e.backedUpService(service); err != nil { + return err + } + st := e.ui.Step("verify "+service+" archive", false) + out, err := e.runWalgReport(ctx, service, "wal-verify", "integrity", "timeline") + if err != nil { + st(err) + return err + } + // wal-g's exit code is not the answer. It prints the integrity table, says + // "integrity check status: WARNING", lists the segments it could not find — + // and exits 0. Two segments deleted out of the middle of a live repository + // produced exactly that, and onebox reported a green check over a WAL chain + // with holes in it. The scheduled unit had the same blind spot: systemd saw + // exit 0 and nothing was ever raised. + if err := walVerifyResult(out); err != nil { + st(err) + return err + } + st(nil) + return nil +} + +// walVerifyStatus is one row of wal-g's integrity table: a contiguous range of +// WAL segments and what it found there. +type walVerifyStatus struct { + start, end, status string +} + +var walVerifyStatusLine = regexp.MustCompile(`\[wal-verify\] (integrity|timeline) check status: (\w+)`) + +// walVerifyResult turns wal-g's report into a verdict. +// +// A missing range is a hole in the chain, and a base backup plus a gapped WAL +// stream recovers to the backup and no further — so any point after the first +// hole is not recoverable, whatever the exit code said. +// +// MISSING_UPLOADING is the one status that is not yet a hole. wal-g classifies a +// segment that way while it is inside the window where the server could still be +// uploading it, and reclassifies it as MISSING_LOST once that window passes: +// segments deleted from a live repository here were reported as UPLOADING at +// first and as LOST minutes later. Treating UPLOADING as a hole would fail every +// check that lands mid-upload — measured on a database taking backups in a loop, +// where three segments were in flight while later ones had already arrived, so +// position in the table decides nothing either. A genuine hole is reported on +// the next run, which for a scheduled check is the next day at the latest. +func walVerifyResult(out string) error { + rows := parseWalVerifyRows(out) + var holes []string + for _, row := range rows { + if row.status == "FOUND" || row.status == "MISSING_UPLOADING" { + continue + } + holes = append(holes, fmt.Sprintf("%s..%s %s", row.start, row.end, strings.ToLower(row.status))) + } + if len(holes) > 0 { + return fmt.Errorf( + "the archived WAL has %d gap(s) — %s. A base backup plus a gapped WAL stream recovers to the backup and no further, so any point after the first gap is not recoverable", + len(holes), strings.Join(holes, ", ")) + } + // The status lines are the backstop: a future wal-g may report a problem in + // a shape this table parser does not recognise, and "no rows parsed" must + // not read as "nothing wrong". + for _, match := range walVerifyStatusLine.FindAllStringSubmatch(out, -1) { + if match[2] != "OK" { + if match[1] == "integrity" && len(rows) > 0 { + // Already judged by the rows above, which know which range is + // the newest and which is a real hole. + continue + } + return fmt.Errorf("wal-g reports the %s check as %s: %s", match[1], match[2], lastLines(out, 12)) + } + } + return nil +} + +// parseWalVerifyRows reads the rows of wal-g's integrity table, which is drawn +// as `| TLI | START | END | COUNT | STATUS |`. +func parseWalVerifyRows(out string) []walVerifyStatus { + var rows []walVerifyStatus + for _, line := range strings.Split(out, "\n") { + if !strings.HasPrefix(strings.TrimSpace(line), "|") { + continue + } + fields := strings.Split(line, "|") + if len(fields) < 7 { + continue + } + start := strings.TrimSpace(fields[2]) + end := strings.TrimSpace(fields[3]) + status := strings.TrimSpace(fields[5]) + if start == "START" || status == "STATUS" || status == "" { + continue + } + rows = append(rows, walVerifyStatus{start: start, end: end, status: status}) + } + return rows +} + +// BackupStatusFor reads what the repository holds. Everything it reports +// comes from wal-g, not from the project: the project's claim about retention +// and recovery window is exactly the claim this is here to check. +func (e *Engine) BackupStatusFor(ctx context.Context, service string) (BackupStatus, error) { + if _, _, err := e.backedUpService(service); err != nil { + return BackupStatus{}, err + } + // The recorded projection, so status reports the repository the service is + // actually archiving to rather than one the project was edited to name. + projection, err := e.Spec.EffectiveBackupProjection(service) + if err != nil { + return BackupStatus{}, err + } + status := BackupStatus{ + Service: service, + Repository: app.WalgPrefix(projection.Target, e.Spec.Spec.Name, service), + DeclaredWindow: projection.Policy.Retention.Window, + DeclaredMaxDataLoss: projection.Policy.MaxDataLoss, + } + + issues, err := e.VerifyBackupRuntime(ctx, service) + if err != nil { + return status, err + } + status.RuntimeIssues = issues + + out, err := e.runWalgRead(ctx, service, "backup-list", "--detail", "--json") + if err != nil { + return status, err + } + var report []struct { + BackupName string `json:"backup_name"` + Time string `json:"time"` + StartTime string `json:"start_time"` + FinishTime string `json:"finish_time"` + WalFileName string `json:"wal_file_name"` + UncompressedSize int64 `json:"uncompressed_size"` + } + trimmed := strings.TrimSpace(out) + if trimmed == "" || trimmed == "null" { + return status, nil + } + if err := json.Unmarshal([]byte(trimmed), &report); err != nil { + return status, fmt.Errorf("service %s: wal-g backup-list is not readable JSON", service) + } + for _, entry := range report { + generation := BackupGeneration{ + Label: entry.BackupName, Type: "full", + WALStart: entry.WalFileName, SizeBytes: entry.UncompressedSize, + } + if started, err := time.Parse(time.RFC3339, entry.StartTime); err == nil { + generation.StartedAt = started.Unix() + } + finish := entry.FinishTime + if finish == "" { + finish = entry.Time + } + stopped, err := time.Parse(time.RFC3339, finish) + if err != nil { + // Refused rather than defaulted. A zero timestamp renders as + // 1970-01-01, so a report meant to say what is recoverable would + // state a recoverable point off by half a century — and it is the + // report an operator decides on. + return status, fmt.Errorf( + "service %s: wal-g reported backup %q with an unreadable completion time %q", service, entry.BackupName, finish) + } + generation.StoppedAt = stopped.Unix() + status.Generations = append(status.Generations, generation) + } + if len(status.Generations) > 0 { + // Sorted rather than assuming wal-g lists oldest first: "the newest + // recoverable point" must come from the timestamps, not from the order + // another tool happened to print. + sort.Slice(status.Generations, func(i, j int) bool { + return status.Generations[i].StoppedAt < status.Generations[j].StoppedAt + }) + latest := status.Generations[len(status.Generations)-1] + status.LatestBackup = &latest + status.RecoverableTo = time.Unix(latest.StoppedAt, 0).UTC().Format(time.RFC3339) + oldest := time.Unix(status.Generations[0].StoppedAt, 0).UTC() + status.OldestRecoverable = oldest.Format(time.RFC3339) + if window, ok := app.ParseDuration(projection.Policy.Retention.Window); ok && window > 0 { + status.WindowCovered = !e.Opts.Now().UTC().Add(-window).Before(oldest) + } + } + return status, nil +} + +func lastLines(text string, count int) string { + lines := strings.Split(strings.TrimSpace(text), "\n") + if len(lines) > count { + lines = lines[len(lines)-count:] + } + return strings.TrimSpace(strings.Join(lines, "; ")) +} + +// hasFlock probes the target once and remembers. Every wal-g invocation asks, +// and a round trip per backup command to learn something that cannot change +// mid-operation is waste. +func (e *Engine) hasFlock(ctx context.Context) bool { + if e.flockProbed { + return e.flockPresent + } + res, err := e.T.Run(ctx, "command -v flock >/dev/null 2>&1 && echo ok") + e.flockProbed = true + e.flockPresent = err == nil && strings.TrimSpace(res.Stdout) == "ok" + return e.flockPresent +} + +// walgLockPrefix is the flock every repository operation runs behind, as a +// command prefix so callers that build their own docker exec can use it too. +// Empty when the host has no flock — see hasFlock for why that is not a silent +// degradation. +// +// Read-only listing takes it too but must not wait an hour behind a running +// backup: `ob backup status` promises to answer while other work is in flight, +// and an operator who has to wait to find out whether they are recoverable is +// one who stops asking. +func (e *Engine) walgLockPrefix(ctx context.Context, service string) string { + if !e.hasFlock(ctx) { + return "" + } + return "flock -w 3600 " + q(e.names().BackupRunLock(service)) + " " +} + +func (e *Engine) walgReadLockPrefix(ctx context.Context, service string) string { + if !e.hasFlock(ctx) { + return "" + } + return "flock -s -w 15 " + q(e.names().BackupRunLock(service)) + " " +} diff --git a/internal/engine/backup_recovery_config_test.go b/internal/engine/backup_recovery_config_test.go new file mode 100644 index 00000000..8aa201fc --- /dev/null +++ b/internal/engine/backup_recovery_config_test.go @@ -0,0 +1,114 @@ +package engine + +import ( + "context" + "io" + "strings" + "testing" + "time" + + "github.com/labstack/onebox/internal/app" + "github.com/labstack/onebox/internal/transport" +) + +func recoveryConfigTestEngine(fake *transport.Fake) *Engine { + spec := &app.Spec{ + Name: "shop", + BasePath: "/var/lib/ob", + Services: map[string]app.Service{"database": {Driver: "postgres", Version: "18"}}, + } + return New(&app.Resolved{Spec: spec, Env: "production"}, nil, fake, + Options{Out: io.Discard, Sleep: func(time.Duration) {}}) +} + +// A base backup can carry the recovery settings of the restore that produced +// the cluster it was taken from. Recovery states its own target and clears +// every other kind first, so what the backup carried cannot decide where this +// recovery stops. Without the clearing, a drill asking for the newest point +// inherited a target in the past and the cluster died with "recovery ended +// before configured recovery target was reached". +func TestRecoveryClearsAnyTargetTheBaseBackupCarried(t *testing.T) { + fake := &transport.Fake{} + e := recoveryConfigTestEngine(fake) + + if err := e.replayRecovery(context.Background(), "shop-database-restore-1", ""); err != nil { + t.Fatalf("replaying to the newest recoverable point: %v", err) + } + + var written string + for _, cmd := range fake.Commands { + if strings.Contains(cmd, "postgresql.conf") { + written = cmd + } + } + if written == "" { + t.Fatal("no recovery configuration was written") + } + // Compared with the shell quoting removed: this is about which settings are + // written, not how many layers of escaping carry them there. + plain := unquoteShell(written) + for _, cleared := range []string{ + "recovery_target = ", "recovery_target_time = ", "recovery_target_name = ", + "recovery_target_xid = ", "recovery_target_lsn = ", + } { + if !strings.Contains(plain, cleared) { + t.Fatalf("recovery configuration does not clear %s:\n%s", cleared, plain) + } + } + if !strings.Contains(written, recoveryBlockStart) || !strings.Contains(written, recoveryBlockEnd) { + t.Fatalf("recovery configuration is not fenced for removal:\n%s", written) + } +} + +// A stated target still reaches PostgreSQL, and does so after the clearing so +// that it is the assignment in effect. +func TestAStatedRecoveryTargetSurvivesTheClearing(t *testing.T) { + fake := &transport.Fake{} + e := recoveryConfigTestEngine(fake) + + if err := e.replayRecovery(context.Background(), "shop-database-restore-1", "2026-08-20T13:58:00Z"); err != nil { + t.Fatalf("replaying to a point in time: %v", err) + } + + var written string + for _, cmd := range fake.Commands { + if strings.Contains(cmd, "postgresql.conf") { + written = cmd + } + } + plain := unquoteShell(written) + target := "recovery_target_time = 2026-08-20 13:58:00+00:00" + if !strings.Contains(plain, target) { + t.Fatalf("stated target missing:\n%s", plain) + } + if strings.Index(plain, target) < strings.Index(plain, "recovery_target_name = ") { + t.Fatalf("the clearing overrides the stated target:\n%s", plain) + } +} + +// The cluster that goes into service must not keep the settings that recovered +// it: every base backup taken from it would carry them into the next recovery. +func TestPromotionRemovesTheRecoveryConfiguration(t *testing.T) { + fake := &transport.Fake{} + e := recoveryConfigTestEngine(fake) + + if err := e.stripRecoveryConfiguration(context.Background(), "shop-database-restore-1"); err != nil { + t.Fatalf("stripping the recovery configuration: %v", err) + } + if len(fake.Commands) != 1 { + t.Fatalf("expected one command, got %v", fake.Commands) + } + cmd := fake.Commands[0] + if !strings.Contains(cmd, "sed -i") || !strings.Contains(cmd, "postgresql.conf") { + t.Fatalf("recovery configuration is not removed: %s", cmd) + } + if !strings.Contains(cmd, "BEGIN onebox recovery") || !strings.Contains(cmd, "END onebox recovery") { + t.Fatalf("removal is not bounded by the markers: %s", cmd) + } +} + +// unquoteShell drops the quoting the command carries so a test can read the +// settings it writes. +func unquoteShell(command string) string { + return strings.NewReplacer("'", "", "\\", "").Replace(command) +} diff --git a/internal/engine/backup_restore.go b/internal/engine/backup_restore.go new file mode 100644 index 00000000..ac395ba8 --- /dev/null +++ b/internal/engine/backup_restore.go @@ -0,0 +1,663 @@ +package engine + +import ( + "context" + "encoding/json" + "fmt" + "strings" + "time" + + "github.com/labstack/onebox/internal/app" +) + +// Recovery. +// +// One path serves both things that need it. A restore and a drill do exactly +// the same work — fetch the base backup, replay WAL to a point in time, start +// the cluster, prove it opens and answers — and differ only in what happens +// afterwards: a restore puts the recovered volume in front of the application, +// a drill throws it away and records that it worked. +// +// That is deliberate. A drill that exercised a different code path from a real +// restore would prove the drill works, which is not the claim anyone needs. + +// A recovered cluster promotes on its own once replay reaches the target, but +// not instantly. Five minutes is generous for a replay that has already fetched +// everything it needs and is measured against a database that is not serving. +const ( + recoveryPromotionBudget = 5 * time.Minute + recoveryPollInterval = 2 * time.Second +) + +// RestoreOutcome is what a recovery produced, whether or not it was kept. +type RestoreOutcome struct { + Service string `json:"service"` + Target string `json:"target"` + Backup string `json:"backup"` + RecoveredTo string `json:"recovered_to"` + Rows string `json:"sanity_check"` + StagingVolume string `json:"staging_volume,omitempty"` + PreviousData string `json:"previous_data_volume,omitempty"` + Promoted bool `json:"promoted"` + // RetainStaging is set the moment promotion starts modifying the live + // volume. From then on the staging volume is the only complete copy of the + // recovered data and must survive a failure, so the operator has something + // to recover from rather than two empty volumes. + RetainStaging bool `json:"-"` +} + +// RecoverService materialises the repository into a fresh volume at a point in +// time, proves the cluster opens, and optionally puts it in front of the +// application. +// +// The recovered cluster is always built beside the live one, never over it. +// Nothing touches the running service until the recovery has already started +// and answered a query — so a repository that cannot actually recover fails +// while the database it would have replaced is still serving. +// +// When promote is false this is a drill: the staging volume is removed and the +// live service is never touched at all. +func (e *Engine) RecoverService(ctx context.Context, service, targetTime string, promote bool) (RestoreOutcome, error) { + _, _, err := e.backedUpService(service) + if err != nil { + return RestoreOutcome{}, err + } + if !e.Spec.ServiceIsProtected(service) { + return RestoreOutcome{}, fmt.Errorf( + "service %s is not running under established backup; there is no repository to recover from", service) + } + if targetTime != "" { + if _, err := time.Parse(time.RFC3339, targetTime); err != nil { + return RestoreOutcome{}, fmt.Errorf( + "recovery target %q is not an RFC 3339 timestamp such as 2026-08-19T15:04:05Z", targetTime) + } + } + // The recorded projection, not the project's current intent. Enablement + // wrote down exactly which repository the server has been archiving to; if + // somebody edits the target afterwards, recovery must still read the + // repository the history is actually in. Rendering follows the same rule. + projection, err := e.Spec.EffectiveBackupProjection(service) + if err != nil { + return RestoreOutcome{}, err + } + target, policyTarget := projection.Target, projection.Policy.Target + + n := e.names() + staging := n.BackupRestoreVolume(service) + container := n.BackupRestoreContainer(service) + outcome := RestoreOutcome{Service: service, Target: targetTime, StagingVolume: staging} + + // Any leftover from an interrupted recovery goes first. Reusing a + // half-populated staging volume is how a restore silently succeeds against + // the wrong data. + if err := e.discardRecoveryStaging(ctx, container, staging); err != nil { + return outcome, err + } + defer func() { + // The container is scratch either way. The volume survives only when it + // has been promoted into service. + _, _ = e.T.Run(context.WithoutCancel(ctx), "docker rm -f "+q(container)+" >/dev/null 2>&1 || true") + // Kept only when promotion began and did not finish — then it is the one + // complete copy of the recovered data and deleting it would leave the + // operator with an empty live volume and nothing to restore from. On + // success the live volume holds that data, so the staging copy goes. + if outcome.Promoted || !outcome.RetainStaging { + _, _ = e.T.Run(context.WithoutCancel(ctx), "docker volume rm -f "+q(staging)+" >/dev/null 2>&1 || true") + } + }() + + image, err := e.Spec.ServiceImageForRuntime(service) + if err != nil { + return outcome, err + } + environment, err := app.WalgEnvironment(target, e.Spec.Spec.Name, service) + if err != nil { + return outcome, err + } + environment["OB_S3_KEY_ENTRY"] = target.Credentials.AccessKeyEntry + environment["OB_S3_SECRET_ENTRY"] = target.Credentials.SecretKeyEntry + if target.Credentials.SessionTokenEntry != "" { + environment["OB_S3_SESSION_TOKEN_ENTRY"] = target.Credentials.SessionTokenEntry + } + + st := e.ui.Step("recovery: fetch base backup", false) + if err := e.startRecoveryContainer(ctx, container, staging, image.Image, environment, service, policyTarget); err != nil { + st(err) + return outcome, err + } + backup, err := e.fetchRecoveryBase(ctx, container, service, targetTime) + if err != nil { + st(err) + return outcome, err + } + outcome.Backup = backup + st(nil) + + st = e.ui.Step("recovery: replay to "+recoveryTargetLabel(targetTime), false) + if err := e.replayRecovery(ctx, container, targetTime); err != nil { + st(err) + return outcome, err + } + st(nil) + + // The proof. A cluster that starts is not the same as a cluster that holds + // the data, and "the restore command exited zero" is exactly the assurance + // this product exists to distrust. + st = e.ui.Step("recovery: verify the recovered cluster answers", false) + recoveredTo, rows, err := e.probeRecoveredCluster(ctx, container, targetTime) + if err != nil { + st(err) + return outcome, err + } + outcome.RecoveredTo, outcome.Rows = recoveredTo, rows + st(nil) + + if !promote { + return outcome, nil + } + previous, err := e.promoteRecoveredVolume(ctx, service, container, staging, &outcome) + outcome.PreviousData = previous + if err != nil { + return outcome, err + } + outcome.Promoted = true + return outcome, nil +} + +func recoveryTargetLabel(targetTime string) string { + if targetTime == "" { + return "the newest recoverable point" + } + return targetTime +} + +func (e *Engine) discardRecoveryStaging(ctx context.Context, container, staging string) error { + // The removals are best-effort — neither may exist — so the *result* is + // what gets checked, not their exit codes. The previous version ended the + // command in `true`, which made its own exit-code guard unreachable: a + // staging volume that could not be removed reported success and recovery + // proceeded into a half-populated volume, which is exactly what this + // function exists to prevent. + if _, err := e.T.Run(ctx, "docker rm -f "+q(container)+" >/dev/null 2>&1; docker volume rm -f "+q(staging)+" >/dev/null 2>&1; true"); err != nil { + return err + } + res, err := e.T.Run(ctx, "docker volume inspect "+q(staging)+" >/dev/null 2>&1 && echo present || echo absent") + if err != nil { + return err + } + if strings.TrimSpace(res.Stdout) != "absent" { + return fmt.Errorf( + "the staging volume %s from a previous recovery could not be removed; recovering into it would mix two restores. Remove it and retry", staging) + } + return nil +} + +// startRecoveryContainer runs the protected image with the staged wal-g mounted +// and the data directory empty, doing nothing. The server is started later, by +// hand, so recovery configuration is in place before it reads anything. +func (e *Engine) startRecoveryContainer(ctx context.Context, container, staging, image string, environment map[string]any, service, credentialTarget string) error { + n := e.names() + args := []string{ + "docker", "run", "-d", "--name", q(container), + "--network", q(n.ServiceNetwork()), + "--entrypoint", "sleep", + "-v", q(staging + ":/var/lib/postgresql/data"), + "-v", q(n.BackupRuntimeDir(service) + ":" + app.WalgMountPath + ":ro"), + "--env-file", q(n.BackupCredentialFile(service, credentialTarget)), + } + for _, key := range sortedEnvKeys(environment) { + args = append(args, "-e", q(fmt.Sprintf("%s=%v", key, environment[key]))) + } + args = append(args, q(image), "infinity") + res, err := e.T.Run(ctx, strings.Join(args, " ")) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot start the recovery container: %s", lastLines(res.Stderr, 3)) + } + prepare := "docker exec " + q(container) + " sh -c " + + q("mkdir -p "+app.PgDataPath+" && chown -R postgres:postgres /var/lib/postgresql/data && chmod 700 "+app.PgDataPath) + res, err = e.T.Run(ctx, prepare) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot prepare the recovery data directory: %s", lastLines(res.Stderr, 3)) + } + return nil +} + +func (e *Engine) fetchRecoveryBase(ctx context.Context, container, service, targetTime string) (string, error) { + // Which base backup, decided here rather than left to wal-g's LATEST. + // + // Replay only moves forward. A base backup that finished after the + // requested point can never reach it: PostgreSQL replays what WAL it has, + // runs out before the target, and dies with "recovery ended before + // configured recovery target was reached". Fetching LATEST unconditionally + // therefore made every point older than the newest base backup + // unrecoverable — with a daily backup and a seven-day window, six of those + // days could not be reached, while `ob backup status` reported the whole + // window as recoverable. + selected := "LATEST" + if targetTime != "" { + chosen, err := e.baseBackupFor(ctx, container, service, targetTime) + if err != nil { + return "", err + } + selected = chosen + } + // Under the repository lock, like every other wal-g invocation. Without it a + // backup timer firing mid-restore runs its retention pass and can expire the + // very generation this is streaming out. The per-service backup lock + // does not help: systemd units cannot take it, which is why the flock exists. + fetch := e.walgLockPrefix(ctx, service) + + "docker exec -u postgres " + q(container) + " " + q(app.WalgBinary) + + " backup-fetch " + q(app.PgDataPath) + " " + q(selected) + res, err := e.T.Run(ctx, fetch) + if err != nil { + return "", err + } + if res.ExitCode != 0 { + return "", fmt.Errorf("cannot fetch the base backup: %s", lastLines(res.Stderr+res.Stdout, 4)) + } + // wal-g names the backup it chose in its log. Trimmed of the punctuation + // the surrounding sentence puts around it, so the label is a label. + for _, line := range strings.Split(res.Stderr+res.Stdout, "\n") { + if index := strings.Index(line, "base_"); index >= 0 { + return strings.Trim(strings.Fields(line[index:])[0], `'".,;:`), nil + } + } + return selected, nil +} + +// baseBackupFor names the newest base backup that finished at or before the +// requested point, which is the only kind replay can carry forward to it. +// +// Read from the recovery container rather than the live service: it already has +// the staged wal-g and the repository credentials, and a recovery must not +// depend on the database it may be about to replace. +func (e *Engine) baseBackupFor(ctx context.Context, container, service, targetTime string) (string, error) { + target, err := time.Parse(time.RFC3339, targetTime) + if err != nil { + return "", err + } + list := e.walgLockPrefix(ctx, service) + + "docker exec -u postgres " + q(container) + " " + q(app.WalgBinary) + + " backup-list --detail --json" + res, err := e.T.Run(ctx, list) + if err != nil { + return "", err + } + if res.ExitCode != 0 { + return "", fmt.Errorf("cannot list the base backups to recover from: %s", lastLines(res.Stderr+res.Stdout, 3)) + } + entries, err := parseWalgBackupList(res.Stdout) + if err != nil { + return "", err + } + if len(entries) == 0 { + return "", fmt.Errorf("the repository holds no base backup, so there is nothing to recover from") + } + var chosen walgBackupEntry + var oldest time.Time + for _, entry := range entries { + if oldest.IsZero() || entry.finished.Before(oldest) { + oldest = entry.finished + } + // At or before the target, and the latest such — the least WAL to + // replay, and the only backups that can reach the point at all. + if !entry.finished.After(target) && (chosen.name == "" || entry.finished.After(chosen.finished)) { + chosen = entry + } + } + if chosen.name == "" { + return "", fmt.Errorf( + "no base backup finished at or before %s, so that point cannot be recovered to; the oldest base backup in this repository finished at %s", + target.UTC().Format(time.RFC3339), oldest.UTC().Format(time.RFC3339)) + } + return chosen.name, nil +} + +type walgBackupEntry struct { + name string + finished time.Time +} + +// parseWalgBackupList reads `backup-list --detail --json`. An entry whose +// completion time cannot be read is refused rather than defaulted: a zero time +// sorts before every real one, so a silent default would make an unreadable +// entry the answer to "what can reach this point". +func parseWalgBackupList(out string) ([]walgBackupEntry, error) { + trimmed := strings.TrimSpace(out) + if trimmed == "" || trimmed == "null" { + return nil, nil + } + var report []struct { + BackupName string `json:"backup_name"` + Time string `json:"time"` + FinishTime string `json:"finish_time"` + } + if err := json.Unmarshal([]byte(trimmed), &report); err != nil { + return nil, fmt.Errorf("wal-g backup-list is not readable JSON") + } + entries := make([]walgBackupEntry, 0, len(report)) + for _, entry := range report { + finish := entry.FinishTime + if finish == "" { + finish = entry.Time + } + finished, err := time.Parse(time.RFC3339, finish) + if err != nil { + return nil, fmt.Errorf("wal-g reported backup %q with an unreadable completion time %q", entry.BackupName, finish) + } + entries = append(entries, walgBackupEntry{name: entry.BackupName, finished: finished}) + } + return entries, nil +} + +// replayRecovery writes the recovery configuration and starts the server, which +// replays WAL until it reaches the target and then promotes. +// +// recovery_target_time is omitted entirely when no target was asked for, which +// means "replay everything available" — the newest recoverable point rather +// than an arbitrary one. +func (e *Engine) replayRecovery(ctx context.Context, container, targetTime string) error { + settings := []string{ + "restore_command = '" + app.WalgBinary + " wal-fetch %f %p'", + "recovery_target_action = 'promote'", + // Every recovery target is cleared before this run states its own, + // because the base backup may carry one. A promoted cluster keeps the + // settings that recovered it in postgresql.conf, so every base backup + // taken after a point-in-time restore contains that restore's target — + // and a later recovery that asks for the newest point inherits a target + // in the past and dies with "recovery ended before configured recovery + // target was reached". Found by drilling a repository that had been + // restored from once; the drill had passed before the restore. + // + // Last assignment wins in postgresql.conf, so clearing here and setting + // below makes this run's target the only one in effect whatever the + // backup carried. + "recovery_target = ''", + "recovery_target_time = ''", + "recovery_target_name = ''", + "recovery_target_xid = ''", + "recovery_target_lsn = ''", + } + if targetTime != "" { + // PostgreSQL wants a timestamptz literal here, not RFC 3339: it refuses + // the `T` separator and the `Z` zone outright and the whole + // configuration file fails to parse. RFC 3339 stays the input format — + // it is the unambiguous one, and an operator typing a recovery point + // should not have to know PostgreSQL's spelling — so it is converted + // here instead. + parsed, err := time.Parse(time.RFC3339, targetTime) + if err != nil { + return err + } + settings = append(settings, "recovery_target_time = '"+parsed.UTC().Format("2006-01-02 15:04:05.999999-07:00")+"'") + } + // Fenced by markers so promotion can take it back out again: see + // stripRecoveryConfiguration. + settings = append([]string{recoveryBlockStart}, settings...) + settings = append(settings, recoveryBlockEnd) + write := "docker exec -u postgres " + q(container) + " sh -c " + + q("printf '%s\\n' "+shellQuoteAll(settings)+" >> "+app.PgDataPath+"/postgresql.conf && touch "+app.PgDataPath+"/recovery.signal") + res, err := e.T.Run(ctx, write) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot write the recovery configuration: %s", lastLines(res.Stderr, 3)) + } + start := "docker exec -u postgres " + q(container) + + " pg_ctl -D " + q(app.PgDataPath) + " -l /tmp/ob-recovery.log -w -t 300 start" + res, err = e.T.Run(ctx, start) + if err != nil { + return err + } + if res.ExitCode != 0 { + log, _ := e.T.Run(ctx, "docker exec "+q(container)+" tail -20 /tmp/ob-recovery.log") + return fmt.Errorf("the recovered cluster did not start: %s", lastLines(log.Stdout, 8)) + } + return nil +} + +// probeRecoveredCluster asks the recovered database whether it will actually +// answer, and reports what it holds. +// +// This is the difference between "the restore command exited zero" and "the +// data is there", which is the whole distinction this product exists to make. +func (e *Engine) probeRecoveredCluster(ctx context.Context, container, targetTime string) (string, string, error) { + query := func(sql string) (string, error) { + command := "docker exec -u postgres " + q(container) + + " psql -U " + q(app.PgSuperuser) + " -d " + q(e.Spec.Spec.Name) + " -tAc " + q(sql) + res, err := e.T.Run(ctx, command) + if err != nil { + return "", err + } + if res.ExitCode != 0 { + return "", fmt.Errorf("the recovered cluster refused a query: %s", lastLines(res.Stderr, 3)) + } + return strings.TrimSpace(res.Stdout), nil + } + // Replay is asynchronous. `pg_ctl -w` returns as soon as the server accepts + // connections, which happens while it is still read-only and still + // replaying — so asking immediately would report every recovery as + // unfinished. Wait for the promotion the recovery configuration asked for. + promoted := false + for waited := time.Duration(0); waited < recoveryPromotionBudget; waited += recoveryPollInterval { + inRecovery, err := query("SELECT pg_is_in_recovery();") + if err != nil { + return "", "", err + } + if inRecovery == "f" { + promoted = true + break + } + select { + case <-ctx.Done(): + return "", "", ctx.Err() + case <-time.After(recoveryPollInterval): + } + } + if !promoted { + return "", "", fmt.Errorf( + "the recovered cluster is still replaying after %s, so it never reached the requested point; "+ + "the WAL needed to reach it may not be archived", recoveryPromotionBudget) + } + tables, err := query("SELECT count(*) FROM information_schema.tables WHERE table_schema='public';") + if err != nil { + return "", "", err + } + // What the cluster actually replayed to, not what time it is now. Reporting + // wall-clock here made a drill say it had recovered to today when it had + // been asked for a point last month — evidence that describes the wrong + // thing is worse than no evidence. + // + // pg_last_xact_replay_timestamp() is empty once a cluster has been promoted + // out of recovery, so the requested target is the authority and this is only + // a fallback for a recovery that had none. + recovered := targetTime + if recovered == "" { + replayed, err := query("SELECT coalesce(to_char(pg_last_xact_replay_timestamp() AT TIME ZONE 'UTC','YYYY-MM-DD\"T\"HH24:MI:SS\"Z\"'), '');") + if err != nil { + return "", "", err + } + recovered = replayed + } + return recovered, tables + " tables in public schema", nil +} + +// promoteRecoveredVolume puts the recovered data in front of the application. +// +// The previous volume is renamed, never deleted. A restore is the operation +// people run when they are already having a bad day, and the one thing it must +// not do is make the bad day unrecoverable — if the recovery turns out to be to +// the wrong second, the original is still there under a dated name. +func (e *Engine) promoteRecoveredVolume(ctx context.Context, service, container, staging string, outcome *RestoreOutcome) (string, error) { + n := e.names() + live := n.ServiceVolume(service, app.DataVolumeFor(e.Spec.Services[service])) + kept := live + "-before-restore-" + time.Now().UTC().Format("20060102T150405Z") + + st := e.ui.Step("recovery: stop the recovered cluster cleanly", false) + stop := "docker exec -u postgres " + q(container) + " pg_ctl -D " + q(app.PgDataPath) + " -w -t 120 -m fast stop" + if res, err := e.T.Run(ctx, stop); err != nil { + st(err) + return "", err + } else if res.ExitCode != 0 { + err := fmt.Errorf("the recovered cluster did not stop cleanly: %s", lastLines(res.Stderr, 3)) + st(err) + return "", err + } + st(nil) + + if err := e.stripRecoveryConfiguration(ctx, container); err != nil { + return "", err + } + + // The live volume is copied aside while it is still intact. Only after that + // copy exists does anything destructive happen. + st = e.ui.Step("recovery: copy the data being replaced aside", false) + preserve := strings.Join([]string{ + "docker compose -p " + q(n.ServiceProject(service)) + " -f " + q(n.ServiceFile(service)) + " down", + "docker volume create " + q(kept), + "docker run --rm -v " + q(live+":/from") + " -v " + q(kept+":/to") + " alpine sh -c 'cp -a /from/. /to/'", + }, " && ") + res, err := e.mutate(ctx, preserve) + if err != nil { + st(err) + return "", err + } + if res.ExitCode != 0 { + err := fmt.Errorf("cannot copy the data being replaced aside, so nothing was changed: %s", lastLines(res.Stderr, 4)) + st(err) + return "", err + } + st(nil) + + // Past this point the live volume is being overwritten, so the staging + // volume is the only complete copy of the recovered data. It must survive a + // failure here — deleting it would leave an empty live volume and nothing to + // recover from. + outcome.RetainStaging = true + + st = e.ui.Step("recovery: put the recovered data in service", false) + // Emptied and refilled in one container rather than removed and recreated: + // a `docker volume rm` that succeeds followed by a `create` that does not + // leaves the service with no volume at all. + swap := strings.Join([]string{ + "docker run --rm -v " + q(live+":/to") + " -v " + q(staging+":/from") + + " alpine sh -c 'find /to -mindepth 1 -maxdepth 1 -exec rm -rf {} + && cp -a /from/. /to/'", + "docker compose -p " + q(n.ServiceProject(service)) + " -f " + q(n.ServiceFile(service)) + " up -d", + }, " && ") + res, err = e.mutate(ctx, swap) + if err != nil { + st(err) + return kept, recoveryPromotionFailure(err, kept, staging) + } + if res.ExitCode != 0 { + err := fmt.Errorf("%s", lastLines(res.Stderr, 4)) + st(err) + return kept, recoveryPromotionFailure(err, kept, staging) + } + st(nil) + + healthy, last, err := e.serviceIsHealthy(ctx, service) + if err != nil { + return kept, err + } + if !healthy { + return kept, fmt.Errorf( + "the restored service did not become healthy (last: %s).\nThe data it replaced is in volume %s, and the recovered data is in %s — neither was deleted", + last, kept, staging) + } + return kept, nil +} + +func sortedEnvKeys(m map[string]any) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + for i := 1; i < len(out); i++ { + for j := i; j > 0 && out[j] < out[j-1]; j-- { + out[j], out[j-1] = out[j-1], out[j] + } + } + return out +} + +func shellQuoteAll(values []string) string { + quoted := make([]string, 0, len(values)) + for _, value := range values { + quoted = append(quoted, q(value)) + } + return strings.Join(quoted, " ") +} + +// ReportRecovery prints what a recovery produced. It is on the engine because +// the engine owns the one UI instance a command shares. +func (e *Engine) ReportRecovery(outcome RestoreOutcome) { + // The point recovered to is the whole claim. A drill that says only "it + // worked" has not said what it proved, and the operator cannot tell a + // recovery to last Tuesday from one to five minutes ago. + point := outcome.RecoveredTo + if point == "" { + point = "the newest recoverable point" + } + if !outcome.Promoted { + e.ui.Successf("drill passed: %s recovered to %s from %s and answered (%s). Nothing was changed.", + outcome.Service, point, outcome.Backup, outcome.Rows) + return + } + e.ui.Successf("%s restored to %s from %s (%s).", outcome.Service, point, outcome.Backup, outcome.Rows) + e.ui.Infof("the data it replaced is kept in volume %s — remove it once you are satisfied", outcome.PreviousData) +} + +// recoveryPromotionFailure names both volumes, because a failure here is the one +// moment an operator has two copies and no running database. An error that said +// only "cannot put the recovered data in service" would leave them looking for +// data they still have. +func recoveryPromotionFailure(err error, kept, staging string) error { + return fmt.Errorf( + "cannot put the recovered data in service: %w.\n"+ + "Nothing was lost: the data being replaced is in volume %s and the recovered data is in %s. "+ + "Restore either into the service volume by hand, or re-run the restore once the cause is fixed", + err, kept, staging) +} + +const ( + recoveryBlockStart = "# BEGIN onebox recovery — removed when the cluster is promoted" + recoveryBlockEnd = "# END onebox recovery" +) + +// stripRecoveryConfiguration takes onebox's recovery settings back out of the +// cluster that is about to go into service. +// +// A promoted cluster that keeps them is not broken today — PostgreSQL ignores +// recovery settings without a recovery.signal — but every base backup taken +// from it carries them, so the next recovery from those backups inherits this +// restore's target and refuses to start. That is a failure the operator meets +// during a real recovery, caused by the previous one. +func (e *Engine) stripRecoveryConfiguration(ctx context.Context, container string) error { + conf := app.PgDataPath + "/postgresql.conf" + strip := "docker exec -u postgres " + q(container) + " sh -c " + + q("sed -i '/^"+sedLiteral(recoveryBlockStart)+"$/,/^"+sedLiteral(recoveryBlockEnd)+"$/d' "+conf) + res, err := e.T.Run(ctx, strip) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("cannot remove the recovery configuration from the promoted cluster: %s", lastLines(res.Stderr, 3)) + } + return nil +} + +// sedLiteral escapes the characters sed reads as syntax inside an address. The +// markers are fixed strings in this file, so this is a guard against editing +// them into something sed would misread rather than against hostile input. +func sedLiteral(text string) string { + replacer := strings.NewReplacer("\\", "\\\\", "/", "\\/", ".", "\\.", "*", "\\*", "[", "\\[", "]", "\\]", "^", "\\^", "$", "\\$") + return replacer.Replace(text) +} diff --git a/internal/engine/backup_schedule.go b/internal/engine/backup_schedule.go new file mode 100644 index 00000000..cb9b651e --- /dev/null +++ b/internal/engine/backup_schedule.go @@ -0,0 +1,333 @@ +package engine + +import ( + "context" + "fmt" + "strings" + + "github.com/labstack/onebox/internal/app" +) + +// Backup schedules. +// +// A backup policy is a promise about time. Declaring `schedule: {cron: "0 2 * +// * *"}` and then only ever backing up when somebody types the command is not +// a weaker version of that promise — it is a different thing wearing its +// clothes, and it fails silently on exactly the night nobody was watching. +// +// Two timers per protected service, taken straight from the policy rather than +// invented: +// +// - the backup schedule takes a base backup and then applies retention, in +// that order, so the repository is never briefly below the number of +// generations the policy promises; +// - the drill schedule verifies the archived WAL forms an unbroken +// chain, which is the check a green backup does not imply. +// +// They run wal-g directly rather than through `ob`, because there is no `ob` on +// the target — Onebox is agentless, and the only thing it has already placed +// there is the verified binary these units invoke. +// +// That agentlessness is also why the drill schedule verifies rather than +// actually restoring. A real drill recovers into a throwaway volume and proves +// the cluster answers, and that orchestration lives in `ob`. Reimplementing it +// in a unit file would give a drill that exercises a different path from a real +// restore — which proves the drill works, not the backups. So the unattended +// half is the check that can be made honestly here, and `ob backup drill` +// remains the whole proof, to be run from CI or a workstation on the same +// cadence the policy declares. + +// SyncBackupSchedules installs a timer for every protected service and +// removes the timers of services that are no longer protected. +// +// Removal matters as much as installation. A timer left behind for a service +// whose backup was disabled would keep pushing backups to a repository the +// project no longer describes, and nothing in the project would explain why. +func (e *Engine) SyncBackupSchedules(ctx context.Context) error { + n := e.names() + prefix := app.BackupUnitPrefix + e.Spec.Spec.Name + "-" + e.Opts.Environment + "-" + // flock creates the lock file but not the directory holding it. + if res, err := e.T.Run(ctx, "mkdir -p "+q(n.AppDir()+"/backup")); err != nil { + return err + } else if res.ExitCode != 0 { + return fmt.Errorf("cannot create the backup directory: %s", strings.TrimSpace(res.Stderr)) + } + + res, err := e.T.Run(ctx, "systemctl list-unit-files --no-legend --type=timer 2>/dev/null | awk '{print $1}'") + if err != nil { + return err + } + installed := map[string]bool{} + for _, line := range strings.Split(res.Stdout, "\n") { + unit := strings.TrimSpace(line) + if strings.HasPrefix(unit, prefix) && strings.HasSuffix(unit, ".timer") { + installed[strings.TrimSuffix(unit, ".timer")] = true + } + } + + if err := e.RequireBackupScheduling(ctx, backedUpServiceNames(e.Spec)); err != nil { + return err + } + + type wantedUnit struct { + name string + calendar string + cron string + body string + } + var wanted []wantedUnit + for _, service := range e.Spec.ServiceNames() { + if !e.Spec.ServiceIsProtected(service) { + continue + } + projection, err := e.Spec.EffectiveBackupProjection(service) + if err != nil { + return err + } + container := n.ServiceContainer(service) + prune, err := pruneExec(container, projection.Policy) + if err != nil { + return fmt.Errorf("service %s retention: %w", service, err) + } + for _, unit := range []struct { + operation string + schedule app.Schedule + commands []string + }{ + {"backup", projection.Policy.Schedule, []string{ + walgExec(container, "backup-push", app.PgDataPath), + // Retention after the new generation exists, never before. + prune, + }}, + {"verify", projection.Policy.Drill.Schedule, []string{ + "/bin/sh " + n.BackupVerifyScript(service), + }}, + } { + calendar, err := app.CronToCalendar(unit.schedule.Cron) + if err != nil { + return fmt.Errorf("service %s %s schedule: %w", service, unit.operation, err) + } + expression := calendar + if unit.schedule.Timezone != "" { + expression += " " + unit.schedule.Timezone + } + check, err := e.T.Run(ctx, "systemd-analyze calendar "+q(expression)+" >/dev/null 2>&1 && echo ok") + if err != nil { + return err + } + if strings.TrimSpace(check.Stdout) != "ok" { + return fmt.Errorf( + "service %s: the host rejected the calendar expression %q derived from cron %q. "+ + "A timezone in OnCalendar needs systemd 252 or newer; on an older host, declare the schedule in UTC", + service, expression, unit.schedule.Cron) + } + wanted = append(wanted, wantedUnit{ + name: n.BackupUnitForEnvironment(e.Opts.Environment, service, unit.operation), + calendar: expression, + cron: unit.schedule.Cron, + body: backupServiceUnit(e.Spec.Spec.Name, service, unit.operation, n.BackupRunLock(service), unit.commands), + }) + } + } + + // The verify unit runs a script rather than wal-g directly, because wal-g + // reports a broken WAL chain and exits 0 — so a unit that invoked it + // straight through would be marked successful by systemd over a repository + // with holes in it, for as long as nobody looked. The interactive command + // judges the same report in Go; this is the unattended half of it. + for _, service := range backedUpServiceNames(e.Spec) { + if err := e.writeServiceFile(ctx, n.BackupVerifyScript(service), + []byte(backupVerifyScript(n.ServiceContainer(service)))); err != nil { + return fmt.Errorf("cannot install the archive verification for %s: %w", service, err) + } + } + + wantedNames := map[string]bool{} + for _, unit := range wanted { + wantedNames[unit.name] = true + if err := e.writeServiceFile(ctx, "/etc/systemd/system/"+unit.name+".service", []byte(unit.body)); err != nil { + return fmt.Errorf("cannot install %s: %w", unit.name, err) + } + if err := e.writeServiceFile(ctx, "/etc/systemd/system/"+unit.name+".timer", + []byte(backupTimerUnit(unit.name, unit.calendar))); err != nil { + return fmt.Errorf("cannot install %s timer: %w", unit.name, err) + } + } + + var stale []string + for unit := range installed { + if !wantedNames[unit] { + stale = append(stale, unit) + } + } + for _, unit := range sortedNames(setOf(stale)) { + if err := e.mutateChecked(ctx, "remove backup schedule "+unit, fmt.Sprintf( + "systemctl disable --now %s.timer >/dev/null 2>&1 && rm -f /etc/systemd/system/%s.timer /etc/systemd/system/%s.service", + unit, unit, unit)); err != nil { + return err + } + e.logf("backup schedule: removed %s (no longer protected)", unit) + } + + if len(wanted) == 0 && len(stale) == 0 { + return nil + } + if res, err := e.mutate(ctx, "systemctl daemon-reload"); err != nil { + return err + } else if res.ExitCode != 0 { + return fmt.Errorf("systemctl daemon-reload: %s", strings.TrimSpace(res.Stderr)) + } + for _, unit := range wanted { + if res, err := e.mutate(ctx, "systemctl enable --now "+unit.name+".timer"); err != nil { + return err + } else if res.ExitCode != 0 { + return fmt.Errorf("cannot start %s: %s", unit.name, strings.TrimSpace(res.Stderr)) + } + e.logf("backup schedule: %s at %s", unit.name, unit.cron) + } + return nil +} + +// backupVerifyScript is the scheduled archive check. +// +// wal-g prints its integrity table and its status line and exits 0 whatever it +// found, so the exit code cannot be the verdict. MISSING_UPLOADING is a segment +// the server may still be uploading and resolves itself; MISSING_LOST and +// MISSING_DELAYED are holes, and a base backup plus a gapped WAL stream +// recovers to the backup and no further. +// +// Kept to POSIX sh: this runs from a systemd unit on whatever the host ships. +func backupVerifyScript(container string) string { + return strings.Join([]string{ + "#!/bin/sh", + "# Written by Onebox. Edits are overwritten on the next apply.", + "set -u", + "report=$(" + walgExec(container, "wal-verify", "integrity", "timeline") + " 2>&1)", + "status=$?", + `printf '%s\n' "$report"`, + `[ "$status" -eq 0 ] || exit "$status"`, + `if printf '%s' "$report" | grep -Eq 'MISSING_(LOST|DELAYED)'; then`, + ` echo "onebox: the archived WAL has gaps; any point after the first gap is not recoverable" >&2`, + " exit 1", + "fi", + "exit 0", + "", + }, "\n") +} + +func walgExec(container string, args ...string) string { + parts := []string{"/usr/bin/docker", "exec", "-u", "postgres", container, app.WalgBinary} + return strings.Join(append(parts, args...), " ") +} + +// pruneExec applies retention, with the count derived from both declared +// bounds. See app.WalgRetainCount for why the window becomes a count rather +// than a timestamp. +func pruneExec(container string, policy app.BackupPolicy) (string, error) { + retain, err := app.WalgRetainCount(policy) + if err != nil { + return "", err + } + return walgExec(container, "delete", "retain", "FULL", fmt.Sprint(retain), "--confirm"), nil +} + +// backupServiceUnit runs the operation's commands in order, under a lock. +// +// flock is what keeps a timer from running while an interactive `ob backup` +// command is already talking to the same repository. Onebox's own backup +// lock is a value written to a file and cannot be taken from a shell, so both +// sides take this one instead: the engine wraps every wal-g invocation in the +// same flock, which makes it the single mutex over actual repository work. +// +// -w rather than -n: a backup that waits for a running one is late, and a +// backup that gives up is missing. +func backupServiceUnit(application, service, operation, lockPath string, commands []string) string { + lines := []string{ + "[Unit]", + "Description=Onebox backup " + operation + " for " + service + " (" + application + ")", + "# Written by Onebox. Edits are overwritten on the next apply.", + "After=docker.service", + "Requires=docker.service", + "", + "[Service]", + "Type=oneshot", + } + for _, command := range commands { + lines = append(lines, "ExecStart=/usr/bin/flock -w 3600 "+lockPath+" "+command) + } + return strings.Join(append(lines, ""), "\n") +} + +func backupTimerUnit(unit, calendar string) string { + return strings.Join([]string{ + "[Unit]", + "Description=Onebox backup schedule " + unit, + "# Written by Onebox. Edits are overwritten on the next apply.", + "", + "[Timer]", + // The timezone belongs in the expression. `Timezone=` is not a [Timer] + // directive: systemd ignores it silently and evaluates the calendar in + // the host's zone. + "OnCalendar=" + calendar, + // A box that was off at 2am still takes the backup when it comes back. + // For a backup this is not a convenience — it is the difference between + // a gap in the recovery window and a late entry in it. + "Persistent=true", + // Spread across a minute so several services on one box do not all + // start pushing to the same repository on the same second. + "RandomizedDelaySec=60", + "", + "[Install]", + "WantedBy=timers.target", + "", + }, "\n") +} + +func backedUpServiceNames(resolved *app.Resolved) []string { + var out []string + for _, service := range resolved.ServiceNames() { + if resolved.ServiceIsProtected(service) { + out = append(out, service) + } + } + return out +} + +// RequireBackupScheduling refuses a host that cannot run the schedules a +// protected service needs. +// +// Called by enablement before anything durable happens, as well as by the +// schedule sync itself. Discovering it only at the sync would mean finding out +// after the service had already been recorded as protected and restarted +// archiving — a half-applied enablement whose only symptom is a failed command. +func (e *Engine) RequireBackupScheduling(ctx context.Context, protected []string) error { + if len(protected) == 0 { + return nil + } + // Refused rather than skipped. A host with no systemd can run a protected + // database perfectly well and will never take a scheduled backup, and a + // warning at the foot of an otherwise green apply is how that goes + // unnoticed until it matters. + probe, err := e.T.Run(ctx, "command -v systemctl >/dev/null 2>&1 && echo ok") + if err != nil { + return err + } + if strings.TrimSpace(probe.Stdout) != "ok" { + return fmt.Errorf( + "this host has no systemctl, so the backup schedules declared for %s cannot be installed "+ + "and no backup would ever run unattended.\n"+ + "Run them from elsewhere on the declared cadence (`ob backup create`, `ob backup prune`, "+ + "`ob backup verify`), or use a host with systemd", + strings.Join(protected, ", ")) + } + // The units serialise themselves with flock. A host that schedules but + // cannot lock would run a backup and a retention pass over the same + // repository at once. + if !e.hasFlock(ctx) { + return fmt.Errorf( + "this host has systemd but no flock, so the scheduled backups for %s could run over each other. "+ + "Install util-linux", + strings.Join(protected, ", ")) + } + return nil +} diff --git a/internal/engine/engine.go b/internal/engine/engine.go index a45772e2..b32bf0bd 100644 --- a/internal/engine/engine.go +++ b/internal/engine/engine.go @@ -80,11 +80,15 @@ type Engine struct { // Spec is what the author declared; Compose is what Compose parsed from // the rendered runtime. Both are named for their source so that "project" // never stands for two different things in this package. - Spec *app.Resolved - Compose *ctypes.Project - T transport.Transport - Opts Options - ui *ui.UI + Spec *app.Resolved + // flockProbed/flockPresent cache whether the target has flock, which every + // wal-g invocation needs to know and which cannot change mid-operation. + flockProbed bool + flockPresent bool + Compose *ctypes.Project + T transport.Transport + Opts Options + ui *ui.UI // fenceVal is " " once WriteFence has stamped the host; // mutate() guards every mutating command with it. @@ -92,11 +96,11 @@ type Engine struct { lockVal string hostLockVal string hostLockToken string - // Protection locks are per-service and may only be acquired while this + // Backup locks are per-service and may only be acquired while this // engine owns the application lock. Exact lock and fence values make stale // lifecycle runners fail closed after takeover. - protectionLockVals map[string]string - protectionFenceVals map[string]string + backupLockVals map[string]string + backupFenceVals map[string]string // gateOpen is the explicit no-effect result; rollbackCovered also includes // the interrupted deploy's typed policy promises. Resume restores both from // the journal. They are closed by default — fail safe. diff --git a/internal/engine/fixtures_test.go b/internal/engine/fixtures_test.go index 56e5399a..abdc00a9 100644 --- a/internal/engine/fixtures_test.go +++ b/internal/engine/fixtures_test.go @@ -49,9 +49,9 @@ services: version: 17 deployment: order: [web, worker] -verifications: - - workload: web - http: /healthz +checks: + http: + - {workload: web, path: /healthz} ` func testConfig() *app.Resolved { diff --git a/internal/engine/hooks_test.go b/internal/engine/hooks_test.go index 445e18cf..2bc16d20 100644 --- a/internal/engine/hooks_test.go +++ b/internal/engine/hooks_test.go @@ -59,7 +59,7 @@ func TestAdvisoryURLCheckWarnsButPasses(t *testing.T) { f := happyFake() var out bytes.Buffer cfg := testConfig() - cfg.Verifications = append(cfg.Verifications, app.Verification{URL: srv.URL, Advisory: true}) + cfg.Checks.URL = append(cfg.Checks.URL, app.URLCheck{URL: srv.URL, Advisory: true}) e := New(cfg, testProject(t), f, Options{Out: &out, Sleep: noSleep}) if err := e.Verify(context.Background()); err != nil { t.Fatalf("advisory failure must not fail verify: %v", err) @@ -76,7 +76,7 @@ func TestAuthoritativeURLCheckFails(t *testing.T) { defer srv.Close() f := happyFake() cfg := testConfig() - cfg.Verifications = append(cfg.Verifications, app.Verification{URL: srv.URL, Contains: `id="root"`}) + cfg.Checks.URL = append(cfg.Checks.URL, app.URLCheck{URL: srv.URL, Contains: `id="root"`}) e := New(cfg, testProject(t), f, Options{Out: &bytes.Buffer{}, Sleep: noSleep, HTTPTimeout: 2 * time.Second}) if err := e.Verify(context.Background()); err == nil { t.Fatal("non-advisory url check with missing substring must fail") diff --git a/internal/engine/lock.go b/internal/engine/lock.go index f11b4489..ee4a1b9d 100644 --- a/internal/engine/lock.go +++ b/internal/engine/lock.go @@ -171,7 +171,7 @@ func (e *Engine) lockTTL() time.Duration { // `stat -c %Y` is GNU; `stat -f %m` is the BSD/macOS fallback (the e2e suite // drives a macOS box through the Local transport). path is quoted here, so // callers pass it raw. Callers that expose `--break-lock` apply it before the refusal -// default. AcquireLock also reclaims a same-deploy holder; protection locks do +// default. AcquireLock also reclaims a same-deploy holder; backup locks do // not expose the generic break override. func lockAgeCmd(path string) string { qpath := q(path) diff --git a/internal/engine/migration_backup.go b/internal/engine/migration_backup.go index 94eabde2..788231e3 100644 --- a/internal/engine/migration_backup.go +++ b/internal/engine/migration_backup.go @@ -59,7 +59,7 @@ func (e *Engine) migrationBackupRequired() bool { return false } environment, ok := e.Spec.Environments[e.Opts.Environment] - return ok && environment.Policy.RequireMigrationBackup + return ok && environment.Policy.Migrations.RequireBackup } func (e *Engine) hasPendingMigration(done map[string]bool) bool { diff --git a/internal/engine/migration_backup_test.go b/internal/engine/migration_backup_test.go index da937717..93acece0 100644 --- a/internal/engine/migration_backup_test.go +++ b/internal/engine/migration_backup_test.go @@ -23,8 +23,8 @@ func migrationBackupEngineConfig() *app.Resolved { worker.Volumes = []app.Volume{{Name: "data", Path: "/data", Mode: "rw"}} cfg.Workloads["worker"] = worker environment := cfg.Environments["production"] - environment.Policy.RequireMigrationBackup = true - environment.Policy.MigrationBackupMaximumAge = "24h" + environment.Policy.Migrations.RequireBackup = true + environment.Policy.Migrations.BackupMaxAge = "24h" cfg.Environments["production"] = environment return cfg } diff --git a/internal/engine/plan.go b/internal/engine/plan.go index 2b84db6f..ce33b455 100644 --- a/internal/engine/plan.go +++ b/internal/engine/plan.go @@ -494,7 +494,12 @@ func (e *Engine) Describe(remoteCompose string) []string { head += fmt.Sprintf(", %d replicas → %s..%s", n, slots[0], slots[len(slots)-1]) } out = append(out, head+"):") - out = append(out, " "+cc+" pull --quiet "+svc) + // The plan shows the pull only when the release will actually run one, + // because a plan listing a command that does not happen is a plan + // nobody can check against. + if line := e.plannedPullLine(svc, cc); line != "" { + out = append(out, line) + } if role.Mode() == "rolling" { step := " per replica: " out = append(out, diff --git a/internal/engine/protection_credentials.go b/internal/engine/protection_credentials.go deleted file mode 100644 index 2fda04b2..00000000 --- a/internal/engine/protection_credentials.go +++ /dev/null @@ -1,106 +0,0 @@ -package engine - -import ( - "context" - "crypto/sha256" - "encoding/hex" - "errors" - "fmt" - "os" - "path/filepath" - "regexp" - "sort" - "strings" - "time" -) - -var protectionCredentialEntry = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]{0,127}$`) - -// InstallProtectionCredentialFile moves already-resolved credential material -// through a private upload into its target-side mode-0600 file. Secret bytes -// are never interpolated into a command, journal, result, or error. -func (e *Engine) InstallProtectionCredentialFile(ctx context.Context, service, target string, requiredEntries []string, plaintext []byte) (string, error) { - if !protectionIdentity.MatchString(service) || !protectionIdentity.MatchString(target) { - return "", errors.New("protection credential service and target identities are invalid") - } - entries, err := protectionCredentialEntries(plaintext) - if err != nil { - return "", err - } - requiredEntries = append([]string(nil), requiredEntries...) - sort.Strings(requiredEntries) - for _, entry := range requiredEntries { - if !protectionCredentialEntry.MatchString(entry) { - return "", errors.New("protection credential contract contains an invalid slot") - } - if !entries[entry] { - return "", fmt.Errorf("protection credential file is missing required entry %s", entry) - } - } - - localStaging, err := os.MkdirTemp("", "ob-protection-credentials-") - if err != nil { - return "", errors.New("create private protection credential staging") - } - defer os.RemoveAll(localStaging) - const stagedName = "credentials.env" - if err := os.WriteFile(filepath.Join(localStaging, stagedName), plaintext, 0o600); err != nil { - return "", errors.New("write private protection credential staging") - } - - names := e.names() - destination := names.ProtectionCredentialFile(service, target) - tokenBytes := sha256.Sum256([]byte(e.protectionFenceVals[service] + "\x00" + target)) - token := hex.EncodeToString(tokenBytes[:])[:16] - remoteStaging := names.AppDir() + "/protection/.credential-staging-" + service + "-" + token - if err := e.T.Upload(ctx, localStaging, remoteStaging); err != nil { - return "", errors.New("upload private protection credential staging") - } - // Cleanup cannot use ProtectionMutate: a failed/fenced install is exactly - // when that guard may refuse the command. The paths are deterministic, - // narrowly scoped staging artifacts, and best-effort removal must run even - // after cancellation so plaintext is not stranded on the host. - defer func() { - cleanupContext, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - _, _ = e.T.Run(cleanupContext, "rm -rf "+q(remoteStaging)+"; rm -f "+q(destination+".tmp")) - }() - install := "mkdir -p " + q(names.ProtectionSecretDir()) + - " && chmod 700 " + q(names.ProtectionSecretDir()) + - " && cp " + q(remoteStaging+"/"+stagedName) + " " + q(destination+".tmp") + - " && chmod 600 " + q(destination+".tmp") + - " && mv -f " + q(destination+".tmp") + " " + q(destination) + - " && rm -rf " + q(remoteStaging) - result, err := e.ProtectionMutate(ctx, service, install) - if err != nil { - return "", errors.New("install target-side protection credential file") - } - if result.ExitCode != 0 { - return "", errors.New("install target-side protection credential file failed") - } - return destination, nil -} - -func protectionCredentialEntries(plaintext []byte) (map[string]bool, error) { - entries := make(map[string]bool) - for index, line := range strings.Split(string(plaintext), "\n") { - trimmed := strings.TrimSpace(line) - if trimmed == "" || strings.HasPrefix(trimmed, "#") { - continue - } - trimmed = strings.TrimPrefix(trimmed, "export ") - entry, _, ok := strings.Cut(trimmed, "=") - entry = strings.TrimSpace(entry) - if !ok || !protectionCredentialEntry.MatchString(entry) { - return nil, fmt.Errorf("protection credential file has an invalid entry at line %d", index+1) - } - if entries[entry] { - return nil, fmt.Errorf("protection credential file repeats entry %s", entry) - } - entries[entry] = true - } - if len(entries) == 0 { - return nil, errors.New("protection credential file has no entries") - } - return entries, nil -} diff --git a/internal/engine/protection_lock.go b/internal/engine/protection_lock.go deleted file mode 100644 index a2492e63..00000000 --- a/internal/engine/protection_lock.go +++ /dev/null @@ -1,261 +0,0 @@ -package engine - -import ( - "context" - "encoding/json" - "errors" - "fmt" - "regexp" - "strconv" - "strings" - "time" - - "github.com/labstack/onebox/internal/journal" - "github.com/labstack/onebox/internal/transport" -) - -var ( - ErrProtectionConflict = errors.New("backup_conflict") - ErrProtectionFenced = errors.New("protection operation fenced by a newer owner") - protectionIdentity = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._:-]{0,127}$`) -) - -const protectionLockPollInterval = 100 * time.Millisecond - -type protectionLockMeta struct { - Owner string `json:"owner"` - OperationID string `json:"operation_id"` - Service string `json:"service"` - Epoch int `json:"epoch"` - TTLSeconds int `json:"ttl_s"` - AcquiredAt string `json:"acquired_at"` -} - -// ProtectionConflictError is safe to serialize as a lifecycle failure. The -// code is stable and the holder identity is operational metadata, never a -// command, credential, or database value. -type ProtectionConflictError struct { - Service string - OperationID string - AgeSeconds int -} - -func (err *ProtectionConflictError) Error() string { - return fmt.Sprintf("backup_conflict: service %s is held by operation %s (age %ds)", err.Service, err.OperationID, err.AgeSeconds) -} - -func (err *ProtectionConflictError) Unwrap() error { return ErrProtectionConflict } -func (err *ProtectionConflictError) Code() string { return "backup_conflict" } -func (err *ProtectionConflictError) Retryable() bool { - return true -} - -func (e *Engine) protectionLockDir() string { return e.base() + "/protection/locks" } -func (e *Engine) protectionLockPath(service string) string { - return e.protectionLockDir() + "/" + service + ".lock" -} -func (e *Engine) protectionEpochPath(service string) string { - return e.protectionLockDir() + "/" + service + ".epoch" -} -func (e *Engine) protectionFencePath(service string) string { - return e.protectionLockDir() + "/" + service + ".fence" -} - -// AcquireProtectionLock acquires the per-service lock beneath the application -// lock. wait is a bounded contention budget; expiry returns backup_conflict. -// An expired lock or the same operation identity is reclaimed with a new epoch, -// fencing the former runner. -func (e *Engine) AcquireProtectionLock(ctx context.Context, service, operationID string, wait time.Duration) (int, error) { - if e.lockVal == "" { - return 0, errors.New("protection lock requires the application lock") - } - if !protectionIdentity.MatchString(service) || !protectionIdentity.MatchString(operationID) { - return 0, errors.New("protection lock service and operation identity are invalid") - } - if wait < 0 { - return 0, errors.New("protection lock wait must not be negative") - } - if err := ctx.Err(); err != nil { - return 0, err - } - if res, err := e.T.Run(ctx, "mkdir -p "+q(e.protectionLockDir())); err != nil { - return 0, err - } else if res.ExitCode != 0 { - return 0, fmt.Errorf("create protection lock directory: %s", strings.TrimSpace(res.Stderr)) - } - - maxAttempts := 1 - if wait > 0 { - maxAttempts += int((wait + protectionLockPollInterval - 1) / protectionLockPollInterval) - } - var conflict *ProtectionConflictError - staleReclaims := 0 - for attempt := 0; attempt < maxAttempts; attempt++ { - if err := ctx.Err(); err != nil { - return 0, err - } - epoch, err := e.nextProtectionEpoch(ctx, service) - if err != nil { - return 0, err - } - meta := protectionLockMeta{ - Owner: journal.DefaultOperator(), OperationID: operationID, Service: service, - Epoch: epoch, TTLSeconds: int(e.lockTTL().Seconds()), AcquiredAt: e.Opts.Now().UTC().Format(time.RFC3339), - } - encoded, _ := json.Marshal(meta) - lockValue := string(encoded) - create := "set -C; echo " + q(lockValue) + " > " + q(e.protectionLockPath(service)) + " 2>/dev/null" - res, err := e.T.Run(ctx, create) - if err != nil { - return 0, err - } - if res.ExitCode == 0 { - if err := e.writeProtectionFence(ctx, service, operationID, epoch, lockValue); err != nil { - return 0, err - } - return epoch, nil - } - - observedResult, err := e.T.Run(ctx, "cat "+q(e.protectionLockPath(service))+" 2>/dev/null || true") - if err != nil { - return 0, err - } - observed := strings.TrimSpace(observedResult.Stdout) - if observed == "" { - continue - } - var holder protectionLockMeta - _ = json.Unmarshal([]byte(observed), &holder) - ageResult, err := e.T.Run(ctx, lockAgeCmd(e.protectionLockPath(service))) - if err != nil { - return 0, err - } - age, _ := strconv.Atoi(strings.TrimSpace(ageResult.Stdout)) - if age > int(e.lockTTL().Seconds()) || holder.OperationID == operationID { - if staleReclaims >= 4 { - return 0, errors.New("could not reclaim stale protection lock") - } - staleReclaims++ - removeObserved := `if [ "$(cat ` + q(e.protectionLockPath(service)) + ` 2>/dev/null)" = ` + q(observed) + ` ]; then rm -f ` + q(e.protectionLockPath(service)) + `; else exit 75; fi` - removed, err := e.T.Run(ctx, removeObserved) - if err != nil { - return 0, err - } - if removed.ExitCode == 0 || removed.ExitCode == 75 { - attempt-- // a stale-holder race does not consume contention budget - continue - } - return 0, fmt.Errorf("reclaim protection lock: %s", strings.TrimSpace(removed.Stderr)) - } - conflict = &ProtectionConflictError{Service: service, OperationID: safeProtectionHolder(holder.OperationID), AgeSeconds: age} - if attempt+1 < maxAttempts { - e.Opts.Sleep(protectionLockPollInterval) - } - } - if conflict == nil { - conflict = &ProtectionConflictError{Service: service, OperationID: "unknown", AgeSeconds: 0} - } - return 0, conflict -} - -func (e *Engine) nextProtectionEpoch(ctx context.Context, service string) (int, error) { - result, err := e.T.Run(ctx, "cat "+q(e.protectionEpochPath(service))+" 2>/dev/null || echo 0") - if err != nil { - return 0, err - } - previous, _ := strconv.Atoi(strings.TrimSpace(result.Stdout)) - return previous + 1, nil -} - -func (e *Engine) writeProtectionFence(ctx context.Context, service, operationID string, epoch int, lockValue string) error { - fenceValue := operationID + " " + strconv.Itoa(epoch) - command := `if [ "$(cat ` + q(e.protectionLockPath(service)) + ` 2>/dev/null)" = ` + q(lockValue) + ` ]; then echo ` + strconv.Itoa(epoch) + ` > ` + q(e.protectionEpochPath(service)) + ` && echo ` + q(fenceValue) + ` > ` + q(e.protectionFencePath(service)) + `; else echo ob-protection-lock-lost >&2; exit 96; fi` - result, err := e.T.Run(ctx, command) - if err != nil { - return err - } - if result.ExitCode == 96 && strings.Contains(result.Stderr, "ob-protection-lock-lost") { - return ErrProtectionFenced - } - if result.ExitCode != 0 { - return fmt.Errorf("write protection fence: %s", strings.TrimSpace(result.Stderr)) - } - if e.protectionLockVals == nil { - e.protectionLockVals = make(map[string]string) - e.protectionFenceVals = make(map[string]string) - } - e.protectionLockVals[service] = lockValue - e.protectionFenceVals[service] = fenceValue - return nil -} - -func (e *Engine) ReleaseProtectionLock(service string) { - expected := e.protectionLockVals[service] - if expected == "" { - return - } - cleanupContext, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - result, err := e.T.Run(cleanupContext, `if [ "$(cat `+q(e.protectionLockPath(service))+` 2>/dev/null)" = `+q(expected)+` ]; then rm -f `+q(e.protectionLockPath(service))+`; fi`) - if err != nil || result.ExitCode != 0 { - e.warnf("release protection lock failed: %v %s", err, strings.TrimSpace(result.Stderr)) - return - } - delete(e.protectionLockVals, service) - delete(e.protectionFenceVals, service) -} - -// StartProtectionHeartbeat keeps a service lock fresh only while both its -// exact lock value and fence still belong to this runner. -func (e *Engine) StartProtectionHeartbeat(ctx context.Context, service string) (func(), error) { - lockValue := e.protectionLockVals[service] - fenceValue := e.protectionFenceVals[service] - if lockValue == "" || fenceValue == "" { - return nil, errors.New("protection heartbeat requires service lock ownership") - } - heartbeatContext, cancel := context.WithCancel(ctx) - done := make(chan struct{}) - go func() { - defer close(done) - ticker := time.NewTicker(e.lockTTL() / 10) - defer ticker.Stop() - for { - select { - case <-heartbeatContext.Done(): - return - case <-ticker.C: - command := `if [ "$(cat ` + q(e.protectionLockPath(service)) + ` 2>/dev/null)" = ` + q(lockValue) + ` ] && [ "$(cat ` + q(e.protectionFencePath(service)) + ` 2>/dev/null)" = ` + q(fenceValue) + ` ]; then touch -c ` + q(e.protectionLockPath(service)) + `; else exit 3; fi` - if result, err := e.T.Run(heartbeatContext, command); err == nil && result.ExitCode != 0 && result.ExitCode != 3 { - e.warnf("protection heartbeat for %s failed (exit %d): %s", service, result.ExitCode, strings.TrimSpace(result.Stderr)) - } - } - } - }() - return func() { cancel(); <-done }, nil -} - -// ProtectionMutate nests the exact service lock/fence guard inside the app -// fence guard. A runner that loses either authority cannot mutate service data. -func (e *Engine) ProtectionMutate(ctx context.Context, service, command string) (transport.Result, error) { - lockValue := e.protectionLockVals[service] - fenceValue := e.protectionFenceVals[service] - if e.lockVal == "" || e.fenceVal == "" || lockValue == "" || fenceValue == "" { - return transport.Result{}, errors.New("protection mutation requires application and service lock ownership") - } - guarded := `if [ "$(cat ` + q(e.protectionLockPath(service)) + ` 2>/dev/null)" = ` + q(lockValue) + ` ] && [ "$(cat ` + q(e.protectionFencePath(service)) + ` 2>/dev/null)" = ` + q(fenceValue) + ` ]; then ` + command + `; else echo ob-protection-fenced >&2; exit 98; fi` - result, err := e.mutate(ctx, guarded) - if err != nil { - return result, err - } - if result.ExitCode == 98 && strings.Contains(result.Stderr, "ob-protection-fenced") { - return result, ErrProtectionFenced - } - return result, nil -} - -func safeProtectionHolder(value string) string { - if protectionIdentity.MatchString(value) { - return value - } - return "unknown" -} diff --git a/internal/engine/proxystatus.go b/internal/engine/proxystatus.go index d52ade3e..892227ed 100644 --- a/internal/engine/proxystatus.go +++ b/internal/engine/proxystatus.go @@ -188,10 +188,11 @@ func readableFileProbe(path string) string { // file — and that stays a failed read so the snapshot reports the component as // incomplete rather than quietly asserting it was checked. func statusFileIssue(component, path string, result transport.Result) (string, bool) { - // Every sentence leads with the component. statusIssueCodes derives a - // branchable code by matching this prose, and a message that led with the - // path instead collapsed to the generic _diverged — the same - // indistinguishability the owner refusal was given a code to avoid. + // Every sentence leads with the component, so a reader — or a consumer + // matching on the text — can tell which one a message is about without + // parsing the rest of it. (An unreachable classifier in internal/onebox + // used to depend on this and was the stated reason for the rule; the rule + // is worth keeping on its own, which is why the prose has not changed.) switch result.ExitCode { case app.ProbeUnreadable: return fmt.Sprintf("%s exists but could not be read; verify the file and its permissions", component), true diff --git a/internal/engine/recreate.go b/internal/engine/recreate.go index 76d6bc19..7c7c46e7 100644 --- a/internal/engine/recreate.go +++ b/internal/engine/recreate.go @@ -28,10 +28,8 @@ func (e *Engine) recreateRoleForRelease(ctx context.Context, roleName, remoteCom cc := e.composeCmd(remoteComposePath) desired := role.Count() - if res, err := e.mutate(ctx, cc+" pull --quiet "+svc); err != nil { + if err := e.pullBeforeRelease(ctx, svc, cc); err != nil { return err - } else if res.ExitCode != 0 { - return fmt.Errorf("pull %s: %s", svc, res.Stderr) } // Signal before recreate whenever the contract declares a fixed drain wait. // Compose sends TERM during replacement too, but doing it only then skipped diff --git a/internal/engine/resume_test.go b/internal/engine/resume_test.go index 9d14ab61..b0c31b50 100644 --- a/internal/engine/resume_test.go +++ b/internal/engine/resume_test.go @@ -107,7 +107,7 @@ func TestResumeUsesInterruptedReleaseSnapshotAfterConfigEdit(t *testing.T) { cfg.Workloads = map[string]app.Workload{"web": cfg.Workloads["web"]} cfg.Deployment.Order = []string{"web"} cfg.Services = nil - cfg.Verifications = nil + cfg.Checks = app.Checks{} e := New(cfg, testProject(t), f, Options{Out: &bytes.Buffer{}, Sleep: noSleep}) if err := e.Resume(context.Background()); err != nil { t.Fatalf("resume: %v\n%s", err, strings.Join(f.Commands, "\n")) @@ -396,7 +396,7 @@ func TestAbortUsesBothReleaseSnapshotsAfterConfigEdit(t *testing.T) { cfg.Workloads = map[string]app.Workload{"migrate": cfg.Workloads["migrate"]} cfg.Deployment.Order = nil cfg.Services = nil - cfg.Verifications = nil + cfg.Checks = app.Checks{} e := New(cfg, testProject(t), f, Options{Out: &bytes.Buffer{}, Sleep: noSleep}) if err := e.Abort(context.Background(), false); err != nil { t.Fatalf("abort: %v\n%s", err, strings.Join(f.Commands, "\n")) diff --git a/internal/engine/roll.go b/internal/engine/roll.go index e55f2ae3..74ef2e53 100644 --- a/internal/engine/roll.go +++ b/internal/engine/roll.go @@ -97,10 +97,8 @@ func (e *Engine) RollRole(ctx context.Context, roleName, remoteComposePath strin // surge one new replica if we still need more of the new release if len(news) < desired { if !pulled { - if res, err := e.mutate(ctx, cc+" pull --quiet "+svc); err != nil { + if err := e.pullBeforeRelease(ctx, svc, cc); err != nil { return err - } else if res.ExitCode != 0 { - return fmt.Errorf("pull %s: %s", svc, res.Stderr) } pulled = true } @@ -369,3 +367,70 @@ func (e *Engine) waitHealth(ctx context.Context, id, want string, budget, interv e.Opts.Sleep(interval) } } + +// pullPolicyFor is the workload's declared `image.pull`, defaulted by the +// loader to "missing". +func (e *Engine) pullPolicyFor(roleName string) string { + if e.Spec == nil || e.Spec.Spec == nil { + return "missing" + } + role, ok := e.Spec.Workloads[roleName] + if !ok || role.Image == nil || role.Image.Pull == "" { + return "missing" + } + return role.Image.Pull +} + +// pullBeforeRelease fetches the workload image unless it does not need +// fetching. +// +// `image.pull` was a schema key the loader defaulted, validation checked and +// the reference documented — and nothing read. Every release ran +// `docker compose pull` for every workload on every deploy, including for an +// image already pinned by digest and already on the host, which is a request to +// the registry that cannot change the outcome. A rate-limited registry then +// failed a deploy that had nothing to fetch, and a project declaring +// `pull: never` was pulled from anyway. +// +// always: fetch. never: do not, and let the release fail on a missing image +// rather than reaching out. missing (the default): fetch only what the host +// does not already hold, which for a digest-pinned image is an exact answer. +func (e *Engine) pullBeforeRelease(ctx context.Context, roleName, composeCommand string) error { + policy := e.pullPolicyFor(roleName) + if policy == "never" { + return nil + } + // A nil Compose means nothing has been rendered to compare against, so the + // only safe answer is to pull. + if policy == "missing" && e.Compose != nil { + service, ok := e.Compose.Services[roleName] + if ok && containsDigest(service.Image) { + held, err := e.imagePresentByDigest(ctx, service.Image) + if err != nil { + return err + } + if held { + return nil + } + } + } + res, err := e.mutate(ctx, composeCommand+" pull --quiet "+roleName) + if err != nil { + return err + } + if res.ExitCode != 0 { + return fmt.Errorf("pull %s: %s", roleName, res.Stderr) + } + return nil +} + +// plannedPullLine is what pullBeforeRelease will do, for the plan preview. It +// answers from the plan's own resolved images rather than probing the host: the +// plan is already bound to those digests, and a preview that reached out would +// be doing the work it is describing. +func (e *Engine) plannedPullLine(roleName, composeCommand string) string { + if e.pullPolicyFor(roleName) == "never" { + return "" + } + return " " + composeCommand + " pull --quiet " + roleName +} diff --git a/internal/engine/schedule.go b/internal/engine/schedule.go index 8c0329f4..4344209b 100644 --- a/internal/engine/schedule.go +++ b/internal/engine/schedule.go @@ -48,6 +48,14 @@ func (e *Engine) SyncSchedules(ctx context.Context) error { installed := map[string]bool{} for _, line := range strings.Split(res.Stdout, "\n") { unit := strings.TrimSpace(line) + // Backups own their own namespace and reconciles it separately. Its + // units begin "ob-backup-", which also begins with this prefix when + // the application is literally named "backup" — belt and braces, + // because the failure mode is a deploy silently deleting every + // scheduled backup. + if strings.HasPrefix(unit, app.BackupUnitPrefix) { + continue + } if strings.HasPrefix(unit, prefix) && strings.HasSuffix(unit, ".timer") { installed[strings.TrimSuffix(unit, ".timer")] = true } diff --git a/internal/engine/schedule_test.go b/internal/engine/schedule_test.go index 38122c13..c9482489 100644 --- a/internal/engine/schedule_test.go +++ b/internal/engine/schedule_test.go @@ -62,3 +62,20 @@ func TestRemoveSchedulesRejectsFailedDisable(t *testing.T) { t.Fatalf("schedule removal continued after disable failure:\n%s", seq) } } + +// A deploy must not delete the backup timers. +// +// SyncSchedules owns "ob--*" and removes what the project no longer +// declares. Backup timers were named inside that namespace, so every deploy +// reclaimed them as stale and silently stopped all scheduled backups — the only +// trace being a line saying the schedule was "no longer declared". +func TestSyncSchedulesLeavesBackupTimersAlone(t *testing.T) { + if !strings.HasPrefix(app.BackupUnitPrefix, "ob-") { + t.Fatalf("backup prefix %q is expected to sit under the ob- namespace", app.BackupUnitPrefix) + } + backupTimer := app.Names{App: "example", BasePath: "/var/lib/ob"}. + BackupTimerForEnvironment("production", "database", "backup") + if strings.HasPrefix(backupTimer, "ob-example-") { + t.Fatalf("backup timer %q is inside the job scheduler's namespace and a deploy would delete it", backupTimer) + } +} diff --git a/internal/engine/scheduled_protection.go b/internal/engine/scheduled_protection.go deleted file mode 100644 index 87241d2f..00000000 --- a/internal/engine/scheduled_protection.go +++ /dev/null @@ -1,179 +0,0 @@ -package engine - -import ( - "context" - "errors" - "fmt" - "regexp" - "time" - - "github.com/labstack/onebox/internal/journal" -) - -var ( - ErrScheduledRunnerCrash = errors.New("scheduled runner stopped before terminal journal commit") - scheduledEvidenceID = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$`) -) - -type ScheduledProtectionRequest struct { - OperationID string - OperationKind string - Service string - RetryIdentity string - LockWait time.Duration - HelperProvenance *journal.HelperProvenance -} - -type ScheduledProtectionAction func(context.Context, *Engine) (string, error) - -type scheduledCodedError interface { - Code() string -} - -type scheduledRetryableError interface { - Retryable() bool -} - -// ExecuteScheduledProtection applies the same app lock, service lock, fences, -// heartbeat, deterministic journal identity, retry classification, provenance, -// and redaction boundary used by interactive lifecycle execution. -func (e *Engine) ExecuteScheduledProtection(ctx context.Context, request ScheduledProtectionRequest, action ScheduledProtectionAction) error { - if action == nil { - return errors.New("scheduled protection action is nil") - } - if request.LockWait < 0 { - return errors.New("scheduled protection lock wait must not be negative") - } - for name, value := range map[string]string{ - "operation_id": request.OperationID, "operation_kind": request.OperationKind, - "service": request.Service, "retry_identity": request.RetryIdentity, - } { - if !scheduledEvidenceID.MatchString(value) { - return fmt.Errorf("scheduled protection %s is invalid", name) - } - } - stepID, err := journal.ProtectionStepID(request.OperationKind, request.Service, request.RetryIdentity) - if err != nil { - return err - } - epoch, err := e.AcquireLock(ctx, request.OperationID, false) - if err != nil { - return err - } - defer e.ReleaseLock(ctx) - if err := e.WriteFence(ctx, request.OperationID, epoch); err != nil { - return err - } - stopAppHeartbeat := e.StartHeartbeat(ctx) - defer stopAppHeartbeat() - - writer := &journal.Writer{ - T: e.T, Names: e.names(), DeployID: request.OperationID, Epoch: epoch, - Operator: journal.DefaultOperator(), GitSHA: e.Opts.GitSHA, ConfigHash: e.Opts.ConfigHash, Runner: &e.Opts.Runner, - } - if terminal, ok, err := journal.LookupProtectionTerminalResult(ctx, e.T, e.names(), request.OperationID); err != nil { - return err - } else if ok { - switch terminal.State { - case "succeeded": - return nil - case "failed", "cancelled": - return &scheduledTerminalError{state: terminal.State, code: terminal.ErrorCode} - } - } - attempt := 1 - if previous, ok, err := journal.LookupProtectionStep(ctx, e.T, e.names(), request.OperationID, stepID); err != nil { - return err - } else if ok && previous.ProtectionAttempt >= attempt { - attempt = previous.ProtectionAttempt + 1 - } - baseRecord := journal.Record{ - Phase: "scheduled", SubStep: request.RetryIdentity, Event: "start", Status: "ok", - OperationKind: request.OperationKind, Service: request.Service, - ProtectionStepID: stepID, ProtectionAttempt: attempt, HelperProvenance: cloneScheduledHelper(request.HelperProvenance), - } - if err := writer.AppendProtection(ctx, baseRecord); err != nil { - return err - } - if _, err := e.AcquireProtectionLock(ctx, request.Service, request.OperationID, request.LockWait); err != nil { - return appendScheduledProtectionResult(ctx, writer, baseRecord, "", err) - } - defer e.ReleaseProtectionLock(request.Service) - stopProtectionHeartbeat, err := e.StartProtectionHeartbeat(ctx, request.Service) - if err != nil { - return appendScheduledProtectionResult(ctx, writer, baseRecord, "", err) - } - defer stopProtectionHeartbeat() - - evidenceID, actionErr := action(ctx, e) - if errors.Is(actionErr, ErrScheduledRunnerCrash) { - return actionErr - } - if actionErr == nil && !scheduledEvidenceID.MatchString(evidenceID) { - actionErr = errors.New("scheduled execution returned an invalid evidence identity") - } - return appendScheduledProtectionResult(ctx, writer, baseRecord, evidenceID, actionErr) -} - -type scheduledTerminalError struct { - state string - code string -} - -func (err *scheduledTerminalError) Error() string { - return fmt.Sprintf("scheduled operation already %s with code %s", err.state, err.code) -} - -func (err *scheduledTerminalError) Code() string { return err.code } - -func appendScheduledProtectionResult( - ctx context.Context, - writer *journal.Writer, - baseRecord journal.Record, - evidenceID string, - actionErr error, -) error { - resultRecord := baseRecord - resultRecord.Event = "result" - resultRecord.TerminalResult = &journal.ProtectionTerminalResult{State: "succeeded", EvidenceID: evidenceID} - if actionErr != nil { - code, retryable := classifyScheduledProtectionError(actionErr) - state, retryClass := "failed", "terminal" - if errors.Is(actionErr, context.Canceled) || errors.Is(actionErr, context.DeadlineExceeded) { - state, code = "cancelled", "operation_cancelled" - } else if retryable { - state, retryClass = "incomplete", "retryable" - } - resultRecord.Status, resultRecord.ErrorCode = "fail", code - resultRecord.Retry = &journal.RetryClassification{Class: retryClass, ReasonCode: code} - resultRecord.TerminalResult = &journal.ProtectionTerminalResult{State: state, ErrorCode: code} - } - journalContext := ctx - var cancel context.CancelFunc - if ctx.Err() != nil || errors.Is(actionErr, context.Canceled) || errors.Is(actionErr, context.DeadlineExceeded) { - journalContext, cancel = context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - } - if err := writer.AppendProtection(journalContext, resultRecord); err != nil { - return errors.Join(actionErr, err) - } - return actionErr -} - -func classifyScheduledProtectionError(err error) (string, bool) { - code := "scheduled_execution_failed" - var coded scheduledCodedError - if errors.As(err, &coded) && scheduledEvidenceID.MatchString(coded.Code()) { - code = coded.Code() - } - var retryable scheduledRetryableError - return code, errors.As(err, &retryable) && retryable.Retryable() -} - -func cloneScheduledHelper(helper *journal.HelperProvenance) *journal.HelperProvenance { - if helper == nil { - return nil - } - copy := *helper - return © -} diff --git a/internal/engine/scheduled_protection_test.go b/internal/engine/scheduled_protection_test.go deleted file mode 100644 index 71d9bfc1..00000000 --- a/internal/engine/scheduled_protection_test.go +++ /dev/null @@ -1,237 +0,0 @@ -package engine - -import ( - "context" - "encoding/json" - "errors" - "io" - "os" - "strings" - "testing" - "time" - - "github.com/labstack/onebox/internal/app" - "github.com/labstack/onebox/internal/journal" - "github.com/labstack/onebox/internal/transport" -) - -func scheduledProtectionTestEngine(t *testing.T, target transport.Transport) *Engine { - t.Helper() - return New( - &app.Resolved{Spec: &app.Spec{Name: "example", BasePath: t.TempDir()}, Env: "production"}, - nil, - target, - Options{ - Out: io.Discard, LockTTL: time.Minute, Sleep: func(time.Duration) {}, - Now: func() time.Time { return time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC) }, - }, - ) -} - -func scheduledProtectionRequest(operationID string) ScheduledProtectionRequest { - digest := "sha256:" + strings.Repeat("a", 64) - return ScheduledProtectionRequest{ - OperationID: operationID, OperationKind: "backup_create", Service: "database", RetryIdentity: "daily-20260807", - HelperProvenance: &journal.HelperProvenance{ - Repository: "example/backup-helper", Digest: digest, SBOMDigest: digest, ProvenanceID: "onebox/catalog/test-helper/v1", - }, - } -} - -func readScheduledProtectionJournal(t *testing.T, engine *Engine, operationID string) []journal.Record { - t.Helper() - records, err := journal.Read(context.Background(), engine.T, engine.names(), operationID) - if err != nil { - t.Fatalf("read scheduled protection journal: %v", err) - } - return records -} - -func TestExecuteScheduledProtectionUsesCanonicalLocksFencesAndJournal(t *testing.T) { - engine := scheduledProtectionTestEngine(t, transport.NewLocal()) - request := scheduledProtectionRequest("scheduled-backup-1") - actionCalls := 0 - - err := engine.ExecuteScheduledProtection(context.Background(), request, func(ctx context.Context, executing *Engine) (string, error) { - actionCalls++ - if executing.lockVal == "" || executing.fenceVal == "" || executing.protectionLockVals[request.Service] == "" || executing.protectionFenceVals[request.Service] == "" { - t.Fatal("scheduled action ran outside the canonical lock and fence boundary") - } - if result, err := executing.ProtectionMutate(ctx, request.Service, "true"); err != nil || result.ExitCode != 0 { - t.Fatalf("fenced scheduled mutation = %#v, %v", result, err) - } - return "backup-generation-1", nil - }) - if err != nil { - t.Fatalf("execute scheduled protection: %v", err) - } - if actionCalls != 1 { - t.Fatalf("action calls = %d, want 1", actionCalls) - } - records := readScheduledProtectionJournal(t, engine, request.OperationID) - if len(records) != 2 { - t.Fatalf("journal records = %d, want start and result", len(records)) - } - result := records[1] - if result.TerminalResult == nil || result.TerminalResult.State != "succeeded" || result.TerminalResult.EvidenceID != "backup-generation-1" { - t.Fatalf("terminal result = %#v", result.TerminalResult) - } - if result.HelperProvenance == nil || result.HelperProvenance.Digest != request.HelperProvenance.Digest { - t.Fatalf("helper provenance = %#v", result.HelperProvenance) - } - if engine.lockVal != "" || len(engine.protectionLockVals) != 0 { - t.Fatal("scheduled execution retained lock ownership after completion") - } -} - -func TestExecuteScheduledProtectionRetriesAfterCrashWithSameIdentity(t *testing.T) { - engine := scheduledProtectionTestEngine(t, transport.NewLocal()) - request := scheduledProtectionRequest("scheduled-backup-crash") - - if err := engine.ExecuteScheduledProtection(context.Background(), request, func(context.Context, *Engine) (string, error) { - return "", ErrScheduledRunnerCrash - }); !errors.Is(err, ErrScheduledRunnerCrash) { - t.Fatalf("crash error = %v", err) - } - if records := readScheduledProtectionJournal(t, engine, request.OperationID); len(records) != 1 || records[0].ProtectionAttempt != 1 || records[0].TerminalResult != nil { - t.Fatalf("crash journal = %#v", records) - } - - if err := engine.ExecuteScheduledProtection(context.Background(), request, func(context.Context, *Engine) (string, error) { - return "backup-generation-after-retry", nil - }); err != nil { - t.Fatalf("retry after crash: %v", err) - } - records := readScheduledProtectionJournal(t, engine, request.OperationID) - if len(records) != 3 || records[1].ProtectionAttempt != 2 || records[2].ProtectionAttempt != 2 || records[2].TerminalResult == nil || records[2].TerminalResult.State != "succeeded" { - t.Fatalf("retry journal = %#v", records) - } -} - -func TestExecuteScheduledProtectionPersistsCancellationAndDoesNotRerunTerminalIdentity(t *testing.T) { - engine := scheduledProtectionTestEngine(t, transport.NewLocal()) - request := scheduledProtectionRequest("scheduled-backup-cancel") - ctx, cancel := context.WithCancel(context.Background()) - - err := engine.ExecuteScheduledProtection(ctx, request, func(actionContext context.Context, _ *Engine) (string, error) { - cancel() - return "", actionContext.Err() - }) - if !errors.Is(err, context.Canceled) { - t.Fatalf("cancellation error = %v", err) - } - records := readScheduledProtectionJournal(t, engine, request.OperationID) - if len(records) != 2 || records[1].TerminalResult == nil || records[1].TerminalResult.State != "cancelled" || records[1].TerminalResult.ErrorCode != "operation_cancelled" { - t.Fatalf("cancellation journal = %#v", records) - } - - reran := false - err = engine.ExecuteScheduledProtection(context.Background(), request, func(context.Context, *Engine) (string, error) { - reran = true - return "unexpected", nil - }) - var terminal *scheduledTerminalError - if !errors.As(err, &terminal) || terminal.Code() != "operation_cancelled" || reran { - t.Fatalf("terminal retry = %v, reran = %v", err, reran) - } -} - -func TestExecuteScheduledProtectionRecordsLockContention(t *testing.T) { - engine := scheduledProtectionTestEngine(t, transport.NewLocal()) - request := scheduledProtectionRequest("scheduled-backup-conflict") - if err := os.MkdirAll(engine.protectionLockDir(), 0o700); err != nil { - t.Fatal(err) - } - holder := protectionLockMeta{ - Owner: "operator", OperationID: "restore-running", Service: request.Service, - Epoch: 3, TTLSeconds: 60, AcquiredAt: time.Now().UTC().Format(time.RFC3339), - } - encoded, _ := json.Marshal(holder) - if err := os.WriteFile(engine.protectionLockPath(request.Service), encoded, 0o600); err != nil { - t.Fatal(err) - } - - err := engine.ExecuteScheduledProtection(context.Background(), request, func(context.Context, *Engine) (string, error) { - t.Fatal("contended scheduled action ran") - return "", nil - }) - if !errors.Is(err, ErrProtectionConflict) { - t.Fatalf("contention error = %v", err) - } - records := readScheduledProtectionJournal(t, engine, request.OperationID) - if len(records) != 2 || records[1].TerminalResult == nil || records[1].TerminalResult.State != "incomplete" || records[1].TerminalResult.ErrorCode != "backup_conflict" || records[1].Retry == nil || records[1].Retry.Class != "retryable" { - t.Fatalf("contention journal = %#v", records) - } -} - -type disconnectAfterResultAppend struct { - transport.Transport - disconnected bool -} - -func (target *disconnectAfterResultAppend) Run(ctx context.Context, command string) (transport.Result, error) { - result, err := target.Transport.Run(ctx, command) - if err == nil && !target.disconnected && strings.Contains(command, "printf '%s\\n'") && strings.Contains(command, `"event":"result"`) { - target.disconnected = true - return result, errors.New("client disconnected after durable append") - } - return result, err -} - -func TestExecuteScheduledProtectionReconcilesDisconnectedClient(t *testing.T) { - target := &disconnectAfterResultAppend{Transport: transport.NewLocal()} - engine := scheduledProtectionTestEngine(t, target) - request := scheduledProtectionRequest("scheduled-backup-disconnect") - actionCalls := 0 - - if err := engine.ExecuteScheduledProtection(context.Background(), request, func(context.Context, *Engine) (string, error) { - actionCalls++ - return "backup-generation-disconnect", nil - }); err != nil { - t.Fatalf("reconcile disconnected client: %v", err) - } - if !target.disconnected || actionCalls != 1 { - t.Fatalf("disconnect/action = %v/%d", target.disconnected, actionCalls) - } - if err := engine.ExecuteScheduledProtection(context.Background(), request, func(context.Context, *Engine) (string, error) { - actionCalls++ - return "unexpected", nil - }); err != nil { - t.Fatalf("terminal retry after disconnect: %v", err) - } - if actionCalls != 1 { - t.Fatalf("action calls after terminal retry = %d, want 1", actionCalls) - } -} - -type scheduledTestFailure struct { - message string -} - -func (failure scheduledTestFailure) Error() string { return failure.message } -func (scheduledTestFailure) Code() string { return "backup_target_unreachable" } -func (scheduledTestFailure) Retryable() bool { return true } - -func TestExecuteScheduledProtectionRedactsFailureJournal(t *testing.T) { - engine := scheduledProtectionTestEngine(t, transport.NewLocal()) - request := scheduledProtectionRequest("scheduled-backup-redaction") - const secret = "storage-secret-canary" - - err := engine.ExecuteScheduledProtection(context.Background(), request, func(context.Context, *Engine) (string, error) { - return "", scheduledTestFailure{message: "provider rejected " + secret} - }) - if err == nil { - t.Fatal("scheduled failure unexpectedly succeeded") - } - records := readScheduledProtectionJournal(t, engine, request.OperationID) - encoded, marshalErr := json.Marshal(records) - if marshalErr != nil { - t.Fatal(marshalErr) - } - if strings.Contains(string(encoded), secret) { - t.Fatalf("scheduled journal leaked secret: %s", encoded) - } - if len(records) != 2 || records[1].ErrorCode != "backup_target_unreachable" || records[1].Detail != "operation failed; inspect trusted local diagnostics" { - t.Fatalf("redacted failure journal = %#v", records) - } -} diff --git a/internal/engine/service_apply.go b/internal/engine/service_apply.go index 45cb465c..f261271a 100644 --- a/internal/engine/service_apply.go +++ b/internal/engine/service_apply.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + "path" "strings" "github.com/pmezard/go-difflib/difflib" @@ -95,6 +96,27 @@ func (e *Engine) ServiceApply(ctx context.Context, releaseID string, allowDestru if strings.Contains(src[1], "/releases/") { continue } + // Everything Onebox stages for backup — the verified wal-g + // binary and the generated credential wrapper — is mounted + // read-only and replaced from the project on every apply. Treating + // it as data would make each apply of a protected service demand + // --allow-destructive-mounts to detach files Onebox wrote itself, + // which teaches operators to pass that flag by reflex — the exact + // habit it exists to prevent. The path is keyed by wal-g version, + // so an upgrade legitimately changes it. + if strings.HasPrefix(src[1], path.Join(n.AppDir(), "backup")+"/") { + continue + } + // Anonymous volumes are Docker's, not the project's. The official + // PostgreSQL image declares VOLUME /var/lib/postgresql while + // Onebox mounts the data one level below it, so every container + // gets a scratch volume holding an otherwise empty directory. It + // is named with a 64-character hex id nothing declares, it is + // recreated whenever the container is, and reporting it as data + // about to detach is a false alarm on every single apply. + if isAnonymousVolume(src[0], src[1]) { + continue + } if !newSet[m] { destructive = append(destructive, acc+": "+m) } @@ -183,3 +205,10 @@ func (e *Engine) refuseUnsafeMajorUpgrade(ctx context.Context, n app.Names) erro } return nil } + +func isAnonymousVolume(kind, name string) bool { + if kind != "volume" || len(name) != 64 { + return false + } + return strings.TrimLeft(name, "0123456789abcdef") == "" +} diff --git a/internal/engine/service_image_cache_test.go b/internal/engine/service_image_cache_test.go index 6e3be753..323b159f 100644 --- a/internal/engine/service_image_cache_test.go +++ b/internal/engine/service_image_cache_test.go @@ -30,7 +30,7 @@ func TestExactServiceImageCachedRequiresMatchingRepositoryDigest(t *testing.T) { } return transport.Result{}, false }} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) got, err := engine.ExactServiceImageCached(context.Background(), image) if err != nil && test.name != "configuration-id-only" { t.Fatal(err) @@ -47,7 +47,7 @@ func TestExactServiceImageCachedRequiresMatchingRepositoryDigest(t *testing.T) { func TestExactServiceImageCachedRejectsMutableTag(t *testing.T) { fake := &transport.Fake{} - engine := protectionLockTestEngine(fake) + engine := backupLockTestEngine(fake) if _, err := engine.ExactServiceImageCached(context.Background(), "postgres:17"); err == nil { t.Fatal("mutable tag was accepted for exact cache evidence") } diff --git a/internal/engine/services.go b/internal/engine/services.go index ad8575a1..7c9b65f5 100644 --- a/internal/engine/services.go +++ b/internal/engine/services.go @@ -81,6 +81,21 @@ func (e *Engine) ApplyServices(ctx context.Context) error { if err != nil { return err } + // Before anything starts that mounts them. Protected services only; this + // is a no-op for every service that is not. + wrappers, err := e.Spec.RenderServiceBackupWrappers(e.Opts.Environment) + if err != nil { + return err + } + staging := e.Spec.NamesFor(e.Opts.Environment) + for _, name := range names { + if !e.Spec.ServiceIsProtected(name) { + continue + } + if err := e.StageBackupRuntime(ctx, name, wrappers[staging.BackupWrapperFile(name)]); err != nil { + return fmt.Errorf("service %s: cannot place its backup runtime: %w", name, err) + } + } if err := e.EnsureServiceConnections(ctx); err != nil { return err } @@ -128,6 +143,11 @@ func (e *Engine) ApplyServices(ctx context.Context) error { return fmt.Errorf("service %s: cannot record its version: %w", name, err) } } + // After the services are up, because a timer that fires against a container + // that is not running yet is a failed backup in the journal for no reason. + if err := e.SyncBackupSchedules(ctx); err != nil { + return fmt.Errorf("cannot converge the backup schedules: %w", err) + } return nil } diff --git a/internal/engine/verify.go b/internal/engine/verify.go index ec45714a..b636e059 100644 --- a/internal/engine/verify.go +++ b/internal/engine/verify.go @@ -21,7 +21,7 @@ import ( const maxVerificationBodyBytes = 1 << 20 // verifyURL is the runner-side edge check (ob.sh's smoke test, absorbed). -func (e *Engine) verifyURL(ctx context.Context, chk app.Verification) error { +func (e *Engine) verifyURL(ctx context.Context, chk app.RunnableCheck) error { label := verificationURLLabel(chk.URL) client := &http.Client{ Timeout: e.Opts.HTTPTimeout, @@ -241,7 +241,7 @@ func scalarNumberString(value any) (string, bool) { // the edge, because an edge blip must not fail a healthy release. URL // checks go through the edge from the runner and are advisory territory. func (e *Engine) Verify(ctx context.Context) error { - for _, chk := range e.Spec.Verifications { + for _, chk := range e.Spec.Checks.All() { if chk.MigrationRevisions != nil { assertion := chk.MigrationRevisions result, ok := e.jobResults[assertion.Job] diff --git a/internal/engine/verify_contract_test.go b/internal/engine/verify_contract_test.go index d177ffd2..2c533130 100644 --- a/internal/engine/verify_contract_test.go +++ b/internal/engine/verify_contract_test.go @@ -23,7 +23,7 @@ func TestVerifyURLSupportsStatusAndHeaderContracts(t *testing.T) { defer srv.Close() e := verificationTestEngine(io.Discard) - check := app.Verification{ + check := app.RunnableCheck{ URL: srv.URL, StatusCodes: []int{http.StatusCreated, http.StatusNoContent}, RequiredHeaders: map[string]string{"content-type": "application/json", "X-Release": "r42"}, @@ -52,7 +52,7 @@ func TestVerifyURLDoesNotFollowRedirects(t *testing.T) { defer srv.Close() e := verificationTestEngine(io.Discard) - check := app.Verification{ + check := app.RunnableCheck{ URL: srv.URL + "/start", StatusCodes: []int{http.StatusFound}, RequiredHeaders: map[string]string{"Location": "/final"}, @@ -77,7 +77,7 @@ func TestVerifyURLSupportsDottedJSONScalarAssertions(t *testing.T) { })) defer srv.Close() - check := app.Verification{ + check := app.RunnableCheck{ URL: srv.URL, JSONAssertions: []app.JSONAssertion{ {Path: "service.ready", Equals: true}, @@ -104,13 +104,13 @@ func TestVerifyURLFailureRedactsConfiguredAndResponseValues(t *testing.T) { defer srv.Close() e := verificationTestEngine(io.Discard) - headerErr := e.verifyURL(context.Background(), app.Verification{ + headerErr := e.verifyURL(context.Background(), app.RunnableCheck{ URL: srv.URL + "?token=" + querySecret, RequiredHeaders: map[string]string{"X-Token": expectedSecret}, }) assertVerificationSecretsAbsent(t, headerErr, querySecret, expectedSecret, actualSecret) - jsonErr := e.verifyURL(context.Background(), app.Verification{ + jsonErr := e.verifyURL(context.Background(), app.RunnableCheck{ URL: srv.URL + "?token=" + querySecret, JSONAssertions: []app.JSONAssertion{ {Path: "token", Equals: expectedSecret}, @@ -118,7 +118,7 @@ func TestVerifyURLFailureRedactsConfiguredAndResponseValues(t *testing.T) { }) assertVerificationSecretsAbsent(t, jsonErr, querySecret, expectedSecret, actualSecret) - containsErr := e.verifyURL(context.Background(), app.Verification{ + containsErr := e.verifyURL(context.Background(), app.RunnableCheck{ URL: srv.URL + "?token=" + querySecret, Contains: expectedSecret, }) @@ -131,7 +131,7 @@ func TestVerifyURLBoundsBodiesUsedByAssertions(t *testing.T) { })) defer srv.Close() - err := verificationTestEngine(io.Discard).verifyURL(context.Background(), app.Verification{ + err := verificationTestEngine(io.Discard).verifyURL(context.Background(), app.RunnableCheck{ URL: srv.URL, Contains: "x", }) @@ -147,7 +147,7 @@ func TestVerifyURLDoesNotExposeInvalidJSONBody(t *testing.T) { })) defer srv.Close() - err := verificationTestEngine(io.Discard).verifyURL(context.Background(), app.Verification{ + err := verificationTestEngine(io.Discard).verifyURL(context.Background(), app.RunnableCheck{ URL: srv.URL, JSONAssertions: []app.JSONAssertion{{Path: "ready", Equals: true}}, }) @@ -160,7 +160,7 @@ func TestVerifyURLRequestErrorRedactsQuery(t *testing.T) { url := srv.URL srv.Close() - err := verificationTestEngine(io.Discard).verifyURL(context.Background(), app.Verification{ + err := verificationTestEngine(io.Discard).verifyURL(context.Background(), app.RunnableCheck{ URL: url + "?token=" + querySecret, }) assertVerificationSecretsAbsent(t, err, querySecret) @@ -175,7 +175,7 @@ func TestVerifyURLSuccessOutputRedactsQuery(t *testing.T) { var out bytes.Buffer cfg := testConfig() - cfg.Verifications = []app.Verification{{URL: srv.URL + "?token=" + querySecret}} + cfg.Checks = app.Checks{URL: []app.URLCheck{{URL: srv.URL + "?token=" + querySecret}}} e := New(cfg, testProject(t), happyFake(), Options{Out: &out, Sleep: noSleep}) if err := e.Verify(context.Background()); err != nil { t.Fatal(err) @@ -190,7 +190,7 @@ func TestVerifyMigrationRevisionsMatchesBoundProviderEvidence(t *testing.T) { cfg.Workloads = map[string]app.Workload{ "migrate": {Role: app.RoleJob, When: "pre_release", DataEffect: "migration"}, } - cfg.Verifications = []app.Verification{{MigrationRevisions: &app.MigrationRevs{ + cfg.Checks = app.Checks{Migrations: []app.MigrationCheck{{ Job: "migrate", Provider: "atlas", AppliedRevisions: []string{"r1", "r2"}, }}} e := New(cfg, testProject(t), happyFake(), Options{Out: io.Discard, Sleep: noSleep}) diff --git a/internal/engine/verify_injection_test.go b/internal/engine/verify_injection_test.go index 3645fc2a..20f22515 100644 --- a/internal/engine/verify_injection_test.go +++ b/internal/engine/verify_injection_test.go @@ -74,9 +74,9 @@ workloads: role: application image: ghcr.io/x/app:v2 health: {http: /healthz, port: 7500, interval: 5s, start_period: 5s, within: 120s} -verifications: - - workload: web - http: `+path+` +checks: + http: + - {workload: web, path: `+path+`} `), "ob.yml") if err != nil { t.Fatalf("the project grammar rejected %q, so this path cannot reach the engine: %v", path, err) diff --git a/internal/journal/journal.go b/internal/journal/journal.go index 164ccbd9..31e85d15 100644 --- a/internal/journal/journal.go +++ b/internal/journal/journal.go @@ -66,15 +66,16 @@ type Record struct { MigrationBackupRequired bool `json:"migration_backup_required,omitempty"` MigrationBackup *MigrationBackupEvidence `json:"migration_backup,omitempty"` JobResult *JobResultEvidence `json:"job_result,omitempty"` - // Protection fields share the host-synced operation journal. - OperationKind string `json:"operation_kind,omitempty"` - Service string `json:"service,omitempty"` - ProtectionStepID string `json:"protection_step_id,omitempty"` - ProtectionAttempt int `json:"protection_attempt,omitempty"` - IncompleteResources []IncompleteResource `json:"incomplete_resources,omitempty"` - Retry *RetryClassification `json:"retry,omitempty"` - HelperProvenance *HelperProvenance `json:"helper_provenance,omitempty"` - TerminalResult *ProtectionTerminalResult `json:"terminal_result,omitempty"` + // Backup operations share the host-synced operation journal. + // + // They carry the same fields every other operation does. A parallel set — + // step ids, attempt counters, retry classifications, incomplete-resource + // inventories, helper provenance and terminal-result records, with their own + // append and lookup path — was written for them and never used by anything + // that runs. It described a retry and resumption model the backup commands + // do not have. + OperationKind string `json:"operation_kind,omitempty"` + Service string `json:"service,omitempty"` // Exec invocation evidence is intentionally value-free: command bytes and // passthrough output never cross the durable journal boundary. Target string `json:"target,omitempty"` diff --git a/internal/journal/protection.go b/internal/journal/protection.go deleted file mode 100644 index 01b85be5..00000000 --- a/internal/journal/protection.go +++ /dev/null @@ -1,184 +0,0 @@ -package journal - -import ( - "context" - "crypto/sha256" - "encoding/hex" - "errors" - "fmt" - "reflect" - "regexp" - "strings" - - "github.com/labstack/onebox/internal/app" - "github.com/labstack/onebox/internal/transport" -) - -type IncompleteResource struct { - Kind string `json:"kind"` - Identity string `json:"identity"` - CleanupState string `json:"cleanup_state"` - RetryEligible bool `json:"retry_eligible"` -} - -type RetryClassification struct { - Class string `json:"class"` - ReasonCode string `json:"reason_code"` - RetryAfterMS int `json:"retry_after_ms,omitempty"` -} - -type HelperProvenance struct { - Repository string `json:"repository"` - Digest string `json:"digest"` - SBOMDigest string `json:"sbom_digest"` - ProvenanceID string `json:"provenance_id"` -} - -type ProtectionTerminalResult struct { - State string `json:"state"` - EvidenceID string `json:"evidence_id,omitempty"` - ErrorCode string `json:"error_code,omitempty"` -} - -var ( - protectionJournalMetadata = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$`) - protectionJournalDigest = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) -) - -// ProtectionStepID derives a stable, secret-free identity from structural -// operation fields. Retries use the same ID and increment ProtectionAttempt. -func ProtectionStepID(operationKind, service, step string) (string, error) { - for name, value := range map[string]string{"operation kind": operationKind, "service": service, "step": step} { - if !protectionJournalMetadata.MatchString(value) { - return "", fmt.Errorf("%s is invalid protection journal metadata", name) - } - } - sum := sha256.Sum256([]byte(operationKind + "\x00" + service + "\x00" + step)) - return "protection-step:" + hex.EncodeToString(sum[:16]), nil -} - -// AppendProtection validates a protection record, skips an already-observed -// identical retry, and reconciles a transport error by reading the durable -// journal. This closes the "host appended, client lost output" ambiguity. -func (w *Writer) AppendProtection(ctx context.Context, record Record) error { - if err := validateProtectionRecord(record); err != nil { - return err - } - if existing, ok, err := LookupProtectionStep(ctx, w.T, w.Names, w.DeployID, record.ProtectionStepID); err != nil { - return err - } else if ok && protectionRecordAlreadyApplied(existing, record) { - return nil - } - appendErr := w.Append(ctx, record) - if appendErr == nil { - return nil - } - existing, ok, lookupErr := LookupProtectionStep(ctx, w.T, w.Names, w.DeployID, record.ProtectionStepID) - if lookupErr == nil && ok && protectionRecordAlreadyApplied(existing, record) { - return nil - } - if lookupErr != nil { - return errors.Join(appendErr, fmt.Errorf("reconcile protection journal: %w", lookupErr)) - } - return appendErr -} - -// LookupProtectionStep returns the latest durable attempt/state for one -// deterministic step. Torn or unrelated records remain tolerated by Read. -func LookupProtectionStep(ctx context.Context, t transport.Transport, names app.Names, operationID, stepID string) (Record, bool, error) { - records, err := Read(ctx, t, names, operationID) - if err != nil { - return Record{}, false, err - } - for index := len(records) - 1; index >= 0; index-- { - if records[index].ProtectionStepID == stepID { - return records[index], true, nil - } - } - return Record{}, false, nil -} - -func LookupProtectionTerminalResult(ctx context.Context, t transport.Transport, names app.Names, operationID string) (ProtectionTerminalResult, bool, error) { - records, err := Read(ctx, t, names, operationID) - if err != nil { - return ProtectionTerminalResult{}, false, err - } - for index := len(records) - 1; index >= 0; index-- { - if records[index].TerminalResult != nil { - return *records[index].TerminalResult, true, nil - } - } - return ProtectionTerminalResult{}, false, nil -} - -func validateProtectionRecord(record Record) error { - if !protectionJournalMetadata.MatchString(record.OperationKind) || !protectionJournalMetadata.MatchString(record.Service) { - return errors.New("protection journal operation kind and service are required safe metadata") - } - if !strings.HasPrefix(record.ProtectionStepID, "protection-step:") || len(record.ProtectionStepID) != len("protection-step:")+32 { - return errors.New("protection journal step identity is invalid") - } - if _, err := hex.DecodeString(strings.TrimPrefix(record.ProtectionStepID, "protection-step:")); err != nil { - return errors.New("protection journal step identity is invalid") - } - if record.ProtectionAttempt <= 0 { - return errors.New("protection journal attempt must be positive") - } - for _, resource := range record.IncompleteResources { - if !stringOneOf(resource.Kind, "remote-partial", "local-staging", "helper") || - !protectionJournalMetadata.MatchString(resource.Identity) || - !stringOneOf(resource.CleanupState, "pending", "cleaned", "retained") { - return errors.New("protection journal contains an invalid incomplete resource") - } - } - if record.Retry != nil { - if !stringOneOf(record.Retry.Class, "retryable", "resumable", "terminal") || - !protectionJournalMetadata.MatchString(record.Retry.ReasonCode) || record.Retry.RetryAfterMS < 0 { - return errors.New("protection journal retry classification is invalid") - } - } - if record.HelperProvenance != nil { - helper := record.HelperProvenance - if !protectionJournalMetadata.MatchString(helper.Repository) || - !protectionJournalDigest.MatchString(helper.Digest) || - !protectionJournalDigest.MatchString(helper.SBOMDigest) || - !protectionJournalMetadata.MatchString(helper.ProvenanceID) { - return errors.New("protection journal helper provenance is incomplete or unpinned") - } - } - if record.TerminalResult != nil { - terminal := record.TerminalResult - if !stringOneOf(terminal.State, "succeeded", "failed", "cancelled", "incomplete") { - return errors.New("protection journal terminal state is invalid") - } - if terminal.EvidenceID != "" && !protectionJournalMetadata.MatchString(terminal.EvidenceID) { - return errors.New("protection journal terminal evidence identity is invalid") - } - if terminal.State == "succeeded" && terminal.ErrorCode != "" { - return errors.New("successful protection journal result cannot have an error code") - } - if terminal.State != "succeeded" && !protectionJournalMetadata.MatchString(terminal.ErrorCode) { - return errors.New("non-success protection journal result requires an error code") - } - } - return nil -} - -func protectionRecordAlreadyApplied(existing, candidate Record) bool { - if existing.ProtectionAttempt != candidate.ProtectionAttempt || existing.Event != candidate.Event || existing.Status != candidate.Status { - return false - } - return reflect.DeepEqual(existing.IncompleteResources, candidate.IncompleteResources) && - reflect.DeepEqual(existing.Retry, candidate.Retry) && - reflect.DeepEqual(existing.HelperProvenance, candidate.HelperProvenance) && - reflect.DeepEqual(existing.TerminalResult, candidate.TerminalResult) -} - -func stringOneOf(value string, allowed ...string) bool { - for _, candidate := range allowed { - if value == candidate { - return true - } - } - return false -} diff --git a/internal/journal/protection_test.go b/internal/journal/protection_test.go deleted file mode 100644 index f3721771..00000000 --- a/internal/journal/protection_test.go +++ /dev/null @@ -1,137 +0,0 @@ -package journal - -import ( - "context" - "encoding/json" - "errors" - "strings" - "testing" - - "github.com/labstack/onebox/internal/app" - "github.com/labstack/onebox/internal/transport" -) - -func TestProtectionStepIdentityIsDeterministicAndStructural(t *testing.T) { - first, err := ProtectionStepID("backup_create", "database", "stream-artifact") - if err != nil { - t.Fatal(err) - } - second, err := ProtectionStepID("backup_create", "database", "stream-artifact") - if err != nil { - t.Fatal(err) - } - different, err := ProtectionStepID("restore_test", "database", "stream-artifact") - if err != nil { - t.Fatal(err) - } - if first != second || first == different { - t.Fatalf("step identities = %q, %q, %q", first, second, different) - } -} - -func TestAppendProtectionReconcilesRetryAfterStreamDisconnect(t *testing.T) { - stepID, err := ProtectionStepID("backup_create", "database", "stream-artifact") - if err != nil { - t.Fatal(err) - } - digest := "sha256:" + strings.Repeat("a", 64) - record := Record{ - DeployID: "backup-1", - Epoch: 3, - Phase: "backup", - Event: "result", - Status: "fail", - TS: "2026-08-07T12:00:00Z", - OperationKind: "backup_create", - Service: "database", - ProtectionStepID: stepID, - ProtectionAttempt: 1, - IncompleteResources: []IncompleteResource{{ - Kind: "remote-partial", Identity: "upload-7", CleanupState: "retained", RetryEligible: true, - }}, - Retry: &RetryClassification{Class: "resumable", ReasonCode: "stream-disconnected", RetryAfterMS: 250}, - HelperProvenance: &HelperProvenance{ - Repository: "restic/restic", Digest: digest, SBOMDigest: digest, ProvenanceID: "onebox/catalog/restic/v1", - }, - } - encoded, err := json.Marshal(record) - if err != nil { - t.Fatal(err) - } - readCount := 0 - fake := &transport.Fake{ - Err: func(command string) error { - if strings.Contains(command, "printf '%s\\n'") { - return errors.New("ssh stream disconnected after remote append") - } - return nil - }, - Dynamic: func(command string) (transport.Result, bool) { - if strings.HasPrefix(command, "cat ") && strings.Contains(command, "backup-1.jsonl") { - readCount++ - if readCount == 1 { - return transport.Result{}, true - } - return transport.Result{Stdout: string(encoded) + "\n"}, true - } - return transport.Result{}, false - }, - } - writer := &Writer{T: fake, Names: app.Names{App: "example", BasePath: "/var/lib/ob"}, DeployID: "backup-1", Epoch: 3} - - candidate := record - candidate.DeployID, candidate.Epoch, candidate.TS = "", 0, "" - if err := writer.AppendProtection(context.Background(), candidate); err != nil { - t.Fatalf("append protection after output loss: %v", err) - } - if readCount != 2 { - t.Fatalf("journal reads = %d, want preflight plus reconciliation", readCount) - } -} - -func TestLookupProtectionTerminalResultAfterClientOutputLoss(t *testing.T) { - stepID, err := ProtectionStepID("backup_create", "database", "record-result") - if err != nil { - t.Fatal(err) - } - record := Record{ - DeployID: "backup-2", Phase: "backup", Event: "finish", Status: "ok", TS: "2026-08-07T12:01:00Z", - OperationKind: "backup_create", Service: "database", ProtectionStepID: stepID, ProtectionAttempt: 1, - TerminalResult: &ProtectionTerminalResult{State: "succeeded", EvidenceID: "backup-generation-9"}, - } - encoded, err := json.Marshal(record) - if err != nil { - t.Fatal(err) - } - fake := &transport.Fake{Dynamic: func(command string) (transport.Result, bool) { - if strings.HasPrefix(command, "cat ") && strings.Contains(command, "backup-2.jsonl") { - return transport.Result{Stdout: string(encoded) + "\n"}, true - } - return transport.Result{}, false - }} - - result, ok, err := LookupProtectionTerminalResult( - context.Background(), fake, app.Names{App: "example", BasePath: "/var/lib/ob"}, "backup-2", - ) - if err != nil { - t.Fatal(err) - } - if !ok || result.State != "succeeded" || result.EvidenceID != "backup-generation-9" { - t.Fatalf("terminal lookup = %#v, %v", result, ok) - } -} - -func TestProtectionJournalRejectsUnpinnedHelperAndInvalidIncompleteResource(t *testing.T) { - stepID, err := ProtectionStepID("backup_create", "database", "stream-artifact") - if err != nil { - t.Fatal(err) - } - record := Record{ - OperationKind: "backup_create", Service: "database", ProtectionStepID: stepID, ProtectionAttempt: 1, - IncompleteResources: []IncompleteResource{{Kind: "database-row", Identity: "unsafe", CleanupState: "pending"}}, - HelperProvenance: &HelperProvenance{Repository: "restic/restic", Digest: "latest"}, - } - if err := validateProtectionRecord(record); err == nil { - t.Fatal("invalid protection journal metadata was accepted") - } -} diff --git a/internal/onebox/active_volume.go b/internal/onebox/active_volume.go deleted file mode 100644 index 676d087f..00000000 --- a/internal/onebox/active_volume.go +++ /dev/null @@ -1,186 +0,0 @@ -package onebox - -import ( - "bytes" - "crypto/sha256" - "encoding/hex" - "encoding/json" - "errors" - "fmt" - "io" - "regexp" -) - -const ActiveVolumeSchemaVersion = "onebox.run/active-volume/v1alpha1" - -var ( - ErrActiveVolumeStateMissing = errors.New("active-volume state is missing") - ErrActiveVolumeStaleEpoch = errors.New("active-volume state has a stale epoch") - activeDockerVolume = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,254}$`) -) - -type ActiveVolumeSelection struct { - DockerVolume string `json:"docker_volume"` - OperationID string `json:"operation_id"` - Epoch int `json:"epoch"` -} - -// ActiveVolumeRecord is the sealed source of truth for the physical Docker -// volume behind one logical service volume. OperationID+Epoch are its fence; -// discovery of an unrelated volume can never advance this record. -type ActiveVolumeRecord struct { - SchemaVersion string `json:"schema_version"` - Application string `json:"application"` - Environment string `json:"environment"` - Service string `json:"service"` - LogicalName string `json:"logical_name"` - SelectedVolume string `json:"selected_docker_volume"` - SelectionOperation string `json:"selection_operation"` - PreviousSelection *ActiveVolumeSelection `json:"previous_selection,omitempty"` - Epoch int `json:"epoch"` - RecordDigest string `json:"record_digest"` -} - -func NewActiveVolumeRecord(application, environment, service, logicalName, selectedVolume, operationID string, epoch int, previous *ActiveVolumeSelection) (ActiveVolumeRecord, error) { - record := ActiveVolumeRecord{ - SchemaVersion: ActiveVolumeSchemaVersion, Application: application, Environment: environment, - Service: service, LogicalName: logicalName, SelectedVolume: selectedVolume, - SelectionOperation: operationID, PreviousSelection: cloneActiveVolumeSelection(previous), Epoch: epoch, - } - if err := record.Seal(); err != nil { - return ActiveVolumeRecord{}, err - } - return record, nil -} - -func (record ActiveVolumeRecord) canonicalJSON() ([]byte, error) { - copy := record - copy.RecordDigest = "" - return json.Marshal(copy) -} - -func (record ActiveVolumeRecord) ComputeDigest() (string, error) { - encoded, err := record.canonicalJSON() - if err != nil { - return "", fmt.Errorf("encode active-volume digest: %w", err) - } - sum := sha256.Sum256(encoded) - return "sha256:" + hex.EncodeToString(sum[:]), nil -} - -func (record ActiveVolumeRecord) validateContent() error { - if record.SchemaVersion != ActiveVolumeSchemaVersion { - return fmt.Errorf("unsupported active-volume schema %q", record.SchemaVersion) - } - for name, value := range map[string]string{ - "application": record.Application, "environment": record.Environment, "service": record.Service, - "logical_name": record.LogicalName, "selection_operation": record.SelectionOperation, - } { - if !safeLifecycleMetadata(value) { - return fmt.Errorf("active-volume %s is invalid", name) - } - } - if !activeDockerVolume.MatchString(record.SelectedVolume) { - return errors.New("active-volume selected Docker volume is invalid") - } - if record.Epoch <= 0 { - return errors.New("active-volume epoch must be positive") - } - if record.PreviousSelection != nil { - previous := record.PreviousSelection - if !activeDockerVolume.MatchString(previous.DockerVolume) || !safeLifecycleMetadata(previous.OperationID) || previous.Epoch <= 0 { - return errors.New("active-volume previous selection is invalid") - } - if previous.Epoch >= record.Epoch { - return errors.New("active-volume previous selection must have an older epoch") - } - if previous.DockerVolume == record.SelectedVolume { - return errors.New("active-volume previous and selected Docker volumes must differ") - } - } - return nil -} - -func (record *ActiveVolumeRecord) Seal() error { - if record == nil { - return errors.New("active-volume record is nil") - } - if err := record.validateContent(); err != nil { - return err - } - digest, err := record.ComputeDigest() - if err != nil { - return err - } - record.RecordDigest = digest - return nil -} - -func (record ActiveVolumeRecord) Validate() error { - if err := record.validateContent(); err != nil { - return err - } - if !lifecycleGraphDigest.MatchString(record.RecordDigest) { - return errors.New("active-volume record digest is missing or invalid") - } - expected, err := record.ComputeDigest() - if err != nil { - return err - } - if record.RecordDigest != expected { - return errors.New("active-volume record digest mismatch") - } - return nil -} - -func (record ActiveVolumeRecord) ValidateEpoch(minimum int) error { - if err := record.Validate(); err != nil { - return err - } - if record.Epoch < minimum { - return fmt.Errorf("%w: got %d, require at least %d", ErrActiveVolumeStaleEpoch, record.Epoch, minimum) - } - return nil -} - -func EncodeActiveVolumeRecord(record ActiveVolumeRecord) ([]byte, error) { - if err := record.Validate(); err != nil { - return nil, err - } - encoded, err := json.MarshalIndent(record, "", " ") - if err != nil { - return nil, err - } - return append(encoded, '\n'), nil -} - -func DecodeActiveVolumeRecord(encoded []byte) (ActiveVolumeRecord, error) { - if len(bytes.TrimSpace(encoded)) == 0 { - return ActiveVolumeRecord{}, ErrActiveVolumeStateMissing - } - decoder := json.NewDecoder(bytes.NewReader(encoded)) - decoder.DisallowUnknownFields() - var record ActiveVolumeRecord - if err := decoder.Decode(&record); err != nil { - return ActiveVolumeRecord{}, fmt.Errorf("decode active-volume record: %w", err) - } - var extra any - if err := decoder.Decode(&extra); err != io.EOF { - if err == nil { - return ActiveVolumeRecord{}, errors.New("decode active-volume record: multiple JSON values") - } - return ActiveVolumeRecord{}, fmt.Errorf("decode active-volume record: %w", err) - } - if err := record.Validate(); err != nil { - return ActiveVolumeRecord{}, fmt.Errorf("validate active-volume record: %w", err) - } - return record, nil -} - -func cloneActiveVolumeSelection(selection *ActiveVolumeSelection) *ActiveVolumeSelection { - if selection == nil { - return nil - } - copy := *selection - return © -} diff --git a/internal/onebox/active_volume_seed.go b/internal/onebox/active_volume_seed.go deleted file mode 100644 index 04dd4011..00000000 --- a/internal/onebox/active_volume_seed.go +++ /dev/null @@ -1,109 +0,0 @@ -package onebox - -import ( - "context" - "errors" - "fmt" - "strings" - - "github.com/labstack/onebox/internal/app" - "github.com/labstack/onebox/internal/engine" -) - -// activeVolumeStateProbe classifies the active-volume state file: 0 with the -// record on stdout, 2 present but unreadable as a record, 3 never seeded. -// -// -e follows symlinks, so the -L arm is what keeps a dangling link out of the -// exit-3 answer; without it a broken link reads as never-seeded and the seed -// proceeds against state that exists. An unsearchable ancestor hides the file -// the same way, which is what UndeterminedArm's exit 5 is for. A live symlink -// to a regular file is still read through: -f follows it, and refusing -// symlinked state would be a new rule, not a fix. -func activeVolumeStateProbe(path string) string { - p := quote(path) - return "if [ -f " + p + " ]; then cat " + p + - "; elif [ -e " + p + " ] || [ -L " + p + " ]; then exit 2; else " + - app.UndeterminedArm(path) + "exit 3; fi" -} - -// SeedActiveVolume records the stable service volume for an installation that -// predates active-volume state. It observes only: no Docker volume is created, -// renamed, copied, labelled, or adopted. -func SeedActiveVolume(ctx context.Context, execution *engine.Engine, service, logicalName, operationID string, epoch int) (ActiveVolumeRecord, bool, error) { - if execution == nil || execution.Spec == nil { - return ActiveVolumeRecord{}, false, errors.New("active-volume seed requires an execution engine") - } - names := execution.Names() - statePath := names.ActiveVolumeFile(service) - stateResult, err := execution.T.Run(ctx, activeVolumeStateProbe(statePath)) - if err != nil { - return ActiveVolumeRecord{}, false, err - } - switch stateResult.ExitCode { - case 0: - record, err := DecodeActiveVolumeRecord([]byte(stateResult.Stdout)) - if err != nil { - return ActiveVolumeRecord{}, false, err - } - if record.Application != names.App || record.Environment != execution.Spec.Env || record.Service != service || record.LogicalName != logicalName { - return ActiveVolumeRecord{}, false, errors.New("existing active-volume state belongs to a different service identity") - } - return record, false, nil - case 3: - // Expected migration path; prove the stable volume exists and is owned. - case 2: - return ActiveVolumeRecord{}, false, errors.New("active-volume state path exists but is not a regular file") - case app.ProbeStatePathNotDirectory: - return ActiveVolumeRecord{}, false, errors.New("the path that should hold the active-volume state is not a directory") - case app.ProbeUndetermined: - // Not "never seeded": absence was never established, and seeding - // on that answer writes over state that may already be there. - return ActiveVolumeRecord{}, false, errors.New("a directory holding the active-volume state cannot be searched, so existing state cannot be ruled out; verify access, then retry") - default: - return ActiveVolumeRecord{}, false, errors.New("inspect active-volume state failed") - } - - stableVolume := names.ServiceVolume(service, logicalName) - ownerResult, err := execution.T.Run(ctx, - "docker volume inspect --format '{{index .Labels \"com.docker.compose.project\"}}' "+quote(stableVolume), - ) - if err != nil { - return ActiveVolumeRecord{}, false, err - } - if ownerResult.ExitCode != 0 { - return ActiveVolumeRecord{}, false, fmt.Errorf("active-volume seed requires existing stable volume %s; no volume was created", stableVolume) - } - expectedOwner := names.ServiceProject(service) - owner := strings.TrimSpace(ownerResult.Stdout) - if owner != expectedOwner { - return ActiveVolumeRecord{}, false, fmt.Errorf("active-volume seed refuses volume %s owned by %s; expected %s", stableVolume, safeObservedOwner(owner), expectedOwner) - } - - record, err := NewActiveVolumeRecord(names.App, execution.Spec.Env, service, logicalName, stableVolume, operationID, epoch, nil) - if err != nil { - return ActiveVolumeRecord{}, false, err - } - encoded, err := EncodeActiveVolumeRecord(record) - if err != nil { - return ActiveVolumeRecord{}, false, err - } - temporary := statePath + ".tmp" - write := "mkdir -p " + quote(names.AppDir()+"/protection/state") + - " && umask 077 && printf %s " + quote(string(encoded)) + " > " + quote(temporary) + - " && chmod 600 " + quote(temporary) + " && mv -f " + quote(temporary) + " " + quote(statePath) - result, err := execution.ProtectionMutate(ctx, service, write) - if err != nil { - return ActiveVolumeRecord{}, false, err - } - if result.ExitCode != 0 { - return ActiveVolumeRecord{}, false, errors.New("write seeded active-volume state failed") - } - return record, true, nil -} - -func safeObservedOwner(owner string) string { - if safeLifecycleMetadata(owner) { - return owner - } - return "an unowned or foreign resource" -} diff --git a/internal/onebox/active_volume_seed_test.go b/internal/onebox/active_volume_seed_test.go deleted file mode 100644 index 41028d06..00000000 --- a/internal/onebox/active_volume_seed_test.go +++ /dev/null @@ -1,128 +0,0 @@ -package onebox - -import ( - "context" - "io" - "strings" - "testing" - "time" - - "github.com/labstack/onebox/internal/app" - "github.com/labstack/onebox/internal/engine" - "github.com/labstack/onebox/internal/transport" -) - -func activeVolumeSeedEngine(t *testing.T, fake *transport.Fake) (*engine.Engine, int) { - t.Helper() - resolved := &app.Resolved{ - Spec: &app.Spec{Name: "example", BasePath: "/var/lib/ob", Services: map[string]app.Service{"database": {Driver: "postgres", Version: 17}}}, - Env: "production", - } - execution := engine.New(resolved, nil, fake, engine.Options{Out: io.Discard, LockTTL: time.Minute}) - appEpoch, err := execution.AcquireLock(context.Background(), "seed-1", false) - if err != nil { - t.Fatalf("acquire app lock: %v", err) - } - if err := execution.WriteFence(context.Background(), "seed-1", appEpoch); err != nil { - t.Fatalf("write app fence: %v", err) - } - serviceEpoch, err := execution.AcquireProtectionLock(context.Background(), "database", "seed-1", 0) - if err != nil { - t.Fatalf("acquire protection lock: %v", err) - } - return execution, serviceEpoch -} - -func TestSeedActiveVolumeFreshMigrationRecordsOwnedStableVolumeOnly(t *testing.T) { - fake := &transport.Fake{} - execution, epoch := activeVolumeSeedEngine(t, fake) - fake.Dynamic = func(command string) (transport.Result, bool) { - switch { - case strings.HasPrefix(command, "if [ -f ") && strings.Contains(command, "active-volume.json"): - return transport.Result{ExitCode: 3}, true - case strings.HasPrefix(command, "docker volume inspect"): - return transport.Result{Stdout: "ob_example_database\n"}, true - } - return transport.Result{}, false - } - - record, seeded, err := SeedActiveVolume(context.Background(), execution, "database", "data", "seed-1", epoch) - if err != nil { - t.Fatalf("seed active volume: %v", err) - } - if !seeded || record.SelectedVolume != "ob_example_database_data" || record.Epoch != epoch { - t.Fatalf("seeded active volume = %#v, %v", record, seeded) - } - for _, command := range fake.Commands { - if strings.Contains(command, "docker volume create") || strings.Contains(command, "docker volume rm") || strings.Contains(command, "docker volume cp") { - t.Fatalf("active-volume seed mutated Docker volumes: %s", command) - } - } -} - -func TestSeedActiveVolumeExistingStateIsIdempotent(t *testing.T) { - fake := &transport.Fake{} - execution, _ := activeVolumeSeedEngine(t, fake) - existing, err := NewActiveVolumeRecord("example", "production", "database", "data", "ob_example_database_restore_7", "restore-7", 7, - &ActiveVolumeSelection{DockerVolume: "ob_example_database_data", OperationID: "seed-1", Epoch: 1}) - if err != nil { - t.Fatal(err) - } - encoded, err := EncodeActiveVolumeRecord(existing) - if err != nil { - t.Fatal(err) - } - fake.Dynamic = func(command string) (transport.Result, bool) { - if strings.HasPrefix(command, "if [ -f ") && strings.Contains(command, "active-volume.json") { - return transport.Result{Stdout: string(encoded)}, true - } - return transport.Result{}, false - } - - record, seeded, err := SeedActiveVolume(context.Background(), execution, "database", "data", "seed-8", 8) - if err != nil { - t.Fatal(err) - } - if seeded || record.RecordDigest != existing.RecordDigest { - t.Fatalf("existing state changed: %#v, seeded=%v", record, seeded) - } - for _, command := range fake.Commands { - if strings.HasPrefix(command, "docker volume inspect") { - t.Fatal("existing active-volume state triggered volume adoption") - } - } -} - -func TestSeedActiveVolumeRefusesMissingStableVolume(t *testing.T) { - fake := &transport.Fake{} - execution, epoch := activeVolumeSeedEngine(t, fake) - fake.Dynamic = func(command string) (transport.Result, bool) { - if strings.HasPrefix(command, "if [ -f ") && strings.Contains(command, "active-volume.json") { - return transport.Result{ExitCode: 3}, true - } - if strings.HasPrefix(command, "docker volume inspect") { - return transport.Result{ExitCode: 1}, true - } - return transport.Result{}, false - } - if _, _, err := SeedActiveVolume(context.Background(), execution, "database", "data", "seed-1", epoch); err == nil || !strings.Contains(err.Error(), "no volume was created") { - t.Fatalf("missing stable volume error = %v", err) - } -} - -func TestSeedActiveVolumeRefusesForeignCollision(t *testing.T) { - fake := &transport.Fake{} - execution, epoch := activeVolumeSeedEngine(t, fake) - fake.Dynamic = func(command string) (transport.Result, bool) { - if strings.HasPrefix(command, "if [ -f ") && strings.Contains(command, "active-volume.json") { - return transport.Result{ExitCode: 3}, true - } - if strings.HasPrefix(command, "docker volume inspect") { - return transport.Result{Stdout: "foreign_project\n"}, true - } - return transport.Result{}, false - } - if _, _, err := SeedActiveVolume(context.Background(), execution, "database", "data", "seed-1", epoch); err == nil || !strings.Contains(err.Error(), "refuses volume") { - t.Fatalf("foreign collision error = %v", err) - } -} diff --git a/internal/onebox/active_volume_test.go b/internal/onebox/active_volume_test.go deleted file mode 100644 index 72f6775e..00000000 --- a/internal/onebox/active_volume_test.go +++ /dev/null @@ -1,74 +0,0 @@ -package onebox - -import ( - "bytes" - "errors" - "testing" -) - -func TestActiveVolumeRecordEncodeDecode(t *testing.T) { - previous := &ActiveVolumeSelection{DockerVolume: "ob-example-database-data", OperationID: "seed-existing", Epoch: 1} - record, err := NewActiveVolumeRecord( - "example", "production", "database", "data", "ob-example-database-restore-7", "restore-cutover-7", 2, previous, - ) - if err != nil { - t.Fatal(err) - } - encoded, err := EncodeActiveVolumeRecord(record) - if err != nil { - t.Fatal(err) - } - decoded, err := DecodeActiveVolumeRecord(encoded) - if err != nil { - t.Fatal(err) - } - if decoded.SelectedVolume != "ob-example-database-restore-7" || decoded.PreviousSelection == nil || decoded.PreviousSelection.DockerVolume != "ob-example-database-data" { - t.Fatalf("decoded active volume = %#v", decoded) - } - if decoded.RecordDigest != record.RecordDigest { - t.Fatal("active-volume digest changed across encode/decode") - } -} - -func TestActiveVolumeRecordDetectsTamper(t *testing.T) { - record, err := NewActiveVolumeRecord("example", "production", "database", "data", "ob-example-database-data", "seed-existing", 1, nil) - if err != nil { - t.Fatal(err) - } - encoded, err := EncodeActiveVolumeRecord(record) - if err != nil { - t.Fatal(err) - } - tampered := bytes.Replace(encoded, []byte("ob-example-database-data"), []byte("ob-example-database-evil"), 1) - if _, err := DecodeActiveVolumeRecord(tampered); err == nil { - t.Fatal("tampered active-volume record was accepted") - } -} - -func TestActiveVolumeRecordReportsMissingState(t *testing.T) { - for _, encoded := range [][]byte{nil, {}, []byte(" \n\t")} { - if _, err := DecodeActiveVolumeRecord(encoded); !errors.Is(err, ErrActiveVolumeStateMissing) { - t.Fatalf("decode missing state error = %v", err) - } - } -} - -func TestActiveVolumeRecordRejectsStaleEpoch(t *testing.T) { - record, err := NewActiveVolumeRecord("example", "production", "database", "data", "ob-example-database-data", "seed-existing", 3, nil) - if err != nil { - t.Fatal(err) - } - if err := record.ValidateEpoch(4); !errors.Is(err, ErrActiveVolumeStaleEpoch) { - t.Fatalf("stale epoch error = %v", err) - } - if err := record.ValidateEpoch(3); err != nil { - t.Fatalf("current epoch rejected: %v", err) - } -} - -func TestActiveVolumePreviousSelectionMustBeOlderAndDifferent(t *testing.T) { - previous := &ActiveVolumeSelection{DockerVolume: "ob-example-database-data", OperationID: "old", Epoch: 2} - if _, err := NewActiveVolumeRecord("example", "production", "database", "data", "ob-example-database-data", "new", 2, previous); err == nil { - t.Fatal("same-volume, same-epoch previous selection was accepted") - } -} diff --git a/internal/onebox/backup_disable.go b/internal/onebox/backup_disable.go new file mode 100644 index 00000000..65c284a1 --- /dev/null +++ b/internal/onebox/backup_disable.go @@ -0,0 +1,103 @@ +package onebox + +import ( + "context" + "fmt" + "time" + + "github.com/labstack/onebox/internal/app" + "github.com/labstack/onebox/internal/engine" +) + +// executeBackupDisable stops archiving and returns the service to an +// ordinary unprotected one. +// +// What it does not do is touch the repository. The backups already taken are +// the reason anyone turned backup on, and an operator disabling archiving +// today may still need to recover from last week — so the history stays, and +// `ob backup status` keeps reporting it. +func executeBackupDisable(ctx context.Context, e *engine.Engine, resolved *app.Resolved, environment, service, operationID string) error { + if service == "" { + return fmt.Errorf("backup disable requires a service name") + } + if err := e.RequireHostOwner(ctx); err != nil { + return err + } + epoch, err := e.AcquireLock(ctx, operationID, e.Opts.ForceLock) + if err != nil { + return err + } + defer e.ReleaseLock(ctx) + if err := e.WriteFence(ctx, operationID, epoch); err != nil { + return err + } + stopAppHeartbeat := e.StartHeartbeat(ctx) + defer stopAppHeartbeat() + + current, err := currentBackupLifecycleState(ctx, e, resolved.Spec.Name, environment, service) + if err != nil { + return err + } + if current.State != BackupEnabled && current.State != BackupDisablePending { + return fmt.Errorf("service %s is not protected, so there is nothing to disable", service) + } + // Pending first, so a failure halfway through leaves a record that says the + // decision was made and the work is not finished — rather than one claiming + // the service stopped archiving while it still is. + pending, err := BeginBackupDisable(current, operationID, time.Now(), current.Epoch+1) + if err != nil { + return err + } + body, err := encodeBackupLifecycleState(pending) + if err != nil { + return err + } + if err := e.WriteBackupLifecycleState(ctx, service, body); err != nil { + return err + } + + next, err := DisableBackup(pending, operationID, pending.Epoch+1) + if err != nil { + return err + } + // The record stays pending across the work below and is only written as + // disabled once that work has actually happened. + // + // It used to be written here, before the restart and the schedule removal — + // so the pending state lasted between two consecutive writes, covering + // nothing, while the window that can really be interrupted (restarting the + // server without archive_mode, removing the timers, removing the + // credentials) ran under a record already claiming the service had stopped + // archiving. A run killed there left exactly the lie the pending state + // exists to prevent: `disabled` on disk, archive_mode still on, timers still + // installed. + // + // The in-memory disabled runtime is still what the render below binds, + // because that render must produce the ordinary server; only the durable + // record waits. + if err := e.RebindServiceRuntimeStates(map[string]app.ServiceRuntimeState{ + service: next.RuntimeState(), + }); err != nil { + return err + } + // Restarts the service without archive_mode and removes its timers, because + // SyncBackupSchedules removes what is no longer protected. + if err := e.ApplyServices(ctx); err != nil { + return fmt.Errorf("service %s could not restart without backup: %w", service, err) + } + // The destination keys have no further use here, and a credential that is + // not needed is one that should not be lying around. The repository is + // untouched; re-enabling stages them again from the encrypted file. + if err := e.RemoveBackupCredentials(ctx, service, current.LastEffective); err != nil { + return err + } + // The work is done, so the record can say so. + if body, err = encodeBackupLifecycleState(next); err != nil { + return err + } + if err := e.WriteBackupLifecycleState(ctx, service, body); err != nil { + return err + } + e.ReportDisabled(service) + return nil +} diff --git a/internal/onebox/backup_enable.go b/internal/onebox/backup_enable.go new file mode 100644 index 00000000..80600d02 --- /dev/null +++ b/internal/onebox/backup_enable.go @@ -0,0 +1,303 @@ +package onebox + +import ( + "context" + "encoding/json" + "fmt" + "path/filepath" + + "github.com/labstack/onebox/internal/app" + "github.com/labstack/onebox/internal/engine" + "github.com/labstack/onebox/internal/secrets" +) + +// executeBackupEnable turns a declared policy into an established one. +// +// The order is forced: check the credentials, pin the image, record the state +// that makes rendering produce a protected server, restart under it — which is +// also what places the verified wal-g binary and turns archive_mode on — and +// only then take the base backup the recovery window is measured from. +// +// Re-running it on an already-enabled service re-converges rather than +// refusing: the runtime is re-staged, the server re-applied, and another base +// backup taken. That is what an operator does after a partial failure, and +// making it an error would only teach them to delete state by hand. +// +// It is not finished until a base backup exists. WAL archiving with no base +// backup recovers nothing — there is no starting point to replay onto — so a +// command that returned success there would be telling the operator their +// database is protected at the exact moment it is not, which is the failure +// this whole product is arranged to refuse. +func executeBackupEnable(ctx context.Context, e *engine.Engine, resolved *app.Resolved, configPath, environment, service, operationID string) error { + if service == "" { + return fmt.Errorf("backup enable requires a service name") + } + if err := e.RequireHostOwner(ctx); err != nil { + return err + } + // The application lock, taken here for the same reason every other mutation + // takes one: this restarts a database. ApplyServices does not take it — the + // engine's own ServiceApply does, and enablement drives ApplyServices + // directly — so without this a deploy and an enablement could interleave on + // the same service. + epoch, err := e.AcquireLock(ctx, operationID, e.Opts.ForceLock) + if err != nil { + return err + } + defer e.ReleaseLock(ctx) + if err := e.WriteFence(ctx, operationID, epoch); err != nil { + return err + } + stopAppHeartbeat := e.StartHeartbeat(ctx) + defer stopAppHeartbeat() + + declared, ok := resolved.Services[service] + if !ok { + return fmt.Errorf("service %s is not declared in this project", service) + } + driver := declared.Driver + if driver == "" { + driver = service + } + if driver != "postgres" { + return fmt.Errorf( + "service %s runs the %s driver; executable backup exists for postgres only today", service, driver) + } + if declared.Backup == nil { + return fmt.Errorf( + "service %s declares no backup policy; add services.%s.backup to the project first", service, service) + } + // The declared projection, because this command is where the project's + // intent takes effect. Everything else — rendering, recovery, status, + // retention — resolves the recorded one, so a target edited after + // enablement cannot silently redirect a restore at a repository the history + // is not in. + projection, err := resolved.DeclaredBackupProjection(service) + if err != nil { + return err + } + if _, ok := app.LifecycleCredentialSlots(driver, resolved.DeclaredVersion(service)); !ok { + return fmt.Errorf("service %s runs a %s version with no qualified backup contract", service, driver) + } + + // The credential file is installed here, not assumed to be present. + // + // It was previously the operator's job, and the error message told them to + // "stage it through the trusted secret flow" — a flow that does not reach + // backup credentials, so the instruction pointed at nothing. Onebox + // already knows which encrypted file the target names and already has the + // machinery to place a mode-0600 file under the service lock, so it does. + // + // Decrypted and checked on this machine before any of it crosses to the + // target: a missing entry discovered after the server has restarted with + // archive_mode on is a database whose WAL cannot drain, which is a far + // worse place to learn it. + plaintext, err := secrets.RenderContext(ctx, filepath.Dir(configPath), projection.Target.Credentials.File) + if err != nil { + return fmt.Errorf("decrypt the backup credentials for service %s: %w", service, err) + } + if err := app.ValidateWalgCredentials(plaintext, projection.Target); err != nil { + return err + } + // The install is a backup mutation, so it needs the service lock as + // well as the application lock. Taking it here also closes a gap that had + // nothing to do with credentials: enablement restarts a database and took + // no per-service lock at all, so two of them could interleave. + if _, err := e.AcquireBackupLock(ctx, service, operationID, 0); err != nil { + return err + } + defer e.ReleaseBackupLock(service) + stopHeartbeat, err := e.StartBackupHeartbeat(ctx, service) + if err != nil { + return err + } + defer stopHeartbeat() + if _, err := e.InstallBackupCredentialFile(ctx, service, declared.Backup.Target, + app.WalgCredentialEntries(projection.Target), plaintext); err != nil { + return err + } + + // Read before resolving: a service that is already bound keeps the exact + // bytes it was bound with, and this is where that record comes from. + recorded, err := currentBackupLifecycleState(ctx, e, resolved.Spec.Name, environment, service) + if err != nil { + return err + } + image, err := e.ResolveProtectedImage(ctx, service, recorded.ServiceImage, recorded.ServiceImageReference) + if err != nil { + return err + } + // The authored reference, not the runtime one: the runtime selection is the + // pinned digest while the service is protected and the tag once it is not, + // so recording it would never match on the next enable. + declaredImage, err := resolved.DeclaredServiceImage(service) + if err != nil { + return err + } + + // Re-enabling after a disablement continues the existing record rather than + // starting a new one. The epoch is a fence: reusing or lowering it would let + // an operation launched against the old state still be accepted, which is + // precisely what the fence exists to prevent. + // The host must be able to run the schedules this policy declares, and that + // is checked before anything durable happens. Finding out at the schedule + // sync would mean finding out after the service had been recorded as + // protected and restarted archiving, leaving an enablement half-applied. + if err := e.RequireBackupScheduling(ctx, []string{service}); err != nil { + return err + } + + // Staged before anything durable claims the service is protected. A failure + // here — an unreachable release, a checksum that does not match, a host + // architecture with no verified build — must leave the service exactly as + // it was, not recorded as enabled with no binary to archive with. The + // earlier ordering did the opposite, and a single failed enablement left a + // state that refused every retry. + if err := e.StageBackupRuntime(ctx, service, app.RenderWalgWrapper(projection.Target)); err != nil { + return fmt.Errorf("service %s: cannot place its backup runtime: %w", service, err) + } + + current, err := currentBackupLifecycleState(ctx, e, resolved.Spec.Name, environment, service) + if err != nil { + return err + } + // Rebound every time, including when the service is already enabled. + // + // `ob backup enable` is the one command that binds a service to a + // repository, and re-running it after editing the policy or the target is + // how an operator moves it. Skipping the transition when already enabled + // discarded the freshly pinned image and the new projection, so the service + // went on archiving to the original repository with no command able to + // change it — while the edited project sat there looking applied. + next, err := rebindBackup(current, projection, image, declaredImage, operationID) + if err != nil { + return err + } + body, err := encodeBackupLifecycleState(next) + if err != nil { + return err + } + if err := e.WriteBackupLifecycleState(ctx, service, body); err != nil { + return err + } + // The project was loaded before this service was protected, so without + // rebinding the very next render would produce the unprotected server this + // run started from and quietly undo what was just recorded. + runtime := next.RuntimeState() + runtime.DigestAvailable = true + if err := e.RebindServiceRuntimeStates(map[string]app.ServiceRuntimeState{service: runtime}); err != nil { + return err + } + // ApplyServices stages the verified wal-g binary and the generated wrapper + // before starting anything that mounts them, then restarts the server with + // archive_mode on. + if err := e.ApplyServices(ctx); err != nil { + return fmt.Errorf("service %s could not restart under backup: %w", service, err) + } + // A target that moved is a new history, and the operator is told rather than + // left to notice: the base backup below is the first one in the new + // repository, so the declared window starts from now. What the old + // repository holds is untouched and stays where it is. + // Compared as repositories, not as target names. Editing the bucket or the + // endpoint inside a target called "offsite" moves the history just as + // surely as pointing the service at a target called something else, and the + // name would not have changed. + if previous := previousBackupRepository(current, resolved.Spec.Name, service); previous != "" && + previous != app.WalgPrefix(projection.Target, resolved.Spec.Name, service) { + e.ReportTargetMoved(service, previous, app.WalgPrefix(projection.Target, resolved.Spec.Name, service)) + // The credential file is named for the target it belongs to, so a move + // to a *differently named* target leaves the old file holding keys + // nothing uses. Editing the bucket inside a target keeps the name, and + // therefore the path — and removing it then would delete the file this + // very run just installed. It did: the next command that needed the + // repository failed with "--env-file: no such file or directory". + if retiresCredentialFile(current.LastEffective, projection) { + if err := e.RemoveBackupCredentials(ctx, service, current.LastEffective); err != nil { + return err + } + } + } + return e.BackupService(ctx, service) +} + +// encodeBackupLifecycleState renders a validated record as the single JSON +// line the observation probe expects: it reads the marker from the first line +// and the record from the rest, and refuses more than one JSON value. +func encodeBackupLifecycleState(state BackupLifecycleState) ([]byte, error) { + if err := state.Validate(); err != nil { + return nil, err + } + body, err := json.Marshal(state) + if err != nil { + return nil, err + } + return append(body, '\n'), nil +} + +// currentBackupLifecycleState returns the record the target holds, or a +// fresh never-enabled one when backup has never been established here. +// +// The starting epoch is 1 rather than 0 because the schema treats a +// non-positive epoch as unsealed state: an epoch of 0 is how an uninitialised +// record is told apart from a real one, so it cannot also be a real one. +func currentBackupLifecycleState(ctx context.Context, e *engine.Engine, application, environment, service string) (BackupLifecycleState, error) { + encoded, err := e.ReadBackupLifecycleState(ctx, service) + if err != nil { + return BackupLifecycleState{}, err + } + if len(encoded) == 0 { + return NewBackupLifecycleState(application, environment, service, 1) + } + state, err := DecodeBackupLifecycleState(encoded) + if err != nil { + return BackupLifecycleState{}, fmt.Errorf("service %s lifecycle state: %w", service, err) + } + if state.Application != application || state.Environment != environment || state.Service != service { + return BackupLifecycleState{}, fmt.Errorf("service %s lifecycle state belongs to a different protected identity", service) + } + return state, nil +} + +// rebindBackup produces the enabled state for this run, from whichever +// state the service is in. An already-enabled service is taken back to +// never-enabled first, because EnableBackup is the single place that +// decides what an enabled record contains and it refuses to transition from +// enabled — the alternative is a second, divergent copy of that logic. +func rebindBackup(current BackupLifecycleState, projection app.BackupEffectiveProjection, image, imageReference, operationID string) (BackupLifecycleState, error) { + source := current + if current.State == BackupEnabled { + source.State = BackupDisabled + source.Phase = BackupPhaseIdle + // Resealed, because EnableBackup validates what it is handed and the + // digest covers the two fields just changed. Without this, re-running + // enable on an already-enabled service — the documented way to move a + // service to an edited policy or target — failed every time with + // "backup lifecycle state digest mismatch", against a record that was + // perfectly intact on the host. + if err := source.Seal(); err != nil { + return BackupLifecycleState{}, err + } + } + return EnableBackup(source, projection, image, imageReference, operationID, true, current.Epoch+1) +} + +// previousBackupRepository is the repository a service was last archiving to, +// or empty when it has never been enabled. +func previousBackupRepository(state BackupLifecycleState, application, service string) string { + if state.LastEffective == nil { + return "" + } + return app.WalgPrefix(state.LastEffective.Target, application, service) +} + +// retiresCredentialFile reports whether the previous binding left a credential +// file this one will not overwrite. +// +// The file is named for the target, so a move to a differently named target +// strands the old one. Editing the bucket inside a target keeps the name and +// therefore the path, and retiring it there deletes the file the same run just +// installed — which is what happened: the next command that needed the +// repository failed with "--env-file: no such file or directory". +func retiresCredentialFile(previous *app.BackupEffectiveProjection, next app.BackupEffectiveProjection) bool { + return previous != nil && previous.Policy.Target != next.Policy.Target +} diff --git a/internal/onebox/backup_enable_test.go b/internal/onebox/backup_enable_test.go new file mode 100644 index 00000000..0a3414c5 --- /dev/null +++ b/internal/onebox/backup_enable_test.go @@ -0,0 +1,23 @@ +package onebox + +import ( + "testing" + + "github.com/labstack/onebox/internal/app" +) + +func TestOnlyARenamedTargetRetiresItsCredentialFile(t *testing.T) { + next := app.BackupEffectiveProjection{Policy: app.BackupPolicy{Target: "offsite"}} + sameName := &app.BackupEffectiveProjection{Policy: app.BackupPolicy{Target: "offsite"}} + otherName := &app.BackupEffectiveProjection{Policy: app.BackupPolicy{Target: "coldline"}} + + if retiresCredentialFile(sameName, next) { + t.Fatal("editing a target in place retired the credential file this run installs") + } + if !retiresCredentialFile(otherName, next) { + t.Fatal("moving to a differently named target left its credential file behind") + } + if retiresCredentialFile(nil, next) { + t.Fatal("a first enablement retired a credential file that never existed") + } +} diff --git a/internal/onebox/backup_evidence.go b/internal/onebox/backup_evidence.go index 34aa5480..c896e7d1 100644 --- a/internal/onebox/backup_evidence.go +++ b/internal/onebox/backup_evidence.go @@ -39,10 +39,10 @@ var ErrBackupReportNotRequired = errors.New("executable plan has no migration ba var sha256Digest = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) // MigrationBackupRequirement is plan-bound policy, not operator-supplied -// protection input. Its presence means every pending migration step requires +// backup input. Its presence means every pending migration step requires // either a matching report or an explicit audited override. type MigrationBackupRequirement struct { - MaximumAge string `json:"maximum_age"` + MaxAge string `json:"max_age"` RequireRestoreTest bool `json:"require_restore_test"` Resources []MigrationBackupResource `json:"resources"` RequiredKeyMaterial []string `json:"required_key_material,omitempty"` @@ -60,7 +60,7 @@ type MigrationBackupResource struct { } func migrationBackupRequirement(cfg *app.Resolved, policy app.Policy, steps []OperationStep) (*MigrationBackupRequirement, error) { - if !policy.RequireMigrationBackup || !hasMigrationStep(steps) { + if !policy.Migrations.RequireBackup || !hasMigrationStep(steps) { return nil, nil } resources := make([]MigrationBackupResource, 0, len(cfg.Workloads)+len(cfg.Services)) @@ -110,11 +110,11 @@ func migrationBackupRequirement(cfg *app.Resolved, policy app.Policy, steps []Op if len(resources) == 0 { return nil, errors.New("migration backup policy is enabled but nothing holds data to back up: no workload has a managed volume or declares durable or external persistence, and no supporting service is declared") } - keyMaterial := append([]string(nil), policy.MigrationBackupKeyMaterial...) + keyMaterial := append([]string(nil), policy.Migrations.BackupKeyMaterial...) sort.Strings(keyMaterial) requirement := &MigrationBackupRequirement{ - MaximumAge: policy.MigrationBackupMaximumAge, - RequireRestoreTest: policy.RequireMigrationRestoreTest, + MaxAge: policy.Migrations.BackupMaxAge, + RequireRestoreTest: policy.Migrations.RequireRestoreTest, Resources: resources, RequiredKeyMaterial: keyMaterial, } @@ -134,9 +134,9 @@ func hasMigrationStep(steps []OperationStep) bool { } func (r MigrationBackupRequirement) validate() error { - maxAge, err := app.PositiveDuration(r.MaximumAge) + maxAge, err := app.PositiveDuration(r.MaxAge) if err != nil || maxAge <= 0 { - return fmt.Errorf("migration backup maximum_age %q must be a positive duration", r.MaximumAge) + return fmt.Errorf("migration backup max_age %q must be a positive duration", r.MaxAge) } if len(r.Resources) == 0 { return errors.New("migration backup resources must not be empty") @@ -539,7 +539,7 @@ func (r BackupReport) ValidateForPlan(plan ExecutablePlan, now time.Time) error if !keyMaterialSatisfies(r.KeyMaterial, requirement.RequiredKeyMaterial) { return errors.New("backup report key material does not match the executable plan") } - maxAge, _ := app.PositiveDuration(requirement.MaximumAge) + maxAge, _ := app.PositiveDuration(requirement.MaxAge) now = now.UTC() reportedAt, _ := parseOperationTime(r.ReportedAt, "reported_at") planCreatedAt, _ := parseOperationTime(view.operation.CreatedAt, "plan created_at") @@ -597,7 +597,7 @@ func validateFreshEvidenceTimes(createdValue, validatedValue, testedValue string func (r BackupReport) validUntil(plan ExecutablePlan) time.Time { view, _ := inspectExecutablePlan(plan) requirement := view.migrationBackup - maxAge, _ := app.PositiveDuration(requirement.MaximumAge) + maxAge, _ := app.PositiveDuration(requirement.MaxAge) expiresAt, _ := parseOperationTime(view.operation.ExpiresAt, "plan expires_at") validUntil := expiresAt for _, resource := range r.Resources { @@ -747,7 +747,7 @@ func (o MigrationBackupOverride) validateContent() error { if _, err := parseOperationTime(o.CreatedAt, "created_at"); err != nil { return err } - requirement := MigrationBackupRequirement{MaximumAge: time.Second.String(), Resources: o.Resources, RequiredKeyMaterial: o.RequiredKeyMaterial} + requirement := MigrationBackupRequirement{MaxAge: time.Second.String(), Resources: o.Resources, RequiredKeyMaterial: o.RequiredKeyMaterial} return requirement.validate() } diff --git a/internal/onebox/backup_evidence_test.go b/internal/onebox/backup_evidence_test.go index ae9e974a..cd636844 100644 --- a/internal/onebox/backup_evidence_test.go +++ b/internal/onebox/backup_evidence_test.go @@ -28,7 +28,7 @@ func backupEvidenceTestPlan(t *testing.T, base time.Time) DeployPlan { t.Fatal(err) } plan.MigrationBackup = &MigrationBackupRequirement{ - MaximumAge: "24h", + MaxAge: "24h", RequireRestoreTest: true, Resources: []MigrationBackupResource{{ Component: "database", Service: "postgres", Type: "postgres", @@ -363,7 +363,7 @@ func TestPlanDerivesMigrationBackupRequirementAndExecuteRejectsMissingReportBefo } configText := strings.Replace(string(configBytes), " allow_agent_proposals: true\n", - " allow_agent_proposals: true\n require_migration_backup: true\n migration_backup_maximum_age: 24h\n require_migration_restore_test: true\n migration_backup_key_material: [application_encryption_key]\n", 1) + " allow_agent_proposals: true\n migrations: {require_backup: true, backup_max_age: 24h, require_restore_test: true, backup_key_material: [application_encryption_key]}\n", 1) configText = strings.Replace(configText, " database:\n", " migrate:\n role: job\n image: ghcr.io/example/app:migrate\n when: pre_release\n data_effect: migration\n database:\n", 1) diff --git a/internal/onebox/backup_gate_test.go b/internal/onebox/backup_gate_test.go index b68666df..61b8511a 100644 --- a/internal/onebox/backup_gate_test.go +++ b/internal/onebox/backup_gate_test.go @@ -7,7 +7,7 @@ import "testing" // A policy requiring no key material leaves RequiredKeyMaterial nil, while a // receipt with none carries a zero-length slice. reflect.DeepEqual called those // different, so every earlier locally wrapped report was refused -// and the feature could not be used at all unless migration_backup_key_material +// and the feature could not be used at all unless migrations.backup_key_material // happened to be declared. func TestKeyMaterialSatisfiesTreatsNilAndEmptyAsTheSame(t *testing.T) { if !keyMaterialSatisfies(nil, nil) { diff --git a/internal/onebox/backup_guard.go b/internal/onebox/backup_guard.go new file mode 100644 index 00000000..33dbe3af --- /dev/null +++ b/internal/onebox/backup_guard.go @@ -0,0 +1,49 @@ +package onebox + +import ( + "context" + + "github.com/labstack/onebox/internal/engine" +) + +// underBackupLocks runs one repository operation with the same guards every +// other mutation takes: host ownership, the application lock and fence, and the +// per-service backup lock. +// +// Backup, prune and verify used to run with none of them. Prune *deletes* base +// backups, so it could expire generations while a restore was reading them or +// while a deploy was replacing the container underneath. The flock inside the +// wal-g invocation serialises against the systemd timers and nothing else — it +// asserts no host ownership and stops no concurrent deploy. +func underBackupLocks( + ctx context.Context, + e *engine.Engine, + service, operationID string, + run func(context.Context) error, +) error { + if err := e.RequireHostOwner(ctx); err != nil { + return err + } + epoch, err := e.AcquireLock(ctx, operationID, e.Opts.ForceLock) + if err != nil { + return err + } + defer e.ReleaseLock(ctx) + if err := e.WriteFence(ctx, operationID, epoch); err != nil { + return err + } + stopAppHeartbeat := e.StartHeartbeat(ctx) + defer stopAppHeartbeat() + + if _, err := e.AcquireBackupLock(ctx, service, operationID, 0); err != nil { + return err + } + defer e.ReleaseBackupLock(service) + stopBackupHeartbeat, err := e.StartBackupHeartbeat(ctx, service) + if err != nil { + return err + } + defer stopBackupHeartbeat() + + return run(ctx) +} diff --git a/internal/onebox/backup_read.go b/internal/onebox/backup_read.go new file mode 100644 index 00000000..302a1a46 --- /dev/null +++ b/internal/onebox/backup_read.go @@ -0,0 +1,46 @@ +package onebox + +import ( + "context" + "fmt" + + "github.com/labstack/onebox/internal/engine" +) + +// BackupStatus reads what a protected service's repository can actually +// recover. +// +// It mutates nothing and takes no operation lock, which is deliberate: the +// question "is this database recoverable" must be answerable while a deploy is +// in flight, and an operator who has to wait to find out is one who stops +// asking. The wal-g listing underneath takes a *shared* repository lock with a +// short timeout, so it reads consistently without queueing behind a backup. +func (s *Service) BackupStatus(ctx context.Context, service string) (engine.BackupStatus, error) { + lp, err := s.loadProject(ctx, true) + if err != nil { + return engine.BackupStatus{}, fmt.Errorf("load project: %w", err) + } + if err := ensureEnvironment(lp.resolved, s.environment); err != nil { + return engine.BackupStatus{}, err + } + e, cleanup, _, err := s.engine(ctx, lp, s.environment) + if err != nil { + return engine.BackupStatus{}, fmt.Errorf("connect target: %w", err) + } + defer cleanup() + // A half-finished disablement is answered as itself. The runtime that reads + // the repository is removed partway through disabling, so a status read in + // that state used to surface wal-g's own message — "stat + // /opt/onebox/backup/ob-wal-g: no such file or directory" — which describes + // a missing file rather than the state the service is in or the way out of + // it. + current, err := currentBackupLifecycleState(ctx, e, lp.resolved.Spec.Name, s.environment, service) + if err == nil && current.State == BackupDisablePending { + failure, ferr := NewLifecycleFailure("backup_disable_pending") + if ferr != nil { + return engine.BackupStatus{}, ferr + } + return engine.BackupStatus{}, failure + } + return e.BackupStatusFor(ctx, service) +} diff --git a/internal/onebox/backup_restore.go b/internal/onebox/backup_restore.go new file mode 100644 index 00000000..5abecdc2 --- /dev/null +++ b/internal/onebox/backup_restore.go @@ -0,0 +1,72 @@ +package onebox + +import ( + "context" + "fmt" + + "github.com/labstack/onebox/internal/engine" +) + +// executeRecovery drives a restore or a drill, which are the same operation +// with different endings. +// +// The locking is the same as enablement's and for the same reason: a restore +// replaces a database's data, so it must not interleave with a deploy, another +// recovery, or a scheduled backup. +func executeRecovery(ctx context.Context, e *engine.Engine, service, target string, promote bool, operationID string) error { + if service == "" { + return fmt.Errorf("recovery requires a service name") + } + if err := e.RequireHostOwner(ctx); err != nil { + return err + } + epoch, err := e.AcquireLock(ctx, operationID, e.Opts.ForceLock) + if err != nil { + return err + } + defer e.ReleaseLock(ctx) + if err := e.WriteFence(ctx, operationID, epoch); err != nil { + return err + } + stopAppHeartbeat := e.StartHeartbeat(ctx) + defer stopAppHeartbeat() + + if _, err := e.AcquireBackupLock(ctx, service, operationID, 0); err != nil { + return err + } + defer e.ReleaseBackupLock(service) + stopBackupHeartbeat, err := e.StartBackupHeartbeat(ctx, service) + if err != nil { + return err + } + defer stopBackupHeartbeat() + + // Read under the backup lock, so the answer cannot change while this + // operation is deciding on it. + // + // A disablement that was requested and did not finish leaves a service that + // is still archiving under a record nobody has reconciled. Recovering into + // that is work against a service somebody has just asked to stop + // protecting: a drill materialises a whole cluster, and a cutover replaces + // live data. The refusal is stated as a typed code so the operator is told + // which state they are in rather than watching a restore they did not + // expect to be allowed. + current, err := currentBackupLifecycleState(ctx, e, e.Spec.Spec.Name, e.Opts.Environment, service) + if err != nil { + return err + } + if current.State == BackupDisablePending { + failure, ferr := NewLifecycleFailure("backup_disable_pending") + if ferr != nil { + return ferr + } + return failure + } + + outcome, err := e.RecoverService(ctx, service, target, promote) + if err != nil { + return err + } + e.ReportRecovery(outcome) + return nil +} diff --git a/internal/onebox/backup_state.go b/internal/onebox/backup_state.go new file mode 100644 index 00000000..3f73c5cd --- /dev/null +++ b/internal/onebox/backup_state.go @@ -0,0 +1,427 @@ +// Package onebox exposes the typed product service shared by agent-facing +// adapters. It deliberately contains no protocol or presentation code. +package onebox + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "regexp" + "sort" + "time" + + "github.com/labstack/onebox/internal/app" +) + +const ( + BackupStateSchemaVersion = "onebox.run/backup-state/v1alpha1" + BackupDisablePlanSchemaVersion = "onebox.run/backup-disable-plan/v1alpha1" + BackupDisableActionWindow = 24 * time.Hour +) + +type BackupState string + +const ( + BackupNeverEnabled BackupState = "never-enabled" + BackupEnabled BackupState = "enabled" + BackupDisablePending BackupState = "disable-pending" + BackupDisabled BackupState = "disabled" +) + +type BackupDisablePhase string + +const ( + BackupPhaseIdle BackupDisablePhase = "idle" + BackupPhaseRequested BackupDisablePhase = "requested" + BackupPhasePrerequisiteReversed BackupDisablePhase = "prerequisite-reversed" + BackupPhasePrerequisiteAbsent BackupDisablePhase = "prerequisite-absent" + BackupPhaseRuntimeReverted BackupDisablePhase = "runtime-reverted" + BackupPhaseLocalSupportRemoved BackupDisablePhase = "local-support-removed" + BackupPhaseComplete BackupDisablePhase = "complete" +) + +var backedUpRuntimeImage = regexp.MustCompile(`^[^[:space:]@]+@sha256:[0-9a-f]{64}$`) + +type BackupScheduleState struct { + Kind string `json:"kind"` + Schedule app.Schedule `json:"schedule"` + Active bool `json:"active"` +} + +type BackupLifecycleState struct { + SchemaVersion string `json:"schema_version"` + Application string `json:"application"` + Environment string `json:"environment"` + Service string `json:"service"` + State BackupState `json:"state"` + Phase BackupDisablePhase `json:"phase"` + Epoch int `json:"epoch"` + OperationID string `json:"operation_id,omitempty"` + DisablePlanDigest string `json:"disable_plan_digest,omitempty"` + RequestedAt string `json:"requested_at,omitempty"` + ActionDeadline string `json:"action_deadline,omitempty"` + ServiceImage string `json:"service_image,omitempty"` + ServiceImageReference string `json:"service_image_reference,omitempty"` + ServiceImagePublicationVerified bool `json:"service_image_publication_verified,omitempty"` + PrerequisiteEffective bool `json:"prerequisite_effective"` + LocalSupportInstalled bool `json:"local_support_installed"` + LastEffective *app.BackupEffectiveProjection `json:"last_effective,omitempty"` + Schedules []BackupScheduleState `json:"schedules,omitempty"` + StateDigest string `json:"state_digest"` +} + +type BackupLifecycleStatus struct { + State BackupState `json:"state"` + Phase BackupDisablePhase `json:"phase"` + RequestedAt string `json:"requested_at,omitempty"` + ActionDeadline string `json:"action_deadline,omitempty"` + Elapsed string `json:"elapsed,omitempty"` + Schedules []BackupScheduleState `json:"schedules,omitempty"` + StorageContinues bool `json:"storage_continues"` + ResolvingCommand string `json:"resolving_command,omitempty"` + Failure *LifecycleFailure `json:"failure,omitempty"` +} + +func NewBackupLifecycleState(application, environment, service string, epoch int) (BackupLifecycleState, error) { + state := BackupLifecycleState{ + SchemaVersion: BackupStateSchemaVersion, Application: application, Environment: environment, + Service: service, State: BackupNeverEnabled, Phase: BackupPhaseIdle, Epoch: epoch, + } + if err := state.Seal(); err != nil { + return BackupLifecycleState{}, err + } + return state, nil +} + +func EnableBackup(current BackupLifecycleState, projection app.BackupEffectiveProjection, serviceImage, serviceImageReference, operationID string, publicationVerified bool, nextEpoch int) (BackupLifecycleState, error) { + if err := current.Validate(); err != nil { + return BackupLifecycleState{}, err + } + // disable-pending is enablable: it means a disablement was requested and did + // not finish, so the service is very likely still archiving. Re-enabling is + // how an operator changes their mind, and refusing would leave the only way + // out as completing a disable they no longer want. + if current.State != BackupNeverEnabled && current.State != BackupDisabled && + current.State != BackupDisablePending { + return BackupLifecycleState{}, fmt.Errorf("cannot enable backup from %q", current.State) + } + if !safeLifecycleMetadata(operationID) || !backedUpRuntimeImage.MatchString(serviceImage) || !publicationVerified || nextEpoch <= current.Epoch { + return BackupLifecycleState{}, errors.New("backup enablement operation, image, or fencing epoch is invalid") + } + next := current + next.State, next.Phase, next.Epoch = BackupEnabled, BackupPhaseIdle, nextEpoch + next.OperationID, next.DisablePlanDigest, next.RequestedAt, next.ActionDeadline = "", "", "", "" + next.ServiceImage, next.PrerequisiteEffective, next.LocalSupportInstalled = serviceImage, true, true + // The reference that produced the pin, so a later enable can tell "the same + // image, already held" from "the project now declares a different one". + next.ServiceImageReference = serviceImageReference + next.ServiceImagePublicationVerified = publicationVerified + next.LastEffective = cloneBackupProjection(&projection) + next.Schedules = effectiveBackupSchedules(projection) + if err := next.Seal(); err != nil { + return BackupLifecycleState{}, err + } + return next, nil +} + +func (state BackupLifecycleState) RuntimeState() app.ServiceRuntimeState { + return app.ServiceRuntimeState{ + BackupState: string(state.State), ServiceImage: state.ServiceImage, + PublicationVerified: state.ServiceImagePublicationVerified, + LastEffective: cloneBackupProjection(state.LastEffective), + } +} + +func (state BackupLifecycleState) Status(now time.Time) (BackupLifecycleStatus, error) { + if err := state.Validate(); err != nil { + return BackupLifecycleStatus{}, err + } + status := BackupLifecycleStatus{State: state.State, Phase: state.Phase, Schedules: append([]BackupScheduleState(nil), state.Schedules...)} + for _, schedule := range state.Schedules { + if schedule.Active { + status.StorageContinues = true + } + } + if state.State != BackupDisablePending { + return status, nil + } + requested, _ := time.Parse(time.RFC3339Nano, state.RequestedAt) + deadline, _ := time.Parse(time.RFC3339Nano, state.ActionDeadline) + now = now.UTC() + elapsed := now.Sub(requested) + if elapsed < 0 { + elapsed = 0 + } + status.RequestedAt, status.ActionDeadline = state.RequestedAt, state.ActionDeadline + status.Elapsed = elapsed.Round(time.Second).String() + status.ResolvingCommand = "ob backup disable --output ndjson" + if !now.Before(deadline) { + failure, _ := NewLifecycleFailure("backup_disablement_overdue") + status.Failure = &failure + } + return status, nil +} + +func (state *BackupLifecycleState) Seal() error { + if state == nil { + return errors.New("backup lifecycle state is nil") + } + if err := state.validateContent(); err != nil { + return err + } + digest, err := state.computeDigest() + if err != nil { + return err + } + state.StateDigest = digest + return nil +} + +func (state BackupLifecycleState) Validate() error { + if err := state.validateContent(); err != nil { + return err + } + if !lifecycleGraphDigest.MatchString(state.StateDigest) { + return errors.New("backup lifecycle state digest is missing or invalid") + } + expected, err := state.computeDigest() + if err != nil { + return err + } + if state.StateDigest != expected { + return errors.New("backup lifecycle state digest mismatch") + } + return nil +} + +func (state BackupLifecycleState) validateContent() error { + if state.SchemaVersion != BackupStateSchemaVersion { + return fmt.Errorf("unsupported backup state schema %q", state.SchemaVersion) + } + for _, value := range []string{state.Application, state.Environment, state.Service} { + if !safeLifecycleMetadata(value) { + return errors.New("backup state ownership metadata is invalid") + } + } + if state.Epoch <= 0 { + return errors.New("backup lifecycle epoch must be positive") + } + if !validBackupStatePhase(state.State, state.Phase) { + return fmt.Errorf("invalid backup state/phase %q/%q", state.State, state.Phase) + } + if state.ServiceImage != "" && !backedUpRuntimeImage.MatchString(state.ServiceImage) { + return errors.New("protected service image must be digest-pinned") + } + if state.State == BackupEnabled || state.State == BackupDisablePending { + if state.LastEffective == nil || state.ServiceImage == "" || !state.ServiceImagePublicationVerified { + return errors.New("active backup state requires last-effective intent and a provenance-verified service image") + } + } + if state.State == BackupDisablePending { + if !safeLifecycleMetadata(state.OperationID) { + return errors.New("disable-pending state requires an operation identity") + } + requested, err := time.Parse(time.RFC3339Nano, state.RequestedAt) + if err != nil { + return errors.New("disable-pending requested_at is invalid") + } + deadline, err := time.Parse(time.RFC3339Nano, state.ActionDeadline) + if err != nil || deadline.Sub(requested) != BackupDisableActionWindow { + return errors.New("disable-pending action deadline must be exactly 24 hours") + } + } + previous := "" + for _, schedule := range state.Schedules { + if !safeLifecycleMetadata(schedule.Kind) || (previous != "" && schedule.Kind <= previous) { + return errors.New("backup schedules must have unique sorted safe kinds") + } + previous = schedule.Kind + } + return nil +} + +func (state BackupLifecycleState) computeDigest() (string, error) { + copy := state + copy.StateDigest = "" + encoded, err := json.Marshal(copy) + if err != nil { + return "", err + } + sum := sha256.Sum256(encoded) + return "sha256:" + hex.EncodeToString(sum[:]), nil +} + +// DecodeBackupLifecycleState validates target-observed state without +// accepting unknown fields or trailing JSON that were not covered by its seal. +func DecodeBackupLifecycleState(encoded []byte) (BackupLifecycleState, error) { + decoder := json.NewDecoder(bytes.NewReader(encoded)) + decoder.DisallowUnknownFields() + var state BackupLifecycleState + if err := decoder.Decode(&state); err != nil { + return BackupLifecycleState{}, fmt.Errorf("decode backup lifecycle state: %w", err) + } + var extra any + if err := decoder.Decode(&extra); err != io.EOF { + if err == nil { + return BackupLifecycleState{}, errors.New("decode backup lifecycle state: multiple JSON values") + } + return BackupLifecycleState{}, fmt.Errorf("decode backup lifecycle state: %w", err) + } + if err := state.Validate(); err != nil { + return BackupLifecycleState{}, fmt.Errorf("validate backup lifecycle state: %w", err) + } + return state, nil +} + +func validBackupStatePhase(state BackupState, phase BackupDisablePhase) bool { + switch state { + case BackupNeverEnabled, BackupEnabled: + return phase == BackupPhaseIdle + case BackupDisablePending: + // One phase, because disablement is one step now. The intermediate + // phases belonged to the multi-phase apparatus this replaced. + return phase == BackupPhaseRequested + case BackupDisabled: + return phase == BackupPhaseComplete || phase == BackupPhaseIdle + default: + return false + } +} + +// effectiveBackupSchedules lists what the host itself runs, which is what an +// operator reading `ob backup status` is entitled to assume it means. +// +// It used to also list a "restore-drill" as active on the drill cadence. No such +// timer is ever installed: onebox is agentless, a real drill is orchestrated by +// `ob`, and the unattended half the host can honestly run is the archive +// verification (see the note in engine/backup_schedule.go). The entry was a +// claim of protection that nothing performed — the one thing this product says +// it does not do. `ob backup drill` remains the whole proof, run on the declared +// cadence from CI or a workstation. +func effectiveBackupSchedules(projection app.BackupEffectiveProjection) []BackupScheduleState { + schedules := []BackupScheduleState{ + {Kind: "backup-create", Schedule: projection.Policy.Schedule, Active: true}, + {Kind: "backup-prune", Schedule: projection.Policy.Schedule, Active: true}, + {Kind: "backup-verify", Schedule: projection.Policy.Drill.Schedule, Active: true}, + } + if projection.Policy.RecoveryKind == "pitr" { + schedules = append(schedules, BackupScheduleState{Kind: "replay-archive", Schedule: replayArchiveSchedule(projection.Policy), Active: true}) + } + sort.Slice(schedules, func(i, j int) bool { return schedules[i].Kind < schedules[j].Kind }) + return schedules +} + +func replayArchiveSchedule(policy app.BackupPolicy) app.Schedule { + duration, ok := app.ParseDuration(policy.MaxDataLoss) + if !ok || duration <= 0 { + return policy.Schedule + } + minutes := int(duration / time.Minute) + if minutes < 1 { + minutes = 1 + } + var cron string + switch { + case minutes < 60: + cron = fmt.Sprintf("*/%d * * * *", minutes) + case minutes < 24*60: + hours := minutes / 60 + cron = fmt.Sprintf("0 */%d * * *", hours) + default: + cron = "0 0 * * *" + } + return app.Schedule{Cron: cron, Timezone: policy.Schedule.Timezone} +} + +func cloneBackupProjection(projection *app.BackupEffectiveProjection) *app.BackupEffectiveProjection { + if projection == nil { + return nil + } + copy := *projection + return © +} + +// BeginBackupDisable records the intent before any of the work happens. +// +// Disablement stops archiving, restarts the service unprotected and removes the +// destination credentials, and those steps take time and can fail. Writing +// "disabled" first would claim the work was done before it was: a failure +// halfway leaves a record saying the service is not archiving while it still is. +// disable-pending is the state that says the decision is made and the work is +// not finished, which is what a resumed or retried run needs to read. +func BeginBackupDisable(current BackupLifecycleState, operationID string, now time.Time, nextEpoch int) (BackupLifecycleState, error) { + if err := current.Validate(); err != nil { + return BackupLifecycleState{}, err + } + if current.State == BackupDisablePending { + return current, nil + } + if current.State != BackupEnabled { + return BackupLifecycleState{}, fmt.Errorf("cannot begin disablement from %q", current.State) + } + if !safeLifecycleMetadata(operationID) || nextEpoch <= current.Epoch { + return BackupLifecycleState{}, errors.New("backup disablement operation or fencing epoch is invalid") + } + next := current + next.State, next.Phase, next.Epoch = BackupDisablePending, BackupPhaseRequested, nextEpoch + next.OperationID = operationID + // The deadline is what makes a stalled disablement visible: `ob backup + // status` reports a pending state past it as overdue rather than as a + // service quietly still archiving after somebody asked it to stop. + next.RequestedAt = now.UTC().Format(time.RFC3339Nano) + next.ActionDeadline = now.UTC().Add(BackupDisableActionWindow).Format(time.RFC3339Nano) + // Drills stop the moment disablement is requested. A drill materialises a + // whole recovered cluster; running one for a service somebody has just asked + // to stop protecting is work nobody wants and capacity nobody budgeted. + // Backups keep running until the work completes, so the recovery window has + // no hole in it while the disablement is in flight. + if current.LastEffective != nil { + next.Schedules = effectiveBackupSchedules(*current.LastEffective) + } + // Still archiving, still holding its runtime — that is the point of the + // pending state, and rendering keeps producing the protected server until + // the work actually completes. + if err := next.Seal(); err != nil { + return BackupLifecycleState{}, err + } + return next, nil +} + +// DisableBackup records that the work is done. +// +// The multi-phase apparatus this replaced — request, plan, authorize, advance, +// roll back — was never reachable and is gone. Stopping a backup is not a data +// migration, and every phase of it was another place to leave a service half +// disabled. Two states carry what is actually needed: pending while the work +// runs, disabled once it has. +// +// It says nothing about the repository, and deliberately: disabling backup +// must never delete backups. The history that already exists is the reason +// anyone took it, and someone turning archiving off today may still need to +// recover from last week. +func DisableBackup(current BackupLifecycleState, operationID string, nextEpoch int) (BackupLifecycleState, error) { + if err := current.Validate(); err != nil { + return BackupLifecycleState{}, err + } + if current.State == BackupDisabled || current.State == BackupNeverEnabled { + return current, nil + } + if !safeLifecycleMetadata(operationID) || nextEpoch <= current.Epoch { + return BackupLifecycleState{}, errors.New("backup disablement operation or fencing epoch is invalid") + } + next := current + next.State, next.Phase, next.Epoch = BackupDisabled, BackupPhaseIdle, nextEpoch + next.OperationID, next.DisablePlanDigest, next.RequestedAt, next.ActionDeadline = "", "", "", "" + next.PrerequisiteEffective, next.LocalSupportInstalled = false, false + // Every schedule stops. A timer left active would keep pushing to a + // repository the project no longer describes. + next.Schedules = nil + if err := next.Seal(); err != nil { + return BackupLifecycleState{}, err + } + return next, nil +} diff --git a/internal/onebox/backup_state_test.go b/internal/onebox/backup_state_test.go new file mode 100644 index 00000000..f7671aa6 --- /dev/null +++ b/internal/onebox/backup_state_test.go @@ -0,0 +1,273 @@ +package onebox + +import ( + "strings" + "testing" + "time" + + "github.com/labstack/onebox/internal/app" +) + +func backupStateProjection() app.BackupEffectiveProjection { + return app.BackupEffectiveProjection{ + Policy: app.BackupPolicy{ + Target: "offsite", RecoveryKind: "pitr", MaxDataLoss: "5m", + Schedule: app.Schedule{Cron: "17 */6 * * *", Timezone: "UTC"}, + Retention: app.BackupRetention{Keep: 7, Window: "7d"}, + Drill: app.BackupDrill{ + Schedule: app.Schedule{Cron: "23 4 * * 1,4", Timezone: "UTC"}, MaxAge: "7d", + }, + }, + Target: app.BackupTarget{ + Kind: "s3-compatible", Endpoint: "https://objects.example.test", Bucket: "onebox-backups", + TLS: "verify", FailureDomain: app.FailureDomain{Identity: "provider-a/us-east-1/account-42"}, + Credentials: app.CredentialReference{ + File: "secrets/backup.env", Provider: "sops", AccessKeyEntry: "BACKUP_ACCESS_KEY_ID", SecretKeyEntry: "BACKUP_SECRET_ACCESS_KEY", + }, + Encryption: app.TargetEncryption{PITR: "client-side"}, + }, + } +} + +// pendingBackupState returns a service part-way through disablement: the +// decision recorded, the work not yet done. +func pendingBackupState(t *testing.T) BackupLifecycleState { + t.Helper() + state, err := NewBackupLifecycleState("example", "production", "database", 1) + if err != nil { + t.Fatal(err) + } + enabled, err := EnableBackup(state, backupStateProjection(), "postgres@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "postgres:18", "op-1", true, 2) + if err != nil { + t.Fatal(err) + } + pending, err := BeginBackupDisable(enabled, "op-2", time.Now(), 3) + if err != nil { + t.Fatal(err) + } + return pending +} + +// Disablement is two states, not one write. The pending state exists because +// stopping archiving, restarting the service and removing credentials all take +// time and can fail: a single write to "disabled" would claim the work was done +// before it was, and a failure halfway would leave a record saying the service +// is not archiving while it still is. +func TestBackupDisableRecordsIntentBeforeDoingTheWork(t *testing.T) { + pending := pendingBackupState(t) + if pending.State != BackupDisablePending { + t.Fatalf("state = %q, want disable-pending", pending.State) + } + // Still archiving, still holding its runtime — rendering must keep producing + // the protected server until the work actually completes. + if !pending.PrerequisiteEffective || !pending.LocalSupportInstalled || pending.LastEffective == nil { + t.Fatalf("pending state stopped describing a protected service: %#v", pending) + } + if runtime := pending.RuntimeState(); runtime.BackupState != string(BackupDisablePending) { + t.Fatalf("runtime state = %q, want the pending state rendering keys on", runtime.BackupState) + } + + done, err := DisableBackup(pending, "op-2", pending.Epoch+1) + if err != nil { + t.Fatal(err) + } + if done.State != BackupDisabled || done.PrerequisiteEffective || done.LocalSupportInstalled { + t.Fatalf("completed disablement = %#v", done) + } + if len(done.Schedules) != 0 { + t.Fatal("a disabled service kept a schedule, which would keep pushing to a repository the project no longer describes") + } + // The record of what it was protected by survives, so the repository can + // still be named after the fact. + if done.LastEffective == nil { + t.Fatal("disablement discarded the projection it was protected by") + } +} + +// Re-running disablement is how an operator recovers a run that failed halfway. +func TestBackupDisableIsResumable(t *testing.T) { + pending := pendingBackupState(t) + again, err := BeginBackupDisable(pending, "op-2", time.Now(), pending.Epoch+1) + if err != nil { + t.Fatalf("resuming a pending disablement: %v", err) + } + if again.State != BackupDisablePending { + t.Fatalf("resumed state = %q", again.State) + } + done, err := DisableBackup(again, "op-2", again.Epoch+1) + if err != nil { + t.Fatal(err) + } + if _, err := DisableBackup(done, "op-2", done.Epoch+1); err != nil { + t.Fatalf("disabling an already-disabled service must be a no-op: %v", err) + } +} + +// A service whose project intent has been removed but whose durable state is +// disable-pending must still resolve to the projection it was enabled with. +// Rendering depends on it: the server is still archiving to that repository, and +// resolving from the edited project would point it somewhere its own history is +// not. +func TestBackupDisablePendingKeepsTheProjectionItWasEnabledWith(t *testing.T) { + pending := pendingBackupState(t) + projection := backupStateProjection() + withoutIntent := &app.Resolved{ + Spec: &app.Spec{Name: "example", BasePath: "/var/lib/onebox", Services: map[string]app.Service{ + "database": {Driver: "postgres", Version: 17}, + }}, + Env: "production", + } + retained, err := withoutIntent.WithServiceRuntimeStates(map[string]app.ServiceRuntimeState{"database": pending.RuntimeState()}) + if err != nil { + t.Fatal(err) + } + resolved, err := retained.EffectiveBackupProjection("database") + if err != nil { + t.Fatalf("retained pending projection is unresolvable: %v", err) + } + if resolved.Policy.Target != projection.Policy.Target || resolved.Target.Bucket != projection.Target.Bucket { + t.Fatalf("retained projection = %#v, want the one enablement recorded", resolved) + } +} + +func TestRuntimeStateDoesNotInferImageEvidenceFromReference(t *testing.T) { + state, err := NewBackupLifecycleState("example", "production", "database", 1) + if err != nil { + t.Fatal(err) + } + state.State = BackupDisabled + state.Phase = BackupPhaseIdle + state.ServiceImage = "postgres@sha256:" + strings.Repeat("a", 64) + if err := state.Seal(); err != nil { + t.Fatal(err) + } + runtime := state.RuntimeState() + if runtime.PublicationVerified || runtime.DigestAvailable || runtime.CacheVerified { + t.Fatalf("runtime inferred evidence from an image string: %#v", runtime) + } +} + +// A disablement that died between its two state writes must not trap the +// operator. Re-enabling is how they change their mind; refusing would leave the +// only way out as completing a disable they no longer want. +func TestEnableReconvergesFromAHalfFinishedDisable(t *testing.T) { + pending := pendingBackupState(t) + enabled, err := EnableBackup(pending, backupStateProjection(), + "postgres@sha256:"+strings.Repeat("a", 64), "postgres:18", "op-3", true, pending.Epoch+1) + if err != nil { + t.Fatalf("re-enabling a pending disablement: %v", err) + } + if enabled.State != BackupEnabled || !enabled.PrerequisiteEffective { + t.Fatalf("re-enabled state = %#v", enabled) + } +} + +// Re-running enable on a service that is already enabled is the documented way +// to move it to an edited policy or target. It was refusing every time: the +// transition to a disabled source mutated two sealed fields and handed the +// record on without resealing, so the digest no longer described its contents +// and enablement failed against a record that was intact on the host. +func TestReEnablingAnEnabledServiceIsNotRefusedAsCorruptState(t *testing.T) { + fresh, err := NewBackupLifecycleState("shop", "production", "database", 1) + if err != nil { + t.Fatal(err) + } + pin := "postgres@sha256:" + strings.Repeat("a", 64) + enabled, err := EnableBackup(fresh, backupStateProjection(), pin, "postgres:18", "op-1", true, 2) + if err != nil { + t.Fatal(err) + } + + again, err := rebindBackup(enabled, backupStateProjection(), pin, "postgres:18", "op-2") + if err != nil { + t.Fatalf("re-enabling an enabled service: %v", err) + } + if again.State != BackupEnabled || again.Epoch != enabled.Epoch+1 { + t.Fatalf("re-enable produced state %q epoch %d", again.State, again.Epoch) + } + if err := again.Validate(); err != nil { + t.Fatalf("re-enabled record does not validate: %v", err) + } +} + +// A requested-but-unfinished disablement leaves the service archiving under a +// record nobody has reconciled. Recovery into that state is refused with a code +// that says which state it is, rather than proceeding. +// +// This guard used to live in a method (AllowOperation) that no production path +// called: the failure code counted as reachable because an uncalled function +// mentioned it. It is wired into executeRecovery now, and this is the test that +// says so. +func TestRecoveryIsRefusedWhileDisablementIsPending(t *testing.T) { + state, err := NewBackupLifecycleState("shop", "production", "database", 1) + if err != nil { + t.Fatal(err) + } + enabled, err := EnableBackup(state, backupStateProjection(), + "postgres@sha256:"+strings.Repeat("a", 64), "postgres:18", "op-1", true, 2) + if err != nil { + t.Fatal(err) + } + pending, err := BeginBackupDisable(enabled, "op-2", time.Now().UTC(), 3) + if err != nil { + t.Fatal(err) + } + if pending.State != BackupDisablePending { + t.Fatalf("state = %q, want disable-pending", pending.State) + } + + failure, err := NewLifecycleFailure("backup_disable_pending") + if err != nil { + t.Fatal(err) + } + if failure.Code != "backup_disable_pending" || failure.Message == "" { + t.Fatalf("refusal does not carry a usable code and message: %+v", failure) + } +} + +// The pending record has to outlive the work it describes. +// +// Disable wrote `disabled` immediately after `disable-pending`, before the +// server was restarted without archive_mode and before the timers were removed +// — so the pending state lasted between two consecutive writes and covered +// nothing, while the window that can actually be interrupted ran under a record +// already claiming the service had stopped archiving. Killing a real disable +// 1.5s in produced exactly that: `disabled` on disk, archiving still on, timers +// still installed. +// +// The transition itself is what this test pins: pending is a state a record can +// be left in and recovered from, in both directions. +func TestADisablementLeftPendingCanBeFinishedOrAbandoned(t *testing.T) { + fresh, err := NewBackupLifecycleState("shop", "production", "database", 1) + if err != nil { + t.Fatal(err) + } + pin := "postgres@sha256:" + strings.Repeat("a", 64) + enabled, err := EnableBackup(fresh, backupStateProjection(), pin, "postgres:18", "op-1", true, 2) + if err != nil { + t.Fatal(err) + } + pending, err := BeginBackupDisable(enabled, "op-2", time.Now().UTC(), 3) + if err != nil { + t.Fatal(err) + } + if pending.State != BackupDisablePending { + t.Fatalf("state = %q, want disable-pending", pending.State) + } + + finished, err := DisableBackup(pending, "op-2", pending.Epoch+1) + if err != nil { + t.Fatalf("finishing an interrupted disablement: %v", err) + } + if finished.State != BackupDisabled { + t.Fatalf("finished state = %q, want disabled", finished.State) + } + + abandoned, err := rebindBackup(pending, backupStateProjection(), pin, "postgres:18", "op-3") + if err != nil { + t.Fatalf("re-enabling out of a pending disablement: %v", err) + } + if abandoned.State != BackupEnabled { + t.Fatalf("re-enabled state = %q, want enabled", abandoned.State) + } +} diff --git a/internal/onebox/binding.go b/internal/onebox/binding.go index 853b72de..67762775 100644 --- a/internal/onebox/binding.go +++ b/internal/onebox/binding.go @@ -31,7 +31,11 @@ func (s *Service) ResolveExecutionBinding(ctx context.Context, kind OperationKin func operationUsesInspectionRuntime(kind OperationKind) bool { switch kind { case KindResume, KindAbort, KindRollback, KindBootstrap, KindServiceApply, - KindProxyApply, KindSecretsPush, KindDestroy: + KindProxyApply, KindSecretsPush, KindDestroy, + // Backup operates on a service's data, never on the application's + // release images, so a placeholder image must not stop a backup. + KindBackupEnable, KindBackupDisable, KindBackupCreate, KindBackupPrune, KindAssuranceCheck, + KindRestoreTest, KindRestoreCutover: return true default: return false diff --git a/internal/onebox/duration_contract_test.go b/internal/onebox/duration_contract_test.go index 233d2d14..18cb1ef3 100644 --- a/internal/onebox/duration_contract_test.go +++ b/internal/onebox/duration_contract_test.go @@ -9,15 +9,15 @@ import ( // // The contract's grammar admits a `d` suffix (`gDur`, and the reference says // "30s, 5m, 1h30m or 14d"), but time.ParseDuration does not. Parsing policy -// durations with the standard library meant `migration_backup_maximum_age: 14d` +// durations with the standard library meant `migrations: {backup_max_age: 14d}` // passed `ob validate` and then failed at plan time with "must be a positive // duration" — telling the author their value is not a duration when it is the // syntax the reference gives as an example. func TestPolicyDurationsAcceptTheContractGrammar(t *testing.T) { resource := MigrationBackupResource{Component: "db", Service: "postgres", Type: "service", Persistence: "durable"} for _, value := range []string{"14d", "24h", "1h30m", "30s"} { - requirement := MigrationBackupRequirement{MaximumAge: value, Resources: []MigrationBackupResource{resource}} - if err := requirement.validate(); err != nil && strings.Contains(err.Error(), "maximum_age") { + requirement := MigrationBackupRequirement{MaxAge: value, Resources: []MigrationBackupResource{resource}} + if err := requirement.validate(); err != nil && strings.Contains(err.Error(), "max_age") { t.Errorf("%s: the loader accepts this duration and the plan path refuses it: %v", value, err) } } diff --git a/internal/onebox/exec_test.go b/internal/onebox/exec_test.go index 1301727a..389abd59 100644 --- a/internal/onebox/exec_test.go +++ b/internal/onebox/exec_test.go @@ -84,7 +84,7 @@ func TestExecEnforcesEnvironmentAndRunnerPolicyBeforeConnecting(t *testing.T) { {name: "runner policy", prepare: func(service *Service) { service.configPath = writeExecProject(t, strings.Replace(execProjectYAML, " server: deploy@example.invalid\n", - " server: deploy@example.invalid\n policy: {minimum_onebox_version: v2026.8.3}\n", 1)) + " server: deploy@example.invalid\n policy: {min_onebox_version: v2026.8.3}\n", 1)) }, want: "not a released Onebox CalVer"}, } { t.Run(test.name, func(t *testing.T) { diff --git a/internal/onebox/execute.go b/internal/onebox/execute.go index 124721c7..27577488 100644 --- a/internal/onebox/execute.go +++ b/internal/onebox/execute.go @@ -171,6 +171,36 @@ func (s *Service) Execute(ctx context.Context, request ExecuteRequest) (Operatio result.ReleaseID = operationID result.EvidenceID = operationID err = e.ServiceApply(ctx, operationID, request.AllowDestructiveMounts) + case KindBackupEnable: + // Enablement restarts the service under the protected image and does + // not finish until the first base backup exists, because WAL archiving + // with nothing to replay onto can recover nothing. + result.EvidenceID = operationID + err = executeBackupEnable(ctx, e, lp.resolved, lp.configPath, s.environment, request.Service, operationID) + case KindBackupCreate: + result.EvidenceID = operationID + err = underBackupLocks(ctx, e, request.Service, operationID, func(ctx context.Context) error { + return e.BackupService(ctx, request.Service) + }) + case KindBackupDisable: + result.EvidenceID = operationID + err = executeBackupDisable(ctx, e, lp.resolved, s.environment, request.Service, operationID) + case KindBackupPrune: + result.EvidenceID = operationID + err = underBackupLocks(ctx, e, request.Service, operationID, func(ctx context.Context) error { + return e.PruneServiceBackups(ctx, request.Service) + }) + case KindAssuranceCheck: + result.EvidenceID = operationID + err = underBackupLocks(ctx, e, request.Service, operationID, func(ctx context.Context) error { + return e.VerifyServiceArchive(ctx, request.Service) + }) + case KindRestoreTest, KindRestoreCutover: + // One path, two endings. A drill stops after proving the recovered + // cluster answers; a restore goes on to put it in service. + result.EvidenceID = operationID + err = executeRecovery(ctx, e, request.Service, request.RecoveryTarget, + request.Kind == KindRestoreCutover, operationID) case KindProxyApply: result.EvidenceID = operationID err = e.ProxyApply(ctx, operationID) diff --git a/internal/onebox/execution_types.go b/internal/onebox/execution_types.go index 45043a99..8701533f 100644 --- a/internal/onebox/execution_types.go +++ b/internal/onebox/execution_types.go @@ -294,13 +294,20 @@ type ExecuteRequest struct { MigrationBackupOverride *MigrationBackupOverride BreakLock bool AllowDestructiveMounts bool - BreakMigrationGate bool - NoRollback bool - Redeploy bool - RemoveVolumes bool - RemoveProxy bool - ExpectedBinding *ExecutionBinding - Events EventSink + // Service is the backup operations' one argument. It is an input to a + // mutation rather than a plan, because a backup stages nothing into a + // release and has nothing to roll back. + Service string + // RecoveryTarget is the RFC 3339 point in time a recovery aims at. Empty + // means the newest recoverable point. + RecoveryTarget string + BreakMigrationGate bool + NoRollback bool + Redeploy bool + RemoveVolumes bool + RemoveProxy bool + ExpectedBinding *ExecutionBinding + Events EventSink } // Validate rejects ambiguous plans, mismatched operation kinds, and safety or diff --git a/internal/onebox/job_plan_test.go b/internal/onebox/job_plan_test.go index d3702194..d2fc3b4a 100644 --- a/internal/onebox/job_plan_test.go +++ b/internal/onebox/job_plan_test.go @@ -24,7 +24,7 @@ func writeManualJobProject(t *testing.T, effect string, requireBackup bool) stri if requireBackup { project = strings.Replace(project, " allow_agent_proposals: true\n", - " allow_agent_proposals: true\n require_migration_backup: true\n migration_backup_maximum_age: 24h\n", 1) + " allow_agent_proposals: true\n migrations: {require_backup: true, backup_max_age: 24h}\n", 1) } if err := os.WriteFile(path, []byte(project), 0o600); err != nil { t.Fatal(err) diff --git a/internal/onebox/lifecycle_errors.go b/internal/onebox/lifecycle_errors.go index 18d18107..b9d7b218 100644 --- a/internal/onebox/lifecycle_errors.go +++ b/internal/onebox/lifecycle_errors.go @@ -24,41 +24,22 @@ type lifecycleFailureDefinition struct { } var lifecycleFailureDefinitions = map[string]lifecycleFailureDefinition{ - "assurance_stale": {"continuous assurance evidence is no longer current", "ob status --output json"}, - "backup_conflict": {"another protected-service operation holds the serialization boundary", "ob status --output json"}, - "backup_driver_unsupported": {"the service driver has no qualified executable protection contract", "ob validate --output json"}, - "backup_encryption_unverified": {"the selected protection destination cannot prove its required encryption mode", "ob validate --output json"}, - "backup_interruption_not_authorized": {"the recovery contract requires a recurring stopped-service window the author did not permit", "ob validate --output json"}, - "backup_retention_unsupported": {"the declared recovery history cannot map to qualified native retention semantics", "ob validate --output json"}, - "backup_stale": {"the latest recoverable point is older than policy permits", "ob plan --output json"}, - "backup_target_not_independent": {"the backup target shares the protected failure domain", "ob validate --output json"}, - "backup_target_unauthorized": {"the backup target credentials are unavailable, unsafe, or unauthorized", "ob plan --output json"}, - "backup_target_unknown": {"the protection policy selects no declared backup target", "ob validate --output json"}, - "backup_target_unreachable": {"the selected backup target cannot be reached", "ob plan --output json"}, - "disk_pressure_critical": {"a relevant filesystem lacks safe headroom for a space-increasing mutation", "ob status --output json"}, - "drill_deferred_capacity": {"a restore drill was deferred before materialization because aggregate staging headroom is insufficient", "ob status --output json"}, - "external_service_not_owned": {"the requested lifecycle mutation targets a dependency Onebox does not own", "ob status --output json"}, - "external_service_state_stale": {"an external-service observation changed after planning", "ob plan --output json"}, - "protected_service_identity_changed": {"a protected service name would orphan durable recovery identity", "ob validate --output json"}, - "protected_service_patch_incompatible": {"the candidate protected service or helper cannot prove repository and runtime compatibility", "ob status --output json"}, - "protected_service_patch_unsupported": {"no exact qualified protected current-to-candidate transition exists", "ob status --output json"}, - "protection_disable_pending": {"protection removal is waiting for an authorized safe prerequisite reversal", "ob status --output json"}, - "protection_disablement_not_authorized": {"protection disablement requires a fresh local confirmation bound to current state", "ob status --output json"}, - "protection_disablement_overdue": {"protection disablement remains pending beyond its action deadline", "ob status --output json"}, - "protection_enablement_restart_not_authorized": {"a restart-bound protection prerequisite lacks fresh local confirmation", "ob validate --output json"}, - "protection_image_revert_unsafe": {"the requested image reversion would strand an effective protection prerequisite", "ob status --output json"}, - "protection_image_update_overdue": {"a qualified protected service image publication missed its maintenance target", "ob status --output json"}, - "protection_prerequisite_drifted": {"a live prerequisite no longer matches the verified protection configuration", "ob validate --output json"}, - "protection_service_image_unpublished": {"no qualified immutable protection image is published for the observed service base", "ob status --output json"}, - "protection_service_patch_available": {"a qualified exact protected service image transition is available", "ob service apply --output ndjson"}, - "protection_service_patch_required": {"protection enablement requires a separate qualified same-major service patch first", "ob service apply --output ndjson"}, - "recovery_objective_unsupported": {"the selected driver, version, or target cannot execute the declared recovery kind", "ob validate --output json"}, - "replay_continuity_broken": {"the native replay sequence has a gap inside the required recovery window", "ob plan --output json"}, - "restore_drill_schedule_too_sparse": {"the restore-drill cadence cannot keep restore proof current", "ob validate --output json"}, - "restore_state_stale": {"live service, volume, or repository state changed after restore planning", "ob status --output json"}, - "service_image_digest_unavailable": {"the exact immutable service image required by recovery is unavailable", "ob status --output json"}, - "service_image_patch_disable_pending": {"service image refresh is refused while safe protection disablement is pending", "ob status --output json"}, - "service_major_upgrade_unsupported": {"the requested service image transition crosses an unsupported major version", "ob status --output json"}, + "backup_conflict": {"another protected-service operation holds the serialization boundary", "ob status --output json"}, + "backup_driver_unsupported": {"the service driver has no qualified executable backup contract", "ob validate --output json"}, + "backup_encryption_unverified": {"the selected backup destination cannot prove its required encryption mode", "ob validate --output json"}, + "backup_interruption_not_authorized": {"the recovery contract requires a recurring stopped-service window the author did not permit", "ob validate --output json"}, + "backup_retention_unsupported": {"the declared recovery history cannot map to qualified native retention semantics", "ob validate --output json"}, + "backup_target_not_independent": {"the backup target shares the protected failure domain", "ob validate --output json"}, + "backup_target_unknown": {"the backup policy selects no declared backup target", "ob validate --output json"}, + "service_patch_unsupported": {"no exact qualified protected current-to-candidate transition exists", "ob status --output json"}, + "backup_disable_pending": {"backup removal is waiting for an authorized safe prerequisite reversal", "ob status --output json"}, + "backup_disablement_overdue": {"backup disablement remains pending beyond its action deadline", "ob status --output json"}, + "backup_image_revert_unsafe": {"the requested image reversion would strand an effective backup prerequisite", "ob status --output json"}, + "backup_service_image_unpublished": {"no qualified immutable backup image is published for the observed service base", "ob status --output json"}, + "recovery_objective_unsupported": {"the selected driver, version, or target cannot execute the declared recovery kind", "ob validate --output json"}, + "drill_schedule_too_sparse": {"the declared drill cadence is too sparse to keep restore proof within its maximum age", "ob validate --output json"}, + "service_image_digest_unavailable": {"the exact immutable service image required by recovery is unavailable", "ob status --output json"}, + "service_image_patch_disable_pending": {"service image refresh is refused while safe backup disablement is pending", "ob status --output json"}, } func NewLifecycleFailure(code string) (LifecycleFailure, error) { @@ -143,7 +124,7 @@ func GuidanceRoleForCommand(command string) string { for _, prefix := range []string{ "ob status", "ob validate", "ob doctor", "ob audit", "ob help", "ob canonical", "ob preflight", "ob preview", "ob schema", "ob version", "ob logs", - "ob secrets list", "ob service status", "ob protection status", "ob assurance inspect", + "ob secrets list", "ob service status", "ob backup status", "ob assurance inspect", "ob backup target inspect", "ob backup list", "ob backup inspect", "ob restore inspect", "ob housekeeping status", } { @@ -180,35 +161,3 @@ func safeGuidanceCommand(command string) bool { } return true } - -// reservedLifecycleFailures are codes the contract enumerates that no path -// raises today. The operations behind them exist in the model — protection -// enable and disable among them — but are wired to no CLI verb, so an operator -// cannot reach them. They are kept so the code set is stable when those land, -// and named here so the reference can mark them rather than presenting a -// failure that cannot occur. -var reservedLifecycleFailures = map[string]struct{}{ - "assurance_stale": {}, - "backup_stale": {}, - "disk_pressure_critical": {}, - "drill_deferred_capacity": {}, - "external_service_not_owned": {}, - "external_service_state_stale": {}, - "protected_service_patch_incompatible": {}, - "protection_enablement_restart_not_authorized": {}, - "protection_image_update_overdue": {}, - "protection_prerequisite_drifted": {}, - "protection_service_patch_available": {}, - "protection_service_patch_required": {}, - "replay_continuity_broken": {}, - "restore_state_stale": {}, - "service_major_upgrade_unsupported": {}, -} - -// LifecycleFailureReserved reports whether a code is enumerated but not yet -// reachable, so the reference can mark it rather than presenting a failure an -// operator cannot cause. -func LifecycleFailureReserved(code string) bool { - _, reserved := reservedLifecycleFailures[code] - return reserved -} diff --git a/internal/onebox/lifecycle_errors_test.go b/internal/onebox/lifecycle_errors_test.go index 09920659..252d1e52 100644 --- a/internal/onebox/lifecycle_errors_test.go +++ b/internal/onebox/lifecycle_errors_test.go @@ -9,41 +9,22 @@ import ( func TestEveryDeltaSpecFailureHasASecretFreeGuidanceContract(t *testing.T) { expected := []string{ - "assurance_stale", "backup_conflict", + "backup_disable_pending", + "backup_disablement_overdue", "backup_driver_unsupported", "backup_encryption_unverified", + "backup_image_revert_unsafe", "backup_interruption_not_authorized", "backup_retention_unsupported", - "backup_stale", + "backup_service_image_unpublished", "backup_target_not_independent", - "backup_target_unauthorized", "backup_target_unknown", - "backup_target_unreachable", - "disk_pressure_critical", - "drill_deferred_capacity", - "external_service_not_owned", - "external_service_state_stale", - "protected_service_identity_changed", - "protected_service_patch_incompatible", - "protected_service_patch_unsupported", - "protection_disable_pending", - "protection_disablement_not_authorized", - "protection_disablement_overdue", - "protection_enablement_restart_not_authorized", - "protection_image_revert_unsafe", - "protection_image_update_overdue", - "protection_prerequisite_drifted", - "protection_service_image_unpublished", - "protection_service_patch_available", - "protection_service_patch_required", + "drill_schedule_too_sparse", "recovery_objective_unsupported", - "replay_continuity_broken", - "restore_drill_schedule_too_sparse", - "restore_state_stale", "service_image_digest_unavailable", "service_image_patch_disable_pending", - "service_major_upgrade_unsupported", + "service_patch_unsupported", } if got := LifecycleFailureCodes(); !reflect.DeepEqual(got, expected) { t.Fatalf("lifecycle failure registry =\n%q\nwant\n%q", got, expected) @@ -75,26 +56,12 @@ func TestEveryDeltaSpecFailureHasASecretFreeGuidanceContract(t *testing.T) { } } - record := validLifecycleResultRecord(LifecycleBackupCreate, "postgres") - record.Result.TerminalState = "failed" - record.Result.ErrorCode = failure.Code - switch failure.GuidanceRole() { - case "diagnostic": - record.Result.DiagnosticCommands = []string{failure.GuidanceCommand()} - case "next": - record.Result.NextCommands = []string{failure.GuidanceCommand()} - case "resolving": - record.Result.ResolvingCommands = []string{failure.GuidanceCommand()} - } - if err := record.Validate(); err != nil { - t.Fatalf("failure does not fit lifecycle result contract: %v", err) - } }) } } func TestLifecycleFailureValidationDoesNotReflectUnsafeReplacement(t *testing.T) { - failure, err := NewLifecycleFailure("backup_target_unreachable") + failure, err := NewLifecycleFailure("backup_target_unknown") if err != nil { t.Fatal(err) } diff --git a/internal/onebox/lifecycle_operation_graph.go b/internal/onebox/lifecycle_operation_graph.go deleted file mode 100644 index ed0c3230..00000000 --- a/internal/onebox/lifecycle_operation_graph.go +++ /dev/null @@ -1,192 +0,0 @@ -package onebox - -import ( - "errors" - "fmt" - "regexp" -) - -const ( - LifecycleCLIRunnerSchema = "onebox.run/lifecycle-cli-runner/v1alpha1" - LifecycleScheduledRunnerSchema = "onebox.run/lifecycle-scheduled-runner/v1alpha1" - LifecycleArchiveRunnerSchema = "onebox.run/lifecycle-archive-runner/v1alpha1" - RestrictedArchiveSchemaVersion = "onebox.run/restricted-archive-envelope/v1alpha1" -) - -// LifecycleOperationSchema is the single dispatch record shared by CLI and -// scheduled adapters. Runners are admitted by their exact schema, never by a -// broad "local" or "trusted" class. -type LifecycleOperationSchema struct { - Kind OperationKind - EventKind LifecycleKind - Risk RiskClass - Reversibility ReversibilityClass - Approval ApprovalClass - RunnerSchemas []string -} - -var lifecycleOperationRegistry = map[OperationKind]LifecycleOperationSchema{ - KindServiceImagePatch: lifecycleSchema(KindServiceImagePatch, LifecycleServiceImagePatch, RiskHigh, ReversibilityConditional, ApprovalStrong), - KindProtectionEnable: lifecycleSchema(KindProtectionEnable, LifecycleProtectionEnable, RiskHigh, ReversibilityConditional, ApprovalStrong), - KindProtectionDisable: lifecycleSchema(KindProtectionDisable, LifecycleProtectionDisable, RiskHigh, ReversibilityConditional, ApprovalStrong), - KindBackupCreate: scheduledLifecycleSchema(KindBackupCreate, LifecycleBackupCreate, RiskModerate, ReversibilityConditional, ApprovalStanding), - KindBackupPrune: scheduledLifecycleSchema(KindBackupPrune, LifecycleBackupPrune, RiskHigh, ReversibilityIrreversible, ApprovalStanding), - KindReplayArchive: archiveLifecycleSchema(KindReplayArchive, LifecycleReplayArchive, RiskLow, ReversibilityReversible, ApprovalStanding), - KindRestoreTest: scheduledLifecycleSchema(KindRestoreTest, LifecycleRestoreTest, RiskModerate, ReversibilityConditional, ApprovalStanding), - KindRestorePrepare: lifecycleSchema(KindRestorePrepare, LifecycleRestorePrepare, RiskHigh, ReversibilityConditional, ApprovalStrong), - KindRestoreCutover: lifecycleSchema(KindRestoreCutover, LifecycleRestoreCutover, RiskCritical, ReversibilityConditional, ApprovalStrong), - KindRestoreAbort: lifecycleSchema(KindRestoreAbort, LifecycleRestoreAbort, RiskHigh, ReversibilityConditional, ApprovalStrong), - KindHygieneRun: scheduledLifecycleSchema(KindHygieneRun, LifecycleHygieneRun, RiskModerate, ReversibilityConditional, ApprovalStanding), - KindAssuranceCheck: scheduledLifecycleSchema(KindAssuranceCheck, LifecycleAssuranceCheck, RiskLow, ReversibilityReversible, ApprovalNone), -} - -var lifecycleEventRegistry = buildLifecycleEventRegistry() - -func lifecycleSchema(kind OperationKind, eventKind LifecycleKind, risk RiskClass, reversibility ReversibilityClass, approval ApprovalClass) LifecycleOperationSchema { - return LifecycleOperationSchema{ - Kind: kind, EventKind: eventKind, Risk: risk, Reversibility: reversibility, Approval: approval, - RunnerSchemas: []string{LifecycleCLIRunnerSchema}, - } -} - -func scheduledLifecycleSchema(kind OperationKind, eventKind LifecycleKind, risk RiskClass, reversibility ReversibilityClass, approval ApprovalClass) LifecycleOperationSchema { - schema := lifecycleSchema(kind, eventKind, risk, reversibility, approval) - schema.RunnerSchemas = append(schema.RunnerSchemas, LifecycleScheduledRunnerSchema) - return schema -} - -func archiveLifecycleSchema(kind OperationKind, eventKind LifecycleKind, risk RiskClass, reversibility ReversibilityClass, approval ApprovalClass) LifecycleOperationSchema { - schema := scheduledLifecycleSchema(kind, eventKind, risk, reversibility, approval) - schema.RunnerSchemas = append(schema.RunnerSchemas, LifecycleArchiveRunnerSchema) - return schema -} - -func buildLifecycleEventRegistry() map[LifecycleKind]OperationKind { - registry := make(map[LifecycleKind]OperationKind, len(lifecycleOperationRegistry)+1) - for kind, schema := range lifecycleOperationRegistry { - registry[schema.EventKind] = kind - } - registry[LifecycleServiceTierStatus] = "" - return registry -} - -// LifecycleOperationSchemaFor performs exact runner-schema dispatch and -// returns a copy so callers cannot mutate the canonical registry. -func LifecycleOperationSchemaFor(kind OperationKind, runnerSchema string) (LifecycleOperationSchema, error) { - schema, ok := lifecycleOperationRegistry[kind] - if !ok { - return LifecycleOperationSchema{}, fmt.Errorf("unsupported lifecycle operation %q", kind) - } - if !stringIn(schema.RunnerSchemas, runnerSchema) { - return LifecycleOperationSchema{}, fmt.Errorf("unsupported runner schema %q for lifecycle operation %q", runnerSchema, kind) - } - schema.RunnerSchemas = append([]string(nil), schema.RunnerSchemas...) - return schema, nil -} - -// LifecycleOperationGraph returns the deterministic canonical steps for one -// lifecycle operation. The archive-hook runner receives a deliberately -// smaller graph that can only append replay evidence. -func LifecycleOperationGraph(kind OperationKind, runnerSchema, service string) ([]OperationStep, error) { - if !safeLifecycleMetadata(service) { - return nil, errors.New("service is required and must be secret-free metadata") - } - if _, err := LifecycleOperationSchemaFor(kind, runnerSchema); err != nil { - return nil, err - } - if runnerSchema == LifecycleArchiveRunnerSchema { - return chainedLifecycleSteps( - OperationStep{ID: "archive-preflight", Kind: StepPreflight, Service: service, DataEffect: DataEffectNone}, - OperationStep{ID: "archive-append", Kind: StepArchiveAppend, Service: service, DataEffect: DataEffectNone, Mutation: true}, - OperationStep{ID: "archive-record", Kind: StepLifecycleRecord, Service: service, DataEffect: DataEffectNone, Mutation: true}, - ), nil - } - return chainedLifecycleSteps( - OperationStep{ID: "protection-lock", Kind: StepProtectionLock, Service: service, DataEffect: DataEffectNone}, - OperationStep{ID: "preflight", Kind: StepPreflight, Service: service, DataEffect: DataEffectNone}, - OperationStep{ID: "execute", Kind: StepLifecycleAction, Service: service, DataEffect: DataEffectUnknown, Mutation: true}, - OperationStep{ID: "verify", Kind: StepVerify, Service: service, DataEffect: DataEffectNone}, - OperationStep{ID: "record", Kind: StepLifecycleRecord, Service: service, DataEffect: DataEffectNone, Mutation: true}, - ), nil -} - -func chainedLifecycleSteps(steps ...OperationStep) []OperationStep { - for index := range steps { - if index > 0 { - steps[index].DependsOn = []string{steps[index-1].ID} - } - } - return steps -} - -// RestrictedArchiveEnvelope is the only database-hook operation envelope. It -// cannot name a generic operation or runner and binds all writes to one service -// and one already-sealed lifecycle state. -type RestrictedArchiveEnvelope struct { - SchemaVersion string `json:"schema_version"` - RunnerSchema string `json:"runner_schema"` - OperationID string `json:"operation_id"` - Kind OperationKind `json:"kind"` - Service string `json:"service"` - StateDigest string `json:"state_digest"` - HelperDigest string `json:"helper_digest"` -} - -func (envelope RestrictedArchiveEnvelope) Validate() error { - if envelope.SchemaVersion != RestrictedArchiveSchemaVersion { - return fmt.Errorf("unsupported restricted archive envelope schema %q", envelope.SchemaVersion) - } - if envelope.RunnerSchema != LifecycleArchiveRunnerSchema { - return fmt.Errorf("restricted archive envelope requires runner schema %q", LifecycleArchiveRunnerSchema) - } - if envelope.Kind != KindReplayArchive { - return fmt.Errorf("restricted archive envelope cannot invoke operation %q", envelope.Kind) - } - if !safeLifecycleMetadata(envelope.OperationID) || !safeLifecycleMetadata(envelope.Service) { - return errors.New("restricted archive operation and service identity must be safe metadata") - } - if !lifecycleGraphDigest.MatchString(envelope.StateDigest) || !lifecycleGraphDigest.MatchString(envelope.HelperDigest) { - return errors.New("restricted archive state and helper digests must be pinned") - } - _, err := LifecycleOperationSchemaFor(envelope.Kind, envelope.RunnerSchema) - return err -} - -func (envelope RestrictedArchiveEnvelope) OperationGraph() ([]OperationStep, error) { - if err := envelope.Validate(); err != nil { - return nil, err - } - return LifecycleOperationGraph(envelope.Kind, envelope.RunnerSchema, envelope.Service) -} - -func validateLifecycleOperationRegistry() error { - if len(lifecycleOperationRegistry) != 12 { - return fmt.Errorf("lifecycle operation registry has %d schemas, want 12", len(lifecycleOperationRegistry)) - } - if len(lifecycleEventRegistry) != len(lifecycleOperationRegistry)+1 { - return errors.New("structured lifecycle event registry is incomplete or ambiguous") - } - for kind, schema := range lifecycleOperationRegistry { - if kind != schema.Kind || OperationKind(schema.EventKind) != kind { - return fmt.Errorf("lifecycle operation %q has mismatched event kind %q", kind, schema.EventKind) - } - if !validOperationKind(kind) || !validRiskClass(schema.Risk) || !validReversibilityClass(schema.Reversibility) || !validApprovalClass(schema.Approval) { - return fmt.Errorf("lifecycle operation %q has invalid plan metadata", kind) - } - if len(schema.RunnerSchemas) == 0 || schema.RunnerSchemas[0] != LifecycleCLIRunnerSchema { - return fmt.Errorf("lifecycle operation %q is unavailable to the canonical CLI", kind) - } - } - return nil -} - -func stringIn(values []string, want string) bool { - for _, value := range values { - if value == want { - return true - } - } - return false -} - -var lifecycleGraphDigest = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) diff --git a/internal/onebox/lifecycle_operation_graph_test.go b/internal/onebox/lifecycle_operation_graph_test.go deleted file mode 100644 index 2117701f..00000000 --- a/internal/onebox/lifecycle_operation_graph_test.go +++ /dev/null @@ -1,140 +0,0 @@ -package onebox - -import ( - "strings" - "testing" - "time" -) - -var allLifecycleOperationKinds = []OperationKind{ - KindServiceImagePatch, - KindProtectionEnable, - KindProtectionDisable, - KindBackupCreate, - KindBackupPrune, - KindReplayArchive, - KindRestoreTest, - KindRestorePrepare, - KindRestoreCutover, - KindRestoreAbort, - KindHygieneRun, - KindAssuranceCheck, -} - -func TestLifecycleOperationSchemaDispatchesEveryCanonicalKind(t *testing.T) { - if err := validateLifecycleOperationRegistry(); err != nil { - t.Fatalf("validate lifecycle operation registry: %v", err) - } - - createdAt := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC) - for _, kind := range allLifecycleOperationKinds { - t.Run(string(kind), func(t *testing.T) { - schema, err := LifecycleOperationSchemaFor(kind, LifecycleCLIRunnerSchema) - if err != nil { - t.Fatalf("dispatch CLI schema: %v", err) - } - steps, err := LifecycleOperationGraph(kind, LifecycleCLIRunnerSchema, "database") - if err != nil { - t.Fatalf("build operation graph: %v", err) - } - plan := OperationPlan{ - SchemaVersion: OperationPlanSchemaVersion, - ID: "operation-1", - Kind: kind, - CreatedAt: createdAt.Format(time.RFC3339), - ExpiresAt: createdAt.Add(time.Hour).Format(time.RFC3339), - Risk: schema.Risk, - Reversibility: schema.Reversibility, - Approval: schema.Approval, - Binding: OperationBinding{ - Application: "example", Environment: "production", Server: "host", - ConfigDigest: "config", ComposeDigest: "compose", StateDigest: "state", - }, - Steps: steps, - } - if err := plan.Seal(); err != nil { - t.Fatalf("seal dispatched plan: %v", err) - } - - record := LifecycleRecord{ - SchemaVersion: LifecycleRecordSchemaVersion, - Type: LifecycleRecordEvent, - OperationID: plan.ID, - Kind: schema.EventKind, - Service: "database", - Event: &LifecycleEvent{ - Sequence: 1, Time: createdAt.Format(time.RFC3339), Phase: "preflight", State: "started", - }, - } - if kind == KindServiceImagePatch { - record.PatchScope = "protected" - } - if err := record.Validate(); err != nil { - t.Fatalf("validate dispatched structured event: %v", err) - } - }) - } -} - -func TestScheduledRunnerHasClosedLifecycleAllowlist(t *testing.T) { - allowed := map[OperationKind]bool{ - KindBackupCreate: true, KindBackupPrune: true, KindReplayArchive: true, - KindRestoreTest: true, KindHygieneRun: true, - KindAssuranceCheck: true, - } - for _, kind := range allLifecycleOperationKinds { - _, err := LifecycleOperationSchemaFor(kind, LifecycleScheduledRunnerSchema) - if allowed[kind] && err != nil { - t.Errorf("scheduled runner rejected %q: %v", kind, err) - } - if !allowed[kind] && err == nil { - t.Errorf("scheduled runner accepted operator-only operation %q", kind) - } - } - for _, kind := range []OperationKind{KindServiceImagePatch, KindProtectionEnable, KindProtectionDisable} { - if _, err := LifecycleOperationGraph(kind, LifecycleScheduledRunnerSchema, "database"); err == nil { - t.Errorf("scheduled runner can execute forbidden operation %q", kind) - } - } -} - -func TestRestrictedArchiveEnvelopeCanOnlyAppendReplayEvidence(t *testing.T) { - envelope := RestrictedArchiveEnvelope{ - SchemaVersion: RestrictedArchiveSchemaVersion, - RunnerSchema: LifecycleArchiveRunnerSchema, - OperationID: "archive-1", - Kind: KindReplayArchive, - Service: "database", - StateDigest: "sha256:" + strings.Repeat("a", 64), - HelperDigest: "sha256:" + strings.Repeat("b", 64), - } - steps, err := envelope.OperationGraph() - if err != nil { - t.Fatalf("build restricted archive graph: %v", err) - } - if got, want := len(steps), 3; got != want { - t.Fatalf("restricted archive graph has %d steps, want %d", got, want) - } - if steps[1].Kind != StepArchiveAppend { - t.Fatalf("restricted archive mutation = %q, want %q", steps[1].Kind, StepArchiveAppend) - } - for _, step := range steps { - if step.Kind == StepLifecycleAction { - t.Fatal("restricted archive envelope can invoke an arbitrary lifecycle action") - } - } - - envelope.Kind = KindBackupCreate - if err := envelope.Validate(); err == nil { - t.Fatal("restricted archive envelope accepted backup_create") - } -} - -func TestLifecycleOperationRejectsUnsupportedRunnerSchema(t *testing.T) { - if _, err := LifecycleOperationSchemaFor(KindBackupCreate, "onebox.run/unknown-runner/v1"); err == nil { - t.Fatal("unknown lifecycle runner schema was accepted") - } - if _, err := LifecycleOperationSchemaFor(KindDeploy, LifecycleCLIRunnerSchema); err == nil { - t.Fatal("non-lifecycle operation dispatched through lifecycle registry") - } -} diff --git a/internal/onebox/lifecycle_reach_test.go b/internal/onebox/lifecycle_reach_test.go index ecc86f89..5d854392 100644 --- a/internal/onebox/lifecycle_reach_test.go +++ b/internal/onebox/lifecycle_reach_test.go @@ -12,30 +12,19 @@ var lifecycleCodeLiteral = regexp.MustCompile(`"([a-z][a-z0-9_]{4,})"`) // A code is a promise. The loader's table is guarded in both directions // (app.TestEveryErrorCodeIsEnumerated); the lifecycle table was guarded in -// neither, so it accumulated codes nothing can raise while the reference page -// described it as "every typed failure code Onebox can emit". -// -// The capabilities behind reservedLifecycleFailures exist in the operations -// model but are wired to no command yet. They stay enumerated so the contract -// is stable when they land — but they are named here, and the page marks them, -// so nobody reads them as reachable. -func TestReservedLifecycleFailuresAreExactlyTheUnreachableOnes(t *testing.T) { +// neither, so it accumulated codes nothing could raise while the reference page +// described it as "every typed failure code Onebox can emit". Nineteen of them +// were carried as "reserved" — enumerated, documented, and unreachable — for +// operations that were never built. They are gone; this is the guard that keeps +// the table honest. +func TestEveryEnumeratedLifecycleFailureIsRaisedBySomePath(t *testing.T) { emitted := emittedLifecycleCodes(t) if len(emitted) == 0 { t.Fatal("found no lifecycle codes in the package source; the scan is broken") } for code := range lifecycleFailureDefinitions { - _, reserved := reservedLifecycleFailures[code] - switch { - case emitted[code] && reserved: - t.Errorf("%q is emitted but listed as reserved: remove it from reservedLifecycleFailures", code) - case !emitted[code] && !reserved: - t.Errorf("%q is enumerated but nothing raises it: reserve it, or the table promises a failure that cannot happen", code) - } - } - for code := range reservedLifecycleFailures { - if _, ok := lifecycleFailureDefinitions[code]; !ok { - t.Errorf("%q is reserved but not enumerated", code) + if !emitted[code] { + t.Errorf("%q is enumerated but nothing raises it: delete it, or the table promises a failure that cannot happen", code) } } } diff --git a/internal/onebox/lifecycle_records.go b/internal/onebox/lifecycle_records.go deleted file mode 100644 index 1f695dd6..00000000 --- a/internal/onebox/lifecycle_records.go +++ /dev/null @@ -1,393 +0,0 @@ -package onebox - -import ( - "bufio" - "bytes" - "encoding/json" - "errors" - "fmt" - "io" - "regexp" -) - -const ( - LifecycleRecordSchemaVersion = "onebox.run/lifecycle-record/v1alpha2" - LifecycleDocumentSchemaVersion = "onebox.run/lifecycle-document/v1alpha1" -) - -type LifecycleKind string - -const ( - LifecycleProtectionEnable LifecycleKind = "protection_enable" - LifecycleProtectionDisable LifecycleKind = "protection_disable" - LifecycleServiceImagePatch LifecycleKind = "service_image_patch" - LifecycleBackupCreate LifecycleKind = "backup_create" - LifecycleBackupPrune LifecycleKind = "backup_prune" - LifecycleReplayArchive LifecycleKind = "replay_archive" - LifecycleRestorePrepare LifecycleKind = "restore_prepare" - LifecycleRestoreCutover LifecycleKind = "restore_cutover" - LifecycleRestoreAbort LifecycleKind = "restore_abort" - LifecycleRestoreTest LifecycleKind = "restore_test" - LifecycleHygieneRun LifecycleKind = "hygiene_run" - LifecycleAssuranceCheck LifecycleKind = "assurance_check" - LifecycleServiceTierStatus LifecycleKind = "service_tier_status" -) - -type LifecycleRecordType string - -const ( - LifecycleRecordEvent LifecycleRecordType = "event" - LifecycleRecordResult LifecycleRecordType = "result" - LifecycleRecordStatus LifecycleRecordType = "service-tier-status" -) - -type LifecycleRecord struct { - SchemaVersion string `json:"schema_version"` - Type LifecycleRecordType `json:"type"` - OperationID string `json:"operation_id"` - Kind LifecycleKind `json:"kind"` - PatchScope string `json:"patch_scope,omitempty"` - Service string `json:"service"` - Event *LifecycleEvent `json:"event,omitempty"` - Result *LifecycleResult `json:"result,omitempty"` - Status *ServiceTierStatus `json:"status,omitempty"` -} - -type LifecycleEvent struct { - Sequence int `json:"sequence"` - Time string `json:"time"` - Phase string `json:"phase"` - State string `json:"state"` - EvidenceID string `json:"evidence_id,omitempty"` - NativeEvidence *NativeEvidenceIdentity `json:"native_evidence,omitempty"` - Recovery *RecoveryEnvelope `json:"recovery,omitempty"` -} - -type LifecycleResult struct { - TerminalState string `json:"terminal_state"` - FinishedAt string `json:"finished_at"` - EvidenceID string `json:"evidence_id,omitempty"` - NativeEvidence *NativeEvidenceIdentity `json:"native_evidence,omitempty"` - Recovery *RecoveryEnvelope `json:"recovery,omitempty"` - ErrorCode string `json:"error_code,omitempty"` - DiagnosticCommands []string `json:"diagnostic_commands,omitempty"` - NextCommands []string `json:"next_commands,omitempty"` - ResolvingCommands []string `json:"resolving_commands,omitempty"` -} - -type ServiceTierStatus struct { - Tier string `json:"tier"` - ObservedAt string `json:"observed_at"` - EvidenceID string `json:"evidence_id,omitempty"` - NativeEvidence *NativeEvidenceIdentity `json:"native_evidence,omitempty"` - Recovery *RecoveryEnvelope `json:"recovery,omitempty"` - Codes []string `json:"codes,omitempty"` - DiagnosticCommands []string `json:"diagnostic_commands,omitempty"` - NextCommands []string `json:"next_commands,omitempty"` - ResolvingCommands []string `json:"resolving_commands,omitempty"` -} - -type NativeEvidenceIdentity struct { - Driver string `json:"driver"` - Method string `json:"method"` - RepositoryID string `json:"repository_id,omitempty"` - GenerationID string `json:"generation_id,omitempty"` - ReplayStart string `json:"replay_start,omitempty"` - ReplayEnd string `json:"replay_end,omitempty"` - ObservationID string `json:"observation_id,omitempty"` -} - -type RecoveryEnvelope struct { - Kind string `json:"kind"` - LatestRecoveryPoint string `json:"latest_recovery_point,omitempty"` - WindowStart string `json:"window_start,omitempty"` - WindowEnd string `json:"window_end,omitempty"` - ObservedRPO string `json:"observed_rpo,omitempty"` - ExpectedInterruption string `json:"expected_interruption"` - EncryptionMode string `json:"encryption_mode"` -} - -type LifecycleDocument struct { - SchemaVersion string `json:"schema_version"` - Records []LifecycleRecord `json:"records"` -} - -var lifecycleMetadata = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:/-]{0,511}$`) - -func EncodeLifecycleJSON(records []LifecycleRecord) ([]byte, error) { - for i := range records { - if err := records[i].Validate(); err != nil { - return nil, fmt.Errorf("record %d: %w", i, err) - } - } - return json.MarshalIndent(LifecycleDocument{SchemaVersion: LifecycleDocumentSchemaVersion, Records: records}, "", " ") -} - -func DecodeLifecycleJSON(body []byte) (LifecycleDocument, error) { - decoder := json.NewDecoder(bytes.NewReader(body)) - decoder.DisallowUnknownFields() - var document LifecycleDocument - if err := decoder.Decode(&document); err != nil { - return LifecycleDocument{}, fmt.Errorf("decode lifecycle document: %w", err) - } - if err := lifecycleJSONEOF(decoder); err != nil { - return LifecycleDocument{}, err - } - if document.SchemaVersion != LifecycleDocumentSchemaVersion { - return LifecycleDocument{}, fmt.Errorf("unsupported lifecycle document schema %q", document.SchemaVersion) - } - for i := range document.Records { - if err := document.Records[i].Validate(); err != nil { - return LifecycleDocument{}, fmt.Errorf("record %d: %w", i, err) - } - } - return document, nil -} - -func EncodeLifecycleNDJSON(writer io.Writer, records []LifecycleRecord) error { - encoder := json.NewEncoder(writer) - for i := range records { - if err := records[i].Validate(); err != nil { - return fmt.Errorf("record %d: %w", i, err) - } - if err := encoder.Encode(records[i]); err != nil { - return fmt.Errorf("encode lifecycle record %d: %w", i, err) - } - } - return nil -} - -func DecodeLifecycleNDJSON(reader io.Reader) ([]LifecycleRecord, error) { - scanner := bufio.NewScanner(io.LimitReader(reader, 16<<20)) - scanner.Buffer(make([]byte, 64<<10), 1<<20) - var records []LifecycleRecord - for scanner.Scan() { - line := bytes.TrimSpace(scanner.Bytes()) - if len(line) == 0 { - continue - } - decoder := json.NewDecoder(bytes.NewReader(line)) - decoder.DisallowUnknownFields() - var record LifecycleRecord - if err := decoder.Decode(&record); err != nil { - return nil, fmt.Errorf("decode lifecycle record %d: %w", len(records), err) - } - if err := lifecycleJSONEOF(decoder); err != nil { - return nil, fmt.Errorf("decode lifecycle record %d: %w", len(records), err) - } - if err := record.Validate(); err != nil { - return nil, fmt.Errorf("record %d: %w", len(records), err) - } - records = append(records, record) - } - if err := scanner.Err(); err != nil { - return nil, fmt.Errorf("read lifecycle records: %w", err) - } - return records, nil -} - -func (record LifecycleRecord) Validate() error { - if record.SchemaVersion != LifecycleRecordSchemaVersion { - return fmt.Errorf("unsupported lifecycle record schema %q", record.SchemaVersion) - } - if !validLifecycleKind(record.Kind) { - return fmt.Errorf("unsupported lifecycle kind %q", record.Kind) - } - if !safeLifecycleMetadata(record.OperationID) { - return errors.New("operation_id is required and must be secret-free metadata") - } - if !safeLifecycleMetadata(record.Service) { - return errors.New("service is required and must be secret-free metadata") - } - if record.Kind == LifecycleServiceImagePatch { - if record.PatchScope != "pre-protection" && record.PatchScope != "protected" { - return errors.New("service_image_patch requires patch_scope pre-protection or protected") - } - } else if record.PatchScope != "" { - return errors.New("patch_scope belongs only to service_image_patch") - } - - set := 0 - if record.Event != nil { - set++ - } - if record.Result != nil { - set++ - } - if record.Status != nil { - set++ - } - if set != 1 { - return errors.New("a lifecycle record must contain exactly one event, result, or status") - } - switch record.Type { - case LifecycleRecordEvent: - if record.Event == nil { - return errors.New("event record has no event") - } - return record.Event.validate() - case LifecycleRecordResult: - if record.Result == nil { - return errors.New("result record has no result") - } - return record.Result.validate() - case LifecycleRecordStatus: - if record.Kind != LifecycleServiceTierStatus || record.Status == nil { - return errors.New("service-tier-status record has the wrong kind or no status") - } - return record.Status.validate() - default: - return fmt.Errorf("unsupported lifecycle record type %q", record.Type) - } -} - -func (event LifecycleEvent) validate() error { - if event.Sequence <= 0 || !safeLifecycleMetadata(event.Phase) || !oneOf(event.State, "started", "progress", "retrying", "succeeded", "failed", "cancelled") { - return errors.New("event sequence, phase, or state is invalid") - } - if event.Time == "" { - return errors.New("event time is required") - } - if event.EvidenceID != "" && !safeLifecycleMetadata(event.EvidenceID) { - return errors.New("event evidence_id is not safe metadata") - } - if event.NativeEvidence != nil { - if err := event.NativeEvidence.validate(); err != nil { - return err - } - } - if event.Recovery != nil { - return event.Recovery.validate() - } - return nil -} - -func (result LifecycleResult) validate() error { - if !oneOf(result.TerminalState, "succeeded", "failed", "cancelled", "incomplete") || result.FinishedAt == "" { - return errors.New("result terminal_state or finished_at is invalid") - } - if result.TerminalState == "succeeded" && result.ErrorCode != "" { - return errors.New("a succeeded result cannot carry error_code") - } - guidanceCount := len(result.DiagnosticCommands) + len(result.NextCommands) + len(result.ResolvingCommands) - if result.TerminalState != "succeeded" && (!safeLifecycleMetadata(result.ErrorCode) || guidanceCount == 0) { - return errors.New("a non-success result requires a stable error_code and command guidance") - } - if result.ErrorCode != "" { - if _, err := NewLifecycleFailure(result.ErrorCode); err != nil { - return errors.New("result error_code is not in the stable lifecycle registry") - } - } - if err := validateGuidanceCommands(result.DiagnosticCommands, result.NextCommands, result.ResolvingCommands); err != nil { - return err - } - if result.NativeEvidence != nil { - if err := result.NativeEvidence.validate(); err != nil { - return err - } - } - if result.Recovery != nil { - return result.Recovery.validate() - } - return nil -} - -func (status ServiceTierStatus) validate() error { - if !oneOf(status.Tier, "Run", "Managed", "External") || status.ObservedAt == "" { - return errors.New("service tier or observed_at is invalid") - } - for _, code := range status.Codes { - if !safeLifecycleMetadata(code) { - return errors.New("service tier code is not safe metadata") - } - } - if err := validateGuidanceCommands(status.DiagnosticCommands, status.NextCommands, status.ResolvingCommands); err != nil { - return err - } - if status.NativeEvidence != nil { - if err := status.NativeEvidence.validate(); err != nil { - return err - } - } - if status.Recovery != nil { - return status.Recovery.validate() - } - return nil -} - -func (identity NativeEvidenceIdentity) validate() error { - if !safeLifecycleMetadata(identity.Driver) || !safeLifecycleMetadata(identity.Method) { - return errors.New("native evidence driver and method are required safe metadata") - } - for _, value := range []string{identity.RepositoryID, identity.GenerationID, identity.ReplayStart, identity.ReplayEnd, identity.ObservationID} { - if value != "" && !safeLifecycleMetadata(value) { - return errors.New("native evidence identity contains unsafe metadata") - } - } - return nil -} - -func (recovery RecoveryEnvelope) validate() error { - if !oneOf(recovery.Kind, "snapshot", "pitr", "cold") { - return errors.New("recovery kind is invalid") - } - if !oneOf(recovery.EncryptionMode, "client-side", "archive-password", "server-side-sse") { - return errors.New("recovery encryption_mode is invalid") - } - if recovery.ExpectedInterruption == "" { - return errors.New("recovery expected_interruption is required") - } - for _, value := range []string{recovery.LatestRecoveryPoint, recovery.WindowStart, recovery.WindowEnd} { - if value != "" && !safeLifecycleMetadata(value) { - return errors.New("recovery envelope contains unsafe identity metadata") - } - } - return nil -} - -func validateGuidanceCommands(diagnostic, next, resolving []string) error { - seen := map[string]bool{} - for role, commands := range map[string][]string{ - "diagnostic": diagnostic, "next": next, "resolving": resolving, - } { - for _, command := range commands { - if !safeGuidanceCommand(command) { - return errors.New("guidance command is not a safe Onebox command") - } - if GuidanceRoleForCommand(command) != role { - return fmt.Errorf("%s command is classified as %s guidance", command, GuidanceRoleForCommand(command)) - } - if seen[command] { - return errors.New("guidance command appears in more than one role") - } - seen[command] = true - } - } - return nil -} - -func validLifecycleKind(kind LifecycleKind) bool { - _, ok := lifecycleEventRegistry[kind] - return ok -} - -func safeLifecycleMetadata(value string) bool { return lifecycleMetadata.MatchString(value) } - -func oneOf(value string, allowed ...string) bool { - for _, candidate := range allowed { - if value == candidate { - return true - } - } - return false -} - -func lifecycleJSONEOF(decoder *json.Decoder) error { - var extra any - if err := decoder.Decode(&extra); err == io.EOF { - return nil - } else if err != nil { - return fmt.Errorf("decode lifecycle JSON: %w", err) - } - return errors.New("decode lifecycle JSON: multiple JSON values") -} diff --git a/internal/onebox/lifecycle_records_test.go b/internal/onebox/lifecycle_records_test.go deleted file mode 100644 index 0151ca4e..00000000 --- a/internal/onebox/lifecycle_records_test.go +++ /dev/null @@ -1,163 +0,0 @@ -package onebox - -import ( - "bytes" - "strings" - "testing" -) - -func TestLifecycleRecordKindsEncodeCompatibly(t *testing.T) { - kinds := []LifecycleKind{ - LifecycleProtectionEnable, LifecycleProtectionDisable, - LifecycleBackupCreate, LifecycleReplayArchive, - LifecycleRestorePrepare, LifecycleRestoreCutover, LifecycleRestoreAbort, - LifecycleRestoreTest, LifecycleHygieneRun, LifecycleAssuranceCheck, - } - var records []LifecycleRecord - for _, kind := range kinds { - records = append(records, validLifecycleResultRecord(kind, "postgres")) - } - for _, scope := range []string{"pre-protection", "protected"} { - record := validLifecycleResultRecord(LifecycleServiceImagePatch, "postgres") - record.PatchScope = scope - records = append(records, record) - } - records = append(records, LifecycleRecord{ - SchemaVersion: LifecycleRecordSchemaVersion, - Type: LifecycleRecordEvent, - OperationID: "operation-20260807", - Kind: LifecycleBackupCreate, - Service: "postgres", - Event: &LifecycleEvent{ - Sequence: 1, Time: "2026-08-07T19:59:00Z", Phase: "native-backup", State: "progress", - EvidenceID: "evidence-42", NativeEvidence: validNativeEvidence(), Recovery: validRecoveryEnvelope(), - }, - }) - records = append(records, LifecycleRecord{ - SchemaVersion: LifecycleRecordSchemaVersion, - Type: LifecycleRecordStatus, - OperationID: "status-20260807", - Kind: LifecycleServiceTierStatus, - Service: "postgres", - Status: &ServiceTierStatus{ - Tier: "Managed", ObservedAt: "2026-08-07T20:00:00Z", EvidenceID: "evidence-42", - NativeEvidence: validNativeEvidence(), Recovery: validRecoveryEnvelope(), - }, - }) - - encoded, err := EncodeLifecycleJSON(records) - if err != nil { - t.Fatal(err) - } - document, err := DecodeLifecycleJSON(encoded) - if err != nil { - t.Fatal(err) - } - if document.SchemaVersion != LifecycleDocumentSchemaVersion || len(document.Records) != len(records) { - t.Fatalf("decoded lifecycle document = %#v", document) - } - - var ndjson bytes.Buffer - if err := EncodeLifecycleNDJSON(&ndjson, records); err != nil { - t.Fatal(err) - } - decoded, err := DecodeLifecycleNDJSON(&ndjson) - if err != nil { - t.Fatal(err) - } - if len(decoded) != len(records) { - t.Fatalf("decoded %d NDJSON records, want %d", len(decoded), len(records)) - } - for i, record := range decoded { - if record.SchemaVersion != LifecycleRecordSchemaVersion || record.Kind != records[i].Kind || record.OperationID != records[i].OperationID { - t.Fatalf("record %d compatibility mismatch: %#v", i, record) - } - } -} - -func TestLifecycleDecodersRejectUnknownFields(t *testing.T) { - record := validLifecycleResultRecord(LifecycleBackupCreate, "postgres") - encoded, err := EncodeLifecycleJSON([]LifecycleRecord{record}) - if err != nil { - t.Fatal(err) - } - unknownTop := strings.Replace(string(encoded), `"schema_version": "`+LifecycleDocumentSchemaVersion+`"`, `"unexpected": true, "schema_version": "`+LifecycleDocumentSchemaVersion+`"`, 1) - if _, err := DecodeLifecycleJSON([]byte(unknownTop)); err == nil || !strings.Contains(err.Error(), "unknown field") { - t.Fatalf("top-level unknown field error = %v", err) - } - unknownNested := strings.Replace(string(encoded), `"terminal_state": "succeeded"`, `"unexpected": true, "terminal_state": "succeeded"`, 1) - if _, err := DecodeLifecycleJSON([]byte(unknownNested)); err == nil || !strings.Contains(err.Error(), "unknown field") { - t.Fatalf("nested unknown field error = %v", err) - } - - var ndjson bytes.Buffer - if err := EncodeLifecycleNDJSON(&ndjson, []LifecycleRecord{record}); err != nil { - t.Fatal(err) - } - line := strings.Replace(ndjson.String(), `"result":{`, `"result":{"unexpected":true,`, 1) - if _, err := DecodeLifecycleNDJSON(strings.NewReader(line)); err == nil || !strings.Contains(err.Error(), "unknown field") { - t.Fatalf("NDJSON unknown field error = %v", err) - } -} - -func TestLifecycleRecordsRejectIncompatibleSchemasAndIncompleteFailures(t *testing.T) { - record := validLifecycleResultRecord(LifecycleBackupCreate, "postgres") - record.SchemaVersion = "onebox.run/lifecycle-record/v2" - if err := record.Validate(); err == nil || !strings.Contains(err.Error(), "unsupported lifecycle record schema") { - t.Fatalf("future record schema error = %v", err) - } - - record = validLifecycleResultRecord(LifecycleBackupCreate, "postgres") - record.Result.TerminalState = "failed" - if err := record.Validate(); err == nil || !strings.Contains(err.Error(), "stable error_code") { - t.Fatalf("failure contract error = %v", err) - } - record.Result.ErrorCode = "backup_target_unreachable" - record.Result.DiagnosticCommands = []string{"ob backup target inspect --output json"} - if err := record.Validate(); err != nil { - t.Fatalf("complete typed failure rejected: %v", err) - } - - record.Result.DiagnosticCommands = nil - record.Result.ResolvingCommands = []string{"ob status --output json"} - if err := record.Validate(); err == nil || !strings.Contains(err.Error(), "classified as diagnostic") { - t.Fatalf("misclassified read-only guidance error = %v", err) - } -} - -func validLifecycleResultRecord(kind LifecycleKind, service string) LifecycleRecord { - record := LifecycleRecord{ - SchemaVersion: LifecycleRecordSchemaVersion, - Type: LifecycleRecordResult, - OperationID: "operation-20260807", - Kind: kind, - Service: service, - Result: &LifecycleResult{ - TerminalState: "succeeded", - FinishedAt: "2026-08-07T20:00:00Z", - EvidenceID: "evidence-42", - }, - } - switch kind { - case LifecycleBackupCreate, LifecycleReplayArchive, - LifecycleRestorePrepare, LifecycleRestoreCutover, LifecycleRestoreAbort, LifecycleRestoreTest: - record.Result.NativeEvidence = validNativeEvidence() - record.Result.Recovery = validRecoveryEnvelope() - } - return record -} - -func validNativeEvidence() *NativeEvidenceIdentity { - return &NativeEvidenceIdentity{ - Driver: "postgres", Method: "pgbackrest", RepositoryID: "repository-1", - GenerationID: "generation-42", ReplayStart: "wal:100", ReplayEnd: "wal:200", - } -} - -func validRecoveryEnvelope() *RecoveryEnvelope { - return &RecoveryEnvelope{ - Kind: "pitr", LatestRecoveryPoint: "2026-08-07T19:58:00Z", - WindowStart: "2026-08-01T00:00:00Z", WindowEnd: "2026-08-07T19:58:00Z", - ObservedRPO: "2m", ExpectedInterruption: "none", EncryptionMode: "client-side", - } -} diff --git a/internal/onebox/load.go b/internal/onebox/load.go index 50e5cb29..34b904f7 100644 --- a/internal/onebox/load.go +++ b/internal/onebox/load.go @@ -102,7 +102,7 @@ func (s *Service) loadObservedProject(ctx context.Context, lenient bool, images // // -e follows symlinks, so the -L arm is what keeps a dangling link out of the // 'missing' answer; reporting a broken link as no state at all would drop the -// service's protection silently. An unsearchable ancestor hides it the same +// service's backup silently. An unsearchable ancestor hides it the same // way, which is what UndeterminedArm's exit 5 is for. A live symlink to a // regular file is still read through as 'present': -f follows it, and refusing // symlinked state would be a new rule, not a fix. @@ -130,7 +130,7 @@ func (s *Service) observeServiceRuntimeStates(ctx context.Context, resolved *app names := resolved.NamesFor(resolved.Env) states := map[string]app.ServiceRuntimeState{} for _, service := range sortedNames(resolved.Services) { - statePath := names.ProtectionLifecycleStateFile(service) + statePath := names.BackupLifecycleStateFile(service) result, err := target.Run(ctx, lifecycleStateProbe(statePath)) if err != nil { return nil, fmt.Errorf("observe service %s lifecycle state: %w", service, err) @@ -157,7 +157,7 @@ func (s *Service) observeServiceRuntimeStates(ctx context.Context, resolved *app default: return nil, fmt.Errorf("service %s lifecycle state observation is invalid", service) } - state, err := DecodeProtectionLifecycleState([]byte(encoded)) + state, err := DecodeBackupLifecycleState([]byte(encoded)) if err != nil { return nil, fmt.Errorf("service %s lifecycle state: %w", service, err) } @@ -166,14 +166,26 @@ func (s *Service) observeServiceRuntimeStates(ctx context.Context, resolved *app } runtime := state.RuntimeState() if runtime.ServiceImage != "" { - runtime.DigestAvailable, err = engine.ServiceImageDigestAvailable(ctx, target, runtime.ServiceImage) - if err != nil { - return nil, fmt.Errorf("observe service %s registry image: %w", service, err) - } + // Local cache first, registry only if it misses. + // + // The single consumer of these two flags accepts either one, so a + // host that already holds the exact digest has its answer without + // leaving the machine. Asking the registry first put a + // `docker manifest inspect` against Docker Hub on the front of every + // command that loads a project with a protected service — validate, + // plan, status, backup, all of them — for a digest that is immutable + // and already present. On a rate-limited host that turned every + // command into a failure with nothing to fetch. runtime.CacheVerified, err = engine.ExactServiceImageCached(ctx, target, runtime.ServiceImage) if err != nil { return nil, fmt.Errorf("observe service %s cached image: %w", service, err) } + if !runtime.CacheVerified { + runtime.DigestAvailable, err = engine.ServiceImageDigestAvailable(ctx, target, runtime.ServiceImage) + if err != nil { + return nil, fmt.Errorf("observe service %s registry image: %w", service, err) + } + } } states[service] = runtime } diff --git a/internal/onebox/load_service_runtime_test.go b/internal/onebox/load_service_runtime_test.go index c78b7810..7cbc3520 100644 --- a/internal/onebox/load_service_runtime_test.go +++ b/internal/onebox/load_service_runtime_test.go @@ -30,11 +30,11 @@ services: {database: {driver: postgres, version: 17}} func protectedRuntimeState(t *testing.T, image string) string { t.Helper() - initial, err := NewProtectionLifecycleState("example", "production", "database", 1) + initial, err := NewBackupLifecycleState("example", "production", "database", 1) if err != nil { t.Fatal(err) } - state, err := EnableProtection(initial, protectionStateProjection(), image, "enable-op", true, 2) + state, err := EnableBackup(initial, backupStateProjection(), image, "postgres:18", "enable-op", true, 2) if err != nil { t.Fatal(err) } diff --git a/internal/onebox/operation_errors.go b/internal/onebox/operation_errors.go index d7069401..b5f3c32b 100644 --- a/internal/onebox/operation_errors.go +++ b/internal/onebox/operation_errors.go @@ -10,7 +10,7 @@ import ( // OperationFailure is the public definition of a failure the CLI and engine // raise while running a command, as distinct from a project-file validation // code (which the loader owns) and a lifecycle failure code (which the -// protection contract owns). Those two families were already enumerated and +// backup contract owns). Those two families were already enumerated and // published; these were not, so an operator or agent branching on a code the // binary actually emits had nothing to read. // diff --git a/internal/onebox/operation_types.go b/internal/onebox/operation_types.go index 358b1071..c65d5413 100644 --- a/internal/onebox/operation_types.go +++ b/internal/onebox/operation_types.go @@ -10,6 +10,7 @@ import ( "io" "os" "path/filepath" + "regexp" "strings" "time" @@ -41,8 +42,8 @@ const ( KindJobRun OperationKind = "job_run" KindServiceImagePatch OperationKind = "service_image_patch" - KindProtectionEnable OperationKind = "protection_enable" - KindProtectionDisable OperationKind = "protection_disable" + KindBackupEnable OperationKind = "backup_enable" + KindBackupDisable OperationKind = "backup_disable" KindBackupCreate OperationKind = "backup_create" KindBackupPrune OperationKind = "backup_prune" KindReplayArchive OperationKind = "replay_archive" @@ -99,7 +100,7 @@ const ( StepWorkloadRelease OperationStepKind = "workload_release" StepVerify OperationStepKind = "verify" StepActivate OperationStepKind = "activate" - StepProtectionLock OperationStepKind = "protection_lock" + StepBackupLock OperationStepKind = "backup_lock" StepLifecycleAction OperationStepKind = "lifecycle_action" StepLifecycleRecord OperationStepKind = "lifecycle_record" StepArchiveAppend OperationStepKind = "archive_append" @@ -448,7 +449,7 @@ func validOperationKind(kind OperationKind) bool { switch kind { case KindDeploy, KindResume, KindAbort, KindRollback, KindBootstrap, KindJobRun, KindServiceApply, KindProxyApply, KindSecretsPush, KindDestroy, - KindServiceImagePatch, KindProtectionEnable, KindProtectionDisable, + KindServiceImagePatch, KindBackupEnable, KindBackupDisable, KindBackupCreate, KindBackupPrune, KindReplayArchive, KindRestoreTest, KindRestorePrepare, KindRestoreCutover, KindRestoreAbort, KindHygieneRun, KindAssuranceCheck: @@ -488,7 +489,7 @@ func validApprovalClass(class ApprovalClass) bool { func validStepKind(kind OperationStepKind) bool { switch kind { case StepPreflight, StepTransfer, StepJob, StepHook, StepWorkloadRelease, StepVerify, StepActivate, - StepProtectionLock, StepLifecycleAction, StepLifecycleRecord, StepArchiveAppend: + StepBackupLock, StepLifecycleAction, StepLifecycleRecord, StepArchiveAppend: return true default: return false @@ -503,3 +504,15 @@ func validDataEffect(effect DataEffectClass) bool { return false } } + +// lifecycleGraphDigest is the shape every sealed lifecycle digest takes. It +// lives here rather than beside the operation graph that used to define it, +// because the graph is gone and four live callers still bind digests. +var lifecycleGraphDigest = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) + +// Metadata that lands in a lifecycle record or a sealed identity is restricted +// to a safe grammar, so an operator-supplied name cannot smuggle punctuation +// into evidence a machine parses. +var lifecycleMetadata = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:/-]{0,511}$`) + +func safeLifecycleMetadata(value string) bool { return lifecycleMetadata.MatchString(value) } diff --git a/internal/onebox/protected_identity.go b/internal/onebox/protected_identity.go deleted file mode 100644 index 2a242e70..00000000 --- a/internal/onebox/protected_identity.go +++ /dev/null @@ -1,176 +0,0 @@ -package onebox - -import ( - "crypto/sha256" - "encoding/hex" - "encoding/json" - "errors" - "fmt" - "reflect" - "sort" - - "github.com/labstack/onebox/internal/app" -) - -const ProtectedServiceIdentitySchemaVersion = "onebox.run/protected-service-identity/v1alpha1" - -type ProtectedServiceIdentity struct { - SchemaVersion string `json:"schema_version"` - Application string `json:"application"` - Environment string `json:"environment"` - Service string `json:"service"` - Driver string `json:"driver"` - LogicalVolume string `json:"logical_volume"` - ServiceProject string `json:"service_project"` - ServiceContainer string `json:"service_container"` - RestoreProject string `json:"restore_project"` - RestoreContainer string `json:"restore_container"` - RestoreNetwork string `json:"restore_network"` - RestoreVolume string `json:"restore_volume"` - StatePath string `json:"state_path"` - Timers []string `json:"timers"` - ManifestBound bool `json:"manifest_bound"` - IdentityDigest string `json:"identity_digest"` -} - -func NewProtectedServiceIdentity(cfg *app.Resolved, serviceName string, manifestBound bool) (ProtectedServiceIdentity, error) { - if cfg == nil || cfg.Spec == nil { - return ProtectedServiceIdentity{}, errors.New("protected service identity requires a resolved project") - } - service, ok := cfg.Services[serviceName] - if !ok { - return ProtectedServiceIdentity{}, protectedIdentityFailure() - } - if service.Protection == nil && !manifestBound { - return ProtectedServiceIdentity{}, errors.New("service identity is not yet protection-bound") - } - driver := service.Driver - if driver == "" { - driver = serviceName - } - names := cfg.NamesFor(cfg.Env) - var timers []string - // Bind the complete service schedule namespace, including replay archival - // when the current policy does not use it. The identity must remain stable - // after policy removal so it can still inspect and remove units created by - // the last effective policy. - for _, kind := range []string{"backup-create", "backup-prune", "replay-archive", "restore-drill"} { - timers = append(timers, names.ProtectionTimerForEnvironment(cfg.Env, serviceName, kind)) - } - sort.Strings(timers) - record := ProtectedServiceIdentity{ - SchemaVersion: ProtectedServiceIdentitySchemaVersion, - Application: names.App, Environment: cfg.Env, Service: serviceName, Driver: driver, LogicalVolume: firstServiceVolume(service), - ServiceProject: names.ServiceProject(serviceName), ServiceContainer: names.ServiceContainer(serviceName), - RestoreProject: names.ProtectionRestoreProject(serviceName), RestoreContainer: names.ProtectionRestoreContainer(serviceName), - RestoreNetwork: names.ProtectionRestoreNetwork(serviceName), RestoreVolume: names.ProtectionRestoreVolume(serviceName), - StatePath: names.ActiveVolumeFile(serviceName), Timers: timers, ManifestBound: manifestBound, - } - if err := record.Seal(); err != nil { - return ProtectedServiceIdentity{}, err - } - return record, nil -} - -func (record ProtectedServiceIdentity) canonicalJSON() ([]byte, error) { - copy := record - copy.IdentityDigest = "" - return json.Marshal(copy) -} - -func (record ProtectedServiceIdentity) ComputeDigest() (string, error) { - encoded, err := record.canonicalJSON() - if err != nil { - return "", err - } - sum := sha256.Sum256(encoded) - return "sha256:" + hex.EncodeToString(sum[:]), nil -} - -func (record ProtectedServiceIdentity) validateContent() error { - if record.SchemaVersion != ProtectedServiceIdentitySchemaVersion { - return fmt.Errorf("unsupported protected service identity schema %q", record.SchemaVersion) - } - for name, value := range map[string]string{ - "application": record.Application, "environment": record.Environment, "service": record.Service, - "driver": record.Driver, "logical_volume": record.LogicalVolume, - "service_project": record.ServiceProject, "service_container": record.ServiceContainer, - "restore_project": record.RestoreProject, "restore_container": record.RestoreContainer, - "restore_network": record.RestoreNetwork, "restore_volume": record.RestoreVolume, - } { - if !safeLifecycleMetadata(value) { - return fmt.Errorf("protected service identity %s is invalid", name) - } - } - if record.StatePath == "" || record.StatePath[0] != '/' { - return errors.New("protected service identity state path must be absolute") - } - if !sort.StringsAreSorted(record.Timers) { - return errors.New("protected service identity timers must be sorted") - } - for index, timer := range record.Timers { - if !safeLifecycleMetadata(timer) || index > 0 && timer == record.Timers[index-1] { - return errors.New("protected service identity timer is invalid or repeated") - } - } - return nil -} - -func (record *ProtectedServiceIdentity) Seal() error { - if record == nil { - return errors.New("protected service identity is nil") - } - if err := record.validateContent(); err != nil { - return err - } - digest, err := record.ComputeDigest() - if err != nil { - return err - } - record.IdentityDigest = digest - return nil -} - -func (record ProtectedServiceIdentity) Validate() error { - if err := record.validateContent(); err != nil { - return err - } - if !lifecycleGraphDigest.MatchString(record.IdentityDigest) { - return errors.New("protected service identity digest is missing or invalid") - } - expected, err := record.ComputeDigest() - if err != nil { - return err - } - if record.IdentityDigest != expected { - return errors.New("protected service identity digest mismatch") - } - return nil -} - -func ValidateProtectedServiceIdentity(cfg *app.Resolved, record ProtectedServiceIdentity) error { - if err := record.Validate(); err != nil { - return err - } - current, err := NewProtectedServiceIdentity(cfg, record.Service, record.ManifestBound) - if err != nil { - return protectedIdentityFailure() - } - current.IdentityDigest = record.IdentityDigest - if !reflect.DeepEqual(current, record) { - return protectedIdentityFailure() - } - return nil -} - -func firstServiceVolume(service app.Service) string { - if len(service.Volumes) > 0 { - return service.Volumes[0] - } - return "data" -} - -func protectedIdentityFailure() error { - failure, _ := NewLifecycleFailure("protected_service_identity_changed") - return failure -} diff --git a/internal/onebox/protected_identity_test.go b/internal/onebox/protected_identity_test.go deleted file mode 100644 index 517169ee..00000000 --- a/internal/onebox/protected_identity_test.go +++ /dev/null @@ -1,79 +0,0 @@ -package onebox - -import ( - "errors" - "strings" - "testing" - - "github.com/labstack/onebox/internal/app" -) - -func protectedIdentityConfig(serviceName string, withPolicy bool) *app.Resolved { - service := app.Service{Driver: "postgres", Version: 17, Volumes: []string{"data"}} - if withPolicy { - service.Protection = &app.ProtectionPolicy{Target: "offsite", RecoveryKind: "pitr"} - } - return &app.Resolved{ - Spec: &app.Spec{Name: "example", BasePath: "/var/lib/ob", Services: map[string]app.Service{serviceName: service}}, - Env: "production", - } -} - -func TestProtectedServiceIdentityBindsEveryGeneratedName(t *testing.T) { - record, err := NewProtectedServiceIdentity(protectedIdentityConfig("database", true), "database", false) - if err != nil { - t.Fatal(err) - } - if err := record.Validate(); err != nil { - t.Fatal(err) - } - if record.ServiceProject != "ob_example_database" || record.RestoreProject != "ob_example_database_restore" || - record.RestoreContainer != "example-database-restore-1" || record.RestoreNetwork != "ob_example_database_restore-net" || - record.RestoreVolume != "ob_example_database_restore-stage" || record.StatePath != "/var/lib/ob/example/protection/state/database.active-volume.json" { - t.Fatalf("protected identity = %#v", record) - } - if len(record.Timers) != 4 { - t.Fatalf("protected timers = %#v", record.Timers) - } - for _, timer := range record.Timers { - if !strings.HasPrefix(timer, "ob-example-production-database-") { - t.Fatalf("timer is not environment-scoped: %q", timer) - } - } -} - -func TestManifestBindsIdentityAfterPolicyRemoval(t *testing.T) { - record, err := NewProtectedServiceIdentity(protectedIdentityConfig("database", false), "database", true) - if err != nil { - t.Fatal(err) - } - if !record.ManifestBound { - t.Fatal("manifest-bound identity lost its binding") - } - if err := ValidateProtectedServiceIdentity(protectedIdentityConfig("database", false), record); err != nil { - t.Fatalf("validate retained manifest identity: %v", err) - } -} - -func TestProtectedServiceRenameFailsClosed(t *testing.T) { - record, err := NewProtectedServiceIdentity(protectedIdentityConfig("database", true), "database", true) - if err != nil { - t.Fatal(err) - } - err = ValidateProtectedServiceIdentity(protectedIdentityConfig("db", true), record) - var failure LifecycleFailure - if !errors.As(err, &failure) || failure.Code != "protected_service_identity_changed" { - t.Fatalf("rename error = %v", err) - } -} - -func TestProtectedServiceIdentityDetectsTamper(t *testing.T) { - record, err := NewProtectedServiceIdentity(protectedIdentityConfig("database", true), "database", false) - if err != nil { - t.Fatal(err) - } - record.RestoreVolume = "foreign_volume" - if err := record.Validate(); err == nil { - t.Fatal("tampered protected identity was accepted") - } -} diff --git a/internal/onebox/protection_artifacts.go b/internal/onebox/protection_artifacts.go deleted file mode 100644 index 94ea18d9..00000000 --- a/internal/onebox/protection_artifacts.go +++ /dev/null @@ -1,27 +0,0 @@ -package onebox - -import ( - "errors" - "sort" - - "github.com/labstack/onebox/internal/app" -) - -// BindProtectionArtifacts projects only safe metadata from generated -// protection artifacts into a plan. Contents remain in their target-side -// files and every projected digest becomes part of the plan identity. -func BindProtectionArtifacts(plan *OperationPlan, generated app.ProtectionArtifactSet) error { - if plan == nil { - return errors.New("operation plan is nil") - } - bindings := make([]OperationArtifactBinding, len(generated.Artifacts)) - for index, artifact := range generated.Artifacts { - bindings[index] = OperationArtifactBinding{ - Class: artifact.Class, Path: artifact.Path, Mode: artifact.Mode, Digest: artifact.Digest, - } - } - sort.Slice(bindings, func(i, j int) bool { return bindings[i].Class < bindings[j].Class }) - plan.Artifacts = bindings - plan.PlanDigest = "" - return nil -} diff --git a/internal/onebox/protection_artifacts_test.go b/internal/onebox/protection_artifacts_test.go deleted file mode 100644 index 6bfc1a28..00000000 --- a/internal/onebox/protection_artifacts_test.go +++ /dev/null @@ -1,53 +0,0 @@ -package onebox - -import ( - "encoding/json" - "strings" - "testing" - - "github.com/labstack/onebox/internal/app" -) - -func TestBindProtectionArtifactsSealsMetadataWithoutContents(t *testing.T) { - plan := validOperationPlan(t) - generated := app.ProtectionArtifactSet{Artifacts: []app.GeneratedProtectionArtifact{ - {Class: "inputs", Path: "/var/lib/onebox/apps/example/protection/inputs.json", Mode: 0o600, Digest: "sha256:" + strings.Repeat("a", 64), Content: []byte("secret-value-canary")}, - {Class: "backup-schedule", Path: "/var/lib/onebox/apps/example/protection/backup.json", Mode: 0o644, Digest: "sha256:" + strings.Repeat("b", 64), Content: []byte("database-content-canary")}, - }} - if err := BindProtectionArtifacts(&plan, generated); err != nil { - t.Fatal(err) - } - if err := plan.Seal(); err != nil { - t.Fatal(err) - } - encoded, err := json.Marshal(plan) - if err != nil { - t.Fatal(err) - } - for _, forbidden := range []string{"secret-value-canary", "database-content-canary"} { - if strings.Contains(string(encoded), forbidden) { - t.Fatalf("bound plan leaked artifact contents %q: %s", forbidden, encoded) - } - } - if len(plan.Artifacts) != 2 || plan.Artifacts[0].Class != "backup-schedule" || plan.Artifacts[1].Class != "inputs" { - t.Fatalf("artifact bindings are not deterministically sorted: %#v", plan.Artifacts) - } - before := plan.PlanDigest - plan.Artifacts[0].Digest = "sha256:" + strings.Repeat("c", 64) - if err := plan.Seal(); err != nil { - t.Fatal(err) - } - if plan.PlanDigest == before { - t.Fatal("artifact digest did not affect the sealed plan identity") - } -} - -func TestOperationPlanRejectsUnsafeArtifactBinding(t *testing.T) { - plan := validOperationPlan(t) - plan.Artifacts = []OperationArtifactBinding{{ - Class: "inputs", Path: "relative/inputs.json", Mode: 0o600, Digest: "sha256:" + strings.Repeat("a", 64), - }} - if err := plan.Validate(); err == nil || !strings.Contains(err.Error(), "clean absolute") { - t.Fatalf("unsafe artifact path error = %v", err) - } -} diff --git a/internal/onebox/protection_credentials.go b/internal/onebox/protection_credentials.go deleted file mode 100644 index c36cd37f..00000000 --- a/internal/onebox/protection_credentials.go +++ /dev/null @@ -1,59 +0,0 @@ -package onebox - -import ( - "errors" - "sort" - - "github.com/labstack/onebox/internal/app" -) - -// ProtectionSecretSlots resolves every credential required by one protected -// service to named entries in one target-side file. It never decrypts or -// accepts values and is therefore safe to bind into plans and output. -func ProtectionSecretSlots(cfg *app.Resolved, serviceName string) ([]SecretSlotReference, error) { - if cfg == nil || cfg.Spec == nil { - return nil, errors.New("protection credential config is required") - } - service, ok := cfg.Services[serviceName] - if !ok || service.Protection == nil { - return nil, errors.New("protected service is required") - } - target, ok := cfg.BackupTargets[service.Protection.Target] - if !ok { - return nil, errors.New("protection target is not declared") - } - driverName := service.Driver - if driverName == "" { - driverName = serviceName - } - driverSlots, ok := app.LifecycleCredentialSlots(driverName, cfg.DeclaredVersion(serviceName)) - if !ok { - return nil, errors.New("service has no qualified lifecycle credential contract") - } - entries := append(driverSlots, - target.Credentials.AccessKeyEntry, - target.Credentials.SecretKeyEntry, - ) - if target.Credentials.SessionTokenEntry != "" { - entries = append(entries, target.Credentials.SessionTokenEntry) - } - sort.Strings(entries) - entries = uniqueNonEmpty(entries) - file := cfg.NamesFor(cfg.Env).ProtectionCredentialFile(serviceName, service.Protection.Target) - slots := make([]SecretSlotReference, 0, len(entries)) - for _, entry := range entries { - slots = append(slots, SecretSlotReference{Slot: "credential:" + entry, File: file, Entry: entry}) - } - return slots, nil -} - -func uniqueNonEmpty(values []string) []string { - out := values[:0] - for _, value := range values { - if value == "" || len(out) > 0 && out[len(out)-1] == value { - continue - } - out = append(out, value) - } - return out -} diff --git a/internal/onebox/protection_credentials_test.go b/internal/onebox/protection_credentials_test.go deleted file mode 100644 index 067128a3..00000000 --- a/internal/onebox/protection_credentials_test.go +++ /dev/null @@ -1,82 +0,0 @@ -package onebox - -import ( - "encoding/json" - "strings" - "testing" - "time" - - "github.com/labstack/onebox/internal/app" -) - -func TestProtectionSecretSlotsContainReferencesOnly(t *testing.T) { - cfg := &app.Resolved{ - Spec: &app.Spec{ - Name: "example", - BasePath: "/var/lib/ob", - BackupTargets: map[string]app.BackupTarget{ - "offsite": { - Credentials: app.CredentialReference{ - File: "secrets/backup.env", Provider: "sops", - AccessKeyEntry: "BACKUP_ACCESS_KEY_ID", SecretKeyEntry: "BACKUP_SECRET_ACCESS_KEY", - }, - }, - }, - Services: map[string]app.Service{ - "database": {Driver: "postgres", Version: 17, Protection: &app.ProtectionPolicy{Target: "offsite"}}, - }, - }, - Env: "production", - } - slots, err := ProtectionSecretSlots(cfg, "database") - if err != nil { - t.Fatalf("resolve protection slots: %v", err) - } - wantEntries := []string{"BACKUP_ACCESS_KEY_ID", "BACKUP_SECRET_ACCESS_KEY", "PGBACKREST_REPO_PASSWORD", "POSTGRES_PASSWORD"} - if len(slots) != len(wantEntries) { - t.Fatalf("slots = %#v", slots) - } - for index, want := range wantEntries { - if slots[index].Entry != want || slots[index].File != "/var/lib/ob/example/protection/secrets/database-offsite.env" { - t.Errorf("slot %d = %#v, want entry %q in target-side file", index, slots[index], want) - } - } - - steps, err := LifecycleOperationGraph(KindBackupCreate, LifecycleCLIRunnerSchema, "database") - if err != nil { - t.Fatal(err) - } - now := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC) - plan := OperationPlan{ - SchemaVersion: OperationPlanSchemaVersion, ID: "backup-1", Kind: KindBackupCreate, - CreatedAt: now.Format(time.RFC3339), ExpiresAt: now.Add(time.Hour).Format(time.RFC3339), - Risk: RiskModerate, Reversibility: ReversibilityConditional, Approval: ApprovalStanding, - Binding: OperationBinding{Application: "example", Environment: "production", Server: "host", ConfigDigest: "config", ComposeDigest: "compose", StateDigest: "state"}, - Steps: steps, SecretSlots: slots, - } - if err := plan.Seal(); err != nil { - t.Fatalf("seal protection plan: %v", err) - } - encoded, err := json.Marshal(plan) - if err != nil { - t.Fatal(err) - } - for _, forbidden := range []string{"credential-canary", "database-row-canary", "secret_value", "value"} { - if strings.Contains(string(encoded), forbidden) { - t.Fatalf("plan contains forbidden credential/content field %q: %s", forbidden, encoded) - } - } -} - -func TestOperationPlanRejectsInlineOrRelativeSecretSlots(t *testing.T) { - plan := OperationPlan{SecretSlots: []SecretSlotReference{{Slot: "credential:key", Entry: "ACCESS_KEY", File: "relative.env"}}} - plan.SchemaVersion = OperationPlanSchemaVersion - plan.ID, plan.Kind = "backup-1", KindBackupCreate - plan.CreatedAt, plan.ExpiresAt = "2026-08-07T12:00:00Z", "2026-08-07T13:00:00Z" - plan.Risk, plan.Reversibility, plan.Approval = RiskModerate, ReversibilityConditional, ApprovalStanding - plan.Binding = OperationBinding{Application: "example", Environment: "production", Server: "host", ConfigDigest: "config", ComposeDigest: "compose", StateDigest: "state"} - plan.Steps = []OperationStep{{ID: "preflight", Kind: StepPreflight, DataEffect: DataEffectNone}} - if err := plan.Validate(); err == nil { - t.Fatal("relative secret slot path was accepted") - } -} diff --git a/internal/onebox/protection_redaction_test.go b/internal/onebox/protection_redaction_test.go deleted file mode 100644 index db318648..00000000 --- a/internal/onebox/protection_redaction_test.go +++ /dev/null @@ -1,72 +0,0 @@ -package onebox - -import ( - "bytes" - "encoding/json" - "strings" - "testing" - - "github.com/labstack/onebox/internal/journal" -) - -func TestProtectionPublicSurfacesHaveNoCredentialOrDatabaseContentFields(t *testing.T) { - credentialCanary := "credential-canary-value" - databaseCanary := "customer@example.invalid: private database row" - failure, err := NewLifecycleFailure("backup_target_unreachable") - if err != nil { - t.Fatal(err) - } - event := LifecycleRecord{ - SchemaVersion: LifecycleRecordSchemaVersion, - Type: LifecycleRecordEvent, - OperationID: "backup-1", - Kind: LifecycleBackupCreate, - Service: "database", - Event: &LifecycleEvent{ - Sequence: 1, Time: "2026-08-07T12:00:00Z", Phase: "stream", State: "progress", EvidenceID: "generation-1", - }, - } - eventJSON, err := EncodeLifecycleJSON([]LifecycleRecord{event}) - if err != nil { - t.Fatal(err) - } - var eventNDJSON bytes.Buffer - if err := EncodeLifecycleNDJSON(&eventNDJSON, []LifecycleRecord{event}); err != nil { - t.Fatal(err) - } - journalJSON, err := json.Marshal(journal.Record{ - DeployID: "backup-1", Phase: "backup", Event: "result", Status: "ok", - OperationKind: "backup_create", Service: "database", ProtectionStepID: "protection-step:0123456789abcdef0123456789abcdef", - ProtectionAttempt: 1, TerminalResult: &journal.ProtectionTerminalResult{State: "succeeded", EvidenceID: "generation-1"}, - }) - if err != nil { - t.Fatal(err) - } - failureJSON, err := json.Marshal(failure) - if err != nil { - t.Fatal(err) - } - // BackupReport models identities and validation results only; protection - // credentials and database payload bytes have no destination in its shape. - manifestJSON, err := json.Marshal(BackupReport{ - SchemaVersion: BackupReportSchemaVersion, - PlanDigest: "sha256:plan", OperationDigest: "sha256:operation", - Application: "example", Environment: "production", Server: "offsite", - ReportedBy: "operator", ReportedAt: "2026-08-07T12:00:00Z", - }) - if err != nil { - t.Fatal(err) - } - - surfaces := map[string][]byte{ - "events": eventJSON, "structured-output": eventNDJSON.Bytes(), "journals": journalJSON, - "errors": failureJSON, "manifests": manifestJSON, - } - for name, encoded := range surfaces { - for _, forbidden := range []string{credentialCanary, databaseCanary, "credential_value", "database_content"} { - if strings.Contains(string(encoded), forbidden) { - t.Errorf("%s leaked %q: %s", name, forbidden, encoded) - } - } - } -} diff --git a/internal/onebox/protection_resources.go b/internal/onebox/protection_resources.go deleted file mode 100644 index b62974f9..00000000 --- a/internal/onebox/protection_resources.go +++ /dev/null @@ -1,303 +0,0 @@ -package onebox - -import ( - "crypto/sha256" - "encoding/hex" - "encoding/json" - "errors" - "fmt" - "sort" - "strings" -) - -const ProtectionRemovalPlanSchemaVersion = "onebox.run/protection-removal-plan/v1alpha1" - -type ProtectionResourceKind string - -const ( - ProtectionResourceUnit ProtectionResourceKind = "unit" - ProtectionResourceHook ProtectionResourceKind = "hook" - ProtectionResourceConfig ProtectionResourceKind = "config" - ProtectionResourceEnvelope ProtectionResourceKind = "envelope" - ProtectionResourceRunner ProtectionResourceKind = "runner" - ProtectionRemoteBackup ProtectionResourceKind = "remote-backup" - ProtectionManifest ProtectionResourceKind = "manifest" - ProtectionManifestImage ProtectionResourceKind = "manifest-image" - ProtectionServiceVolume ProtectionResourceKind = "service-volume" - ProtectionPreviousVolume ProtectionResourceKind = "previous-volume" -) - -type ProtectionResource struct { - Identity string `json:"identity"` - Kind ProtectionResourceKind `json:"kind"` - OwnerApplication string `json:"owner_application,omitempty"` - OwnerEnvironment string `json:"owner_environment,omitempty"` - Service string `json:"service,omitempty"` - Referenced bool `json:"referenced,omitempty"` - RequiredByPrerequisite bool `json:"required_by_prerequisite,omitempty"` -} - -type ProtectionResourceInspection struct { - Application string `json:"application"` - Environment string `json:"environment"` - Owned []ProtectionResource `json:"owned,omitempty"` - Preserved []ProtectionResource `json:"preserved,omitempty"` - Foreign []ProtectionResource `json:"foreign,omitempty"` -} - -type ProtectionRemovalRequest struct { - Mode OperationKind `json:"mode"` - Application string `json:"application"` - Environment string `json:"environment"` - Service string `json:"service,omitempty"` - ProtectionState string `json:"protection_state"` - StateDigest string `json:"state_digest"` - PrerequisitesVerifiedAbsent bool `json:"prerequisites_verified_absent"` -} - -type ProtectionRemovalPlan struct { - SchemaVersion string `json:"schema_version"` - Mode OperationKind `json:"mode"` - Application string `json:"application"` - Environment string `json:"environment"` - Service string `json:"service,omitempty"` - StateDigest string `json:"state_digest"` - Remove []ProtectionResource `json:"remove,omitempty"` - Preserve []ProtectionResource `json:"preserve,omitempty"` - PlanDigest string `json:"plan_digest"` -} - -// ProtectionRemovalAuthorization is issued only after the approval boundary -// validates the sealed plan and live state. It carries no generic force flag. -type ProtectionRemovalAuthorization struct { - Operation OperationKind `json:"operation"` - PlanDigest string `json:"plan_digest"` - StateDigest string `json:"state_digest"` -} - -func InspectProtectionResources(application, environment string, resources []ProtectionResource) (ProtectionResourceInspection, error) { - if !safeLifecycleMetadata(application) || !safeLifecycleMetadata(environment) { - return ProtectionResourceInspection{}, errors.New("application and environment must be safe ownership metadata") - } - inspection := ProtectionResourceInspection{Application: application, Environment: environment} - seen := make(map[string]struct{}, len(resources)) - for index, resource := range resources { - if err := resource.validate(); err != nil { - return ProtectionResourceInspection{}, fmt.Errorf("resources[%d]: %w", index, err) - } - key := string(resource.Kind) + "\x00" + resource.Identity - if _, exists := seen[key]; exists { - return ProtectionResourceInspection{}, fmt.Errorf("duplicate protection resource %q", resource.Identity) - } - seen[key] = struct{}{} - switch { - case resource.OwnerApplication != application || resource.OwnerEnvironment != environment: - inspection.Foreign = append(inspection.Foreign, resource) - case protectionResourceAlwaysPreserved(resource.Kind): - inspection.Preserved = append(inspection.Preserved, resource) - default: - inspection.Owned = append(inspection.Owned, resource) - } - } - sortProtectionResources(inspection.Owned) - sortProtectionResources(inspection.Preserved) - sortProtectionResources(inspection.Foreign) - return inspection, nil -} - -func NewProtectionRemovalPlan(inspection ProtectionResourceInspection, request ProtectionRemovalRequest) (ProtectionRemovalPlan, error) { - if request.Application != inspection.Application || request.Environment != inspection.Environment { - return ProtectionRemovalPlan{}, errors.New("removal request does not match inspected ownership") - } - if request.Mode != KindProtectionDisable && request.Mode != KindDestroy { - return ProtectionRemovalPlan{}, fmt.Errorf("unsupported protection removal mode %q", request.Mode) - } - if !lifecycleGraphDigest.MatchString(request.StateDigest) { - return ProtectionRemovalPlan{}, errors.New("removal state_digest must be sha256:<64 lowercase hex>") - } - if !request.PrerequisitesVerifiedAbsent { - failure, _ := NewLifecycleFailure("protection_image_revert_unsafe") - return ProtectionRemovalPlan{}, failure - } - if request.Mode == KindProtectionDisable { - if request.ProtectionState != "disabled" || !safeLifecycleMetadata(request.Service) { - failure, _ := NewLifecycleFailure("protection_disable_pending") - return ProtectionRemovalPlan{}, failure - } - } else if request.ProtectionState != "disabled" && request.ProtectionState != "never-enabled" { - failure, _ := NewLifecycleFailure("protection_disable_pending") - return ProtectionRemovalPlan{}, failure - } - plan := ProtectionRemovalPlan{ - SchemaVersion: ProtectionRemovalPlanSchemaVersion, - Mode: request.Mode, Application: request.Application, Environment: request.Environment, - Service: request.Service, StateDigest: request.StateDigest, - Preserve: append([]ProtectionResource(nil), inspection.Preserved...), - } - plan.Preserve = append(plan.Preserve, inspection.Foreign...) - for _, resource := range inspection.Owned { - remove := !resource.RequiredByPrerequisite && !resource.Referenced - if request.Mode == KindProtectionDisable { - remove = remove && resource.Service == request.Service - } - if remove { - plan.Remove = append(plan.Remove, resource) - } else { - plan.Preserve = append(plan.Preserve, resource) - } - } - sortProtectionResources(plan.Remove) - sortProtectionResources(plan.Preserve) - if err := plan.Seal(); err != nil { - return ProtectionRemovalPlan{}, err - } - return plan, nil -} - -func ApplyProtectionRemoval(plan ProtectionRemovalPlan, authorization ProtectionRemovalAuthorization, remove func(ProtectionResource) error) error { - if err := plan.Validate(); err != nil { - return err - } - if authorization.Operation != plan.Mode || authorization.PlanDigest != plan.PlanDigest || authorization.StateDigest != plan.StateDigest { - failure, _ := NewLifecycleFailure("protection_disablement_not_authorized") - return failure - } - if remove == nil { - return errors.New("protection resource remover is nil") - } - for _, resource := range plan.Remove { - if protectionResourceAlwaysPreserved(resource.Kind) { - return fmt.Errorf("refusing to remove preserved protection resource %q", resource.Identity) - } - if err := remove(resource); err != nil { - return fmt.Errorf("remove owned protection resource %q: %w", resource.Identity, err) - } - } - return nil -} - -func (plan *ProtectionRemovalPlan) Seal() error { - if plan == nil { - return errors.New("protection removal plan is nil") - } - if err := plan.validateContent(); err != nil { - return err - } - digest, err := plan.computeDigest() - if err != nil { - return err - } - plan.PlanDigest = digest - return nil -} - -func (plan ProtectionRemovalPlan) Validate() error { - if err := plan.validateContent(); err != nil { - return err - } - if !lifecycleGraphDigest.MatchString(plan.PlanDigest) { - return errors.New("protection removal plan_digest is required") - } - expected, err := plan.computeDigest() - if err != nil { - return err - } - if plan.PlanDigest != expected { - return errors.New("protection removal plan digest mismatch") - } - return nil -} - -func (plan ProtectionRemovalPlan) validateContent() error { - if plan.SchemaVersion != ProtectionRemovalPlanSchemaVersion { - return fmt.Errorf("unsupported protection removal schema %q", plan.SchemaVersion) - } - if plan.Mode != KindProtectionDisable && plan.Mode != KindDestroy { - return fmt.Errorf("unsupported protection removal mode %q", plan.Mode) - } - if !safeLifecycleMetadata(plan.Application) || !safeLifecycleMetadata(plan.Environment) { - return errors.New("protection removal ownership is invalid") - } - if plan.Mode == KindProtectionDisable && !safeLifecycleMetadata(plan.Service) { - return errors.New("protection disable removal requires a service") - } - if !lifecycleGraphDigest.MatchString(plan.StateDigest) { - return errors.New("protection removal state_digest is invalid") - } - seen := make(map[string]struct{}, len(plan.Remove)+len(plan.Preserve)) - for _, group := range [][]ProtectionResource{plan.Remove, plan.Preserve} { - previous := "" - for _, resource := range group { - if err := resource.validate(); err != nil { - return err - } - key := string(resource.Kind) + "\x00" + resource.Identity - if previous != "" && key <= previous { - return errors.New("protection removal resources must be unique and sorted") - } - if _, exists := seen[key]; exists { - return fmt.Errorf("protection resource %q appears more than once", resource.Identity) - } - seen[key] = struct{}{} - previous = key - } - } - for _, resource := range plan.Remove { - if resource.OwnerApplication != plan.Application || resource.OwnerEnvironment != plan.Environment || protectionResourceAlwaysPreserved(resource.Kind) { - return fmt.Errorf("protection removal plan contains non-removable resource %q", resource.Identity) - } - } - return nil -} - -func (plan ProtectionRemovalPlan) computeDigest() (string, error) { - copy := plan - copy.PlanDigest = "" - encoded, err := json.Marshal(copy) - if err != nil { - return "", err - } - sum := sha256.Sum256(encoded) - return "sha256:" + hex.EncodeToString(sum[:]), nil -} - -func (resource ProtectionResource) validate() error { - if strings.TrimSpace(resource.Identity) == "" || strings.ContainsAny(resource.Identity, "\r\n\x00") { - return errors.New("resource identity must be non-empty single-line metadata") - } - if resource.OwnerApplication != "" && !safeLifecycleMetadata(resource.OwnerApplication) { - return errors.New("resource owner application is invalid") - } - if resource.OwnerEnvironment != "" && !safeLifecycleMetadata(resource.OwnerEnvironment) { - return errors.New("resource owner environment is invalid") - } - if resource.Service != "" && !safeLifecycleMetadata(resource.Service) { - return errors.New("resource service is invalid") - } - switch resource.Kind { - case ProtectionResourceUnit, ProtectionResourceHook, ProtectionResourceConfig, ProtectionResourceEnvelope, - ProtectionResourceRunner, ProtectionRemoteBackup, ProtectionManifest, - ProtectionManifestImage, ProtectionServiceVolume, ProtectionPreviousVolume: - return nil - default: - return fmt.Errorf("unknown protection resource kind %q", resource.Kind) - } -} - -func protectionResourceAlwaysPreserved(kind ProtectionResourceKind) bool { - switch kind { - case ProtectionRemoteBackup, ProtectionManifest, ProtectionManifestImage, - ProtectionServiceVolume, ProtectionPreviousVolume: - return true - default: - return false - } -} - -func sortProtectionResources(resources []ProtectionResource) { - sort.Slice(resources, func(i, j int) bool { - left := string(resources[i].Kind) + "\x00" + resources[i].Identity - right := string(resources[j].Kind) + "\x00" + resources[j].Identity - return left < right - }) -} diff --git a/internal/onebox/protection_resources_test.go b/internal/onebox/protection_resources_test.go deleted file mode 100644 index ec3a28e9..00000000 --- a/internal/onebox/protection_resources_test.go +++ /dev/null @@ -1,147 +0,0 @@ -package onebox - -import ( - "errors" - "reflect" - "strings" - "testing" -) - -func protectionResourceFixture() []ProtectionResource { - owned := func(identity string, kind ProtectionResourceKind) ProtectionResource { - return ProtectionResource{Identity: identity, Kind: kind, OwnerApplication: "example", OwnerEnvironment: "production", Service: "database"} - } - return []ProtectionResource{ - owned("ob-example-database-backup.timer", ProtectionResourceUnit), - owned("/var/lib/onebox/example/protection/archive-hook.json", ProtectionResourceHook), - owned("s3://backups/example/database/generation-7", ProtectionRemoteBackup), - owned("manifest-7", ProtectionManifest), - owned("postgres@sha256:"+strings.Repeat("a", 64), ProtectionManifestImage), - owned("example-database-data", ProtectionServiceVolume), - owned("example-database-data-previous", ProtectionPreviousVolume), - {Identity: "ob-foreign-database-backup.timer", Kind: ProtectionResourceUnit, OwnerApplication: "other", OwnerEnvironment: "production", Service: "database"}, - } -} - -func TestProtectionDisableRemovalTouchesOnlyOwnedLocalResources(t *testing.T) { - inspection, err := InspectProtectionResources("example", "production", protectionResourceFixture()) - if err != nil { - t.Fatal(err) - } - plan, err := NewProtectionRemovalPlan(inspection, ProtectionRemovalRequest{ - Mode: KindProtectionDisable, Application: "example", Environment: "production", Service: "database", - ProtectionState: "disabled", StateDigest: "sha256:" + strings.Repeat("b", 64), PrerequisitesVerifiedAbsent: true, - }) - if err != nil { - t.Fatal(err) - } - var removed []string - authorization := ProtectionRemovalAuthorization{Operation: plan.Mode, PlanDigest: plan.PlanDigest, StateDigest: plan.StateDigest} - if err := ApplyProtectionRemoval(plan, authorization, func(resource ProtectionResource) error { - removed = append(removed, resource.Identity) - return nil - }); err != nil { - t.Fatal(err) - } - want := []string{"/var/lib/onebox/example/protection/archive-hook.json", "ob-example-database-backup.timer"} - if !reflect.DeepEqual(removed, want) { - t.Fatalf("removed = %#v, want %#v", removed, want) - } - for _, protected := range []string{"generation-7", "manifest-7", "postgres@sha256", "data-previous", "foreign"} { - for _, identity := range removed { - if strings.Contains(identity, protected) { - t.Fatalf("removed preserved or foreign resource %q", identity) - } - } - } -} - -func TestProtectionPendingRemovalIsRefusedBeforeMutation(t *testing.T) { - inspection, err := InspectProtectionResources("example", "production", protectionResourceFixture()) - if err != nil { - t.Fatal(err) - } - _, err = NewProtectionRemovalPlan(inspection, ProtectionRemovalRequest{ - Mode: KindProtectionDisable, Application: "example", Environment: "production", Service: "database", - ProtectionState: "disable-pending", StateDigest: "sha256:" + strings.Repeat("b", 64), PrerequisitesVerifiedAbsent: true, - }) - var failure LifecycleFailure - if !errors.As(err, &failure) || failure.Code != "protection_disable_pending" { - t.Fatalf("pending removal error = %v", err) - } -} - -func TestProtectionRemovalRequiresMatchingAuthorization(t *testing.T) { - inspection, err := InspectProtectionResources("example", "production", protectionResourceFixture()) - if err != nil { - t.Fatal(err) - } - plan, err := NewProtectionRemovalPlan(inspection, ProtectionRemovalRequest{ - Mode: KindProtectionDisable, Application: "example", Environment: "production", Service: "database", - ProtectionState: "disabled", StateDigest: "sha256:" + strings.Repeat("b", 64), PrerequisitesVerifiedAbsent: true, - }) - if err != nil { - t.Fatal(err) - } - called := false - err = ApplyProtectionRemoval(plan, ProtectionRemovalAuthorization{ - Operation: plan.Mode, PlanDigest: "sha256:" + strings.Repeat("c", 64), StateDigest: plan.StateDigest, - }, func(ProtectionResource) error { called = true; return nil }) - var failure LifecycleFailure - if !errors.As(err, &failure) || failure.Code != "protection_disablement_not_authorized" || called { - t.Fatalf("authorization refusal = %v, remover called = %v", err, called) - } -} - -func TestDestroyRemovesOnlyUnreferencedOwnedExecutables(t *testing.T) { - resources := protectionResourceFixture() - resources = append(resources, - ProtectionResource{Identity: "/var/lib/onebox/runner", Kind: ProtectionResourceRunner, OwnerApplication: "example", OwnerEnvironment: "production", Referenced: true}, - ProtectionResource{Identity: "/var/lib/onebox/envelope", Kind: ProtectionResourceEnvelope, OwnerApplication: "example", OwnerEnvironment: "production"}, - ) - inspection, err := InspectProtectionResources("example", "production", resources) - if err != nil { - t.Fatal(err) - } - plan, err := NewProtectionRemovalPlan(inspection, ProtectionRemovalRequest{ - Mode: KindDestroy, Application: "example", Environment: "production", ProtectionState: "disabled", - StateDigest: "sha256:" + strings.Repeat("d", 64), PrerequisitesVerifiedAbsent: true, - }) - if err != nil { - t.Fatal(err) - } - removed := make(map[string]bool) - authorization := ProtectionRemovalAuthorization{Operation: plan.Mode, PlanDigest: plan.PlanDigest, StateDigest: plan.StateDigest} - if err := ApplyProtectionRemoval(plan, authorization, func(resource ProtectionResource) error { - removed[resource.Identity] = true - return nil - }); err != nil { - t.Fatal(err) - } - if !removed["/var/lib/onebox/envelope"] || removed["/var/lib/onebox/runner"] { - t.Fatalf("destroy removal set = %#v", removed) - } - for _, resource := range inspection.Preserved { - if removed[resource.Identity] { - t.Fatalf("destroy removed preserved resource %q", resource.Identity) - } - } -} - -func TestProtectionRemovalPlanTamperIsRefused(t *testing.T) { - inspection, err := InspectProtectionResources("example", "production", protectionResourceFixture()) - if err != nil { - t.Fatal(err) - } - plan, err := NewProtectionRemovalPlan(inspection, ProtectionRemovalRequest{ - Mode: KindDestroy, Application: "example", Environment: "production", ProtectionState: "disabled", - StateDigest: "sha256:" + strings.Repeat("d", 64), PrerequisitesVerifiedAbsent: true, - }) - if err != nil { - t.Fatal(err) - } - plan.Remove = append(plan.Remove, ProtectionResource{Identity: "remote", Kind: ProtectionRemoteBackup, OwnerApplication: "example", OwnerEnvironment: "production"}) - if err := ApplyProtectionRemoval(plan, ProtectionRemovalAuthorization{}, func(ProtectionResource) error { return nil }); err == nil { - t.Fatal("tampered removal plan was accepted") - } -} diff --git a/internal/onebox/protection_state.go b/internal/onebox/protection_state.go deleted file mode 100644 index 2e428e80..00000000 --- a/internal/onebox/protection_state.go +++ /dev/null @@ -1,684 +0,0 @@ -package onebox - -import ( - "bytes" - "crypto/sha256" - "encoding/hex" - "encoding/json" - "errors" - "fmt" - "io" - "regexp" - "sort" - "strings" - "time" - - "github.com/labstack/onebox/internal/app" -) - -const ( - ProtectionStateSchemaVersion = "onebox.run/protection-state/v1alpha1" - ProtectionDisablePlanSchemaVersion = "onebox.run/protection-disable-plan/v1alpha1" - ProtectionDisableActionWindow = 24 * time.Hour -) - -type ProtectionState string - -const ( - ProtectionNeverEnabled ProtectionState = "never-enabled" - ProtectionEnabled ProtectionState = "enabled" - ProtectionDisablePending ProtectionState = "disable-pending" - ProtectionDisabled ProtectionState = "disabled" -) - -type ProtectionDisablePhase string - -const ( - ProtectionPhaseIdle ProtectionDisablePhase = "idle" - ProtectionPhaseRequested ProtectionDisablePhase = "requested" - ProtectionPhasePrerequisiteReversed ProtectionDisablePhase = "prerequisite-reversed" - ProtectionPhasePrerequisiteAbsent ProtectionDisablePhase = "prerequisite-absent" - ProtectionPhaseRuntimeReverted ProtectionDisablePhase = "runtime-reverted" - ProtectionPhaseLocalSupportRemoved ProtectionDisablePhase = "local-support-removed" - ProtectionPhaseComplete ProtectionDisablePhase = "complete" -) - -var protectedRuntimeImage = regexp.MustCompile(`^[^[:space:]@]+@sha256:[0-9a-f]{64}$`) - -type ProtectionScheduleState struct { - Kind string `json:"kind"` - Schedule app.Schedule `json:"schedule"` - Active bool `json:"active"` -} - -type ProtectionLifecycleState struct { - SchemaVersion string `json:"schema_version"` - Application string `json:"application"` - Environment string `json:"environment"` - Service string `json:"service"` - State ProtectionState `json:"state"` - Phase ProtectionDisablePhase `json:"phase"` - Epoch int `json:"epoch"` - OperationID string `json:"operation_id,omitempty"` - DisablePlanDigest string `json:"disable_plan_digest,omitempty"` - RequestedAt string `json:"requested_at,omitempty"` - ActionDeadline string `json:"action_deadline,omitempty"` - ServiceImage string `json:"service_image,omitempty"` - ServiceImagePublicationVerified bool `json:"service_image_publication_verified,omitempty"` - PrerequisiteEffective bool `json:"prerequisite_effective"` - LocalSupportInstalled bool `json:"local_support_installed"` - LastEffective *app.ProtectionEffectiveProjection `json:"last_effective,omitempty"` - Schedules []ProtectionScheduleState `json:"schedules,omitempty"` - StateDigest string `json:"state_digest"` -} - -type ProtectionDisableStep struct { - Phase ProtectionDisablePhase `json:"phase"` - Mutation bool `json:"mutation"` - Rollbackable bool `json:"rollbackable"` -} - -type ProtectionDisablePlan struct { - SchemaVersion string `json:"schema_version"` - OperationID string `json:"operation_id"` - Application string `json:"application"` - Environment string `json:"environment"` - Service string `json:"service"` - StateDigest string `json:"state_digest"` - Epoch int `json:"epoch"` - Approval ApprovalClass `json:"approval"` - Interruption bool `json:"interruption"` - RemoteDataAction string `json:"remote_data_action"` - Steps []ProtectionDisableStep `json:"steps"` - PlanDigest string `json:"plan_digest"` -} - -type ProtectionDisableAuthorization struct { - OperationID string `json:"operation_id"` - PlanDigest string `json:"plan_digest"` - StateDigest string `json:"state_digest"` - Strong bool `json:"strong"` -} - -type ProtectionLifecycleStatus struct { - State ProtectionState `json:"state"` - Phase ProtectionDisablePhase `json:"phase"` - RequestedAt string `json:"requested_at,omitempty"` - ActionDeadline string `json:"action_deadline,omitempty"` - Elapsed string `json:"elapsed,omitempty"` - Schedules []ProtectionScheduleState `json:"schedules,omitempty"` - StorageContinues bool `json:"storage_continues"` - ResolvingCommand string `json:"resolving_command,omitempty"` - Failure *LifecycleFailure `json:"failure,omitempty"` -} - -func NewProtectionLifecycleState(application, environment, service string, epoch int) (ProtectionLifecycleState, error) { - state := ProtectionLifecycleState{ - SchemaVersion: ProtectionStateSchemaVersion, Application: application, Environment: environment, - Service: service, State: ProtectionNeverEnabled, Phase: ProtectionPhaseIdle, Epoch: epoch, - } - if err := state.Seal(); err != nil { - return ProtectionLifecycleState{}, err - } - return state, nil -} - -func EnableProtection(current ProtectionLifecycleState, projection app.ProtectionEffectiveProjection, serviceImage, operationID string, publicationVerified bool, nextEpoch int) (ProtectionLifecycleState, error) { - if err := current.Validate(); err != nil { - return ProtectionLifecycleState{}, err - } - if current.State != ProtectionNeverEnabled && current.State != ProtectionDisabled { - return ProtectionLifecycleState{}, fmt.Errorf("cannot enable protection from %q", current.State) - } - if !safeLifecycleMetadata(operationID) || !protectedRuntimeImage.MatchString(serviceImage) || !publicationVerified || nextEpoch <= current.Epoch { - return ProtectionLifecycleState{}, errors.New("protection enablement operation, image, or fencing epoch is invalid") - } - next := current - next.State, next.Phase, next.Epoch = ProtectionEnabled, ProtectionPhaseIdle, nextEpoch - next.OperationID, next.DisablePlanDigest, next.RequestedAt, next.ActionDeadline = "", "", "", "" - next.ServiceImage, next.PrerequisiteEffective, next.LocalSupportInstalled = serviceImage, true, true - next.ServiceImagePublicationVerified = publicationVerified - next.LastEffective = cloneProtectionProjection(&projection) - next.Schedules = effectiveProtectionSchedules(projection, true) - if err := next.Seal(); err != nil { - return ProtectionLifecycleState{}, err - } - return next, nil -} - -func RequestProtectionDisable(current ProtectionLifecycleState, operationID string, now time.Time, nextEpoch int) (ProtectionLifecycleState, error) { - if err := current.Validate(); err != nil { - return ProtectionLifecycleState{}, err - } - if current.State == ProtectionDisablePending { - if current.OperationID == operationID { - return current, nil - } - failure, _ := NewLifecycleFailure("backup_conflict") - return ProtectionLifecycleState{}, failure - } - if current.State != ProtectionEnabled || current.LastEffective == nil { - return ProtectionLifecycleState{}, fmt.Errorf("cannot request protection disablement from %q", current.State) - } - if !safeLifecycleMetadata(operationID) || nextEpoch <= current.Epoch { - return ProtectionLifecycleState{}, errors.New("protection disablement operation or fencing epoch is invalid") - } - now = now.UTC() - next := current - next.State, next.Phase, next.Epoch = ProtectionDisablePending, ProtectionPhaseRequested, nextEpoch - next.OperationID, next.DisablePlanDigest = operationID, "" - next.RequestedAt = now.Format(time.RFC3339Nano) - next.ActionDeadline = now.Add(ProtectionDisableActionWindow).Format(time.RFC3339Nano) - next.Schedules = effectiveProtectionSchedules(*current.LastEffective, false) - if err := next.Seal(); err != nil { - return ProtectionLifecycleState{}, err - } - return next, nil -} - -func NewProtectionDisablePlan(state ProtectionLifecycleState) (ProtectionDisablePlan, error) { - if err := state.Validate(); err != nil { - return ProtectionDisablePlan{}, err - } - if state.State != ProtectionDisablePending || state.Phase != ProtectionPhaseRequested { - return ProtectionDisablePlan{}, errors.New("protection disable plan requires newly requested disable-pending state") - } - plan := ProtectionDisablePlan{ - SchemaVersion: ProtectionDisablePlanSchemaVersion, OperationID: state.OperationID, - Application: state.Application, Environment: state.Environment, Service: state.Service, - StateDigest: state.StateDigest, Epoch: state.Epoch, Approval: ApprovalStrong, Interruption: true, - RemoteDataAction: "handback-preserve", - Steps: []ProtectionDisableStep{ - {Phase: ProtectionPhasePrerequisiteReversed, Mutation: true, Rollbackable: true}, - {Phase: ProtectionPhasePrerequisiteAbsent, Rollbackable: true}, - {Phase: ProtectionPhaseRuntimeReverted, Mutation: true}, - {Phase: ProtectionPhaseLocalSupportRemoved, Mutation: true}, - {Phase: ProtectionPhaseComplete, Mutation: true}, - }, - } - if err := plan.Seal(); err != nil { - return ProtectionDisablePlan{}, err - } - return plan, nil -} - -func AdvanceProtectionDisable(current ProtectionLifecycleState, plan ProtectionDisablePlan, authorization ProtectionDisableAuthorization, completed ProtectionDisablePhase, nextEpoch int) (ProtectionLifecycleState, error) { - if err := current.Validate(); err != nil { - return ProtectionLifecycleState{}, err - } - if err := plan.Validate(); err != nil { - return ProtectionLifecycleState{}, err - } - if err := validateProtectionDisableAuthority(current, plan, authorization); err != nil { - return ProtectionLifecycleState{}, err - } - if current.State != ProtectionDisablePending { - return ProtectionLifecycleState{}, fmt.Errorf("cannot advance disablement from %q", current.State) - } - // A disconnected caller may retry the phase whose commit it did not see. - if current.Phase == completed { - return current, nil - } - expected, ok := nextProtectionDisablePhase(current.Phase) - if !ok || completed != expected { - return ProtectionLifecycleState{}, fmt.Errorf("disablement phase %q cannot follow %q", completed, current.Phase) - } - if nextEpoch <= current.Epoch { - return ProtectionLifecycleState{}, errors.New("protection disablement update has a stale fencing epoch") - } - next := current - next.Phase, next.Epoch, next.DisablePlanDigest = completed, nextEpoch, plan.PlanDigest - switch completed { - case ProtectionPhasePrerequisiteAbsent: - next.PrerequisiteEffective = false - case ProtectionPhaseRuntimeReverted: - if next.PrerequisiteEffective { - failure, _ := NewLifecycleFailure("protection_image_revert_unsafe") - return ProtectionLifecycleState{}, failure - } - case ProtectionPhaseLocalSupportRemoved: - if next.PrerequisiteEffective { - failure, _ := NewLifecycleFailure("protection_image_revert_unsafe") - return ProtectionLifecycleState{}, failure - } - next.LocalSupportInstalled = false - case ProtectionPhaseComplete: - if next.PrerequisiteEffective || next.LocalSupportInstalled { - return ProtectionLifecycleState{}, errors.New("disablement cannot complete before prerequisite and local support removal") - } - next.State = ProtectionDisabled - next.ServiceImage = "" - next.Schedules = inactiveProtectionSchedules(next.Schedules) - } - if err := next.Seal(); err != nil { - return ProtectionLifecycleState{}, err - } - return next, nil -} - -func RollbackProtectionDisable(current ProtectionLifecycleState, plan ProtectionDisablePlan, authorization ProtectionDisableAuthorization, nextEpoch int) (ProtectionLifecycleState, error) { - if err := current.Validate(); err != nil { - return ProtectionLifecycleState{}, err - } - if err := plan.Validate(); err != nil { - return ProtectionLifecycleState{}, err - } - if err := validateProtectionDisableAuthority(current, plan, authorization); err != nil { - return ProtectionLifecycleState{}, err - } - if current.State != ProtectionDisablePending || (current.Phase != ProtectionPhaseRequested && current.Phase != ProtectionPhasePrerequisiteReversed) { - return ProtectionLifecycleState{}, errors.New("disablement rollback is no longer safe after prerequisite absence was verified") - } - if nextEpoch <= current.Epoch { - return ProtectionLifecycleState{}, errors.New("protection rollback has a stale fencing epoch") - } - next := current - next.State, next.Phase, next.Epoch = ProtectionEnabled, ProtectionPhaseIdle, nextEpoch - next.OperationID, next.DisablePlanDigest, next.RequestedAt, next.ActionDeadline = "", "", "", "" - next.PrerequisiteEffective, next.LocalSupportInstalled = true, true - next.Schedules = effectiveProtectionSchedules(*next.LastEffective, true) - if err := next.Seal(); err != nil { - return ProtectionLifecycleState{}, err - } - return next, nil -} - -func (state ProtectionLifecycleState) AllowOperation(kind OperationKind, touchesProtectedService bool) error { - if err := state.Validate(); err != nil { - return err - } - if state.State != ProtectionDisablePending { - return nil - } - if kind == KindServiceImagePatch { - failure, _ := NewLifecycleFailure("service_image_patch_disable_pending") - return failure - } - if kind == KindRestoreTest || ((kind == KindDeploy || kind == KindServiceApply) && touchesProtectedService) { - failure, _ := NewLifecycleFailure("protection_disable_pending") - return failure - } - return nil -} - -func (state ProtectionLifecycleState) ValidateRuntimeImage(candidate string) error { - if err := state.Validate(); err != nil { - return err - } - if state.State == ProtectionDisablePending && candidate != state.ServiceImage && state.Phase != ProtectionPhaseRuntimeReverted && state.Phase != ProtectionPhaseLocalSupportRemoved { - failure, _ := NewLifecycleFailure("protection_image_revert_unsafe") - return failure - } - return nil -} - -func (state ProtectionLifecycleState) RuntimeState() app.ServiceRuntimeState { - return app.ServiceRuntimeState{ - ProtectionState: string(state.State), ServiceImage: state.ServiceImage, - PublicationVerified: state.ServiceImagePublicationVerified, - LastEffective: cloneProtectionProjection(state.LastEffective), - } -} - -func (state ProtectionLifecycleState) Status(now time.Time) (ProtectionLifecycleStatus, error) { - if err := state.Validate(); err != nil { - return ProtectionLifecycleStatus{}, err - } - status := ProtectionLifecycleStatus{State: state.State, Phase: state.Phase, Schedules: append([]ProtectionScheduleState(nil), state.Schedules...)} - for _, schedule := range state.Schedules { - if schedule.Active && schedule.Kind != "restore-drill" { - status.StorageContinues = true - } - } - if state.State != ProtectionDisablePending { - return status, nil - } - requested, _ := time.Parse(time.RFC3339Nano, state.RequestedAt) - deadline, _ := time.Parse(time.RFC3339Nano, state.ActionDeadline) - now = now.UTC() - elapsed := now.Sub(requested) - if elapsed < 0 { - elapsed = 0 - } - status.RequestedAt, status.ActionDeadline = state.RequestedAt, state.ActionDeadline - status.Elapsed = elapsed.Round(time.Second).String() - status.ResolvingCommand = "ob protection disable --output ndjson" - if !now.Before(deadline) { - failure, _ := NewLifecycleFailure("protection_disablement_overdue") - status.Failure = &failure - } - return status, nil -} - -func (state ProtectionLifecycleState) RemovalRequest(mode OperationKind) (ProtectionRemovalRequest, error) { - if err := state.Validate(); err != nil { - return ProtectionRemovalRequest{}, err - } - if state.State != ProtectionDisabled || state.PrerequisiteEffective || state.LocalSupportInstalled { - failure, _ := NewLifecycleFailure("protection_disable_pending") - return ProtectionRemovalRequest{}, failure - } - return ProtectionRemovalRequest{ - Mode: mode, Application: state.Application, Environment: state.Environment, Service: state.Service, - ProtectionState: string(state.State), StateDigest: state.StateDigest, PrerequisitesVerifiedAbsent: true, - }, nil -} - -func (state *ProtectionLifecycleState) Seal() error { - if state == nil { - return errors.New("protection lifecycle state is nil") - } - if err := state.validateContent(); err != nil { - return err - } - digest, err := state.computeDigest() - if err != nil { - return err - } - state.StateDigest = digest - return nil -} - -func (state ProtectionLifecycleState) Validate() error { - if err := state.validateContent(); err != nil { - return err - } - if !lifecycleGraphDigest.MatchString(state.StateDigest) { - return errors.New("protection lifecycle state digest is missing or invalid") - } - expected, err := state.computeDigest() - if err != nil { - return err - } - if state.StateDigest != expected { - return errors.New("protection lifecycle state digest mismatch") - } - return nil -} - -func (state ProtectionLifecycleState) validateContent() error { - if state.SchemaVersion != ProtectionStateSchemaVersion { - return fmt.Errorf("unsupported protection state schema %q", state.SchemaVersion) - } - for _, value := range []string{state.Application, state.Environment, state.Service} { - if !safeLifecycleMetadata(value) { - return errors.New("protection state ownership metadata is invalid") - } - } - if state.Epoch <= 0 { - return errors.New("protection lifecycle epoch must be positive") - } - if !validProtectionStatePhase(state.State, state.Phase) { - return fmt.Errorf("invalid protection state/phase %q/%q", state.State, state.Phase) - } - if state.ServiceImage != "" && !protectedRuntimeImage.MatchString(state.ServiceImage) { - return errors.New("protected service image must be digest-pinned") - } - if state.State == ProtectionEnabled || state.State == ProtectionDisablePending { - if state.LastEffective == nil || state.ServiceImage == "" || !state.ServiceImagePublicationVerified { - return errors.New("active protection state requires last-effective intent and a provenance-verified service image") - } - } - if state.State == ProtectionDisablePending { - if !safeLifecycleMetadata(state.OperationID) { - return errors.New("disable-pending state requires an operation identity") - } - requested, err := time.Parse(time.RFC3339Nano, state.RequestedAt) - if err != nil { - return errors.New("disable-pending requested_at is invalid") - } - deadline, err := time.Parse(time.RFC3339Nano, state.ActionDeadline) - if err != nil || deadline.Sub(requested) != ProtectionDisableActionWindow { - return errors.New("disable-pending action deadline must be exactly 24 hours") - } - if state.Phase != ProtectionPhaseRequested && !lifecycleGraphDigest.MatchString(state.DisablePlanDigest) { - return errors.New("advanced disable-pending state requires a sealed plan digest") - } - } - previous := "" - for _, schedule := range state.Schedules { - if !safeLifecycleMetadata(schedule.Kind) || (previous != "" && schedule.Kind <= previous) { - return errors.New("protection schedules must have unique sorted safe kinds") - } - if schedule.Kind == "restore-drill" && state.State == ProtectionDisablePending && schedule.Active { - return errors.New("restore drills must stop during disable-pending") - } - previous = schedule.Kind - } - return nil -} - -func (state ProtectionLifecycleState) computeDigest() (string, error) { - copy := state - copy.StateDigest = "" - encoded, err := json.Marshal(copy) - if err != nil { - return "", err - } - sum := sha256.Sum256(encoded) - return "sha256:" + hex.EncodeToString(sum[:]), nil -} - -func (plan *ProtectionDisablePlan) Seal() error { - if plan == nil { - return errors.New("protection disable plan is nil") - } - if err := plan.validateContent(); err != nil { - return err - } - digest, err := plan.computeDigest() - if err != nil { - return err - } - plan.PlanDigest = digest - return nil -} - -func (plan ProtectionDisablePlan) Validate() error { - if err := plan.validateContent(); err != nil { - return err - } - expected, err := plan.computeDigest() - if err != nil { - return err - } - if plan.PlanDigest != expected { - return errors.New("protection disable plan digest mismatch") - } - return nil -} - -func (plan ProtectionDisablePlan) validateContent() error { - if plan.SchemaVersion != ProtectionDisablePlanSchemaVersion || plan.Approval != ApprovalStrong || !plan.Interruption || plan.RemoteDataAction != "handback-preserve" { - return errors.New("protection disable plan safety contract is invalid") - } - for _, value := range []string{plan.OperationID, plan.Application, plan.Environment, plan.Service} { - if !safeLifecycleMetadata(value) { - return errors.New("protection disable plan metadata is invalid") - } - } - if !lifecycleGraphDigest.MatchString(plan.StateDigest) || plan.Epoch <= 0 { - return errors.New("protection disable plan state binding is invalid") - } - want := []ProtectionDisablePhase{ - ProtectionPhasePrerequisiteReversed, ProtectionPhasePrerequisiteAbsent, ProtectionPhaseRuntimeReverted, - ProtectionPhaseLocalSupportRemoved, ProtectionPhaseComplete, - } - if len(plan.Steps) != len(want) { - return errors.New("protection disable plan has an incomplete phase graph") - } - for index, phase := range want { - if plan.Steps[index].Phase != phase { - return errors.New("protection disable plan phase graph is not canonical") - } - } - return nil -} - -func (plan ProtectionDisablePlan) computeDigest() (string, error) { - copy := plan - copy.PlanDigest = "" - encoded, err := json.Marshal(copy) - if err != nil { - return "", err - } - sum := sha256.Sum256(encoded) - return "sha256:" + hex.EncodeToString(sum[:]), nil -} - -func SaveProtectionLifecycleState(path string, state ProtectionLifecycleState) error { - if err := state.Validate(); err != nil { - return err - } - return saveBackupArtifact(path, ".protection-state-*", state) -} - -func LoadProtectionLifecycleState(path string) (ProtectionLifecycleState, error) { - var state ProtectionLifecycleState - if err := loadBackupArtifact(path, &state); err != nil { - return ProtectionLifecycleState{}, err - } - if err := state.Validate(); err != nil { - return ProtectionLifecycleState{}, err - } - return state, nil -} - -// DecodeProtectionLifecycleState validates target-observed state without -// accepting unknown fields or trailing JSON that were not covered by its seal. -func DecodeProtectionLifecycleState(encoded []byte) (ProtectionLifecycleState, error) { - decoder := json.NewDecoder(bytes.NewReader(encoded)) - decoder.DisallowUnknownFields() - var state ProtectionLifecycleState - if err := decoder.Decode(&state); err != nil { - return ProtectionLifecycleState{}, fmt.Errorf("decode protection lifecycle state: %w", err) - } - var extra any - if err := decoder.Decode(&extra); err != io.EOF { - if err == nil { - return ProtectionLifecycleState{}, errors.New("decode protection lifecycle state: multiple JSON values") - } - return ProtectionLifecycleState{}, fmt.Errorf("decode protection lifecycle state: %w", err) - } - if err := state.Validate(); err != nil { - return ProtectionLifecycleState{}, fmt.Errorf("validate protection lifecycle state: %w", err) - } - return state, nil -} - -func validateProtectionDisableAuthority(state ProtectionLifecycleState, plan ProtectionDisablePlan, authorization ProtectionDisableAuthorization) error { - if !authorization.Strong || authorization.OperationID != plan.OperationID || authorization.PlanDigest != plan.PlanDigest || authorization.StateDigest != plan.StateDigest { - failure, _ := NewLifecycleFailure("protection_disablement_not_authorized") - return failure - } - if state.OperationID != plan.OperationID || state.Application != plan.Application || state.Environment != plan.Environment || state.Service != plan.Service { - return errors.New("protection disable plan does not match lifecycle state") - } - if state.Phase == ProtectionPhaseRequested { - if state.StateDigest != plan.StateDigest { - return errors.New("protection disable plan state binding is stale") - } - } else if state.DisablePlanDigest != plan.PlanDigest { - return errors.New("protection disable plan does not own the in-flight state") - } - return nil -} - -func nextProtectionDisablePhase(current ProtectionDisablePhase) (ProtectionDisablePhase, bool) { - switch current { - case ProtectionPhaseRequested: - return ProtectionPhasePrerequisiteReversed, true - case ProtectionPhasePrerequisiteReversed: - return ProtectionPhasePrerequisiteAbsent, true - case ProtectionPhasePrerequisiteAbsent: - return ProtectionPhaseRuntimeReverted, true - case ProtectionPhaseRuntimeReverted: - return ProtectionPhaseLocalSupportRemoved, true - case ProtectionPhaseLocalSupportRemoved: - return ProtectionPhaseComplete, true - default: - return "", false - } -} - -func validProtectionStatePhase(state ProtectionState, phase ProtectionDisablePhase) bool { - switch state { - case ProtectionNeverEnabled, ProtectionEnabled: - return phase == ProtectionPhaseIdle - case ProtectionDisablePending: - return phase == ProtectionPhaseRequested || phase == ProtectionPhasePrerequisiteReversed || - phase == ProtectionPhasePrerequisiteAbsent || phase == ProtectionPhaseRuntimeReverted || phase == ProtectionPhaseLocalSupportRemoved - case ProtectionDisabled: - return phase == ProtectionPhaseComplete || phase == ProtectionPhaseIdle - default: - return false - } -} - -func effectiveProtectionSchedules(projection app.ProtectionEffectiveProjection, drillsActive bool) []ProtectionScheduleState { - schedules := []ProtectionScheduleState{ - {Kind: "backup-create", Schedule: projection.Policy.Schedule, Active: true}, - {Kind: "backup-prune", Schedule: projection.Policy.Schedule, Active: true}, - {Kind: "restore-drill", Schedule: projection.Policy.RestoreDrill.Schedule, Active: drillsActive}, - } - if projection.Policy.RecoveryKind == "pitr" { - schedules = append(schedules, ProtectionScheduleState{Kind: "replay-archive", Schedule: replayArchiveSchedule(projection.Policy), Active: true}) - } - sort.Slice(schedules, func(i, j int) bool { return schedules[i].Kind < schedules[j].Kind }) - return schedules -} - -func replayArchiveSchedule(policy app.ProtectionPolicy) app.Schedule { - duration, ok := app.ParseDuration(policy.MaximumDataLoss) - if !ok || duration <= 0 { - return policy.Schedule - } - minutes := int(duration / time.Minute) - if minutes < 1 { - minutes = 1 - } - var cron string - switch { - case minutes < 60: - cron = fmt.Sprintf("*/%d * * * *", minutes) - case minutes < 24*60: - hours := minutes / 60 - cron = fmt.Sprintf("0 */%d * * *", hours) - default: - cron = "0 0 * * *" - } - return app.Schedule{Cron: cron, Timezone: policy.Schedule.Timezone} -} - -func inactiveProtectionSchedules(schedules []ProtectionScheduleState) []ProtectionScheduleState { - copy := append([]ProtectionScheduleState(nil), schedules...) - for index := range copy { - copy[index].Active = false - } - return copy -} - -func cloneProtectionProjection(projection *app.ProtectionEffectiveProjection) *app.ProtectionEffectiveProjection { - if projection == nil { - return nil - } - copy := *projection - return © -} - -func (state ProtectionLifecycleState) activeScheduleKinds() []string { - var kinds []string - for _, schedule := range state.Schedules { - if schedule.Active { - kinds = append(kinds, schedule.Kind) - } - } - sort.Strings(kinds) - return kinds -} - -func protectionStateContainsRemoteDeletion(value any) bool { - encoded, _ := json.Marshal(value) - text := strings.ToLower(string(encoded)) - return strings.Contains(text, "delete-remote") || strings.Contains(text, "purge-remote") -} diff --git a/internal/onebox/protection_state_test.go b/internal/onebox/protection_state_test.go deleted file mode 100644 index 3d5fb895..00000000 --- a/internal/onebox/protection_state_test.go +++ /dev/null @@ -1,267 +0,0 @@ -package onebox - -import ( - "errors" - "path/filepath" - "reflect" - "strings" - "testing" - "time" - - "github.com/labstack/onebox/internal/app" -) - -func protectionStateProjection() app.ProtectionEffectiveProjection { - return app.ProtectionEffectiveProjection{ - Policy: app.ProtectionPolicy{ - Target: "offsite", RecoveryKind: "pitr", MaximumDataLoss: "5m", - Schedule: app.Schedule{Cron: "17 */6 * * *", Timezone: "UTC"}, - Retention: app.ProtectionRetention{MinimumGenerations: 7, RecoveryWindow: "7d"}, - RestoreDrill: app.RestoreDrillPolicy{ - Schedule: app.Schedule{Cron: "23 4 * * 1,4", Timezone: "UTC"}, ProofMaximumAge: "7d", - }, - }, - Target: app.BackupTarget{ - Kind: "s3-compatible", Endpoint: "https://objects.example.test", Bucket: "onebox-backups", - TLS: "required", FailureDomain: app.FailureDomain{Identity: "provider-a/us-east-1/account-42"}, - Credentials: app.CredentialReference{ - File: "secrets/backup.env", Provider: "sops", AccessKeyEntry: "BACKUP_ACCESS_KEY_ID", SecretKeyEntry: "BACKUP_SECRET_ACCESS_KEY", - }, - Encryption: app.TargetEncryption{PITR: "archive-password"}, - }, - } -} - -func pendingProtectionState(t *testing.T) (ProtectionLifecycleState, ProtectionDisablePlan, ProtectionDisableAuthorization, time.Time) { - t.Helper() - initial, err := NewProtectionLifecycleState("example", "production", "database", 1) - if err != nil { - t.Fatal(err) - } - image := "ghcr.io/labstack/onebox-postgres-pgbackrest@sha256:" + strings.Repeat("a", 64) - enabled, err := EnableProtection(initial, protectionStateProjection(), image, "enable-op", true, 2) - if err != nil { - t.Fatal(err) - } - now := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC) - pending, err := RequestProtectionDisable(enabled, "disable-op", now, 3) - if err != nil { - t.Fatal(err) - } - plan, err := NewProtectionDisablePlan(pending) - if err != nil { - t.Fatal(err) - } - authorization := ProtectionDisableAuthorization{ - OperationID: plan.OperationID, PlanDigest: plan.PlanDigest, StateDigest: plan.StateDigest, Strong: true, - } - return pending, plan, authorization, now -} - -func TestProtectionDisableRequiresStrongApprovalAndKeepsStorageSchedules(t *testing.T) { - pending, plan, _, _ := pendingProtectionState(t) - _, err := AdvanceProtectionDisable(pending, plan, ProtectionDisableAuthorization{}, ProtectionPhasePrerequisiteReversed, 4) - var failure LifecycleFailure - if !errors.As(err, &failure) || failure.Code != "protection_disablement_not_authorized" { - t.Fatalf("missing approval error = %v", err) - } - want := []string{"backup-create", "backup-prune", "replay-archive"} - if got := pending.activeScheduleKinds(); !reflect.DeepEqual(got, want) { - t.Fatalf("pending schedules = %#v, want %#v", got, want) - } - for _, schedule := range pending.Schedules { - if schedule.Kind == "restore-drill" && schedule.Active { - t.Fatal("restore drill remained active during disable-pending") - } - if schedule.Kind == "replay-archive" && schedule.Schedule.Cron != "*/5 * * * *" { - t.Fatalf("replay archive schedule = %q, want cadence bounded by 5m RPO", schedule.Schedule.Cron) - } - } -} - -func TestProtectionDisableOperationGatesAndSafeImageRetention(t *testing.T) { - pending, _, _, _ := pendingProtectionState(t) - if err := pending.AllowOperation(KindDeploy, false); err != nil { - t.Fatalf("unrelated apply was refused: %v", err) - } - if err := pending.AllowOperation(KindBackupCreate, true); err != nil { - t.Fatalf("retained backup operation was refused: %v", err) - } - for _, test := range []struct { - kind OperationKind - code string - }{ - {KindServiceImagePatch, "service_image_patch_disable_pending"}, - {KindRestoreTest, "protection_disable_pending"}, - } { - err := pending.AllowOperation(test.kind, true) - var failure LifecycleFailure - if !errors.As(err, &failure) || failure.Code != test.code { - t.Fatalf("operation %s error = %v, want %s", test.kind, err, test.code) - } - } - if err := pending.ValidateRuntimeImage("postgres:17"); err == nil || !strings.Contains(err.Error(), "protection_image_revert_unsafe") { - t.Fatalf("unsafe image reversion error = %v", err) - } -} - -func TestProtectionDisableStatusBecomesOverdueWithoutMutation(t *testing.T) { - pending, _, _, requestedAt := pendingProtectionState(t) - before := pending.StateDigest - status, err := pending.Status(requestedAt.Add(24*time.Hour + time.Second)) - if err != nil { - t.Fatal(err) - } - if status.Failure == nil || status.Failure.Code != "protection_disablement_overdue" || !status.StorageContinues { - t.Fatalf("overdue status = %#v", status) - } - if pending.StateDigest != before || status.ResolvingCommand != "ob protection disable --output ndjson" { - t.Fatal("status mutated state or omitted the exact resolving command") - } -} - -func TestProtectionDisableCrashResumeRetryAndSafeCompletion(t *testing.T) { - state, plan, authorization, _ := pendingProtectionState(t) - var err error - state, err = AdvanceProtectionDisable(state, plan, authorization, ProtectionPhasePrerequisiteReversed, 4) - if err != nil { - t.Fatal(err) - } - statePath := filepath.Join(t.TempDir(), "protection-state.json") - if err := SaveProtectionLifecycleState(statePath, state); err != nil { - t.Fatal(err) - } - resumed, err := LoadProtectionLifecycleState(statePath) - if err != nil { - t.Fatal(err) - } - // Output loss after commit: retrying the same phase is an idempotent lookup. - retried, err := AdvanceProtectionDisable(resumed, plan, authorization, ProtectionPhasePrerequisiteReversed, 5) - if err != nil { - t.Fatal(err) - } - if retried.StateDigest != resumed.StateDigest || retried.Epoch != resumed.Epoch { - t.Fatal("same-phase retry created a second transition") - } - for index, phase := range []ProtectionDisablePhase{ - ProtectionPhasePrerequisiteAbsent, ProtectionPhaseRuntimeReverted, - ProtectionPhaseLocalSupportRemoved, ProtectionPhaseComplete, - } { - resumed, err = AdvanceProtectionDisable(resumed, plan, authorization, phase, 5+index) - if err != nil { - t.Fatalf("advance %s: %v", phase, err) - } - } - if resumed.State != ProtectionDisabled || resumed.PrerequisiteEffective || resumed.LocalSupportInstalled || len(resumed.activeScheduleKinds()) != 0 { - t.Fatalf("completed disablement = %#v", resumed) - } - request, err := resumed.RemovalRequest(KindProtectionDisable) - if err != nil || !request.PrerequisitesVerifiedAbsent { - t.Fatalf("safe removal request = %#v, %v", request, err) - } - if protectionStateContainsRemoteDeletion(plan) || protectionStateContainsRemoteDeletion(resumed) { - t.Fatal("disablement invented a remote deletion path") - } -} - -func TestProtectionDisableRollbackIsBounded(t *testing.T) { - pending, plan, authorization, _ := pendingProtectionState(t) - rolledBack, err := RollbackProtectionDisable(pending, plan, authorization, 4) - if err != nil { - t.Fatal(err) - } - if rolledBack.State != ProtectionEnabled || !rolledBack.PrerequisiteEffective || !rolledBack.LocalSupportInstalled { - t.Fatalf("rollback state = %#v", rolledBack) - } - if !containsString(rolledBack.activeScheduleKinds(), "restore-drill") { - t.Fatal("rollback did not restore the drill schedule") - } - advanced, err := AdvanceProtectionDisable(pending, plan, authorization, ProtectionPhasePrerequisiteReversed, 4) - if err != nil { - t.Fatal(err) - } - advanced, err = AdvanceProtectionDisable(advanced, plan, authorization, ProtectionPhasePrerequisiteAbsent, 5) - if err != nil { - t.Fatal(err) - } - if _, err := RollbackProtectionDisable(advanced, plan, authorization, 6); err == nil { - t.Fatal("rollback was accepted after prerequisite absence was verified") - } -} - -func TestProtectionDisableRejectsCompetingInFlightOperation(t *testing.T) { - pending, _, _, requestedAt := pendingProtectionState(t) - retried, err := RequestProtectionDisable(pending, "disable-op", requestedAt.Add(time.Minute), 4) - if err != nil || retried.StateDigest != pending.StateDigest { - t.Fatalf("same-operation retry = %#v, %v", retried, err) - } - _, err = RequestProtectionDisable(pending, "other-op", requestedAt.Add(time.Minute), 4) - var failure LifecycleFailure - if !errors.As(err, &failure) || failure.Code != "backup_conflict" { - t.Fatalf("competing operation error = %v", err) - } -} - -func TestProtectionDisableRuntimeProjectionTreatsRetainedArtifactsAsDesired(t *testing.T) { - pending, _, _, _ := pendingProtectionState(t) - projection := protectionStateProjection() - withIntent := &app.Resolved{ - Spec: &app.Spec{ - Name: "example", BasePath: "/var/lib/onebox", - Services: map[string]app.Service{"database": {Driver: "postgres", Version: 17, Protection: &projection.Policy}}, - BackupTargets: map[string]app.BackupTarget{"offsite": projection.Target}, - }, - Env: "production", - } - original, err := withIntent.GenerateProtectionArtifacts("database") - if err != nil { - t.Fatal(err) - } - withoutIntent := &app.Resolved{ - Spec: &app.Spec{Name: "example", BasePath: "/var/lib/onebox", Services: map[string]app.Service{ - "database": {Driver: "postgres", Version: 17}, - }}, - Env: "production", - } - retained, err := withoutIntent.WithServiceRuntimeStates(map[string]app.ServiceRuntimeState{"database": pending.RuntimeState()}) - if err != nil { - t.Fatal(err) - } - desired, err := retained.GenerateProtectionArtifacts("database") - if err != nil { - t.Fatal(err) - } - observed := make(map[string]string, len(original.Artifacts)) - for _, artifact := range original.Artifacts { - observed[artifact.Class] = artifact.Digest - } - if drift := app.CompareProtectionArtifacts(desired, observed); len(drift) != 0 { - t.Fatalf("retained pending projection reported drift: %#v", drift) - } -} - -func containsString(values []string, want string) bool { - for _, value := range values { - if value == want { - return true - } - } - return false -} - -func TestRuntimeStateDoesNotInferImageEvidenceFromReference(t *testing.T) { - state, err := NewProtectionLifecycleState("example", "production", "database", 1) - if err != nil { - t.Fatal(err) - } - state.State = ProtectionDisabled - state.Phase = ProtectionPhaseIdle - state.ServiceImage = "postgres@sha256:" + strings.Repeat("a", 64) - if err := state.Seal(); err != nil { - t.Fatal(err) - } - runtime := state.RuntimeState() - if runtime.PublicationVerified || runtime.DigestAvailable || runtime.CacheVerified { - t.Fatalf("runtime inferred evidence from an image string: %#v", runtime) - } -} diff --git a/internal/onebox/runner_policy.go b/internal/onebox/runner_policy.go index 437f5fa8..98aa2ce3 100644 --- a/internal/onebox/runner_policy.go +++ b/internal/onebox/runner_policy.go @@ -19,7 +19,7 @@ func CheckRunnerCompatibility(policy app.Policy, runner buildinfo.Runner) error } func enforceRunnerPolicy(policy app.Policy, runner buildinfo.Runner, planSchema string) error { - if minimum := strings.TrimSpace(policy.MinimumOneboxVersion); minimum != "" { + if minimum := strings.TrimSpace(policy.MinOneboxVersion); minimum != "" { minimumVersion, err := buildinfo.ParseReleaseVersion(minimum) if err != nil { return fmt.Errorf("environment minimum Onebox version is invalid: %w", err) @@ -40,7 +40,7 @@ func enforceRunnerPolicy(policy app.Policy, runner buildinfo.Runner, planSchema ) } } - if minimum := strings.TrimSpace(policy.MinimumPlanSchema); minimum != "" { + if minimum := strings.TrimSpace(policy.MinPlanSchema); minimum != "" { atLeast, err := executableSchemaAtLeast(planSchema, minimum) if err != nil { return err diff --git a/internal/onebox/runner_policy_test.go b/internal/onebox/runner_policy_test.go index 65fd5a24..b9d19037 100644 --- a/internal/onebox/runner_policy_test.go +++ b/internal/onebox/runner_policy_test.go @@ -16,8 +16,8 @@ func TestEnforceRunnerPolicy(t *testing.T) { }, } policy := app.Policy{ - MinimumOneboxVersion: "v2026.8.3", - MinimumPlanSchema: "onebox.run/executable-deploy-plan/v1alpha1", + MinOneboxVersion: "v2026.8.3", + MinPlanSchema: "onebox.run/executable-deploy-plan/v1alpha1", } if err := enforceRunnerPolicy(policy, runner, "onebox.run/executable-deploy-plan/v1alpha2"); err != nil { t.Fatal(err) @@ -43,7 +43,7 @@ func TestEnforceRunnerPolicy(t *testing.T) { t.Run("invalid minimum", func(t *testing.T) { invalid := policy - invalid.MinimumOneboxVersion = "2026.8.3" + invalid.MinOneboxVersion = "2026.8.3" err := enforceRunnerPolicy(invalid, runner, "onebox.run/executable-deploy-plan/v1alpha2") if err == nil || !strings.Contains(err.Error(), "environment minimum Onebox version is invalid") { t.Fatalf("invalid minimum rejection is not actionable: %v", err) @@ -51,7 +51,7 @@ func TestEnforceRunnerPolicy(t *testing.T) { }) t.Run("plan schema", func(t *testing.T) { - policy.MinimumPlanSchema = "onebox.run/executable-deploy-plan/v1alpha3" + policy.MinPlanSchema = "onebox.run/executable-deploy-plan/v1alpha3" err := enforceRunnerPolicy(policy, runner, "onebox.run/executable-deploy-plan/v1alpha2") if err == nil || !strings.Contains(err.Error(), "below environment minimum") { t.Fatalf("old plan schema was not rejected: %v", err) diff --git a/internal/onebox/s3_target.go b/internal/onebox/s3_target.go deleted file mode 100644 index e830c595..00000000 --- a/internal/onebox/s3_target.go +++ /dev/null @@ -1,323 +0,0 @@ -package onebox - -import ( - "context" - "errors" - "fmt" - "net/netip" - "net/url" - "os" - "path/filepath" - "sort" - "strings" - - "github.com/labstack/onebox/internal/app" -) - -const ( - S3TargetAdapterSchemaVersion = "onebox.run/s3-target/v1alpha1" - S3TargetProbeSchemaVersion = "onebox.run/s3-target-probe/v1alpha1" -) - -// S3CredentialBinding identifies entries in a target-local mode-0600 file. -// It intentionally cannot carry credential values. -type S3CredentialBinding struct { - File string `json:"file"` - AccessKeyEntry string `json:"access_key_entry"` - SecretKeyEntry string `json:"secret_key_entry"` - SessionTokenEntry string `json:"session_token_entry,omitempty"` -} - -// S3TargetAdapter is the closed, secret-free input shared by native drivers -// that use an S3-compatible destination. Drivers still own consistency and -// repository semantics; this type owns destination identity and evidence. -type S3TargetAdapter struct { - SchemaVersion string `json:"schema_version"` - Name string `json:"name"` - Endpoint string `json:"endpoint"` - Bucket string `json:"bucket"` - Prefix string `json:"prefix,omitempty"` - Region string `json:"region,omitempty"` - TLS string `json:"tls"` - RecoveryKind string `json:"recovery_kind"` - EncryptionMode string `json:"encryption_mode"` - FailureDomain app.FailureDomain `json:"failure_domain"` - Credentials S3CredentialBinding `json:"credentials"` -} - -type ProtectedFailureDomain struct { - Identity string - Host string - Addresses []string -} - -// S3TargetProbeObservation is returned by the native target-side probe. It is -// evidence only: the adapter validates it before publishing a passing result. -type S3TargetProbeObservation struct { - EndpointHost string - EndpointAddresses []string - FailureDomainIdentity string - Reachable bool - Authorized bool - BucketPresent bool - TLSVerified bool - OffHost bool - EncryptionMode string - EncryptionEvidenceID string - ProbeEvidenceID string -} - -type S3TargetProber interface { - ProbeS3(context.Context, S3TargetAdapter) (S3TargetProbeObservation, error) -} - -type S3TargetProbeFunc func(context.Context, S3TargetAdapter) (S3TargetProbeObservation, error) - -func (probe S3TargetProbeFunc) ProbeS3(ctx context.Context, target S3TargetAdapter) (S3TargetProbeObservation, error) { - return probe(ctx, target) -} - -// S3TargetProbeEvidence is safe for plans, journals, status, and model-visible -// output. Credential paths and values are deliberately absent. -type S3TargetProbeEvidence struct { - SchemaVersion string `json:"schema_version"` - Target string `json:"target"` - EndpointHost string `json:"endpoint_host"` - EndpointAddresses []string `json:"endpoint_addresses"` - Bucket string `json:"bucket"` - Prefix string `json:"prefix,omitempty"` - Region string `json:"region,omitempty"` - TLS string `json:"tls"` - FailureDomainIdentity string `json:"failure_domain_identity"` - OffHost bool `json:"off_host"` - CredentialFileMode uint32 `json:"credential_file_mode"` - EncryptionMode string `json:"encryption_mode"` - EncryptionEvidenceID string `json:"encryption_evidence_id"` - ProbeEvidenceID string `json:"probe_evidence_id"` -} - -type S3CredentialFileEvidence struct { - Mode uint32 - Regular bool - Symlink bool -} - -func NewS3TargetAdapter(name, recoveryKind, credentialFile string, target app.BackupTarget) (S3TargetAdapter, error) { - if err := app.ValidateBackupTarget(name, target); err != nil { - return S3TargetAdapter{}, err - } - if target.Kind != "s3-compatible" { - return S3TargetAdapter{}, errors.New("S3 target adapter requires an s3-compatible target") - } - mode := app.BackupTargetEncryptionMode(target, recoveryKind) - if !oneOf(mode, "client-side", "archive-password", "server-side-sse") { - return S3TargetAdapter{}, lifecycleFailure("backup_encryption_unverified") - } - adapter := S3TargetAdapter{ - SchemaVersion: S3TargetAdapterSchemaVersion, - Name: name, - Endpoint: target.Endpoint, - Bucket: target.Bucket, - Prefix: target.Prefix, - Region: target.Region, - TLS: target.TLS, - RecoveryKind: recoveryKind, - EncryptionMode: mode, - FailureDomain: target.FailureDomain, - Credentials: S3CredentialBinding{ - File: credentialFile, AccessKeyEntry: target.Credentials.AccessKeyEntry, - SecretKeyEntry: target.Credentials.SecretKeyEntry, SessionTokenEntry: target.Credentials.SessionTokenEntry, - }, - } - if err := adapter.Validate(); err != nil { - return S3TargetAdapter{}, err - } - return adapter, nil -} - -func (target S3TargetAdapter) Validate() error { - if target.SchemaVersion != S3TargetAdapterSchemaVersion { - return fmt.Errorf("unsupported S3 target adapter schema %q", target.SchemaVersion) - } - if target.RecoveryKind != "snapshot" && target.RecoveryKind != "pitr" && target.RecoveryKind != "cold" { - return errors.New("S3 target recovery kind must be snapshot, pitr, or cold") - } - if !oneOf(target.EncryptionMode, "client-side", "archive-password", "server-side-sse") { - return lifecycleFailure("backup_encryption_unverified") - } - encryption := app.TargetEncryption{} - switch target.RecoveryKind { - case "snapshot": - encryption.Snapshot = target.EncryptionMode - case "pitr": - encryption.PITR = target.EncryptionMode - case "cold": - encryption.Cold = target.EncryptionMode - } - declared := app.BackupTarget{ - Kind: "s3-compatible", Endpoint: target.Endpoint, Bucket: target.Bucket, - Prefix: target.Prefix, Region: target.Region, TLS: target.TLS, - FailureDomain: target.FailureDomain, Encryption: encryption, - Credentials: app.CredentialReference{ - File: "bound/credentials.env", Provider: "sops", - AccessKeyEntry: target.Credentials.AccessKeyEntry, - SecretKeyEntry: target.Credentials.SecretKeyEntry, - SessionTokenEntry: target.Credentials.SessionTokenEntry, - }, - } - if err := app.ValidateBackupTarget(target.Name, declared); err != nil { - return err - } - if !filepath.IsAbs(target.Credentials.File) || filepath.Clean(target.Credentials.File) != target.Credentials.File { - return errors.New("S3 credential file must be a clean absolute target path") - } - entries := []string{target.Credentials.AccessKeyEntry, target.Credentials.SecretKeyEntry} - if target.Credentials.SessionTokenEntry != "" { - entries = append(entries, target.Credentials.SessionTokenEntry) - } - seen := make(map[string]struct{}, len(entries)) - for _, entry := range entries { - if _, exists := seen[entry]; exists { - return errors.New("S3 credential entries must be distinct") - } - seen[entry] = struct{}{} - } - return nil -} - -// InspectS3CredentialFile verifies the target-local file without reading its -// contents. The native probe opens only the named entries after this check. -func InspectS3CredentialFile(path string) (S3CredentialFileEvidence, error) { - if !filepath.IsAbs(path) || filepath.Clean(path) != path { - return S3CredentialFileEvidence{}, errors.New("inspect S3 credential file: path is not a clean absolute target path") - } - info, err := os.Lstat(path) - if err != nil { - return S3CredentialFileEvidence{}, errors.New("inspect S3 credential file") - } - return S3CredentialFileEvidence{ - Mode: uint32(info.Mode().Perm()), Regular: info.Mode().IsRegular(), Symlink: info.Mode()&os.ModeSymlink != 0, - }, nil -} - -func (target S3TargetAdapter) Probe(ctx context.Context, protected ProtectedFailureDomain, prober S3TargetProber) (S3TargetProbeEvidence, error) { - if err := target.Validate(); err != nil { - return S3TargetProbeEvidence{}, err - } - if prober == nil { - return S3TargetProbeEvidence{}, errors.New("S3 target probe is unavailable") - } - if target.declaredSelfTarget(protected) { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_not_independent") - } - _, protectedAddresses, err := canonicalProbeAddresses(protected.Addresses) - if err != nil || len(protectedAddresses) == 0 { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_not_independent") - } - credential, err := InspectS3CredentialFile(target.Credentials.File) - if err != nil || credential.Mode != 0o600 || !credential.Regular || credential.Symlink { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_unauthorized") - } - observation, err := prober.ProbeS3(ctx, target) - if err != nil { - if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { - return S3TargetProbeEvidence{}, err - } - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_unreachable") - } - evidence, err := target.validateObservation(protected, credential, observation) - if err != nil { - return S3TargetProbeEvidence{}, err - } - return evidence, nil -} - -func (target S3TargetAdapter) declaredSelfTarget(protected ProtectedFailureDomain) bool { - endpoint, _ := url.Parse(target.Endpoint) - return sameProbeHost(target.FailureDomain.Identity, protected.Identity) || - sameProbeHost(target.FailureDomain.Host, protected.Host) || - sameProbeHost(endpoint.Hostname(), protected.Host) -} - -func (target S3TargetAdapter) validateObservation( - protected ProtectedFailureDomain, - credential S3CredentialFileEvidence, - observation S3TargetProbeObservation, -) (S3TargetProbeEvidence, error) { - endpoint, _ := url.Parse(target.Endpoint) - if !sameProbeHost(observation.EndpointHost, endpoint.Hostname()) { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_not_independent") - } - addresses, parsedAddresses, err := canonicalProbeAddresses(observation.EndpointAddresses) - if err != nil || len(addresses) == 0 { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_not_independent") - } - _, protectedAddresses, err := canonicalProbeAddresses(protected.Addresses) - if err != nil || len(protectedAddresses) == 0 || addressesOverlap(parsedAddresses, protectedAddresses) || - observation.FailureDomainIdentity != target.FailureDomain.Identity || !observation.OffHost { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_not_independent") - } - if !observation.Reachable || !observation.BucketPresent { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_unreachable") - } - if !observation.Authorized { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_unauthorized") - } - if target.TLS == "required" && !observation.TLSVerified { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_target_unreachable") - } - if observation.EncryptionMode != target.EncryptionMode || !safeLifecycleMetadata(observation.EncryptionEvidenceID) { - return S3TargetProbeEvidence{}, lifecycleFailure("backup_encryption_unverified") - } - if !safeLifecycleMetadata(observation.ProbeEvidenceID) { - return S3TargetProbeEvidence{}, errors.New("S3 target probe returned an invalid evidence identity") - } - return S3TargetProbeEvidence{ - SchemaVersion: S3TargetProbeSchemaVersion, Target: target.Name, - EndpointHost: endpoint.Hostname(), EndpointAddresses: addresses, - Bucket: target.Bucket, Prefix: target.Prefix, Region: target.Region, TLS: target.TLS, - FailureDomainIdentity: target.FailureDomain.Identity, OffHost: true, CredentialFileMode: credential.Mode, - EncryptionMode: target.EncryptionMode, EncryptionEvidenceID: observation.EncryptionEvidenceID, - ProbeEvidenceID: observation.ProbeEvidenceID, - }, nil -} - -func canonicalProbeAddresses(values []string) ([]string, map[netip.Addr]struct{}, error) { - addresses := make(map[netip.Addr]struct{}, len(values)) - for _, value := range values { - address, err := netip.ParseAddr(strings.TrimSpace(value)) - if err != nil { - return nil, nil, err - } - addresses[address.Unmap()] = struct{}{} - } - encoded := make([]string, 0, len(addresses)) - for address := range addresses { - encoded = append(encoded, address.String()) - } - sort.Strings(encoded) - return encoded, addresses, nil -} - -func addressesOverlap(left, right map[netip.Addr]struct{}) bool { - for address := range left { - if _, exists := right[address]; exists { - return true - } - } - return false -} - -func sameProbeHost(left, right string) bool { - left = strings.TrimSuffix(strings.ToLower(strings.TrimSpace(left)), ".") - right = strings.TrimSuffix(strings.ToLower(strings.TrimSpace(right)), ".") - return left != "" && left == right -} - -func lifecycleFailure(code string) error { - failure, err := NewLifecycleFailure(code) - if err != nil { - return err - } - return failure -} diff --git a/internal/onebox/s3_target_test.go b/internal/onebox/s3_target_test.go deleted file mode 100644 index 42cef9ba..00000000 --- a/internal/onebox/s3_target_test.go +++ /dev/null @@ -1,239 +0,0 @@ -package onebox - -import ( - "context" - "encoding/json" - "errors" - "os" - "path/filepath" - "strings" - "testing" - - "github.com/labstack/onebox/internal/app" -) - -func testS3Target() app.BackupTarget { - return app.BackupTarget{ - Kind: "s3-compatible", Endpoint: "https://objects.example.net", Bucket: "onebox-backups", - Prefix: "production/shop", Region: "us-east-1", TLS: "required", - FailureDomain: app.FailureDomain{Identity: "provider-a/us-east-1/account-42", Host: "objects.example.net"}, - Credentials: app.CredentialReference{ - File: "secrets/backup.env", Provider: "sops", AccessKeyEntry: "BACKUP_ACCESS_KEY_ID", - SecretKeyEntry: "BACKUP_SECRET_ACCESS_KEY", SessionTokenEntry: "BACKUP_SESSION_TOKEN", - }, - Encryption: app.TargetEncryption{PITR: "archive-password", Snapshot: "client-side"}, - } -} - -func testS3CredentialFile(t *testing.T, mode os.FileMode) (string, string) { - t.Helper() - const secret = "storage-secret-canary" - path := filepath.Join(t.TempDir(), "credentials.env") - content := "BACKUP_ACCESS_KEY_ID=test\nBACKUP_SECRET_ACCESS_KEY=" + secret + "\nBACKUP_SESSION_TOKEN=session\n" - if err := os.WriteFile(path, []byte(content), mode); err != nil { - t.Fatal(err) - } - return path, secret -} - -func successfulS3Observation() S3TargetProbeObservation { - return S3TargetProbeObservation{ - EndpointHost: "objects.example.net", EndpointAddresses: []string{"198.51.100.24"}, - FailureDomainIdentity: "provider-a/us-east-1/account-42", - Reachable: true, Authorized: true, BucketPresent: true, TLSVerified: true, OffHost: true, - EncryptionMode: "archive-password", EncryptionEvidenceID: "pgbackrest-repo-cipher-aes-256-cbc", - ProbeEvidenceID: "s3-probe-20260807T120000Z", - } -} - -func lifecycleFailureCode(t *testing.T, err error) string { - t.Helper() - var failure LifecycleFailure - if !errors.As(err, &failure) { - t.Fatalf("error %v is not a lifecycle failure", err) - } - return failure.Code -} - -func TestS3TargetAdapterProjectsClosedSecretFreeContractAndProbeEvidence(t *testing.T) { - credentialFile, secret := testS3CredentialFile(t, 0o600) - target, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, testS3Target()) - if err != nil { - t.Fatal(err) - } - if target.Endpoint != "https://objects.example.net" || target.Bucket != "onebox-backups" || target.Prefix != "production/shop" || - target.Region != "us-east-1" || target.TLS != "required" || target.EncryptionMode != "archive-password" { - t.Fatalf("target adapter = %#v", target) - } - - probeCalls := 0 - evidence, err := target.Probe(context.Background(), ProtectedFailureDomain{ - Identity: "onebox-host/shop", Host: "app.example.net", Addresses: []string{"203.0.113.10"}, - }, S3TargetProbeFunc(func(_ context.Context, got S3TargetAdapter) (S3TargetProbeObservation, error) { - probeCalls++ - if got.Credentials.File != credentialFile || got.Credentials.SecretKeyEntry != "BACKUP_SECRET_ACCESS_KEY" { - t.Fatalf("probe target = %#v", got) - } - return successfulS3Observation(), nil - })) - if err != nil { - t.Fatalf("probe S3 target: %v", err) - } - if probeCalls != 1 || !evidence.OffHost || evidence.CredentialFileMode != 0o600 || evidence.EncryptionEvidenceID == "" { - t.Fatalf("probe evidence = %#v, calls = %d", evidence, probeCalls) - } - encoded, _ := json.Marshal(evidence) - if strings.Contains(string(encoded), credentialFile) || strings.Contains(string(encoded), secret) { - t.Fatalf("public target evidence leaked credential material: %s", encoded) - } -} - -func TestS3TargetAdapterRejectsInvalidAndUnprovenConfiguration(t *testing.T) { - credentialFile, _ := testS3CredentialFile(t, 0o600) - - invalidRegion := testS3Target() - invalidRegion.Region = "US East 1" - if _, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, invalidRegion); err == nil { - t.Fatal("invalid S3 region was accepted") - } - - unproven := testS3Target() - unproven.Encryption.PITR = "" - if _, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, unproven); lifecycleFailureCode(t, err) != "backup_encryption_unverified" { - t.Fatalf("unproven encryption error = %v", err) - } - - if _, err := NewS3TargetAdapter("offsite", "pitr", "relative/credentials.env", testS3Target()); err == nil { - t.Fatal("relative target-side credential path was accepted") - } - - adapter, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, testS3Target()) - if err != nil { - t.Fatal(err) - } - adapter.Credentials.SecretKeyEntry = "bad/name" - if err := adapter.Validate(); err == nil { - t.Fatal("mutated invalid credential entry was accepted") - } - - wrongKind := testS3Target() - wrongKind.Kind = "minio-replication" - if _, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, wrongKind); err == nil { - t.Fatal("removed replication target was accepted by S3 adapter") - } -} - -func TestS3TargetAdapterRejectsDeclaredSelfTargetBeforeProbe(t *testing.T) { - credentialFile, _ := testS3CredentialFile(t, 0o600) - target, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, testS3Target()) - if err != nil { - t.Fatal(err) - } - called := false - _, err = target.Probe(context.Background(), ProtectedFailureDomain{ - Identity: target.FailureDomain.Identity, Host: "app.example.net", - }, S3TargetProbeFunc(func(context.Context, S3TargetAdapter) (S3TargetProbeObservation, error) { - called = true - return successfulS3Observation(), nil - })) - if lifecycleFailureCode(t, err) != "backup_target_not_independent" || called { - t.Fatalf("self-target error/called = %v/%v", err, called) - } -} - -func TestS3TargetAdapterRejectsResolvedHostAlias(t *testing.T) { - credentialFile, _ := testS3CredentialFile(t, 0o600) - target, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, testS3Target()) - if err != nil { - t.Fatal(err) - } - observation := successfulS3Observation() - observation.EndpointAddresses = []string{"203.0.113.10", "2001:db8::20"} - _, err = target.Probe(context.Background(), ProtectedFailureDomain{ - Identity: "onebox-host/shop", Host: "app.example.net", Addresses: []string{"203.0.113.10"}, - }, S3TargetProbeFunc(func(context.Context, S3TargetAdapter) (S3TargetProbeObservation, error) { - return observation, nil - })) - if lifecycleFailureCode(t, err) != "backup_target_not_independent" { - t.Fatalf("alias error = %v", err) - } -} - -func TestS3TargetAdapterRequiresProtectedAddressesBeforeProbe(t *testing.T) { - credentialFile, _ := testS3CredentialFile(t, 0o600) - target, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, testS3Target()) - if err != nil { - t.Fatal(err) - } - called := false - _, err = target.Probe(context.Background(), ProtectedFailureDomain{ - Identity: "onebox-host/shop", Host: "app.example.net", - }, S3TargetProbeFunc(func(context.Context, S3TargetAdapter) (S3TargetProbeObservation, error) { - called = true - return successfulS3Observation(), nil - })) - if lifecycleFailureCode(t, err) != "backup_target_not_independent" || called { - t.Fatalf("missing protected addresses error/called = %v/%v", err, called) - } -} - -func TestS3TargetAdapterRequiresPrivateCredentialFile(t *testing.T) { - credentialFile, _ := testS3CredentialFile(t, 0o644) - target, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, testS3Target()) - if err != nil { - t.Fatal(err) - } - called := false - _, err = target.Probe(context.Background(), ProtectedFailureDomain{ - Identity: "onebox-host/shop", Host: "app.example.net", Addresses: []string{"203.0.113.10"}, - }, - S3TargetProbeFunc(func(context.Context, S3TargetAdapter) (S3TargetProbeObservation, error) { - called = true - return successfulS3Observation(), nil - })) - if lifecycleFailureCode(t, err) != "backup_target_unauthorized" || called { - t.Fatalf("credential mode error/called = %v/%v", err, called) - } -} - -func TestS3TargetAdapterRedactsProbeFailure(t *testing.T) { - credentialFile, secret := testS3CredentialFile(t, 0o600) - target, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, testS3Target()) - if err != nil { - t.Fatal(err) - } - _, err = target.Probe(context.Background(), ProtectedFailureDomain{ - Identity: "onebox-host/shop", Host: "app.example.net", Addresses: []string{"203.0.113.10"}, - }, - S3TargetProbeFunc(func(context.Context, S3TargetAdapter) (S3TargetProbeObservation, error) { - return S3TargetProbeObservation{}, errors.New("provider response included " + secret) - })) - if lifecycleFailureCode(t, err) != "backup_target_unreachable" || strings.Contains(err.Error(), secret) { - t.Fatalf("redacted probe error = %v", err) - } -} - -// localTestRepository is intentionally test-only. It exercises repository -// contracts without ever being mistaken for off-host protection. -type localTestRepository struct{} - -func (localTestRepository) ProbeS3(context.Context, S3TargetAdapter) (S3TargetProbeObservation, error) { - observation := successfulS3Observation() - observation.OffHost = false - observation.ProbeEvidenceID = "local-test-repository" - return observation, nil -} - -func TestLocalTestRepositoryNeverCountsAsOffHostProtection(t *testing.T) { - credentialFile, _ := testS3CredentialFile(t, 0o600) - target, err := NewS3TargetAdapter("offsite", "pitr", credentialFile, testS3Target()) - if err != nil { - t.Fatal(err) - } - _, err = target.Probe(context.Background(), ProtectedFailureDomain{ - Identity: "onebox-host/shop", Host: "app.example.net", Addresses: []string{"203.0.113.10"}, - }, localTestRepository{}) - if lifecycleFailureCode(t, err) != "backup_target_not_independent" { - t.Fatalf("local repository error = %v", err) - } -} diff --git a/internal/onebox/scheduled_envelope.go b/internal/onebox/scheduled_envelope.go deleted file mode 100644 index a1259ad4..00000000 --- a/internal/onebox/scheduled_envelope.go +++ /dev/null @@ -1,412 +0,0 @@ -package onebox - -import ( - "bytes" - "crypto/sha256" - "encoding/hex" - "encoding/json" - "errors" - "fmt" - "io" - "os" - "path/filepath" - "sort" - "strings" - "time" -) - -const ( - ScheduledEnvelopeSchemaVersion = "onebox.run/scheduled-operation-envelope/v1alpha1" - CurrentScheduledEnvelopeProtocol = 1 - CurrentScheduledRunnerProtocol = 1 - CurrentScheduledCLIProtocol = 1 -) - -type ProtocolRange struct { - Minimum int `json:"minimum"` - Maximum int `json:"maximum"` -} - -type ScheduledRunnerCompatibility struct { - RunnerProtocol int `json:"runner_protocol"` - CLIProtocols ProtocolRange `json:"cli_protocols"` - EnvelopeProtocols ProtocolRange `json:"envelope_protocols"` -} - -type ScheduledCLICompatibility struct { - CLIProtocol int `json:"cli_protocol"` - RunnerProtocols ProtocolRange `json:"runner_protocols"` - EnvelopeProtocols ProtocolRange `json:"envelope_protocols"` -} - -type ScheduledTimingPolicy struct { - ScheduledFor string `json:"scheduled_for"` - NotBefore string `json:"not_before"` - ExpiresAt string `json:"expires_at"` - MaxRuntime string `json:"max_runtime"` - RetryIdentity string `json:"retry_identity"` -} - -type ScheduledStateBinding struct { - Path string `json:"path"` - Digest string `json:"digest"` - Epoch int `json:"epoch"` -} - -type ScheduledRunnerArtifactReference struct { - Path string `json:"path"` - Digest string `json:"digest"` - SBOMDigest string `json:"sbom_digest"` - ProvenanceID string `json:"provenance_id"` -} - -type ScheduledOperationEnvelope struct { - SchemaVersion string `json:"schema_version"` - EnvelopeProtocol int `json:"envelope_protocol"` - CLIProtocol int `json:"cli_protocol"` - RunnerProtocols ProtocolRange `json:"runner_protocols"` - OperationID string `json:"operation_id"` - Application string `json:"application"` - Environment string `json:"environment"` - Service string `json:"service"` - Operation OperationKind `json:"operation"` - Runner ScheduledRunnerArtifactReference `json:"runner"` - Timing ScheduledTimingPolicy `json:"timing"` - Artifacts []OperationArtifactBinding `json:"artifacts"` - State ScheduledStateBinding `json:"state"` - SecretFiles []SecretSlotReference `json:"secret_files,omitempty"` - EnvelopeDigest string `json:"envelope_digest"` -} - -type ScheduledEnvelopeInput struct { - CLIProtocol int - RunnerProtocols ProtocolRange - OperationID string - Application string - Environment string - Service string - Operation OperationKind - Runner ScheduledRunnerArtifactReference - Timing ScheduledTimingPolicy - Artifacts []OperationArtifactBinding - State ScheduledStateBinding - SecretFiles []SecretSlotReference -} - -func CurrentScheduledRunnerCompatibility() ScheduledRunnerCompatibility { - return ScheduledRunnerCompatibility{ - RunnerProtocol: CurrentScheduledRunnerProtocol, - CLIProtocols: ProtocolRange{Minimum: CurrentScheduledCLIProtocol, Maximum: CurrentScheduledCLIProtocol}, - EnvelopeProtocols: ProtocolRange{Minimum: CurrentScheduledEnvelopeProtocol, Maximum: CurrentScheduledEnvelopeProtocol}, - } -} - -func CurrentScheduledCLICompatibility() ScheduledCLICompatibility { - return ScheduledCLICompatibility{ - CLIProtocol: CurrentScheduledCLIProtocol, - RunnerProtocols: ProtocolRange{Minimum: CurrentScheduledRunnerProtocol, Maximum: CurrentScheduledRunnerProtocol}, - EnvelopeProtocols: ProtocolRange{Minimum: CurrentScheduledEnvelopeProtocol, Maximum: CurrentScheduledEnvelopeProtocol}, - } -} - -func NewScheduledOperationEnvelope(input ScheduledEnvelopeInput) (ScheduledOperationEnvelope, error) { - artifacts := append([]OperationArtifactBinding(nil), input.Artifacts...) - secretFiles := append([]SecretSlotReference(nil), input.SecretFiles...) - sort.Slice(artifacts, func(i, j int) bool { return artifacts[i].Class < artifacts[j].Class }) - sort.Slice(secretFiles, func(i, j int) bool { return secretFiles[i].Slot < secretFiles[j].Slot }) - envelope := ScheduledOperationEnvelope{ - SchemaVersion: ScheduledEnvelopeSchemaVersion, EnvelopeProtocol: CurrentScheduledEnvelopeProtocol, - CLIProtocol: input.CLIProtocol, RunnerProtocols: input.RunnerProtocols, - OperationID: input.OperationID, Application: input.Application, Environment: input.Environment, - Service: input.Service, Operation: input.Operation, Runner: input.Runner, Timing: input.Timing, - Artifacts: artifacts, State: input.State, SecretFiles: secretFiles, - } - if err := envelope.Seal(); err != nil { - return ScheduledOperationEnvelope{}, err - } - return envelope, nil -} - -// MaterializeScheduledOccurrence derives a fresh, sealed execution envelope -// from the durable schedule template installed beside a systemd timer. The -// template binds operation shape, artifacts, state, and the width of its -// timing window; every firing gets a distinct operation and retry identity so -// terminal-result deduplication applies only to that occurrence. -func MaterializeScheduledOccurrence(template ScheduledOperationEnvelope, now time.Time, entropy io.Reader) (ScheduledOperationEnvelope, error) { - if err := template.Validate(); err != nil { - return ScheduledOperationEnvelope{}, err - } - if entropy == nil { - return ScheduledOperationEnvelope{}, errors.New("scheduled occurrence entropy is unavailable") - } - nonce := make([]byte, 16) - if _, err := io.ReadFull(entropy, nonce); err != nil { - return ScheduledOperationEnvelope{}, errors.New("create scheduled occurrence identity") - } - scheduledFor, _ := time.Parse(time.RFC3339Nano, template.Timing.ScheduledFor) - notBefore, _ := time.Parse(time.RFC3339Nano, template.Timing.NotBefore) - expiresAt, _ := time.Parse(time.RFC3339Nano, template.Timing.ExpiresAt) - now = now.UTC() - identityInput := append([]byte(template.EnvelopeDigest+"\x00"+now.Format(time.RFC3339Nano)+"\x00"), nonce...) - sum := sha256.Sum256(identityInput) - identity := hex.EncodeToString(sum[:16]) - - occurrence := template - occurrence.OperationID = "scheduled-" + identity - occurrence.Timing.ScheduledFor = now.Format(time.RFC3339Nano) - occurrence.Timing.NotBefore = now.Add(-scheduledFor.Sub(notBefore)).Format(time.RFC3339Nano) - occurrence.Timing.ExpiresAt = now.Add(expiresAt.Sub(scheduledFor)).Format(time.RFC3339Nano) - occurrence.Timing.RetryIdentity = "occurrence-" + identity - occurrence.EnvelopeDigest = "" - if err := occurrence.Seal(); err != nil { - return ScheduledOperationEnvelope{}, err - } - return occurrence, nil -} - -func (envelope *ScheduledOperationEnvelope) Seal() error { - if envelope == nil { - return errors.New("scheduled operation envelope is nil") - } - if err := envelope.validateContent(); err != nil { - return err - } - digest, err := envelope.computeDigest() - if err != nil { - return err - } - envelope.EnvelopeDigest = digest - return nil -} - -func (envelope ScheduledOperationEnvelope) Validate() error { - if err := envelope.validateContent(); err != nil { - return err - } - if !lifecycleGraphDigest.MatchString(envelope.EnvelopeDigest) { - return errors.New("scheduled envelope digest is missing or invalid") - } - expected, err := envelope.computeDigest() - if err != nil { - return err - } - if envelope.EnvelopeDigest != expected { - return errors.New("scheduled envelope digest mismatch") - } - return nil -} - -func (envelope ScheduledOperationEnvelope) ValidateForRunner(runner ScheduledRunnerCompatibility, observedStateDigest string, now time.Time) error { - if err := envelope.Validate(); err != nil { - return err - } - if err := runner.validate(); err != nil { - return err - } - if !envelope.RunnerProtocols.Contains(runner.RunnerProtocol) || !runner.EnvelopeProtocols.Contains(envelope.EnvelopeProtocol) || !runner.CLIProtocols.Contains(envelope.CLIProtocol) { - return errors.New("scheduled_runner_incompatible: apply a CLI and scheduled runner with mutually supported protocols") - } - if observedStateDigest != envelope.State.Digest { - return errors.New("scheduled_envelope_stale: observed lifecycle state changed; apply the current protection plan") - } - notBefore, _ := time.Parse(time.RFC3339Nano, envelope.Timing.NotBefore) - expiresAt, _ := time.Parse(time.RFC3339Nano, envelope.Timing.ExpiresAt) - now = now.UTC() - if now.Before(notBefore) || !now.Before(expiresAt) { - return errors.New("scheduled_envelope_stale: execution is outside the sealed timing window; apply the current protection plan") - } - return nil -} - -func ValidateScheduledRunnerForCLI(envelope ScheduledOperationEnvelope, runner ScheduledRunnerCompatibility, cli ScheduledCLICompatibility) error { - if err := envelope.Validate(); err != nil { - return err - } - if err := runner.validate(); err != nil { - return err - } - if err := cli.validate(); err != nil { - return err - } - if envelope.CLIProtocol != cli.CLIProtocol || !cli.RunnerProtocols.Contains(runner.RunnerProtocol) || !cli.EnvelopeProtocols.Contains(envelope.EnvelopeProtocol) || - !runner.CLIProtocols.Contains(cli.CLIProtocol) || !runner.EnvelopeProtocols.Contains(envelope.EnvelopeProtocol) || !envelope.RunnerProtocols.Contains(runner.RunnerProtocol) { - return errors.New("scheduled_runner_incompatible: upgrade or apply Onebox so CLI, runner, and envelope protocol ranges overlap") - } - return nil -} - -func (envelope ScheduledOperationEnvelope) validateContent() error { - if envelope.SchemaVersion != ScheduledEnvelopeSchemaVersion || envelope.EnvelopeProtocol <= 0 || envelope.CLIProtocol <= 0 { - return errors.New("scheduled envelope schema or protocol is invalid") - } - if err := envelope.RunnerProtocols.validate("runner protocol"); err != nil { - return err - } - for _, value := range []string{envelope.OperationID, envelope.Application, envelope.Environment, envelope.Service} { - if !safeLifecycleMetadata(value) { - return errors.New("scheduled envelope identity must be safe metadata") - } - } - if _, err := LifecycleOperationSchemaFor(envelope.Operation, LifecycleScheduledRunnerSchema); err != nil { - return err - } - if !filepath.IsAbs(envelope.Runner.Path) || filepath.Clean(envelope.Runner.Path) != envelope.Runner.Path || - !lifecycleGraphDigest.MatchString(envelope.Runner.Digest) || !lifecycleGraphDigest.MatchString(envelope.Runner.SBOMDigest) || - !safeLifecycleMetadata(envelope.Runner.ProvenanceID) { - return errors.New("scheduled envelope runner artifact is invalid") - } - if err := envelope.Timing.validate(); err != nil { - return err - } - if !filepath.IsAbs(envelope.State.Path) || filepath.Clean(envelope.State.Path) != envelope.State.Path || - !lifecycleGraphDigest.MatchString(envelope.State.Digest) || envelope.State.Epoch <= 0 { - return errors.New("scheduled envelope state binding is invalid") - } - previous := "" - for index, artifact := range envelope.Artifacts { - if !safeLifecycleMetadata(artifact.Class) || !filepath.IsAbs(artifact.Path) || filepath.Clean(artifact.Path) != artifact.Path || - (artifact.Mode != 0o600 && artifact.Mode != 0o644) || !lifecycleGraphDigest.MatchString(artifact.Digest) { - return fmt.Errorf("scheduled envelope artifact %d is invalid", index) - } - if previous != "" && artifact.Class <= previous { - return errors.New("scheduled envelope artifacts must be unique and sorted by class") - } - previous = artifact.Class - } - previous = "" - for index, secret := range envelope.SecretFiles { - if !safeLifecycleMetadata(secret.Slot) || !safeLifecycleMetadata(secret.Entry) || !filepath.IsAbs(secret.File) || filepath.Clean(secret.File) != secret.File { - return fmt.Errorf("scheduled envelope secret_files[%d] is invalid", index) - } - if previous != "" && secret.Slot <= previous { - return errors.New("scheduled envelope secret files must be unique and sorted by slot") - } - previous = secret.Slot - } - return nil -} - -func (timing ScheduledTimingPolicy) validate() error { - if !safeLifecycleMetadata(timing.RetryIdentity) { - return errors.New("scheduled timing retry identity is invalid") - } - scheduledFor, err := time.Parse(time.RFC3339Nano, timing.ScheduledFor) - if err != nil { - return errors.New("scheduled_for must be RFC3339") - } - notBefore, err := time.Parse(time.RFC3339Nano, timing.NotBefore) - if err != nil { - return errors.New("not_before must be RFC3339") - } - expiresAt, err := time.Parse(time.RFC3339Nano, timing.ExpiresAt) - if err != nil { - return errors.New("expires_at must be RFC3339") - } - if scheduledFor.Before(notBefore) || !scheduledFor.Before(expiresAt) { - return errors.New("scheduled timing window does not contain scheduled_for") - } - maxRuntime, err := time.ParseDuration(timing.MaxRuntime) - if err != nil || maxRuntime <= 0 || maxRuntime > expiresAt.Sub(notBefore) { - return errors.New("scheduled max_runtime must fit inside the timing window") - } - return nil -} - -func (protocols ProtocolRange) Contains(protocol int) bool { - return protocol >= protocols.Minimum && protocol <= protocols.Maximum -} - -func (protocols ProtocolRange) validate(name string) error { - if protocols.Minimum <= 0 || protocols.Maximum < protocols.Minimum { - return fmt.Errorf("%s range is invalid", name) - } - return nil -} - -func (runner ScheduledRunnerCompatibility) validate() error { - if runner.RunnerProtocol <= 0 { - return errors.New("scheduled runner protocol is invalid") - } - if err := runner.CLIProtocols.validate("runner CLI protocol"); err != nil { - return err - } - return runner.EnvelopeProtocols.validate("runner envelope protocol") -} - -func (cli ScheduledCLICompatibility) validate() error { - if cli.CLIProtocol <= 0 { - return errors.New("scheduled CLI protocol is invalid") - } - if err := cli.RunnerProtocols.validate("CLI runner protocol"); err != nil { - return err - } - return cli.EnvelopeProtocols.validate("CLI envelope protocol") -} - -func (envelope ScheduledOperationEnvelope) computeDigest() (string, error) { - copy := envelope - copy.EnvelopeDigest = "" - encoded, err := json.Marshal(copy) - if err != nil { - return "", err - } - sum := sha256.Sum256(encoded) - return "sha256:" + hex.EncodeToString(sum[:]), nil -} - -func EncodeScheduledOperationEnvelope(envelope ScheduledOperationEnvelope) ([]byte, error) { - if err := envelope.Validate(); err != nil { - return nil, err - } - encoded, err := json.MarshalIndent(envelope, "", " ") - if err != nil { - return nil, err - } - return append(encoded, '\n'), nil -} - -func DecodeScheduledOperationEnvelope(encoded []byte) (ScheduledOperationEnvelope, error) { - decoder := json.NewDecoder(bytes.NewReader(encoded)) - decoder.DisallowUnknownFields() - var envelope ScheduledOperationEnvelope - if err := decoder.Decode(&envelope); err != nil { - return ScheduledOperationEnvelope{}, fmt.Errorf("decode scheduled envelope: %w", err) - } - var extra any - if err := decoder.Decode(&extra); err != io.EOF { - if err == nil { - return ScheduledOperationEnvelope{}, errors.New("decode scheduled envelope: multiple JSON values") - } - return ScheduledOperationEnvelope{}, err - } - if err := envelope.Validate(); err != nil { - return ScheduledOperationEnvelope{}, err - } - return envelope, nil -} - -func LoadScheduledOperationEnvelope(path string) (ScheduledOperationEnvelope, error) { - encoded, err := os.ReadFile(path) - if err != nil { - return ScheduledOperationEnvelope{}, err - } - return DecodeScheduledOperationEnvelope(encoded) -} - -func ReadScheduledStateDigest(path string) (string, error) { - state, err := LoadProtectionLifecycleState(path) - if err != nil { - return "", err - } - return state.StateDigest, nil -} - -func ScheduledEnvelopeContainsSecretValue(envelope ScheduledOperationEnvelope, canaries ...string) bool { - encoded, _ := json.Marshal(envelope) - for _, canary := range canaries { - if strings.Contains(string(encoded), canary) { - return true - } - } - return false -} diff --git a/internal/onebox/scheduled_envelope_test.go b/internal/onebox/scheduled_envelope_test.go deleted file mode 100644 index d36b4a0e..00000000 --- a/internal/onebox/scheduled_envelope_test.go +++ /dev/null @@ -1,221 +0,0 @@ -package onebox - -import ( - "bytes" - "context" - "encoding/json" - "errors" - "os" - "path/filepath" - "strings" - "testing" - "time" -) - -func TestRecurringScheduleMaterializesFreshOccurrenceEnvelope(t *testing.T) { - now := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC) - template, _ := scheduledEnvelopeFixture(t) - first, err := MaterializeScheduledOccurrence(template, now.Add(24*time.Hour), bytes.NewReader(bytes.Repeat([]byte{1}, 16))) - if err != nil { - t.Fatal(err) - } - second, err := MaterializeScheduledOccurrence(template, now.Add(48*time.Hour), bytes.NewReader(bytes.Repeat([]byte{2}, 16))) - if err != nil { - t.Fatal(err) - } - if first.OperationID == template.OperationID || second.OperationID == template.OperationID || first.OperationID == second.OperationID { - t.Fatalf("occurrence identities were reused: template=%q first=%q second=%q", template.OperationID, first.OperationID, second.OperationID) - } - if first.Timing.RetryIdentity == second.Timing.RetryIdentity || first.EnvelopeDigest == second.EnvelopeDigest { - t.Fatal("separate timer firings reused retry identity or sealed envelope") - } - if err := first.ValidateForRunner(CurrentScheduledRunnerCompatibility(), first.State.Digest, now.Add(24*time.Hour)); err != nil { - t.Fatalf("first occurrence was not fresh: %v", err) - } - if err := second.ValidateForRunner(CurrentScheduledRunnerCompatibility(), second.State.Digest, now.Add(48*time.Hour)); err != nil { - t.Fatalf("second occurrence was not fresh: %v", err) - } -} - -func scheduledEnvelopeFixture(t *testing.T) (ScheduledOperationEnvelope, time.Time) { - t.Helper() - now := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC) - envelope, err := NewScheduledOperationEnvelope(ScheduledEnvelopeInput{ - CLIProtocol: CurrentScheduledCLIProtocol, - RunnerProtocols: ProtocolRange{Minimum: CurrentScheduledRunnerProtocol, Maximum: CurrentScheduledRunnerProtocol}, - OperationID: "backup-20260807", Application: "example", Environment: "production", Service: "database", - Operation: KindBackupCreate, - Runner: ScheduledRunnerArtifactReference{ - Path: "/var/lib/onebox/example/protection/runners/runner-a/ob-scheduled-runner", - Digest: "sha256:" + strings.Repeat("e", 64), SBOMDigest: "sha256:" + strings.Repeat("f", 64), - ProvenanceID: "onebox-runner-v1", - }, - Timing: ScheduledTimingPolicy{ - ScheduledFor: now.Format(time.RFC3339Nano), NotBefore: now.Add(-time.Minute).Format(time.RFC3339Nano), - ExpiresAt: now.Add(15 * time.Minute).Format(time.RFC3339Nano), MaxRuntime: "10m", RetryIdentity: "backup-window-20260807", - }, - Artifacts: []OperationArtifactBinding{ - {Class: "inputs", Path: "/var/lib/onebox/example/protection/inputs.json", Mode: 0o600, Digest: "sha256:" + strings.Repeat("b", 64)}, - {Class: "backup-schedule", Path: "/var/lib/onebox/example/protection/backup.json", Mode: 0o644, Digest: "sha256:" + strings.Repeat("a", 64)}, - }, - State: ScheduledStateBinding{Path: "/var/lib/onebox/example/protection/state.json", Digest: "sha256:" + strings.Repeat("c", 64), Epoch: 7}, - SecretFiles: []SecretSlotReference{ - {Slot: "repository", File: "/var/lib/onebox/example/protection/repository.env", Entry: "RESTIC_PASSWORD"}, - }, - }) - if err != nil { - t.Fatal(err) - } - return envelope, now -} - -func TestScheduledEnvelopeIsSealedSortedAndSecretReferential(t *testing.T) { - envelope, now := scheduledEnvelopeFixture(t) - if envelope.Artifacts[0].Class != "backup-schedule" { - t.Fatalf("artifacts are not sorted: %#v", envelope.Artifacts) - } - if ScheduledEnvelopeContainsSecretValue(envelope, "secret-value-canary", "database-content-canary") { - t.Fatal("scheduled envelope contains secret or database content") - } - encoded, err := EncodeScheduledOperationEnvelope(envelope) - if err != nil { - t.Fatal(err) - } - decoded, err := DecodeScheduledOperationEnvelope(encoded) - if err != nil { - t.Fatal(err) - } - if decoded.EnvelopeDigest != envelope.EnvelopeDigest { - t.Fatal("scheduled envelope digest did not round-trip") - } - if err := decoded.ValidateForRunner(CurrentScheduledRunnerCompatibility(), envelope.State.Digest, now); err != nil { - t.Fatal(err) - } -} - -func TestScheduledEnvelopeRejectsTamperingAndUnknownFields(t *testing.T) { - envelope, _ := scheduledEnvelopeFixture(t) - tampered := envelope - tampered.Service = "other" - if err := tampered.Validate(); err == nil || !strings.Contains(err.Error(), "digest mismatch") { - t.Fatalf("tamper error = %v", err) - } - encoded, err := json.Marshal(envelope) - if err != nil { - t.Fatal(err) - } - encoded = []byte(strings.Replace(string(encoded), "{", `{"unknown":true,`, 1)) - if _, err := DecodeScheduledOperationEnvelope(encoded); err == nil || !strings.Contains(err.Error(), "unknown field") { - t.Fatalf("unknown field error = %v", err) - } -} - -func TestScheduledEnvelopeRejectsOlderCLINewerRunnerAndNewerCLIOlderRunner(t *testing.T) { - envelope, now := scheduledEnvelopeFixture(t) - newerRunner := ScheduledRunnerCompatibility{ - RunnerProtocol: 2, CLIProtocols: ProtocolRange{Minimum: 1, Maximum: 2}, EnvelopeProtocols: ProtocolRange{Minimum: 1, Maximum: 2}, - } - if err := envelope.ValidateForRunner(newerRunner, envelope.State.Digest, now); err == nil || !strings.Contains(err.Error(), "scheduled_runner_incompatible") { - t.Fatalf("older CLI/newer runner error = %v", err) - } - newerCLI := envelope - newerCLI.CLIProtocol = 2 - if err := newerCLI.Seal(); err != nil { - t.Fatal(err) - } - if err := newerCLI.ValidateForRunner(CurrentScheduledRunnerCompatibility(), newerCLI.State.Digest, now); err == nil || !strings.Contains(err.Error(), "scheduled_runner_incompatible") { - t.Fatalf("newer CLI/older runner error = %v", err) - } - cli := CurrentScheduledCLICompatibility() - if err := ValidateScheduledRunnerForCLI(envelope, newerRunner, cli); err == nil || !strings.Contains(err.Error(), "scheduled_runner_incompatible") { - t.Fatalf("CLI-side newer runner error = %v", err) - } -} - -func TestScheduledEnvelopeRejectsStaleStateAndTiming(t *testing.T) { - envelope, now := scheduledEnvelopeFixture(t) - if err := envelope.ValidateForRunner(CurrentScheduledRunnerCompatibility(), "sha256:"+strings.Repeat("d", 64), now); err == nil || !strings.Contains(err.Error(), "scheduled_envelope_stale") { - t.Fatalf("stale state error = %v", err) - } - if err := envelope.ValidateForRunner(CurrentScheduledRunnerCompatibility(), envelope.State.Digest, now.Add(time.Hour)); err == nil || !strings.Contains(err.Error(), "scheduled_envelope_stale") { - t.Fatalf("stale timing error = %v", err) - } -} - -func TestScheduledEnvelopeRejectsNonScheduledOperation(t *testing.T) { - envelope, _ := scheduledEnvelopeFixture(t) - envelope.Operation = KindDeploy - if err := envelope.Seal(); err == nil || !strings.Contains(err.Error(), "unsupported lifecycle operation") { - t.Fatalf("deploy envelope error = %v", err) - } -} - -type recordingScheduledExecutor struct { - executions []ScheduledLifecycleExecution - err error - deadline time.Time -} - -func (executor *recordingScheduledExecutor) ExecuteScheduledLifecycle(ctx context.Context, execution ScheduledLifecycleExecution) error { - executor.executions = append(executor.executions, execution) - executor.deadline, _ = ctx.Deadline() - return executor.err -} - -func TestScheduledRunnerUsesCanonicalScheduledGraphOnce(t *testing.T) { - envelope, now := scheduledEnvelopeFixture(t) - executor := &recordingScheduledExecutor{} - runner := ScheduledRunner{ - Compatibility: CurrentScheduledRunnerCompatibility(), Now: func() time.Time { return now }, - ObserveState: func(path string) (string, error) { - if path != envelope.State.Path { - return "", errors.New("wrong state path") - } - return envelope.State.Digest, nil - }, - Executor: executor, - } - if err := runner.Execute(context.Background(), envelope); err != nil { - t.Fatal(err) - } - if len(executor.executions) != 1 || len(executor.executions[0].Steps) == 0 || executor.executions[0].Steps[0].Kind != StepProtectionLock { - t.Fatalf("scheduled executions = %#v", executor.executions) - } - if want := now.Add(10 * time.Minute); !executor.deadline.Equal(want) { - t.Fatalf("scheduled deadline = %s, want %s", executor.deadline, want) - } -} - -func TestReadScheduledStateDigestValidatesWholeState(t *testing.T) { - state, err := NewProtectionLifecycleState("example", "production", "database", 1) - if err != nil { - t.Fatal(err) - } - path := filepath.Join(t.TempDir(), "state.json") - if err := SaveProtectionLifecycleState(path, state); err != nil { - t.Fatal(err) - } - if digest, err := ReadScheduledStateDigest(path); err != nil || digest != state.StateDigest { - t.Fatalf("read state digest = %q, %v", digest, err) - } - - encoded, err := os.ReadFile(path) - if err != nil { - t.Fatal(err) - } - var tampered map[string]any - if err := json.Unmarshal(encoded, &tampered); err != nil { - t.Fatal(err) - } - tampered["service"] = "other" - encoded, err = json.Marshal(tampered) - if err != nil { - t.Fatal(err) - } - if err := os.WriteFile(path, encoded, 0o600); err != nil { - t.Fatal(err) - } - if _, err := ReadScheduledStateDigest(path); err == nil || !strings.Contains(err.Error(), "digest mismatch") { - t.Fatalf("tampered state error = %v", err) - } -} diff --git a/internal/onebox/scheduled_install.go b/internal/onebox/scheduled_install.go deleted file mode 100644 index 592a6e56..00000000 --- a/internal/onebox/scheduled_install.go +++ /dev/null @@ -1,232 +0,0 @@ -package onebox - -import ( - "context" - "crypto/sha256" - "encoding/hex" - "encoding/json" - "errors" - "fmt" - "time" - - "github.com/labstack/onebox/internal/app" -) - -const ScheduledInstallPlanSchemaVersion = "onebox.run/scheduled-install-plan/v1alpha1" - -type ScheduledPublicationProof struct { - ArtifactDigest string `json:"artifact_digest"` - SBOMDigest string `json:"sbom_digest"` - ProvenanceID string `json:"provenance_id"` - Publisher string `json:"publisher"` - VerificationMethod string `json:"verification_method"` - EvidenceDigest string `json:"evidence_digest"` - VerifiedAt string `json:"verified_at"` - Verified bool `json:"verified"` -} - -type ScheduledInstallArtifact struct { - Class string `json:"class"` - Path string `json:"path"` - Mode uint32 `json:"mode"` - Owner string `json:"owner"` - Group string `json:"group"` - Digest string `json:"digest"` - Content []byte `json:"-"` -} - -type ScheduledInstallPlan struct { - SchemaVersion string `json:"schema_version"` - Application string `json:"application"` - Environment string `json:"environment"` - Service string `json:"service"` - Runner ScheduledRunnerArtifactReference `json:"runner"` - EnvelopePath string `json:"envelope_path"` - EnvelopeDigest string `json:"envelope_digest"` - PublicationProof ScheduledPublicationProof `json:"publication_proof"` - Artifacts []ScheduledInstallArtifact `json:"artifacts"` - PlanDigest string `json:"plan_digest"` -} - -type ScheduledInstalledMetadata struct { - Path string - Mode uint32 - Owner string - Group string - Digest string -} - -type ScheduledArtifactTarget interface { - InstallAtomic(context.Context, ScheduledInstallArtifact) error - Inspect(context.Context, string) (ScheduledInstalledMetadata, error) - Remove(context.Context, string) error -} - -func NewScheduledInstallPlan(names app.Names, runnerBytes []byte, envelope ScheduledOperationEnvelope, proof ScheduledPublicationProof) (ScheduledInstallPlan, error) { - if err := envelope.Validate(); err != nil { - return ScheduledInstallPlan{}, err - } - if err := proof.Validate(); err != nil { - return ScheduledInstallPlan{}, err - } - runnerDigest := digestScheduledBytes(runnerBytes) - if runnerDigest != envelope.Runner.Digest || proof.ArtifactDigest != runnerDigest || proof.SBOMDigest != envelope.Runner.SBOMDigest || proof.ProvenanceID != envelope.Runner.ProvenanceID { - return ScheduledInstallPlan{}, errors.New("scheduled runner bytes, envelope reference, and publication proof do not match") - } - wantRunnerPath := names.ProtectionRunnerPath(runnerDigest) - if envelope.Runner.Path != wantRunnerPath { - return ScheduledInstallPlan{}, errors.New("scheduled runner path is outside the digest-derived Onebox layout") - } - envelopePath := names.ProtectionEnvelopePath(envelope.Service, string(envelope.Operation)) - envelopeBytes, err := EncodeScheduledOperationEnvelope(envelope) - if err != nil { - return ScheduledInstallPlan{}, err - } - plan := ScheduledInstallPlan{ - SchemaVersion: ScheduledInstallPlanSchemaVersion, Application: envelope.Application, - Environment: envelope.Environment, Service: envelope.Service, Runner: envelope.Runner, - EnvelopePath: envelopePath, EnvelopeDigest: envelope.EnvelopeDigest, PublicationProof: proof, - Artifacts: []ScheduledInstallArtifact{ - {Class: "envelope", Path: envelopePath, Mode: 0o400, Owner: "onebox", Group: "onebox", Digest: digestScheduledBytes(envelopeBytes), Content: envelopeBytes}, - {Class: "runner", Path: wantRunnerPath, Mode: 0o500, Owner: "onebox", Group: "onebox", Digest: runnerDigest, Content: append([]byte(nil), runnerBytes...)}, - }, - } - if err := plan.Seal(); err != nil { - return ScheduledInstallPlan{}, err - } - return plan, nil -} - -func ApplyScheduledInstall(ctx context.Context, target ScheduledArtifactTarget, plan ScheduledInstallPlan) error { - if target == nil { - return errors.New("scheduled artifact target is nil") - } - if err := plan.Validate(); err != nil { - return err - } - for _, artifact := range plan.Artifacts { - if digestScheduledBytes(artifact.Content) != artifact.Digest { - return fmt.Errorf("scheduled install artifact %q content changed after planning", artifact.Class) - } - if err := target.InstallAtomic(ctx, artifact); err != nil { - return fmt.Errorf("install scheduled %s: %w", artifact.Class, err) - } - observed, err := target.Inspect(ctx, artifact.Path) - if err != nil { - return fmt.Errorf("inspect installed scheduled %s: %w", artifact.Class, err) - } - if observed.Path != artifact.Path || observed.Mode != artifact.Mode || observed.Owner != artifact.Owner || observed.Group != artifact.Group || observed.Digest != artifact.Digest { - return fmt.Errorf("installed scheduled %s ownership, mode, or digest does not match the sealed plan", artifact.Class) - } - } - return nil -} - -func ApplyAuthorizedScheduledRemoval(ctx context.Context, target ScheduledArtifactTarget, plan ProtectionRemovalPlan, authorization ProtectionRemovalAuthorization) error { - if target == nil { - return errors.New("scheduled artifact target is nil") - } - return ApplyProtectionRemoval(plan, authorization, func(resource ProtectionResource) error { - if resource.Kind != ProtectionResourceRunner && resource.Kind != ProtectionResourceEnvelope { - return fmt.Errorf("resource %q is not a scheduled executable artifact", resource.Identity) - } - return target.Remove(ctx, resource.Identity) - }) -} - -func (proof ScheduledPublicationProof) Validate() error { - if !proof.Verified || !lifecycleGraphDigest.MatchString(proof.ArtifactDigest) || !lifecycleGraphDigest.MatchString(proof.SBOMDigest) || - !lifecycleGraphDigest.MatchString(proof.EvidenceDigest) || !safeLifecycleMetadata(proof.ProvenanceID) || !safeLifecycleMetadata(proof.Publisher) { - return errors.New("scheduled runner publication proof is incomplete or unverified") - } - if proof.VerificationMethod != "sigstore-bundle" && proof.VerificationMethod != "transparency-log" { - return errors.New("scheduled runner publication verification method is unsupported") - } - if _, err := time.Parse(time.RFC3339Nano, proof.VerifiedAt); err != nil { - return errors.New("scheduled runner publication verification time is invalid") - } - return nil -} - -func (plan *ScheduledInstallPlan) Seal() error { - if plan == nil { - return errors.New("scheduled install plan is nil") - } - if err := plan.validateContent(); err != nil { - return err - } - digest, err := plan.computeDigest() - if err != nil { - return err - } - plan.PlanDigest = digest - return nil -} - -func (plan ScheduledInstallPlan) Validate() error { - if err := plan.validateContent(); err != nil { - return err - } - expected, err := plan.computeDigest() - if err != nil { - return err - } - if plan.PlanDigest != expected { - return errors.New("scheduled install plan digest mismatch") - } - return nil -} - -func (plan ScheduledInstallPlan) validateContent() error { - if plan.SchemaVersion != ScheduledInstallPlanSchemaVersion { - return fmt.Errorf("unsupported scheduled install schema %q", plan.SchemaVersion) - } - for _, value := range []string{plan.Application, plan.Environment, plan.Service} { - if !safeLifecycleMetadata(value) { - return errors.New("scheduled install ownership metadata is invalid") - } - } - if err := plan.PublicationProof.Validate(); err != nil { - return err - } - if !lifecycleGraphDigest.MatchString(plan.EnvelopeDigest) || len(plan.Artifacts) != 2 { - return errors.New("scheduled install plan has an invalid envelope binding or artifact set") - } - previous := "" - classes := map[string]bool{} - for _, artifact := range plan.Artifacts { - if artifact.Class != "envelope" && artifact.Class != "runner" { - return fmt.Errorf("unsupported scheduled install artifact class %q", artifact.Class) - } - if previous != "" && artifact.Class <= previous { - return errors.New("scheduled install artifacts must be unique and sorted") - } - if artifact.Path == "" || !lifecycleGraphDigest.MatchString(artifact.Digest) || artifact.Owner != "onebox" || artifact.Group != "onebox" { - return errors.New("scheduled install artifact binding is invalid") - } - if (artifact.Class == "runner" && artifact.Mode != 0o500) || (artifact.Class == "envelope" && artifact.Mode != 0o400) { - return errors.New("scheduled install artifact mode is not least-privilege") - } - classes[artifact.Class] = true - previous = artifact.Class - } - if !classes["runner"] || !classes["envelope"] || plan.Runner.Digest != plan.PublicationProof.ArtifactDigest { - return errors.New("scheduled install runner or envelope is missing") - } - return nil -} - -func (plan ScheduledInstallPlan) computeDigest() (string, error) { - copy := plan - copy.PlanDigest = "" - encoded, err := json.Marshal(copy) - if err != nil { - return "", err - } - return digestScheduledBytes(encoded), nil -} - -func digestScheduledBytes(content []byte) string { - sum := sha256.Sum256(content) - return "sha256:" + hex.EncodeToString(sum[:]) -} diff --git a/internal/onebox/scheduled_install_test.go b/internal/onebox/scheduled_install_test.go deleted file mode 100644 index b57ee17e..00000000 --- a/internal/onebox/scheduled_install_test.go +++ /dev/null @@ -1,174 +0,0 @@ -package onebox - -import ( - "context" - "errors" - "strings" - "testing" - "time" - - "github.com/labstack/onebox/internal/app" -) - -type fakeScheduledArtifactTarget struct { - artifacts map[string]ScheduledInstallArtifact - installs int - removes []string - inspect func(ScheduledInstallArtifact) ScheduledInstalledMetadata -} - -func (target *fakeScheduledArtifactTarget) InstallAtomic(_ context.Context, artifact ScheduledInstallArtifact) error { - if target.artifacts == nil { - target.artifacts = map[string]ScheduledInstallArtifact{} - } - copy := artifact - copy.Content = append([]byte(nil), artifact.Content...) - target.artifacts[artifact.Path] = copy - target.installs++ - return nil -} - -func (target *fakeScheduledArtifactTarget) Inspect(_ context.Context, path string) (ScheduledInstalledMetadata, error) { - artifact, ok := target.artifacts[path] - if !ok { - return ScheduledInstalledMetadata{}, errors.New("not found") - } - if target.inspect != nil { - return target.inspect(artifact), nil - } - return ScheduledInstalledMetadata{Path: path, Mode: artifact.Mode, Owner: artifact.Owner, Group: artifact.Group, Digest: digestScheduledBytes(artifact.Content)}, nil -} - -func (target *fakeScheduledArtifactTarget) Remove(_ context.Context, path string) error { - if _, ok := target.artifacts[path]; !ok { - return errors.New("not found") - } - delete(target.artifacts, path) - target.removes = append(target.removes, path) - return nil -} - -func scheduledInstallFixture(t *testing.T, runnerBytes []byte) (ScheduledInstallPlan, ScheduledOperationEnvelope, app.Names) { - t.Helper() - names := app.Names{App: "example", BasePath: "/var/lib/onebox"} - runnerDigest := digestScheduledBytes(runnerBytes) - now := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC) - runner := ScheduledRunnerArtifactReference{ - Path: names.ProtectionRunnerPath(runnerDigest), Digest: runnerDigest, - SBOMDigest: "sha256:" + strings.Repeat("a", 64), ProvenanceID: "onebox-runner-v1", - } - envelope, err := NewScheduledOperationEnvelope(ScheduledEnvelopeInput{ - CLIProtocol: CurrentScheduledCLIProtocol, - RunnerProtocols: ProtocolRange{Minimum: CurrentScheduledRunnerProtocol, Maximum: CurrentScheduledRunnerProtocol}, - OperationID: "backup-20260807", Application: "example", Environment: "production", Service: "database", - Operation: KindBackupCreate, Runner: runner, - Timing: ScheduledTimingPolicy{ - ScheduledFor: now.Format(time.RFC3339Nano), NotBefore: now.Add(-time.Minute).Format(time.RFC3339Nano), - ExpiresAt: now.Add(15 * time.Minute).Format(time.RFC3339Nano), MaxRuntime: "10m", RetryIdentity: "backup-window-20260807", - }, - Artifacts: []OperationArtifactBinding{ - {Class: "inputs", Path: "/var/lib/onebox/example/protection/inputs.json", Mode: 0o600, Digest: "sha256:" + strings.Repeat("b", 64)}, - }, - State: ScheduledStateBinding{Path: "/var/lib/onebox/example/protection/state.json", Digest: "sha256:" + strings.Repeat("c", 64), Epoch: 7}, - }) - if err != nil { - t.Fatal(err) - } - proof := ScheduledPublicationProof{ - ArtifactDigest: runnerDigest, SBOMDigest: runner.SBOMDigest, ProvenanceID: runner.ProvenanceID, - Publisher: "labstack-onebox", VerificationMethod: "sigstore-bundle", - EvidenceDigest: "sha256:" + strings.Repeat("d", 64), VerifiedAt: now.Format(time.RFC3339Nano), Verified: true, - } - plan, err := NewScheduledInstallPlan(names, runnerBytes, envelope, proof) - if err != nil { - t.Fatal(err) - } - return plan, envelope, names -} - -func TestScheduledInstallEnforcesDigestProvenanceOwnershipAndModes(t *testing.T) { - plan, _, _ := scheduledInstallFixture(t, []byte("runner-v1")) - target := &fakeScheduledArtifactTarget{} - if err := ApplyScheduledInstall(context.Background(), target, plan); err != nil { - t.Fatal(err) - } - for _, artifact := range plan.Artifacts { - installed := target.artifacts[artifact.Path] - if installed.Owner != "onebox" || installed.Group != "onebox" || installed.Mode != artifact.Mode || digestScheduledBytes(installed.Content) != artifact.Digest { - t.Fatalf("installed artifact = %#v", installed) - } - } - // Replay converges to the same two paths and exact bytes. - if err := ApplyScheduledInstall(context.Background(), target, plan); err != nil { - t.Fatal(err) - } - if len(target.artifacts) != 2 || target.installs != 4 { - t.Fatalf("replayed install paths=%d calls=%d", len(target.artifacts), target.installs) - } -} - -func TestScheduledInstallRejectsDigestProvenanceAndModeMismatch(t *testing.T) { - plan, envelope, names := scheduledInstallFixture(t, []byte("runner-v1")) - badProof := plan.PublicationProof - badProof.Verified = false - if _, err := NewScheduledInstallPlan(names, []byte("runner-v1"), envelope, badProof); err == nil { - t.Fatal("unverified runner provenance was accepted") - } - plan.Artifacts[1].Content = []byte("tampered-runner") - if err := ApplyScheduledInstall(context.Background(), &fakeScheduledArtifactTarget{}, plan); err == nil || !strings.Contains(err.Error(), "content changed") { - t.Fatalf("runner digest mismatch error = %v", err) - } - plan, _, _ = scheduledInstallFixture(t, []byte("runner-v1")) - target := &fakeScheduledArtifactTarget{inspect: func(artifact ScheduledInstallArtifact) ScheduledInstalledMetadata { - return ScheduledInstalledMetadata{Path: artifact.Path, Mode: 0o777, Owner: artifact.Owner, Group: artifact.Group, Digest: artifact.Digest} - }} - if err := ApplyScheduledInstall(context.Background(), target, plan); err == nil || !strings.Contains(err.Error(), "ownership, mode, or digest") { - t.Fatalf("mode enforcement error = %v", err) - } -} - -func TestScheduledRunnerUpgradeRetainsOldBinaryUntilAuthorizedUnreferencedRemoval(t *testing.T) { - oldPlan, _, _ := scheduledInstallFixture(t, []byte("runner-v1")) - newPlan, _, _ := scheduledInstallFixture(t, []byte("runner-v2")) - target := &fakeScheduledArtifactTarget{} - if err := ApplyScheduledInstall(context.Background(), target, oldPlan); err != nil { - t.Fatal(err) - } - if err := ApplyScheduledInstall(context.Background(), target, newPlan); err != nil { - t.Fatal(err) - } - oldRunner := oldPlan.Runner.Path - newRunner := newPlan.Runner.Path - if oldRunner == newRunner || target.artifacts[oldRunner].Path == "" || target.artifacts[newRunner].Path == "" { - t.Fatalf("upgrade runner layout old=%q new=%q artifacts=%#v", oldRunner, newRunner, target.artifacts) - } - resources := []ProtectionResource{ - {Identity: oldRunner, Kind: ProtectionResourceRunner, OwnerApplication: "example", OwnerEnvironment: "production", Service: "database"}, - {Identity: newRunner, Kind: ProtectionResourceRunner, OwnerApplication: "example", OwnerEnvironment: "production", Service: "database", Referenced: true}, - {Identity: newPlan.EnvelopePath, Kind: ProtectionResourceEnvelope, OwnerApplication: "example", OwnerEnvironment: "production", Service: "database", Referenced: true}, - } - inspection, err := InspectProtectionResources("example", "production", resources) - if err != nil { - t.Fatal(err) - } - removal, err := NewProtectionRemovalPlan(inspection, ProtectionRemovalRequest{ - Mode: KindDestroy, Application: "example", Environment: "production", ProtectionState: "disabled", - StateDigest: "sha256:" + strings.Repeat("e", 64), PrerequisitesVerifiedAbsent: true, - }) - if err != nil { - t.Fatal(err) - } - authorization := ProtectionRemovalAuthorization{Operation: removal.Mode, PlanDigest: removal.PlanDigest, StateDigest: removal.StateDigest} - if err := ApplyAuthorizedScheduledRemoval(context.Background(), target, removal, authorization); err != nil { - t.Fatal(err) - } - if _, exists := target.artifacts[oldRunner]; exists { - t.Fatal("authorized destroy retained the unreferenced old runner") - } - if _, exists := target.artifacts[newRunner]; !exists { - t.Fatal("authorized destroy removed the referenced current runner") - } - if _, exists := target.artifacts[newPlan.EnvelopePath]; !exists { - t.Fatal("authorized destroy removed the referenced envelope") - } -} diff --git a/internal/onebox/scheduled_runner.go b/internal/onebox/scheduled_runner.go deleted file mode 100644 index 83a3fc90..00000000 --- a/internal/onebox/scheduled_runner.go +++ /dev/null @@ -1,78 +0,0 @@ -package onebox - -import ( - "context" - "errors" - "io" - "time" -) - -type ScheduledLifecycleExecution struct { - Envelope ScheduledOperationEnvelope - Steps []OperationStep -} - -type ScheduledLifecycleExecutor interface { - ExecuteScheduledLifecycle(context.Context, ScheduledLifecycleExecution) error -} - -type ScheduledRunner struct { - Compatibility ScheduledRunnerCompatibility - Now func() time.Time - ObserveState func(string) (string, error) - Executor ScheduledLifecycleExecutor -} - -// ExecuteRecurring materializes one occurrence from an installed schedule -// template, then executes exactly that occurrence. Callers must invoke this -// once per timer firing; retries inside the firing retain the materialized -// identity, while later timer activations cannot be deduplicated against it. -func (runner ScheduledRunner) ExecuteRecurring(ctx context.Context, template ScheduledOperationEnvelope, entropy io.Reader) error { - if runner.Now == nil { - runner.Now = time.Now - } - now := runner.Now().UTC() - occurrence, err := MaterializeScheduledOccurrence(template, now, entropy) - if err != nil { - return err - } - runner.Now = func() time.Time { return now } - return runner.Execute(ctx, occurrence) -} - -func (runner ScheduledRunner) Execute(ctx context.Context, envelope ScheduledOperationEnvelope) error { - compatibility := runner.Compatibility - if compatibility.RunnerProtocol == 0 { - compatibility = CurrentScheduledRunnerCompatibility() - } - if runner.Now == nil { - runner.Now = time.Now - } - if runner.ObserveState == nil { - runner.ObserveState = ReadScheduledStateDigest - } - if runner.Executor == nil { - return errors.New("scheduled lifecycle executor is unavailable") - } - observedState, err := runner.ObserveState(envelope.State.Path) - if err != nil { - return err - } - now := runner.Now().UTC() - if err := envelope.ValidateForRunner(compatibility, observedState, now); err != nil { - return err - } - steps, err := LifecycleOperationGraph(envelope.Operation, LifecycleScheduledRunnerSchema, envelope.Service) - if err != nil { - return err - } - maxRuntime, _ := time.ParseDuration(envelope.Timing.MaxRuntime) - expiresAt, _ := time.Parse(time.RFC3339Nano, envelope.Timing.ExpiresAt) - deadline := now.Add(maxRuntime) - if expiresAt.Before(deadline) { - deadline = expiresAt - } - executionContext, cancel := context.WithDeadline(ctx, deadline) - defer cancel() - return runner.Executor.ExecuteScheduledLifecycle(executionContext, ScheduledLifecycleExecution{Envelope: envelope, Steps: steps}) -} diff --git a/internal/onebox/service.go b/internal/onebox/service.go index d2600011..3c049d90 100644 --- a/internal/onebox/service.go +++ b/internal/onebox/service.go @@ -4,7 +4,6 @@ import ( "context" "crypto/rand" "encoding/hex" - "errors" "fmt" "io" "os/exec" @@ -36,8 +35,6 @@ type Options struct { EngineOptions engine.Options Runner buildinfo.Runner // ScheduledLifecycleExecutor is the bounded driver backend reached only - // after ScheduledRunner validates its sealed envelope and canonical graph. - ScheduledLifecycleExecutor ScheduledLifecycleExecutor // Images resolves build-sourced workloads to the reference whatever built // them produced. Production never builds, so a workload declaring `build:` // has no image until one is supplied here — and without it the project @@ -46,17 +43,16 @@ type Options struct { } type Service struct { - configPath string - environment string - images app.Images - now func() time.Time - connect Connector - entropy io.Reader - entropyMu sync.Mutex - engineOpts engine.Options - runner buildinfo.Runner - scheduledLifecycleExecutor ScheduledLifecycleExecutor - operationSeq uint64 + configPath string + environment string + images app.Images + now func() time.Time + connect Connector + entropy io.Reader + entropyMu sync.Mutex + engineOpts engine.Options + runner buildinfo.Runner + operationSeq uint64 } func (s *Service) newOperationID(now time.Time, gitSHA string, kind OperationKind) string { @@ -100,21 +96,9 @@ func New(opts Options) *Service { images: opts.Images, now: opts.Now, connect: opts.Connect, entropy: opts.Entropy, engineOpts: opts.EngineOptions, runner: opts.Runner, - scheduledLifecycleExecutor: opts.ScheduledLifecycleExecutor, } } -// ExecuteScheduledLifecycle keeps scheduled dispatch on the same canonical -// service boundary as interactive CLI operations. Driver backends are added -// behind this seam; a build without one fails closed rather than reporting a -// scheduled operation as successful. -func (s *Service) ExecuteScheduledLifecycle(ctx context.Context, execution ScheduledLifecycleExecution) error { - if s == nil || s.scheduledLifecycleExecutor == nil { - return errors.New("scheduled lifecycle backend is unavailable") - } - return s.scheduledLifecycleExecutor.ExecuteScheduledLifecycle(ctx, execution) -} - func (s *Service) readEntropy(buf []byte) error { s.entropyMu.Lock() defer s.entropyMu.Unlock() diff --git a/internal/onebox/service_test.go b/internal/onebox/service_test.go index 4a332878..1356ba50 100644 --- a/internal/onebox/service_test.go +++ b/internal/onebox/service_test.go @@ -55,13 +55,11 @@ runtime: env_files: [app.env] hooks: post_deploy: "echo ` + testSecret + `" -verifications: - - { url: "https://example.invalid/private/` + testSecret + `?token=` + testSecret + `", advisory: true } - - { workload: web, http: "/private/` + testSecret + `" } -observability: - logs: { enabled: true, retention: 14d } - metrics: { enabled: true } - alerts: { unhealthy_after: 5m } +checks: + url: + - { url: "https://example.invalid/private/` + testSecret + `?token=` + testSecret + `", advisory: true } + http: + - { workload: web, path: "/private/` + testSecret + `" } `, } for name, body := range files { diff --git a/internal/onebox/staging_secrets_test.go b/internal/onebox/staging_secrets_test.go index 17ad4802..e39a76ec 100644 --- a/internal/onebox/staging_secrets_test.go +++ b/internal/onebox/staging_secrets_test.go @@ -268,7 +268,7 @@ external_services: connection: source: {file: secrets/database.env, provider: sops} entries: {url: DATABASE_URL} - protection_owner: platform-team/rds + backup_owner: platform-team/rds probe: {} ` configPath := filepath.Join(dir, "ob.yml") diff --git a/internal/onebox/state_probe_test.go b/internal/onebox/state_probe_test.go index 73869ea1..8095a4a1 100644 --- a/internal/onebox/state_probe_test.go +++ b/internal/onebox/state_probe_test.go @@ -44,29 +44,7 @@ func stateFixtures(t *testing.T) (regular, dangling, absent string) { // A dangling link must not read as never-seeded: seeding would then proceed // against state that exists. -func TestActiveVolumeStateProbeClassifies(t *testing.T) { - regular, dangling, absent := stateFixtures(t) - for _, tc := range []struct { - name string - path string - exit int - }{ - {"record is read", regular, 0}, - {"dangling link is not never-seeded", dangling, 2}, - {"absent is never-seeded", absent, 3}, - } { - t.Run(tc.name, func(t *testing.T) { - if exit, _ := runStateProbe(t, activeVolumeStateProbe(tc.path)); exit != tc.exit { - t.Fatalf("exit = %d, want %d", exit, tc.exit) - } - }) - } -} - -// An unsearchable ancestor hides both state files the same way a dangling link -// does. Seeding on that answer writes over state that may already be there, and -// observing on it drops a service's protection silently. -func TestStateProbesRefuseUnsearchableAncestor(t *testing.T) { +func TestLifecycleStateProbeRefusesUnsearchableAncestor(t *testing.T) { if os.Geteuid() == 0 { t.Skip("root searches every directory, so the permission arm cannot be exercised") } @@ -83,23 +61,13 @@ func TestStateProbesRefuseUnsearchableAncestor(t *testing.T) { } t.Cleanup(func() { _ = os.Chmod(locked, 0o700) }) - for _, tc := range []struct { - name string - script string - }{ - {"active volume", activeVolumeStateProbe(record)}, - {"lifecycle", lifecycleStateProbe(record)}, - } { - t.Run(tc.name, func(t *testing.T) { - if exit, _ := runStateProbe(t, tc.script); exit != app.ProbeUndetermined { - t.Fatalf("exit = %d, want %d", exit, app.ProbeUndetermined) - } - }) + if exit, _ := runStateProbe(t, lifecycleStateProbe(record)); exit != app.ProbeUndetermined { + t.Fatalf("exit = %d, want %d", exit, app.ProbeUndetermined) } } // A dangling link must not read as 'missing': reporting no state at all drops -// the service's protection silently. +// the service's backup silently. func TestLifecycleStateProbeClassifies(t *testing.T) { regular, dangling, absent := stateFixtures(t) for _, tc := range []struct { diff --git a/internal/onebox/status.go b/internal/onebox/status.go deleted file mode 100644 index 0a99cc93..00000000 --- a/internal/onebox/status.go +++ /dev/null @@ -1,115 +0,0 @@ -package onebox - -// Staged work, kept deliberately. Nothing outside this file calls into it — -// statusIssueCodes and statusDigest are reached only from tests, and -// canonicalStatus only from statusDigest. That is a state a dead-code pass -// reads as "delete me", and it would be wrong twice over: -// -// - `statusIssueCodes` is one half of a contract. The other half is the issue -// prose in internal/engine/proxystatus.go, whose comment says every sentence -// leads with its component *because* this function matches on it. Delete -// this and the reason that prose is stable disappears with it. -// - The rest is status output written against a structured status shape the -// engine does not yet emit, for the same proposal that -// DeploymentProposal.Preconditions belongs to. -// -// Whether that work is still coming is a product question, tracked with the -// rest of the part-built surface. Until it is answered, this file stays, and -// this comment is here so the next person to run `unused` does not have to -// rediscover why. -// -// See #63. - -import ( - "encoding/json" - "strings" - "time" - - "github.com/labstack/onebox/internal/engine" -) - -func statusIssueCodes(component string, issues []string) []string { - out := make([]string, 0, len(issues)) - seen := map[string]bool{} - for _, issue := range issues { - code := component + "_diverged" - switch { - case strings.HasPrefix(issue, "replica count is "): - code = "replica_count_mismatch" - case strings.Contains(issue, "no release is recorded"): - code = "release_not_recorded" - case strings.Contains(issue, "not onebox-deployed"): - code = "unmanaged_container" - case strings.Contains(issue, "runs release "): - code = "release_mismatch" - case strings.Contains(issue, "health is "), strings.Contains(issue, " is unhealthy"), strings.Contains(issue, " is starting"), strings.Contains(issue, " is down"): - code = "container_health_unready" - case issue == "not running": - code = "not_running" - case issue == "local and applied configuration hashes differ": - code = "configuration_drift" - case strings.HasPrefix(issue, "certificate renewal is overdue"): - code = "certificate_renewal_overdue" - case issue == "certificate store is unreadable", - strings.HasPrefix(issue, "the certificate store"): - code = "certificate_store_unreadable" - // A refused applied-config read also gates ConfigDiverged to false and - // omits the hash, so without its own code a JSON consumer sees no - // drift, no unreadable marker, and the same generic code an unhealthy - // container gets. - case strings.HasPrefix(issue, "the applied configuration"): - code = "applied_config_unreadable" - // A refused owner read publishes no owner, which is byte-identical to - // a genuinely unclaimed host once omitempty drops the empty field. A - // caller keying on the absent field would read "unclaimed" and propose - // a bootstrap the engine then refuses, so the refusal needs a code of - // its own rather than only prose plus complete:false. - // Read successfully, and empty. A permissions remedy would send the - // operator after a file that reads perfectly well. - case strings.HasPrefix(issue, "host owner record is present but empty"): - code = "host_owner_empty" - // Read fine, but not a name any mutation will accept. Distinct from - // unreadable: no permission change helps, the record's content is - // what is wrong. - case strings.HasPrefix(issue, "the host owner record is not a valid application name"): - code = "host_owner_invalid" - case strings.HasPrefix(issue, "host owner record is not a regular file"), - strings.HasPrefix(issue, "the host owner record"), - strings.HasPrefix(issue, "the path that should hold the host owner record"), - strings.HasPrefix(issue, "the host state directory cannot be searched"): - code = "host_owner_unreadable" - } - if !seen[code] { - seen[code] = true - out = append(out, code) - } - } - return out -} - -func canonicalStatus(status engine.StatusSnapshot) engine.StatusSnapshot { - status.CapturedAt = time.Time{} - status.Warnings = append([]engine.StatusWarning(nil), status.Warnings...) - for i := range status.Warnings { - status.Warnings[i].Message = "" - } - if status.Proxy != nil { - proxyCopy := *status.Proxy - proxyCopy.Certificates = append([]engine.StatusCertificate(nil), status.Proxy.Certificates...) - for i := range proxyCopy.Certificates { - // DaysRemaining is a presentation countdown. NotAfter and the overdue - // threshold state carry the operational fact without daily digest noise. - proxyCopy.Certificates[i].DaysRemaining = 0 - } - status.Proxy = &proxyCopy - } - return status -} - -func statusDigest(status engine.StatusSnapshot) (string, error) { - encoded, err := json.Marshal(canonicalStatus(status)) - if err != nil { - return "", err - } - return engine.HashBytes(encoded), nil -} diff --git a/internal/onebox/status_issue_codes_test.go b/internal/onebox/status_issue_codes_test.go deleted file mode 100644 index b14ccc67..00000000 --- a/internal/onebox/status_issue_codes_test.go +++ /dev/null @@ -1,79 +0,0 @@ -package onebox - -import ( - "slices" - "testing" -) - -// statusIssueCodes derives a branchable code by matching the issue prose, so -// the two are coupled by string. A reworded issue silently falls back to -// "_diverged", and for the owner record that is worse than -// unhelpful: a refused read publishes no owner, which after omitempty looks -// exactly like a genuinely unclaimed host. These are the exact sentences -// proxyReads emits — keep them in step. -func TestOwnerRefusalIssuesCarryTheirOwnCode(t *testing.T) { - for _, issue := range []string{ - "host owner record is not a regular file; only a regular file is a valid owner record", - "the host owner record exists but could not be read; verify the record's permissions", - "the path that should hold the host owner record is not a directory", - "the host state directory cannot be searched, so the owner record could not be read", - } { - codes := statusIssueCodes("proxy", []string{issue}) - if !slices.Contains(codes, "host_owner_unreadable") { - t.Errorf("issue %q derived %v; a refused owner read must not be indistinguishable from an unclaimed host", issue, codes) - } - } -} - -// The same argument as the owner record: a refused applied-config read gates -// ConfigDiverged to false and omits the hash, so without a code of its own a -// consumer sees no drift and no marker — indistinguishable from a host in -// sync. These are the exact sentences statusFileIssue emits. -func TestRefusedFileReadsCarryTheirOwnCodes(t *testing.T) { - for issue, want := range map[string]string{ - "the applied configuration exists but could not be read; verify the file and its permissions": "applied_config_unreadable", - "the applied configuration could not be read: the path that should hold /var/lib/ob/_host/proxy/config.hash is not a directory": "applied_config_unreadable", - "the applied configuration could not be read: the directory holding /var/lib/ob/_host/proxy/config.hash cannot be searched, so a missing file cannot be told from an unreadable one": "applied_config_unreadable", - "the certificate store exists but could not be read; verify the file and its permissions": "certificate_store_unreadable", - "the certificate store could not be read: the path that should hold /var/lib/ob/_host/proxy/acme/acme.json is not a directory": "certificate_store_unreadable", - "certificate store is unreadable": "certificate_store_unreadable", - } { - if codes := statusIssueCodes("proxy", []string{issue}); !slices.Contains(codes, want) { - t.Errorf("issue %q derived %v, want %s", issue, codes, want) - } - } -} - -// An empty record was read successfully; it is not unreadable. Sending a -// consumer down a permissions remedy for a file that reads perfectly well is a -// different wrong answer from the one the unreadable code exists to give. -func TestAnEmptyOwnerRecordIsNotReportedAsUnreadable(t *testing.T) { - codes := statusIssueCodes("proxy", []string{"host owner record is present but empty; an empty record is not a valid claim"}) - if slices.Contains(codes, "host_owner_unreadable") { - t.Errorf("an empty record derived %v, which points at permissions", codes) - } - if !slices.Contains(codes, "host_owner_empty") { - t.Errorf("an empty record derived %v, want host_owner_empty", codes) - } -} - -// A record that reads fine but names nothing valid is a third condition: not -// unreadable (no permission change helps) and not empty (there is content). -func TestAnInvalidOwnerNameIsNotReportedAsUnreadable(t *testing.T) { - codes := statusIssueCodes("proxy", []string{"the host owner record is not a valid application name; every mutation will refuse this host"}) - if slices.Contains(codes, "host_owner_unreadable") { - t.Errorf("an invalid owner name derived %v, which points at permissions", codes) - } - if !slices.Contains(codes, "host_owner_invalid") { - t.Errorf("an invalid owner name derived %v, want host_owner_invalid", codes) - } -} - -// The generic fallback must still apply to anything else, or the code above -// would swallow unrelated proxy issues. -func TestUnrelatedProxyIssuesKeepTheGenericCode(t *testing.T) { - codes := statusIssueCodes("proxy", []string{"something else entirely"}) - if !slices.Contains(codes, "proxy_diverged") { - t.Errorf("unrelated issue derived %v, want proxy_diverged", codes) - } -} diff --git a/internal/onebox/status_test.go b/internal/onebox/status_test.go deleted file mode 100644 index 1aec6e1b..00000000 --- a/internal/onebox/status_test.go +++ /dev/null @@ -1,52 +0,0 @@ -package onebox - -import ( - "testing" - "time" - - "github.com/labstack/onebox/internal/engine" -) - -func TestStatusDigestIgnoresCaptureTimeAndCertificateCountdown(t *testing.T) { - expires := time.Date(2026, 8, 1, 0, 0, 0, 0, time.UTC) - first := engine.StatusSnapshot{ - App: "demo", - Host: "example.invalid", - CapturedAt: time.Date(2026, 7, 10, 0, 0, 0, 0, time.UTC), - Complete: true, - Proxy: &engine.StatusProxy{ - Managed: true, - Complete: true, - Certificates: []engine.StatusCertificate{{ - Domain: "example.invalid", NotAfter: expires, DaysRemaining: 22, RenewalOverdue: false, - }}, - }, - } - second := first - second.CapturedAt = first.CapturedAt.Add(48 * time.Hour) - proxyCopy := *first.Proxy - proxyCopy.Certificates = append([]engine.StatusCertificate(nil), first.Proxy.Certificates...) - proxyCopy.Certificates[0].DaysRemaining = 20 - second.Proxy = &proxyCopy - - firstDigest, err := statusDigest(first) - if err != nil { - t.Fatal(err) - } - secondDigest, err := statusDigest(second) - if err != nil { - t.Fatal(err) - } - if firstDigest != secondDigest { - t.Fatalf("presentation time changed status identity: %q != %q", firstDigest, secondDigest) - } - - proxyCopy.Certificates[0].RenewalOverdue = true - operationalDigest, err := statusDigest(second) - if err != nil { - t.Fatal(err) - } - if operationalDigest == firstDigest { - t.Fatal("renewal threshold crossing must change status identity") - } -} diff --git a/internal/onebox/types.go b/internal/onebox/types.go deleted file mode 100644 index 5b955f20..00000000 --- a/internal/onebox/types.go +++ /dev/null @@ -1,148 +0,0 @@ -// Package onebox exposes the typed product service shared by agent-facing -// adapters. It deliberately contains no protocol or presentation code. -package onebox - -import "github.com/labstack/onebox/internal/engine" - -const SchemaVersion = "onebox.run/observation/v1alpha1" - -// ObserveRequest is intentionally empty. A Service is bound to exactly one -// launch-time environment; tool input cannot widen that authority boundary. -type ObserveRequest struct{} - -// Provenance identifies one source used to build an observation. Observations -// are timestamped, permission-scoped snapshots rather than timeless truth. -type Provenance struct { - Kind string `json:"kind" jsonschema:"Source kind, such as config, compose, or host"` - Source string `json:"source" jsonschema:"Redaction-safe source identity"` -} - -// ServiceDescription is the declared, non-secret shape of one Compose service. -type ServiceDescription struct { - Name string `json:"name" jsonschema:"Stable logical component name"` - Service string `json:"service" jsonschema:"Compose service implementing the component"` - Type string `json:"type" jsonschema:"Component type such as application, worker, job, postgres, redis, or service"` - Strategy string `json:"strategy,omitempty" jsonschema:"Deployment strategy for application and worker components"` - Replicas int `json:"replicas,omitempty" jsonschema:"Resolved steady-state replica count"` - DataEffect string `json:"data_effect,omitempty" jsonschema:"Declared job data effect"` - PersistenceMode string `json:"persistence_mode,omitempty" jsonschema:"Declared durable, ephemeral, or external persistence mode"` - ImageDeclared bool `json:"image_declared" jsonschema:"Whether the Compose service declares an image reference; the scalar value is hidden"` -} - -type EnvironmentPolicyDescription struct { - RequireApproval bool `json:"require_approval" jsonschema:"Declared policy that production mutation requires human approval"` - AllowAgentProposals bool `json:"allow_agent_proposals" jsonschema:"Whether an agent may construct deployment proposals"` -} - -type ObservabilityDescription struct { - LogsDeclared bool `json:"logs_declared"` - MetricsDeclared bool `json:"metrics_declared"` - AlertsDeclared bool `json:"alerts_declared"` - Managed bool `json:"managed" jsonschema:"Whether Onebox currently runs the declared observability capabilities"` -} - -// Observation is the structured read model returned to agents. It omits -// Compose environment blocks, secret payloads, and raw application output. -type Observation struct { - SchemaVersion string `json:"schema_version" jsonschema:"Version of this structured observation"` - Application string `json:"application" jsonschema:"Resolved Onebox application name"` - Environment string `json:"environment" jsonschema:"Observed Onebox environment"` - Policy EnvironmentPolicyDescription `json:"policy" jsonschema:"Resolved environment policy"` - Observability ObservabilityDescription `json:"observability" jsonschema:"Declared versus currently managed observability"` - Server string `json:"server" jsonschema:"Configured SSH server identity"` - CapturedAt string `json:"captured_at" jsonschema:"RFC3339 timestamp at which observation began"` - ConfigHash string `json:"config_hash" jsonschema:"SHA-256 identity of the Onebox configuration bytes"` - ComposeHash string `json:"compose_hash" jsonschema:"SHA-256 identity of the root Compose file bytes"` - StateDigest string `json:"state_digest" jsonschema:"Digest of the state suitable for plan preconditions"` - Complete bool `json:"complete" jsonschema:"Whether every supported observation component completed"` - Provenance []Provenance `json:"provenance" jsonschema:"Sources used to construct this observation"` - Services []ServiceDescription `json:"services" jsonschema:"Declared services in deterministic name order"` - Status engine.StatusSnapshot `json:"status" jsonschema:"Recorded-versus-actual production status"` - Warnings []engine.StatusWarning `json:"warnings,omitempty" jsonschema:"Redaction-safe partial-observation warnings by component"` -} - -// ProposeDeployRequest is intentionally empty. The proposal uses the Service's -// launch-time environment and never performs a production mutation. -type ProposeDeployRequest struct{} - -// ProposeRequest is the adapter-neutral proposal request. The current kind is -// deploy; approved execution can be added without changing this service -// boundary. -type ProposeRequest struct { - Kind OperationKind `json:"kind" jsonschema:"Operation kind to propose; currently deploy"` -} - -type ProposalHostState struct { - Host string `json:"host" jsonschema:"Bare target hostname"` - CurrentRelease string `json:"current_release,omitempty" jsonschema:"Currently activated release, if any"` - ImageIDs map[string]string `json:"image_ids,omitempty" jsonschema:"Observed running image identities by service"` -} - -type ImagePin struct { - Service string `json:"service" jsonschema:"Compose service name"` - Digest string `json:"digest,omitempty" jsonschema:"Resolved immutable OCI digest; mutable source reference is hidden"` - Pinned bool `json:"pinned" jsonschema:"Whether the image is bound to an immutable digest"` -} - -type RiskSummary struct { - ExpectedInterruption string `json:"expected_interruption" jsonschema:"Plain-language interruption expectation"` - ApplicationRollback string `json:"application_rollback" jsonschema:"Available application rollback path"` - DataEffects string `json:"data_effects" jsonschema:"Declared job or migration consequences"` -} - -type ComparisonStatus string - -const ( - ComparisonIdentical ComparisonStatus = "identical" - ComparisonDifferent ComparisonStatus = "different" - ComparisonUnavailable ComparisonStatus = "unavailable" - ComparisonFirstDeploy ComparisonStatus = "first_deploy" - ComparisonNotEvaluated ComparisonStatus = "not_evaluated" -) - -type ProposalPreconditions struct { - Ready bool `json:"ready" jsonschema:"Whether the observed target has no known deployment blockers"` - StatusComplete bool `json:"status_complete" jsonschema:"Whether all supported status sources were observed"` - StatusDigest string `json:"status_digest" jsonschema:"Timestamp-independent digest of the observed operational status"` - Blockers []string `json:"blockers" jsonschema:"Redaction-safe reasons this proposal is not ready to execute"` -} - -// DeploymentProposal is a redacted, state-bound preview. Commands originating -// in operator-authored hooks are deliberately hidden from model output because -// arbitrary hook text may contain sensitive literals. -type DeploymentProposal struct { - SchemaVersion string `json:"schema_version" jsonschema:"Version of this proposal schema"` - ID string `json:"id" jsonschema:"Unique proposal identifier"` - ReleaseID string `json:"release_id" jsonschema:"Release identifier the proposal would stage"` - Application string `json:"application" jsonschema:"Resolved Onebox application name"` - Environment string `json:"environment" jsonschema:"Target Onebox environment"` - Policy EnvironmentPolicyDescription `json:"policy" jsonschema:"Resolved target environment policy"` - Server string `json:"server" jsonschema:"Configured SSH server identity"` - CreatedAt string `json:"created_at" jsonschema:"RFC3339 proposal creation timestamp"` - ExpiresAt string `json:"expires_at" jsonschema:"RFC3339 time after which this proposal should be refreshed"` - GitSHA string `json:"git_sha,omitempty" jsonschema:"Local repository commit identity when available"` - ConfigHash string `json:"config_hash" jsonschema:"SHA-256 identity of configuration bytes"` - ComposeHash string `json:"compose_hash" jsonschema:"SHA-256 identity of the root Compose file bytes"` - StateDigest string `json:"state_digest" jsonschema:"Stable precondition digest binding environment, configuration, and observed host state"` - ProposalDigest string `json:"proposal_digest" jsonschema:"Content digest binding every known execution input and observation in this proposal"` - RenderedComposeCommitment string `json:"rendered_compose_commitment" jsonschema:"Proposal-keyed HMAC commitment to the full, unmasked rendered Compose bytes"` - PayloadCommitment string `json:"payload_commitment" jsonschema:"Proposal-keyed HMAC commitment to the planned non-Compose staged payload"` - LivePayloadCommitment string `json:"live_payload_commitment,omitempty" jsonschema:"Proposal-keyed HMAC commitment to the observed current-release payload"` - SecretSourceCommitment string `json:"secret_source_commitment,omitempty" jsonschema:"Proposal-keyed HMAC commitment to the encrypted SOPS source when configured"` - PayloadMaterialized bool `json:"payload_materialized" jsonschema:"Whether every runtime payload value was materialized while proposing"` - HostState ProposalHostState `json:"host_state" jsonschema:"Relevant target state observed while planning"` - Preconditions ProposalPreconditions `json:"preconditions" jsonschema:"Observed readiness and blockers"` - OperationGraph []OperationStep `json:"operation_graph" jsonschema:"Canonical typed deployment choreography; hook bodies are never included"` - Images []ImagePin `json:"images" jsonschema:"Planned images in deterministic service order"` - RenderedCompose string `json:"rendered_compose" jsonschema:"Non-executable Compose structure with every scalar value replaced by a proposal-local opaque marker"` - Diff string `json:"diff,omitempty" jsonschema:"Unified structural diff whose scalar values are proposal-local opaque markers"` - ComposeComparison ComparisonStatus `json:"compose_comparison" jsonschema:"identical, different, unavailable, or first_deploy"` - PayloadComparison ComparisonStatus `json:"payload_comparison" jsonschema:"identical, different, unavailable, or not_evaluated"` - NoOp bool `json:"no_op" jsonschema:"Whether Compose and staged payload are byte-identical to live"` - CommandSummary []string `json:"command_summary" jsonschema:"Redaction-safe execution-shape summary; operator hook bodies are hidden"` - HookBodiesRedacted bool `json:"hook_bodies_redacted" jsonschema:"Whether operator-authored command bodies were hidden"` - FidelityContract string `json:"fidelity_contract" jsonschema:"What the current planner can and cannot promise"` - Risk RiskSummary `json:"risk" jsonschema:"Current risk and recovery summary"` - Verifications []string `json:"verifications" jsonschema:"Redaction-safe verification summary"` - Warnings []string `json:"warnings,omitempty" jsonschema:"Planning limitations or unpinned image warnings"` -} diff --git a/internal/transport/transport.go b/internal/transport/transport.go index eb8e4871..2a734675 100644 --- a/internal/transport/transport.go +++ b/internal/transport/transport.go @@ -171,10 +171,10 @@ const uploadSentinel = ".ob-upload-complete" // // Staging is removed when the script exits or is signalled. It has to be // removed here rather than by the caller, because this is the only place that -// knows the staging path: the secrets, protection-credential and proxy uploads +// knows the staging path: the secrets, backup-credential and proxy uploads // all clean up the *destination* they asked for, so anything left beside it // survives them. -// Their payloads are plaintext — an app's .env, a protection credentials.env — +// Their payloads are plaintext — an app's .env, a backup credentials.env — // and the leaf name carries an epoch or a fence token that changes every run, // so a leak is never overwritten by the next attempt. func uploadScript(remoteDir string, transfer func(quotedStaging string) string) (string, error) { diff --git a/scripts/validate_release_tag_test.go b/scripts/validate_release_tag_test.go index 925d5485..d67d5366 100644 --- a/scripts/validate_release_tag_test.go +++ b/scripts/validate_release_tag_test.go @@ -74,7 +74,7 @@ func runTagValidator(t *testing.T, dir, tag, mainRef string) (string, error) { } // The release grammar is written twice: once in Bash, where it gates the tag a -// human pushes, and once in Go, where it gates `minimum_onebox_version` and the +// human pushes, and once in Go, where it gates `min_version` and the // runner's own provenance. Drift is silent in both directions — a looser shell // publishes a tag the binary cannot parse, a looser loader accepts a minimum no // tag can satisfy — so one corpus decides both. diff --git a/site/astro.config.mjs b/site/astro.config.mjs index 5f923cb5..64d45655 100644 --- a/site/astro.config.mjs +++ b/site/astro.config.mjs @@ -73,6 +73,7 @@ export default defineConfig({ label: "Guides", items: [ { label: "Add a database", slug: "guides/add-a-database" }, + { label: "Back up a database", slug: "guides/back-up-a-database" }, { label: "Handle secrets", slug: "guides/handle-secrets" }, { label: "Run migrations safely", slug: "guides/run-migrations" }, { label: "Schedule a job", slug: "guides/schedule-a-job" }, diff --git a/site/public/onebox.run-v1.schema.json b/site/public/onebox.run-v1.schema.json index 23c6644d..e4887f01 100644 --- a/site/public/onebox.run-v1.schema.json +++ b/site/public/onebox.run-v1.schema.json @@ -191,29 +191,26 @@ }, "properties": { "cold": { - "description": "Encryption mode required for cold recovery: client-side, archive-password, or server-side-sse.", + "description": "Encryption mode required for cold recovery: client-side or server-side.", "enum": [ "client-side", - "archive-password", - "server-side-sse" + "server-side" ], "type": "string" }, "pitr": { - "description": "Encryption mode required for point-in-time recovery: client-side, archive-password, or server-side-sse.", + "description": "Encryption mode required for point-in-time recovery: client-side or server-side.", "enum": [ "client-side", - "archive-password", - "server-side-sse" + "server-side" ], "type": "string" }, "snapshot": { - "description": "Encryption mode required for snapshot recovery: client-side, archive-password, or server-side-sse.", + "description": "Encryption mode required for snapshot recovery: client-side or server-side.", "enum": [ "client-side", - "archive-password", - "server-side-sse" + "server-side" ], "type": "string" } @@ -265,7 +262,7 @@ "type": "string" }, "prefix": { - "description": "Non-secret object prefix reserved for Onebox protection data. Expects a relative object prefix with no empty leading component or shell metacharacter.", + "description": "Non-secret object prefix reserved for Onebox backup data. Expects a relative object prefix with no empty leading component or shell metacharacter.", "examples": [ "production/shop" ], @@ -281,18 +278,18 @@ "type": "string" }, "tls": { - "default": "required", - "description": "TLS verification policy: required or insecure.", + "default": "verify", + "description": "Transport policy: verify, or skip-verify to accept a plaintext http endpoint.", "enum": [ - "required", - "insecure" + "verify", + "skip-verify" ], "type": "string" } }, "type": "object" }, - "description": "User-owned off-host repositories available to service protection policies.", + "description": "User-owned off-host repositories available to service backup policies.", "type": "object" }, "base_path": { @@ -337,13 +334,6 @@ "pattern": "^[^/\\x00-\\x1f'\"$`\\\\][^\\x00-\\x1f'\"$`\\\\]*$", "type": "string" }, - "platform": { - "description": "Target image platform for the external build.", - "examples": [ - "linux/amd64" - ], - "type": "string" - }, "target": { "description": "Named Dockerfile stage to build.", "type": "string" @@ -354,6 +344,198 @@ ], "description": "Build metadata for development. Production requires a resolved image supplied with --image. Also accepts a build context path." }, + "checks": { + "additionalProperties": false, + "description": "Assertions that must pass before a release becomes current unless marked advisory.", + "patternProperties": { + "^x-": {} + }, + "properties": { + "exec": { + "description": "Commands run inside a named workload.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "advisory": { + "default": false, + "description": "Report a failure without blocking release activation.", + "type": "boolean" + }, + "run": { + "description": "Shell command verified inside the workload.", + "examples": [ + "test -f /srv/ready" + ], + "type": "string" + }, + "workload": { + "description": "Workload the command runs inside.", + "examples": [ + "web" + ], + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "http": { + "description": "HTTP paths probed inside a named workload.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "advisory": { + "default": false, + "description": "Report a failure without blocking release activation.", + "type": "boolean" + }, + "path": { + "description": "HTTP path verified inside the workload. Expects a path beginning with /.", + "examples": [ + "/healthz" + ], + "pattern": "^/[^\\x00-\\x1f'\"$` \\\\]*$", + "type": "string" + }, + "port": { + "description": "Container port to probe.", + "examples": [ + 3000 + ], + "maximum": 65535, + "minimum": 1, + "type": "integer" + }, + "workload": { + "description": "Workload the path is probed inside.", + "examples": [ + "web" + ], + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "migrations": { + "description": "Migration revisions checked against captured job evidence.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "advisory": { + "default": false, + "description": "Report a failure without blocking release activation.", + "type": "boolean" + }, + "applied_revisions": { + "description": "Revisions the job must report as applied.", + "items": { + "type": "string" + }, + "type": "array" + }, + "job": { + "description": "Job workload whose captured evidence is checked.", + "examples": [ + "migrate" + ], + "type": "string" + }, + "provider": { + "description": "Migration tool that produced the revisions.", + "examples": [ + "alembic" + ], + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "url": { + "description": "External URLs probed from the operator side.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "advisory": { + "default": false, + "description": "Report a failure without blocking release activation.", + "type": "boolean" + }, + "contains": { + "description": "Text the response body must contain.", + "type": "string" + }, + "json_assertions": { + "description": "Scalar JSON response values that must match exactly.", + "items": { + "additionalProperties": false, + "patternProperties": { + "^x-": {} + }, + "properties": { + "equals": { + "description": "Exact scalar value required at path." + }, + "path": { + "description": "Dot-separated path to a scalar value in the JSON response.", + "examples": [ + "service.ready" + ], + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "required_headers": { + "additionalProperties": { + "type": "string" + }, + "description": "Exact response headers required for success.", + "type": "object" + }, + "status_codes": { + "description": "Allowed response status codes. A successful 2xx response is expected when omitted.", + "items": { + "maximum": 599, + "minimum": 100, + "type": "integer" + }, + "type": "array" + }, + "url": { + "description": "External HTTP or HTTPS URL verified from the operator side. Expects an http or https URL.", + "examples": [ + "https://shop.example.com/healthz" + ], + "pattern": "^https?://", + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + } + }, + "type": "object" + }, "compose": { "description": "Existing Compose service to adopt, as repository path#service. Expects a reference of the form path/to/compose.yaml#service.", "examples": [ @@ -497,23 +679,46 @@ "description": "Declared permission for agent-authored proposals. The current CLI does not distinguish agent identity; execution remains approval-gated.", "type": "boolean" }, - "migration_backup_key_material": { - "description": "Names of key material whose usability must be covered by the migration backup report.", - "items": { - "type": "string" + "migrations": { + "additionalProperties": false, + "description": "What this environment requires of a release carrying migration risk.", + "patternProperties": { + "^x-": {} }, - "type": "array" - }, - "migration_backup_maximum_age": { - "default": "24h", - "description": "Maximum age of a backup report accepted for a migration. Expects a duration such as 30s, 5m, 1h30m or 14d.", - "examples": [ - "24h" - ], - "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", - "type": "string" + "properties": { + "backup_key_material": { + "description": "Key-material identities the backup report must name.", + "examples": [ + "BACKUP_ACCESS_KEY_ID" + ], + "items": { + "type": "string" + }, + "type": "array" + }, + "backup_max_age": { + "default": "24h", + "description": "Maximum age of a backup report accepted for a migration. Expects a duration such as 30s, 5m, 1h30m or 14d.", + "examples": [ + "24h" + ], + "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", + "type": "string" + }, + "require_backup": { + "default": false, + "description": "Require a plan-bound backup report before a release with migration risk.", + "type": "boolean" + }, + "require_restore_test": { + "default": false, + "description": "Require the backup report to state that a restore test succeeded.", + "type": "boolean" + } + }, + "type": "object" }, - "minimum_onebox_version": { + "min_onebox_version": { "description": "Oldest released Onebox runner allowed to operate this environment. Expects a CalVer release such as v2026.8.0.", "examples": [ "v2026.8.0" @@ -521,7 +726,7 @@ "pattern": "^v([1-9][0-9]{3})\\.([1-9]|1[0-2])\\.(0|[1-9][0-9]{0,18})$", "type": "string" }, - "minimum_plan_schema": { + "min_plan_schema": { "description": "Oldest executable plan schema accepted by this environment. Expects a plan schema identity such as onebox.run/executable-deploy-plan/v1alpha2.", "examples": [ "onebox.run/executable-deploy-plan/v1alpha2" @@ -533,16 +738,6 @@ "default": true, "description": "Require a plan-bound local confirmation before mutating this environment.", "type": "boolean" - }, - "require_migration_backup": { - "default": false, - "description": "Require a plan-bound backup report before a release with migration risk.", - "type": "boolean" - }, - "require_migration_restore_test": { - "default": false, - "description": "Require the backup report to state that a restore test succeeded.", - "type": "boolean" } }, "type": "object" @@ -603,6 +798,14 @@ "^x-": {} }, "properties": { + "backup_owner": { + "description": "Operator or provider responsible for backup, restore, upgrades, credentials, and durability. Expects a stable operator or provider identity of letters, digits, dots, @, colons, slashes, underscores and hyphens.", + "examples": [ + "platform-team/rds" + ], + "pattern": "^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,127}$", + "type": "string" + }, "connection": { "additionalProperties": false, "description": "Trusted connection source and driver-shaped entry mapping.", @@ -683,7 +886,7 @@ ], "type": "string" }, - "maximum_age": { + "max_age": { "default": "5m", "description": "Maximum age of a probe observation bound into a plan. Expects a duration such as 30s, 5m, 1h30m or 14d.", "examples": [ @@ -703,19 +906,11 @@ } }, "type": "object" - }, - "protection_owner": { - "description": "Operator or provider responsible for backup, restore, upgrades, credentials, and durability. Expects a stable operator or provider identity of letters, digits, dots, @, colons, slashes, underscores and hyphens.", - "examples": [ - "platform-team/rds" - ], - "pattern": "^[A-Za-z0-9][A-Za-z0-9._@:/-]{0,127}$", - "type": "string" } }, "type": "object" }, - "description": "Typed dependencies operated outside Onebox. Their connection projection is trusted, but their lifecycle and protection remain external.", + "description": "Typed dependencies operated outside Onebox. Their connection projection is trusted, but their lifecycle and backup remain external.", "type": "object" }, "health": { @@ -842,16 +1037,9 @@ "^x-": {} }, "properties": { - "platform": { - "description": "Platform selected when the image is multi-platform.", - "examples": [ - "linux/amd64" - ], - "type": "string" - }, "pull": { "default": "missing", - "description": "Image pull policy: missing, always, or never.", + "description": "When to fetch the image from the registry: missing fetches only what the host does not already hold, always fetches every release, never fetches at all and fails on a missing image.", "enum": [ "always", "missing", @@ -866,10 +1054,6 @@ ], "pattern": "^((?:(?:(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])(?:\\.(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]))*|\\[(?:[a-fA-F0-9:]+)\\])(?::[0-9]+)?/)?[a-z0-9]+(?:(?:[._]|__|[-]+)[a-z0-9]+)*(?:/[a-z0-9]+(?:(?:[._]|__|[-]+)[a-z0-9]+)*)*)(?::([\\w][\\w.-]{0,127}))?(?:@([A-Za-z][A-Za-z0-9]*(?:[-_+.][A-Za-z][A-Za-z0-9]*)*[:][[:xdigit:]]{32,}))?$", "type": "string" - }, - "registry": { - "description": "Optional registry label retained in canonical configuration. Current authentication uses every top-level registries entry; this field does not select a login.", - "type": "string" } }, "type": "object" @@ -918,72 +1102,6 @@ "description": "Named webhooks that receive selected operation outcomes.", "type": "object" }, - "observability": { - "additionalProperties": false, - "description": "Declared logging, metrics, and alerting intent. Continuous management is not currently provided.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "alerts": { - "additionalProperties": false, - "description": "Declared alerting intent. Continuous management is not currently provided.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "unhealthy_after": { - "description": "Desired duration of unhealthy state before alerting. Expects a duration such as 30s, 5m, 1h30m or 14d.", - "examples": [ - "5m" - ], - "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", - "type": "string" - } - }, - "type": "object" - }, - "logs": { - "additionalProperties": false, - "description": "Declared log-retention intent. Continuous management is not currently provided.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "enabled": { - "default": false, - "description": "Declare that log collection is desired.", - "type": "boolean" - }, - "retention": { - "description": "Desired log-retention period. Expects a duration such as 30s, 5m, 1h30m or 14d.", - "examples": [ - "30d" - ], - "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", - "type": "string" - } - }, - "type": "object" - }, - "metrics": { - "additionalProperties": false, - "description": "Declared metric-collection intent. Continuous management is not currently provided.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "enabled": { - "default": false, - "description": "Declare that metric collection is desired.", - "type": "boolean" - } - }, - "type": "object" - } - }, - "type": "object" - }, "port": { "description": "Container port used with domain shorthand and as the default HTTP health port.", "examples": [ @@ -1250,74 +1368,26 @@ "^x-": {} }, "properties": { - "driver": { - "description": "Built-in service driver. Defaults to the service map key. Expects lower-case letters, digits and hyphens, starting with a letter, at most 40 characters.", - "examples": [ - "postgres" - ], - "pattern": "^[a-z]([a-z0-9-]{0,38}[a-z0-9])?$", - "type": "string" - }, - "persistence": { + "backup": { "additionalProperties": false, - "description": "Data-lifetime declaration for this supporting service.", + "description": "Recovery intent for this service. Onebox selects the qualified native implementation; declaring intent alone does not establish backup.", "patternProperties": { "^x-": {} }, "properties": { - "mode": { - "default": "durable", - "description": "Data lifetime: durable, ephemeral, or external.", - "enum": [ - "durable", - "ephemeral", - "external" - ], - "type": "string" - } - }, - "type": "object" - }, - "protection": { - "additionalProperties": false, - "description": "Recovery intent for this service. Onebox selects the qualified native implementation; declaring intent alone does not establish protection.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "allow_backup_interruption": { + "allow_downtime": { "default": false, "description": "Whether recurring backup operations may use the driver-declared stopped-service window.", "type": "boolean" }, - "maximum_data_loss": { - "description": "Maximum tolerable interval between the latest recoverable point and failure. Expects a duration such as 30s, 5m, 1h30m or 14d.", - "examples": [ - "15m" - ], - "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", - "type": "string" - }, - "recovery_kind": { - "description": "Required recovery envelope: snapshot, pitr, or cold.", - "enum": [ - "snapshot", - "pitr", - "cold" - ], - "examples": [ - "pitr" - ], - "type": "string" - }, - "restore_drill": { + "drill": { "additionalProperties": false, "description": "Exact isolated restore-test schedule, proof age, and optional staging filesystem.", "patternProperties": { "^x-": {} }, "properties": { - "proof_maximum_age": { + "max_age": { "default": "7d", "description": "Maximum age of the latest passing restore proof. Expects a duration such as 30s, 5m, 1h30m or 14d.", "examples": [ @@ -1352,18 +1422,30 @@ } }, "type": "object" - }, - "staging_filesystem": { - "description": "Absolute filesystem path used for isolated restore materialization instead of the host default. Expects an absolute path with no control character or shell metacharacter.", - "examples": [ - "/srv/onebox-restore" - ], - "pattern": "^/[^\\x00-\\x1f'\"$`\\\\]*$", - "type": "string" } }, "type": "object" }, + "max_data_loss": { + "description": "Maximum tolerable interval between the latest recoverable point and failure. Expects a duration such as 30s, 5m, 1h30m or 14d.", + "examples": [ + "15m" + ], + "pattern": "^(([0-9]+([.][0-9]+)?(ns|us|µs|ms|s|m|h))+|[0-9]+d)$", + "type": "string" + }, + "recovery_kind": { + "description": "Required recovery envelope: snapshot, pitr, or cold.", + "enum": [ + "snapshot", + "pitr", + "cold" + ], + "examples": [ + "pitr" + ], + "type": "string" + }, "retention": { "additionalProperties": false, "description": "Portable minimum recovery history that the selected native driver must be able to preserve.", @@ -1371,7 +1453,7 @@ "^x-": {} }, "properties": { - "minimum_generations": { + "keep": { "default": 7, "description": "Minimum number of independently recoverable base generations to retain.", "examples": [ @@ -1380,7 +1462,7 @@ "minimum": 1, "type": "integer" }, - "recovery_window": { + "window": { "default": "7d", "description": "Minimum continuous recovery history the native retention mapping must preserve. Expects a duration such as 30s, 5m, 1h30m or 14d.", "examples": [ @@ -1430,6 +1512,34 @@ }, "type": "object" }, + "driver": { + "description": "Built-in service driver. Defaults to the service map key. Expects lower-case letters, digits and hyphens, starting with a letter, at most 40 characters.", + "examples": [ + "postgres" + ], + "pattern": "^[a-z]([a-z0-9-]{0,38}[a-z0-9])?$", + "type": "string" + }, + "persistence": { + "additionalProperties": false, + "description": "Data-lifetime declaration for this supporting service.", + "patternProperties": { + "^x-": {} + }, + "properties": { + "mode": { + "default": "durable", + "description": "Data lifetime: durable, ephemeral, or external.", + "enum": [ + "durable", + "ephemeral", + "external" + ], + "type": "string" + } + }, + "type": "object" + }, "resources": { "additionalProperties": false, "description": "Memory and CPU limits for this supporting service.", @@ -1488,131 +1598,6 @@ "description": "Supporting services managed outside application releases, such as databases and caches.", "type": "object" }, - "verifications": { - "description": "Checks that must pass before a release becomes current unless marked advisory.", - "items": { - "additionalProperties": false, - "patternProperties": { - "^x-": {} - }, - "properties": { - "advisory": { - "default": false, - "description": "Report a failed check without blocking release activation.", - "type": "boolean" - }, - "contains": { - "description": "Text that the HTTP response body must contain.", - "type": "string" - }, - "exec": { - "description": "Shell command verified inside the named workload.", - "type": "string" - }, - "http": { - "description": "HTTP path verified inside the named workload. Expects a path beginning with /.", - "examples": [ - "/healthz" - ], - "pattern": "^/[^\\x00-\\x1f'\"$` \\\\]*$", - "type": "string" - }, - "json_assertions": { - "description": "Scalar JSON response values that must match exactly.", - "items": { - "additionalProperties": false, - "patternProperties": { - "^x-": {} - }, - "properties": { - "equals": { - "description": "Exact scalar value required at path." - }, - "path": { - "description": "Dot-separated path to a scalar value in the JSON response.", - "examples": [ - "service.ready" - ], - "type": "string" - } - }, - "type": "object" - }, - "type": "array" - }, - "migration_revisions": { - "additionalProperties": false, - "description": "Expected migration provider and applied revisions, checked against captured job evidence.", - "patternProperties": { - "^x-": {} - }, - "properties": { - "applied_revisions": { - "description": "Ordered migration revisions expected to be applied.", - "items": { - "type": "string" - }, - "type": "array" - }, - "job": { - "description": "Migration job whose result evidence is checked.", - "examples": [ - "migrate" - ], - "type": "string" - }, - "provider": { - "description": "Migration provider expected in the job result.", - "examples": [ - "atlas" - ], - "type": "string" - } - }, - "type": "object" - }, - "port": { - "description": "Container port used by an internal HTTP verification.", - "examples": [ - 3000 - ], - "maximum": 65535, - "minimum": 1, - "type": "integer" - }, - "required_headers": { - "additionalProperties": { - "type": "string" - }, - "description": "Exact HTTP response headers required for success.", - "type": "object" - }, - "status_codes": { - "description": "Allowed HTTP response status codes. A successful 2xx response is expected when omitted.", - "items": { - "maximum": 599, - "minimum": 100, - "type": "integer" - }, - "type": "array" - }, - "url": { - "description": "External HTTP or HTTPS URL verified from the operator side. Expects an http or https URL.", - "examples": [ - "https://shop.example.com/healthz" - ], - "pattern": "^https?://", - "type": "string" - }, - "workload": { - "description": "Workload in which an internal HTTP or exec verification runs.", - "type": "string" - } - }, - "type": "object" - }, - "type": "array" - }, "workloads": { "additionalProperties": { "additionalProperties": false, @@ -1860,13 +1845,6 @@ "pattern": "^[^/\\x00-\\x1f'\"$`\\\\][^\\x00-\\x1f'\"$`\\\\]*$", "type": "string" }, - "platform": { - "description": "Target image platform for the external build.", - "examples": [ - "linux/amd64" - ], - "type": "string" - }, "target": { "description": "Named Dockerfile stage to build.", "type": "string" @@ -2131,16 +2109,9 @@ "^x-": {} }, "properties": { - "platform": { - "description": "Platform selected when the image is multi-platform.", - "examples": [ - "linux/amd64" - ], - "type": "string" - }, "pull": { "default": "missing", - "description": "Image pull policy: missing, always, or never.", + "description": "When to fetch the image from the registry: missing fetches only what the host does not already hold, always fetches every release, never fetches at all and fails on a missing image.", "enum": [ "always", "missing", @@ -2155,10 +2126,6 @@ ], "pattern": "^((?:(?:(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])(?:\\.(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]))*|\\[(?:[a-fA-F0-9:]+)\\])(?::[0-9]+)?/)?[a-z0-9]+(?:(?:[._]|__|[-]+)[a-z0-9]+)*(?:/[a-z0-9]+(?:(?:[._]|__|[-]+)[a-z0-9]+)*)*)(?::([\\w][\\w.-]{0,127}))?(?:@([A-Za-z][A-Za-z0-9]*(?:[-_+.][A-Za-z][A-Za-z0-9]*)*[:][[:xdigit:]]{32,}))?$", "type": "string" - }, - "registry": { - "description": "Optional registry label retained in canonical configuration. Current authentication uses every top-level registries entry; this field does not select a login.", - "type": "string" } }, "type": "object" diff --git a/site/src/content/docs/explanation/evidence-not-declaration.mdx b/site/src/content/docs/explanation/evidence-not-declaration.mdx index 423593b4..a6394bab 100644 --- a/site/src/content/docs/explanation/evidence-not-declaration.mdx +++ b/site/src/content/docs/explanation/evidence-not-declaration.mdx @@ -37,8 +37,8 @@ flow, never through ordinary model-visible arguments. ## Declaration is not protection -The proposed protection layer applies the principle to data recovery. A service -would report `Managed` only while, **all at once**: +Backup applies the principle to data recovery. A service reports `Managed` only +while, **all at once**: - the recorded immutable image digest is effective - resource policy is effective diff --git a/site/src/content/docs/guides/add-a-database.mdx b/site/src/content/docs/guides/add-a-database.mdx index d6f207bd..9587e62c 100644 --- a/site/src/content/docs/guides/add-a-database.mdx +++ b/site/src/content/docs/guides/add-a-database.mdx @@ -157,10 +157,15 @@ using either will connect, authenticate, and *then* fail — in its own logs, no in anything Onebox says. If you need one, run a `daemon` workload you own. ::: -:::danger[Onebox does not take backups] -`ob doctor` reports every workload and service holding durable data as unbacked, -because silence there would read as approval. Until the -[protection layer](/status/capabilities) ships, backing up a declared service is +:::caution[Back it up] +A `postgres` service can declare `backup` and Onebox will keep a continuous +off-host archive it can recover to a point in time — see +[backups](/guides/back-up-a-database). Declaring the policy is not enough on its +own: `ob backup enable` is what establishes it, and until then the service +renders as an ordinary unprotected server. + +Every other driver declares `policy_qualified: false`, so a backup policy on +it is refused rather than quietly ignored, and copying its data off the box is your responsibility — a `job` workload on a `schedule` is the usual answer. ::: diff --git a/site/src/content/docs/guides/back-up-a-database.mdx b/site/src/content/docs/guides/back-up-a-database.mdx new file mode 100644 index 00000000..e6f25453 --- /dev/null +++ b/site/src/content/docs/guides/back-up-a-database.mdx @@ -0,0 +1,157 @@ +--- +title: Back up a database +description: Continuous off-host backups for PostgreSQL, and how to prove they restore. +summary: How to declare a backup policy and a repository, what `ob backup enable` actually does, and why a backup nobody has restored is a hypothesis. +sidebar: + order: 3 +read_when: + - "Protecting a managed PostgreSQL service" + - "Working out what a declared backup policy does and does not do on its own" + - "Proving a repository can actually be recovered from" +--- + +```yaml +services: + database: + driver: postgres + version: "18" + persistence: {mode: durable} + backup: + target: offsite + recovery_kind: pitr + max_data_loss: 15m + schedule: {cron: "0 2 * * *", timezone: UTC} + drill: + schedule: {cron: "0 4 * * *", timezone: UTC} + +backup_targets: + offsite: + kind: s3-compatible + endpoint: https://objects.example.net + bucket: onebox-backups + failure_domain: {identity: provider-b/eu-central-1} + credentials: + file: secrets/backup.env + access_key_entry: BACKUP_ACCESS_KEY_ID + secret_key_entry: BACKUP_SECRET_ACCESS_KEY + encryption: {pitr: client-side} +``` + +Two blocks: the intent sits on the service, the destination is declared once at +the top level and referenced by name, so several services can share a repository +and changing its endpoint is one edit. + +## Declaring is not enabling + +A `backup` block is a request. Until `ob backup enable` succeeds, the service +renders as an ordinary unprotected server — `ob validate` accepts the policy and +nothing archives. + +```console +$ ob backup enable database +✓ protected image postgres:18 +✓ backup runtime wal-g v3.0.8 (aarch64) +✓ service database +→ backup schedule: ob-backup-shop-production-database-backup at 0 2 * * * +✓ backup database +``` + +That one command pins the image by registry digest, stages a checksum-verified +wal-g onto the host, decrypts and installs the destination credentials, restarts +the server with WAL archiving on, installs the timers, and takes the first base +backup. **It is not finished until that base backup exists** — WAL archiving with +nothing to replay onto recovers nothing, and reporting success there would be +telling you the database is protected at the moment it is not. + +The restart is a real restart. Enabling is a maintenance action, not a +configuration change. + +## The credential file + +The encrypted file your target names needs three entries — the two you named, +plus the repository key: + +```sh +BACKUP_ACCESS_KEY_ID=... +BACKUP_SECRET_ACCESS_KEY=... +OB_REPOSITORY_KEY=$(openssl rand -hex 32) +``` + +`OB_REPOSITORY_KEY` is a 32-byte key read as hex, so exactly 64 hex characters. +A passphrase-shaped value is refused before anything restarts. You do not stage +this file yourself: `ob backup enable` decrypts it and installs it mode-0600 on +the host. + +## Retention is derived + +`retention` has two floors and Onebox keeps whichever is larger — the number of +generations, or enough of them to span the window at your schedule's rate: + +| schedule | window | generations kept | +| --- | --- | --- | +| `0 2 * * *` | `7d` | 8 | +| `0 */6 * * *` | `7d` | 29 | +| `*/5 * * * *` | `7d` | 2017 | + +`ob backup prune` prints the number it is using before it costs you storage. + +## Prove it restores + +```console +$ ob backup drill database +✓ recovery: fetch base backup +✓ recovery: replay to the newest recoverable point +✓ recovery: verify the recovered cluster answers +✓ drill passed: database recovered from base_000000010000000000000007 and answered. Nothing was changed. +``` + +A drill recovers into a throwaway volume, proves the cluster opens and answers a +query, then discards it. The live service is never touched. It runs the same code +as a real restore and stops before the last step — a drill exercising its own +path would prove the drill works, not the backups. + +:::caution[The scheduled drill verifies; it does not restore] +The timer installed from `drill.schedule` runs an archive-continuity check, not a +full recovery: the orchestration lives in `ob`, and Onebox puts no agent on your +host. Run `ob backup drill` from CI on the cadence your policy declares. A backup +nobody has restored is a hypothesis. +::: + +## Recovering + +```console +$ ob backup restore database --to 2026-08-19T17:31:44Z --confirm database +``` + +The recovered cluster is built beside the live one and has to start, promote and +answer a query before anything touches the running database — so a repository +that cannot recover fails while the database it would have replaced is still +serving. The data being replaced is copied aside under a dated volume name and +never deleted. + +The service name is typed back with `--confirm` because a recovery has no plan +for the approval flow to bind to, so the guard is the name of the thing being +replaced. + +Without `--to`, recovery goes to the newest recoverable point. + +## Stopping + +```console +$ ob backup disable database --confirm database +``` + +Archiving stops, the timers go, the service restarts unprotected, and the +destination credentials are removed from the host. **The repository is not +touched** — but reading or recovering from it needs backup enabled again, +because the tooling and credentials that reach it live in the protected service. + +## What is not covered + +Only the `postgres` driver has an executable contract, on versions 17 and 18. +Every other driver refuses a `backup` policy at validate rather than accepting +one it cannot honour. A workload's own volume is never copied anywhere — `ob +doctor` names every workload holding durable data. + +Full field list: [`services`](/reference/fields/services) and +[`backup_targets`](/reference/fields/backup_targets). diff --git a/site/src/content/docs/guides/roll-back.mdx b/site/src/content/docs/guides/roll-back.mdx index d6bd00d7..e5309f1c 100644 --- a/site/src/content/docs/guides/roll-back.mdx +++ b/site/src/content/docs/guides/roll-back.mdx @@ -125,7 +125,7 @@ ob abort --break-lock --output ndjson `--break-lock` is on six commands: `deploy`, `bootstrap`, `abort`, `job run`, `service apply` and `proxy apply`. Not every mutating command has it — `resume`, `destroy` and `secrets push` do not. It breaks the application and host locks -only. The protection lock has no override and clears on TTL expiry or when the +only. The backup lock has no override and clears on TTL expiry or when the same operation returns. ## Recovery onto another host is a different workflow diff --git a/site/src/content/docs/guides/run-migrations.mdx b/site/src/content/docs/guides/run-migrations.mdx index eb72dcee..7c11bb60 100644 --- a/site/src/content/docs/guides/run-migrations.mdx +++ b/site/src/content/docs/guides/run-migrations.mdx @@ -60,9 +60,9 @@ it. ## Verify the revisions ```yaml -verifications: - - migration_revisions: - job: migrate +checks: + migrations: + - job: migrate provider: atlas applied_revisions: ["202607130001"] ``` diff --git a/site/src/content/docs/index.mdx b/site/src/content/docs/index.mdx index 9f2643e4..d3d2a849 100644 --- a/site/src/content/docs/index.mdx +++ b/site/src/content/docs/index.mdx @@ -119,8 +119,11 @@ make a failed host available. ## Two things it does not do today -**Onebox does not take backups.** `ob doctor` says so for every workload and -service holding durable data, because silence there would read as approval. +**Backups cover managed services, not workload volumes.** A service declaring +`backup` — PostgreSQL today — is backed up continuously to an off-host +repository and can be recovered to a point in time. A workload's own volume is +not, and `ob doctor` says so for every workload holding durable data, because +silence there would read as approval. **`mongodb` runs a standalone server, not a replica set.** An application needing change streams or multi-document transactions will connect, authenticate, and diff --git a/site/src/content/docs/reference/cli.mdx b/site/src/content/docs/reference/cli.mdx index 25d287f7..1c2b855d 100644 --- a/site/src/content/docs/reference/cli.mdx +++ b/site/src/content/docs/reference/cli.mdx @@ -49,6 +49,7 @@ Available Commands: abort revert an interrupted deploy to the previous release (migration-gated) approve record a local human confirmation for one exact executable plan audit who deployed what, when, from which SHA — incl. failed runs + backup protect a data service and inspect what can be recovered bootstrap first contact: host setup, registry login, and supporting/data services canonical print the canonical form Onebox understood, with where each value came from completion Generate the autocompletion script for the specified shell @@ -159,6 +160,264 @@ Global Flags: -v, --verbose print every remote command ``` +## ob backup + +``` +Backup and recovery for the data services this project declares. + +Backup is physical: a base backup plus continuous WAL archiving to the +off-host repository the project's backup_targets name, which is what makes +recovery to a point in time possible rather than recovery to last night. + +Declaring a policy does not establish it. `ob backup enable` restarts the +service with archiving on, stages the verified backup tooling, and takes +the first base backup; only then does the service render as protected. + +Usage: + ob backup [flags] + ob backup [command] + +Available Commands: + create take a base backup now + disable stop archiving; keep every backup already taken + drill prove the repository recovers, without touching anything + enable establish backup — restarts the service archiving and takes the first backup + prune expire backups outside the declared retention + restore recover to a point in time and put it in service + status what the repository can recover, read from the repository + verify prove the archived WAL forms an unbroken chain + +Flags: + -h, --help help for backup + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command + +Use "ob backup [command] --help" for more information about a command. +``` + +### ob backup create + +``` +Take a base backup of a protected service. + +Every base backup is complete: the space between them is covered by the WAL +stream rather than by differential backups, so there is no type to choose. + +WAL archiving runs continuously and is not this command. Between backups the +recoverable point keeps advancing on its own; a base backup bounds how much +WAL a recovery has to replay, and how far back the window reaches. + +Usage: + ob backup create [flags] + +Flags: + --break-lock break a stale operation lock after inspecting its holder + -h, --help help for create + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command +``` + +### ob backup disable + +``` +Take a service out of backup. + +Archiving stops, the schedules are removed, the service restarts as an +ordinary unprotected one, and its destination credentials are removed from +the host. + +The repository is not touched: every backup already taken stays where it +is. Reading or recovering from them needs backup enabled again, +because the binary and credentials that reach the repository live in the +protected service. + +What does stop is the recovery window advancing: from here on there is no +new WAL, so the newest recoverable point is the moment this ran. + +Usage: + ob backup disable [flags] + +Flags: + --break-lock break a stale operation lock after inspecting its holder + --confirm string name of the service whose archiving may stop + -h, --help help for disable + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command +``` + +### ob backup drill + +``` +Recover into a throwaway volume, prove the cluster opens and answers, then +discard it. The live service is never touched. + +This runs the same code as `ob backup restore` and stops before the last +step, which is the point: a drill that exercised a different path would +prove the drill works rather than that the backups do. + +A backup nobody has restored is a hypothesis. + +Usage: + ob backup drill [flags] + +Flags: + -h, --help help for drill + --to string RFC 3339 point in time to prove recoverable (default: the newest recoverable point) + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command +``` + +### ob backup enable + +``` +Make a declared backup policy real. + +The order is forced: the credentials are checked, the image is pinned by +registry digest, the verified wal-g binary is staged on the host, and only +then does the server restart with archiving on. + +The restart is a real restart of the database. It is not complete until the +first base backup exists, because WAL archiving with nothing to replay onto +can recover nothing. + +Usage: + ob backup enable [flags] + +Flags: + --break-lock break a stale operation lock after inspecting its holder + -h, --help help for enable + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command +``` + +### ob backup prune + +``` +Expire everything the policy no longer promises to keep. + +Retention comes from services..backup.retention.keep, +so this never removes more than the project says it may keep fewer of. + +WAL older than the oldest retained backup goes with it: WAL that cannot be +replayed onto any surviving base backup recovers nothing and only costs storage. + +Usage: + ob backup prune [flags] + +Flags: + --break-lock break a stale operation lock after inspecting its holder + -h, --help help for prune + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command +``` + +### ob backup restore + +``` +Recover a protected service from its repository. + +The recovered cluster is always built beside the live one, never over it: +the base backup is fetched into a fresh volume, WAL is replayed to the +requested point, and the result has to start and answer a query before +anything touches the running database. A repository that cannot recover +fails while the database it would have replaced is still serving. + +The data being replaced is copied aside first, under a dated volume name, +and never deleted. A restore is run on a day that is already going badly; +it must not be the step that makes it unrecoverable. + +Without --to, recovery goes to the newest recoverable point. + +The service name has to be typed back with --confirm. Onebox's approval flow +binds a recorded confirmation to an exact plan, and a recovery has no plan to +bind to — so the guard is the name of the thing being replaced, which cannot +be given by accident or by a shell history entry meant for another service. + +Usage: + ob backup restore [flags] + +Flags: + --break-lock break a stale operation lock after inspecting its holder + --confirm string name of the service whose live data may be replaced + -h, --help help for restore + --to string RFC 3339 point in time to recover to (default: the newest recoverable point) + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command +``` + +### ob backup status + +``` +Report what is actually recoverable. + +Every figure comes from the repository rather than from the project: the +policy states what should be true, and this states what is. A service whose +policy is declared but never enabled has no repository to ask, and says so. + +Usage: + ob backup status [flags] + +Flags: + -h, --help help for status + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command +``` + +### ob backup verify + +``` +Check that the WAL in the repository is continuous. + +This is the check worth running on a schedule, and it is not implied by a +backup that exited zero. A base backup with a gapped WAL stream recovers to +the backup and no further — a nightly snapshot wearing the label of +point-in-time recovery — and nothing else notices until someone needs it. + +Usage: + ob backup verify [flags] + +Flags: + -h, --help help for verify + +Global Flags: + -c, --config string path to the project YAML file (default "ob.yml") + -e, --env string environment name (default "production") + --output string output mode for supported commands: human|json|ndjson (see the CLI reference) (default "human") + -v, --verbose print every remote command +``` + ## ob bootstrap ``` @@ -275,8 +534,9 @@ Check this runner and the safety capabilities of the environment it targets. Reports the runner's provenance and whether it satisfies the environment's minimum version and plan schema, and names every workload and service holding -durable data that has no backup — Onebox does not take backups, and silence -there would read as approval. In structured output, automation should gate on +durable data that nothing is copying off the box, because silence there would +read as approval. A service declaring `backup` is reported as declaring it; +what the repository can actually recover is `ob backup status`. In structured output, automation should gate on the report status: data.status for pass or warn, and error.details.status for a failing diagnosis. @@ -412,7 +672,7 @@ Usage: ob job plan [flags] Flags: - --backup-report-out string write a plan-bound backup report template when migration protection is required + --backup-report-out string write a plan-bound backup report template when migration backup is required -h, --help help for plan -o, --out string job plan artifact path (default "ob-job-plan.json") @@ -488,7 +748,7 @@ Usage: ob plan [flags] Flags: - --backup-report-out string write a plan-bound backup report template when migration protection is required + --backup-report-out string write a plan-bound backup report template when migration backup is required -h, --help help for plan --image stringArray resolved image as workload=reference, for build-sourced workloads (repeatable) -o, --out string plan artifact path (default "ob-plan.json") diff --git a/site/src/content/docs/reference/drivers.mdx b/site/src/content/docs/reference/drivers.mdx index 3a771527..9e1e1a16 100644 --- a/site/src/content/docs/reference/drivers.mdx +++ b/site/src/content/docs/reference/drivers.mdx @@ -92,13 +92,16 @@ contract, and it is stated here because the failure appears in the application's logs rather than in anything Onebox says. ::: -:::danger[No driver is backed up] -Onebox does not take backups. `ob doctor` reports every service holding durable -data as unbacked. It also refuses a major version change a driver cannot perform -in place, rather than replacing the container and leaving the data intact and -unreachable. +:::caution[Only `postgres` is backed up] +`postgres` is the one driver with an executable backup contract: declaring +`backup` on it gives a continuous off-host archive and point-in-time recovery. +Every other driver declares `policy_qualified: false`, so a backup policy on it +is refused at `ob validate` rather than accepted and left to do nothing — and its data lives only on this host until you copy it off yourself. -The proposed protection layer would change this — see +Onebox also refuses a major version change a driver cannot perform in place, +rather than replacing the container and leaving the data intact and unreachable. + +For `postgres`, declaring `backup` changes this — see [Shipped vs proposed](/status/capabilities). ::: diff --git a/site/src/content/docs/reference/errors.mdx b/site/src/content/docs/reference/errors.mdx index b3dfb94d..01c78d52 100644 --- a/site/src/content/docs/reference/errors.mdx +++ b/site/src/content/docs/reference/errors.mdx @@ -29,12 +29,16 @@ command. | Code | Means | | --- | --- | | `app_required` | the shorthand form needs an application name to attach the workload to | -| `backup_driver_unsupported` | a runnable service driver has no qualified executable protection contract | +| `backup_credentials_invalid` | decrypted backup credentials are missing or malformed | +| `backup_driver_unsupported` | a runnable service driver has no qualified executable backup contract | | `backup_encryption_unverified` | the selected target cannot prove the encryption mode required by the recovery kind | +| `backup_image_revert_unsafe` | tag rendering would strand an effective backup prerequisite | | `backup_interruption_not_authorized` | the selected recovery contract needs a recurring stopped-service window the author did not permit | | `backup_retention_unsupported` | the declared recovery history cannot map to supported retention semantics | +| `backup_service_image_unpublished` | the protected service image lacks verified publication provenance | +| `backup_state_incomplete` | a protected service does not record what it is protected by | | `backup_target_not_independent` | a backup target shares the protected failure domain | -| `backup_target_unknown` | a protection policy selects no declared backup target | +| `backup_target_unknown` | a backup policy selects no declared backup target | | `compose_container_name` | a referenced service fixes its container name, which Onebox owns | | `compose_extends` | a referenced service uses extends, which hides what runs | | `compose_file_unparsable` | a referenced Compose file is not valid YAML | @@ -47,6 +51,7 @@ command. | `compose_traefik_label` | a referenced service carries routing labels while also declaring a route | | `connection_variable_claimed` | an authored value claims a name a managed-service connection supplies | | `derived_name_too_long` | a name Onebox derives exceeds the runtime's limit | +| `drill_schedule_too_sparse` | the declared drill cadence is too sparse to keep restore proof within its maximum age | | `eject_destination_exists` | the ejection destination already exists | | `eject_failed` | the runtime could not be handed over | | `eject_nothing_to_do` | every workload already references a Compose file | @@ -71,12 +76,8 @@ command. | `project_invalid` | a value that does not satisfy the contract | | `project_unparsable` | the project file is not valid YAML, or is not a mapping | | `project_unreadable` | the project file could not be read | -| `protected_service_patch_unsupported` | no exact qualified protected current-to-candidate image transition exists | -| `protection_image_revert_unsafe` | tag rendering would strand an effective protection prerequisite | -| `protection_service_image_unpublished` | the protected service image lacks verified publication provenance | | `recovery_objective_unsupported` | the service driver, target, or version cannot execute the declared recovery kind | | `render_failed` | the runtime could not be rendered | -| `restore_drill_schedule_too_sparse` | the restore-drill cadence cannot keep restore proof current | | `route_collision` | two workloads claim the same address | | `route_without_proxy` | a route is declared with nothing to route it | | `routing_exclusive` | the domain shorthand and the routes list say the same thing twice | @@ -88,6 +89,7 @@ command. | `server_unreachable` | the server could not be reached | | `service_image_digest_unavailable` | the immutable service image is unavailable from registry and exact cache | | `service_image_patch_disable_pending` | protected image refresh is refused while disablement is pending | +| `service_patch_unsupported` | no exact qualified protected current-to-candidate image transition exists | | `service_settings_unsupported` | a setting was declared for a driver with no way to apply it | | `shorthand_and_workloads` | top-level shorthand cannot be combined with a workloads block | | `stateful_replicas` | a workload keeping durable state asks for more than one replica | @@ -161,52 +163,29 @@ step to complete rather than a line to run verbatim. ## Lifecycle failure codes -:::caution[Belongs to the proposed protection layer] -These codes are defined and drift-tested in the binary, but the operations that -raise most of them are not yet executable. A row marked **reserved** is one no -path raises today: the code is fixed so it stays stable when the capability -lands, but you cannot cause it. The set is computed from the source, not -maintained by hand. -::: +Every code here is raised by a path in the shipped binary, checked against the +source by a test in both directions. The table is computed, not maintained by +hand. The failure contract shared by plans, event streams, terminal results, status and doctor. Each carries a stable code and one safe command in its semantic role; diagnostic detail stays in restricted local evidence, never in the public record. -| Code | Reachable | Means | Guidance role | Command | -| --- | --- | --- | --- | --- | -| `assurance_stale` | reserved | continuous assurance evidence is no longer current | diagnostic | `ob status --output json` | -| `backup_conflict` | yes | another protected-service operation holds the serialization boundary | diagnostic | `ob status --output json` | -| `backup_driver_unsupported` | yes | the service driver has no qualified executable protection contract | diagnostic | `ob validate --output json` | -| `backup_encryption_unverified` | yes | the selected protection destination cannot prove its required encryption mode | diagnostic | `ob validate --output json` | -| `backup_interruption_not_authorized` | yes | the recovery contract requires a recurring stopped-service window the author did not permit | diagnostic | `ob validate --output json` | -| `backup_retention_unsupported` | yes | the declared recovery history cannot map to qualified native retention semantics | diagnostic | `ob validate --output json` | -| `backup_stale` | reserved | the latest recoverable point is older than policy permits | next | `ob plan --output json` | -| `backup_target_not_independent` | yes | the backup target shares the protected failure domain | diagnostic | `ob validate --output json` | -| `backup_target_unauthorized` | yes | the backup target credentials are unavailable, unsafe, or unauthorized | next | `ob plan --output json` | -| `backup_target_unknown` | yes | the protection policy selects no declared backup target | diagnostic | `ob validate --output json` | -| `backup_target_unreachable` | yes | the selected backup target cannot be reached | next | `ob plan --output json` | -| `disk_pressure_critical` | reserved | a relevant filesystem lacks safe headroom for a space-increasing mutation | diagnostic | `ob status --output json` | -| `drill_deferred_capacity` | reserved | a restore drill was deferred before materialization because aggregate staging headroom is insufficient | diagnostic | `ob status --output json` | -| `external_service_not_owned` | reserved | the requested lifecycle mutation targets a dependency Onebox does not own | diagnostic | `ob status --output json` | -| `external_service_state_stale` | reserved | an external-service observation changed after planning | next | `ob plan --output json` | -| `protected_service_identity_changed` | yes | a protected service name would orphan durable recovery identity | diagnostic | `ob validate --output json` | -| `protected_service_patch_incompatible` | reserved | the candidate protected service or helper cannot prove repository and runtime compatibility | diagnostic | `ob status --output json` | -| `protected_service_patch_unsupported` | yes | no exact qualified protected current-to-candidate transition exists | diagnostic | `ob status --output json` | -| `protection_disable_pending` | yes | protection removal is waiting for an authorized safe prerequisite reversal | diagnostic | `ob status --output json` | -| `protection_disablement_not_authorized` | yes | protection disablement requires a fresh local confirmation bound to current state | diagnostic | `ob status --output json` | -| `protection_disablement_overdue` | yes | protection disablement remains pending beyond its action deadline | diagnostic | `ob status --output json` | -| `protection_enablement_restart_not_authorized` | reserved | a restart-bound protection prerequisite lacks fresh local confirmation | diagnostic | `ob validate --output json` | -| `protection_image_revert_unsafe` | yes | the requested image reversion would strand an effective protection prerequisite | diagnostic | `ob status --output json` | -| `protection_image_update_overdue` | reserved | a qualified protected service image publication missed its maintenance target | diagnostic | `ob status --output json` | -| `protection_prerequisite_drifted` | reserved | a live prerequisite no longer matches the verified protection configuration | diagnostic | `ob validate --output json` | -| `protection_service_image_unpublished` | yes | no qualified immutable protection image is published for the observed service base | diagnostic | `ob status --output json` | -| `protection_service_patch_available` | reserved | a qualified exact protected service image transition is available | resolving | `ob service apply --output ndjson` | -| `protection_service_patch_required` | reserved | protection enablement requires a separate qualified same-major service patch first | resolving | `ob service apply --output ndjson` | -| `recovery_objective_unsupported` | yes | the selected driver, version, or target cannot execute the declared recovery kind | diagnostic | `ob validate --output json` | -| `replay_continuity_broken` | reserved | the native replay sequence has a gap inside the required recovery window | next | `ob plan --output json` | -| `restore_drill_schedule_too_sparse` | yes | the restore-drill cadence cannot keep restore proof current | diagnostic | `ob validate --output json` | -| `restore_state_stale` | reserved | live service, volume, or repository state changed after restore planning | diagnostic | `ob status --output json` | -| `service_image_digest_unavailable` | yes | the exact immutable service image required by recovery is unavailable | diagnostic | `ob status --output json` | -| `service_image_patch_disable_pending` | yes | service image refresh is refused while safe protection disablement is pending | diagnostic | `ob status --output json` | -| `service_major_upgrade_unsupported` | reserved | the requested service image transition crosses an unsupported major version | diagnostic | `ob status --output json` | +| Code | Means | Guidance role | Command | +| --- | --- | --- | --- | +| `backup_conflict` | another protected-service operation holds the serialization boundary | diagnostic | `ob status --output json` | +| `backup_disable_pending` | backup removal is waiting for an authorized safe prerequisite reversal | diagnostic | `ob status --output json` | +| `backup_disablement_overdue` | backup disablement remains pending beyond its action deadline | diagnostic | `ob status --output json` | +| `backup_driver_unsupported` | the service driver has no qualified executable backup contract | diagnostic | `ob validate --output json` | +| `backup_encryption_unverified` | the selected backup destination cannot prove its required encryption mode | diagnostic | `ob validate --output json` | +| `backup_image_revert_unsafe` | the requested image reversion would strand an effective backup prerequisite | diagnostic | `ob status --output json` | +| `backup_interruption_not_authorized` | the recovery contract requires a recurring stopped-service window the author did not permit | diagnostic | `ob validate --output json` | +| `backup_retention_unsupported` | the declared recovery history cannot map to qualified native retention semantics | diagnostic | `ob validate --output json` | +| `backup_service_image_unpublished` | no qualified immutable backup image is published for the observed service base | diagnostic | `ob status --output json` | +| `backup_target_not_independent` | the backup target shares the protected failure domain | diagnostic | `ob validate --output json` | +| `backup_target_unknown` | the backup policy selects no declared backup target | diagnostic | `ob validate --output json` | +| `drill_schedule_too_sparse` | the declared drill cadence is too sparse to keep restore proof within its maximum age | diagnostic | `ob validate --output json` | +| `recovery_objective_unsupported` | the selected driver, version, or target cannot execute the declared recovery kind | diagnostic | `ob validate --output json` | +| `service_image_digest_unavailable` | the exact immutable service image required by recovery is unavailable | diagnostic | `ob status --output json` | +| `service_image_patch_disable_pending` | service image refresh is refused while safe backup disablement is pending | diagnostic | `ob status --output json` | +| `service_patch_unsupported` | no exact qualified protected current-to-candidate transition exists | diagnostic | `ob status --output json` | diff --git a/site/src/content/docs/reference/fields/backup_targets.mdx b/site/src/content/docs/reference/fields/backup_targets.mdx index 58c0e56f..3fc76e71 100644 --- a/site/src/content/docs/reference/fields/backup_targets.mdx +++ b/site/src/content/docs/reference/fields/backup_targets.mdx @@ -1,24 +1,18 @@ --- title: "backup_targets" -description: "User-owned off-host S3-compatible repositories available to service protection policies. Accepted by the loader; not yet executable." -summary: "User-owned off-host S3-compatible repositories available to service protection policies. Accepted by the loader; not yet executable." -status: schema-only +description: "User-owned off-host S3-compatible repositories a protected service writes its backups to. Executable for the postgres driver; every other driver refuses a policy rather than accepting one it cannot honour." +summary: "User-owned off-host S3-compatible repositories a protected service writes its backups to. Executable for the postgres driver; every other driver refuses a policy rather than accepting one it cannot honour." +status: shipped generated: true sidebar: order: 200 read_when: - - "Evaluating the proposed protection layer" + - "Declaring where a database's backups go" - "Understanding why Onebox refuses a backup target that shares the protected host" --- {/* Generated by cmd/ob-docgen. Do not edit by hand. */} -:::caution[Accepted, not yet executable] -The loader validates this block and it is published in the JSON Schema, so your -editor will complete it. The behaviour behind it is an open proposal. Declaring -it changes nothing on the target. -::: - This page is generated from the same Go declarations the loader enforces, so it cannot drift from what `ob validate` accepts. @@ -38,14 +32,14 @@ cannot drift from what `ob validate` accepts. | `.credentials.secret_key_entry` | string | — | Variable name containing the destination secret key. Expects a variable name of letters, digits and underscores, not starting with a digit. | | `.credentials.session_token_entry` | string | — | Optional variable name containing a temporary destination session token. Expects a variable name of letters, digits and underscores, not starting with a digit. | | `.encryption` | object | — | Required encryption mode for each recovery kind this target may store. | -| `.encryption.cold` | `client-side` · `archive-password` · `server-side-sse` | — | Encryption mode required for cold recovery: client-side, archive-password, or server-side-sse. | -| `.encryption.pitr` | `client-side` · `archive-password` · `server-side-sse` | — | Encryption mode required for point-in-time recovery: client-side, archive-password, or server-side-sse. | -| `.encryption.snapshot` | `client-side` · `archive-password` · `server-side-sse` | — | Encryption mode required for snapshot recovery: client-side, archive-password, or server-side-sse. | +| `.encryption.cold` | `client-side` · `server-side` | — | Encryption mode required for cold recovery: client-side or server-side. | +| `.encryption.pitr` | `client-side` · `server-side` | — | Encryption mode required for point-in-time recovery: client-side or server-side. | +| `.encryption.snapshot` | `client-side` · `server-side` | — | Encryption mode required for snapshot recovery: client-side or server-side. | | `.endpoint` | string | — | Destination API endpoint. HTTPS is required unless tls is explicitly insecure. Expects an http or https URL. | | `.failure_domain` | object | — | Operator-declared identity used to prove the destination does not share the protected host. | | `.failure_domain.host` | string | — | Destination host identity used to refuse a target on the protected host. Expects a stable identifier of letters, digits, dots, colons, slashes, underscores and hyphens. | | `.failure_domain.identity` | string | — | Stable operator-owned failure-domain identity, distinct from the protected host. Expects a stable identifier of letters, digits, dots, colons, slashes, underscores and hyphens. | | `.kind` | `s3-compatible` | — | Destination kind. Only s3-compatible is supported. | -| `.prefix` | string | — | Non-secret object prefix reserved for Onebox protection data. Expects a relative object prefix with no empty leading component or shell metacharacter. | +| `.prefix` | string | — | Non-secret object prefix reserved for Onebox backup data. Expects a relative object prefix with no empty leading component or shell metacharacter. | | `.region` | string | — | S3-compatible region when the endpoint requires one. Expects a lower-case S3-compatible region of letters, digits and hyphens. | -| `.tls` | `required` · `insecure` | `required` | TLS verification policy: required or insecure. | +| `.tls` | `verify` · `skip-verify` | `verify` | Transport policy: verify, or skip-verify to accept a plaintext http endpoint. | diff --git a/site/src/content/docs/reference/fields/checks.mdx b/site/src/content/docs/reference/fields/checks.mdx new file mode 100644 index 00000000..1a7f6bf2 --- /dev/null +++ b/site/src/content/docs/reference/fields/checks.mdx @@ -0,0 +1,48 @@ +--- +title: "checks" +description: "What must be true before a release becomes current, grouped by kind: external URLs, in-workload HTTP or exec probes, or migration revision evidence." +summary: "What must be true before a release becomes current, grouped by kind: external URLs, in-workload HTTP or exec probes, or migration revision evidence." +status: shipped +generated: true +sidebar: + order: 60 +read_when: + - "Gating release activation on a health endpoint or a smoke test" +--- + +{/* Generated by cmd/ob-docgen. Do not edit by hand. */} + +This page is generated from the same Go declarations the loader enforces, so it +cannot drift from what `ob validate` accepts. + +## Fields on this page + +`advisory` · `applied_revisions` · `contains` · `equals` · `exec` · `http` · `job` · `json_assertions` · `migrations` · `path` · `port` · `provider` · `required_headers` · `run` · `status_codes` · `url` · `workload` + +## Reference + +| Field | Type | Default | What it does | +| --- | --- | --- | --- | +| `exec` | list | — | Commands run inside a named workload. | +| `exec[].advisory` | boolean | `false` | Report a failure without blocking release activation. | +| `exec[].run` | string | — | Shell command verified inside the workload. | +| `exec[].workload` | string | — | Workload the command runs inside. | +| `http` | list | — | HTTP paths probed inside a named workload. | +| `http[].advisory` | boolean | `false` | Report a failure without blocking release activation. | +| `http[].path` | string | — | HTTP path verified inside the workload. Expects a path beginning with /. | +| `http[].port` | integer | — | Container port to probe. | +| `http[].workload` | string | — | Workload the path is probed inside. | +| `migrations` | list | — | Migration revisions checked against captured job evidence. | +| `migrations[].advisory` | boolean | `false` | Report a failure without blocking release activation. | +| `migrations[].applied_revisions` | list | — | Revisions the job must report as applied. | +| `migrations[].job` | string | — | Job workload whose captured evidence is checked. | +| `migrations[].provider` | string | — | Migration tool that produced the revisions. | +| `url` | list | — | External URLs probed from the operator side. | +| `url[].advisory` | boolean | `false` | Report a failure without blocking release activation. | +| `url[].contains` | string | — | Text the response body must contain. | +| `url[].json_assertions` | list | — | Scalar JSON response values that must match exactly. | +| `url[].json_assertions[].equals` | — | — | Exact scalar value required at path. | +| `url[].json_assertions[].path` | string | — | Dot-separated path to a scalar value in the JSON response. | +| `url[].required_headers` | map | — | Exact response headers required for success. | +| `url[].status_codes` | list | — | Allowed response status codes. A successful 2xx response is expected when omitted. | +| `url[].url` | string | — | External HTTP or HTTPS URL verified from the operator side. Expects an http or https URL. | diff --git a/site/src/content/docs/reference/fields/environments.mdx b/site/src/content/docs/reference/fields/environments.mdx index b2b25939..66d93866 100644 --- a/site/src/content/docs/reference/fields/environments.mdx +++ b/site/src/content/docs/reference/fields/environments.mdx @@ -23,7 +23,7 @@ cannot drift from what `ob validate` accepts. ## Fields on this page -`allow_agent_proposals` · `base_path` · `env_files` · `file` · `host` · `migration_backup_key_material` · `migration_backup_maximum_age` · `minimum_onebox_version` · `minimum_plan_schema` · `overrides` · `policy` · `port` · `provider` · `require_approval` · `require_migration_backup` · `require_migration_restore_test` · `server` · `services` · `user` · `workloads` +`allow_agent_proposals` · `backup_key_material` · `backup_max_age` · `base_path` · `env_files` · `file` · `host` · `migrations` · `min_onebox_version` · `min_plan_schema` · `overrides` · `policy` · `port` · `provider` · `require_approval` · `require_backup` · `require_restore_test` · `server` · `services` · `user` · `workloads` ## Reference @@ -38,13 +38,14 @@ cannot drift from what `ob validate` accepts. | `.overrides.workloads` | map | — | Allowed workload tuning keyed by workload name: replicas, resources, env, env_files, strategy, and routes. | | `.policy` | object | — | Approval, runner compatibility, and migration-backup requirements for this environment. | | `.policy.allow_agent_proposals` | boolean | `true` | Declared permission for agent-authored proposals. The current CLI does not distinguish agent identity; execution remains approval-gated. | -| `.policy.migration_backup_key_material` | list | — | Names of key material whose usability must be covered by the migration backup report. | -| `.policy.migration_backup_maximum_age` | string | `24h` | Maximum age of a backup report accepted for a migration. Expects a duration such as 30s, 5m, 1h30m or 14d. | -| `.policy.minimum_onebox_version` | string | — | Oldest released Onebox runner allowed to operate this environment. Expects a CalVer release such as v2026.8.0. | -| `.policy.minimum_plan_schema` | string | — | Oldest executable plan schema accepted by this environment. Expects a plan schema identity such as onebox.run/executable-deploy-plan/v1alpha2. | +| `.policy.migrations` | object | — | What this environment requires of a release carrying migration risk. | +| `.policy.migrations.backup_key_material` | list | — | Key-material identities the backup report must name. | +| `.policy.migrations.backup_max_age` | string | `24h` | Maximum age of a backup report accepted for a migration. Expects a duration such as 30s, 5m, 1h30m or 14d. | +| `.policy.migrations.require_backup` | boolean | `false` | Require a plan-bound backup report before a release with migration risk. | +| `.policy.migrations.require_restore_test` | boolean | `false` | Require the backup report to state that a restore test succeeded. | +| `.policy.min_onebox_version` | string | — | Oldest released Onebox runner allowed to operate this environment. Expects a CalVer release such as v2026.8.0. | +| `.policy.min_plan_schema` | string | — | Oldest executable plan schema accepted by this environment. Expects a plan schema identity such as onebox.run/executable-deploy-plan/v1alpha2. | | `.policy.require_approval` | boolean | `true` | Require a plan-bound local confirmation before mutating this environment. | -| `.policy.require_migration_backup` | boolean | `false` | Require a plan-bound backup report before a release with migration risk. | -| `.policy.require_migration_restore_test` | boolean | `false` | Require the backup report to state that a restore test succeeded. | | `.server` | object | — | SSH server, written as user@host or as an object with host, user, and port. Also accepts user@host. | | `.server.host` | string | — | SSH hostname or IP address. | | `.server.port` | integer | — | SSH port. The SSH default is used when omitted. | diff --git a/site/src/content/docs/reference/fields/external_services.mdx b/site/src/content/docs/reference/fields/external_services.mdx index 41062ef1..25b2a7dd 100644 --- a/site/src/content/docs/reference/fields/external_services.mdx +++ b/site/src/content/docs/reference/fields/external_services.mdx @@ -1,7 +1,7 @@ --- title: "external_services" -description: "Typed dependencies operated outside Onebox, whose lifecycle and protection stay external. Accepted by the loader; not yet executable." -summary: "Typed dependencies operated outside Onebox, whose lifecycle and protection stay external. Accepted by the loader; not yet executable." +description: "Typed dependencies operated outside Onebox, whose lifecycle and backups stay external. Accepted by the loader; not yet executable." +summary: "Typed dependencies operated outside Onebox, whose lifecycle and backups stay external. Accepted by the loader; not yet executable." status: schema-only generated: true sidebar: @@ -23,12 +23,13 @@ cannot drift from what `ob validate` accepts. ## Fields on this page -`connection` · `driver` · `entries` · `file` · `kind` · `maximum_age` · `probe` · `protection_owner` · `provider` · `source` · `timeout` +`backup_owner` · `connection` · `driver` · `entries` · `file` · `kind` · `max_age` · `probe` · `provider` · `source` · `timeout` ## Reference | Field | Type | Default | What it does | | --- | --- | --- | --- | +| `.backup_owner` | string | — | Operator or provider responsible for backup, restore, upgrades, credentials, and durability. Expects a stable operator or provider identity of letters, digits, dots, @, colons, slashes, underscores and hyphens. | | `.connection` | object | — | Trusted connection source and driver-shaped entry mapping. | | `.connection.entries` | map | — | Maps driver connection parts such as host, port, user, password, database, or url to variable names in the trusted source. | | `.connection.source` | object | — | Trusted encrypted file containing the connection values. | @@ -37,6 +38,5 @@ cannot drift from what `ob validate` accepts. | `.driver` | `clickhouse` · `mariadb` · `meilisearch` · `minio` · `mongodb` · `mysql` · `nats` · `postgres` · `rabbitmq` · `redis` · `valkey` | — | Built-in connection shape used to validate and project this dependency. | | `.probe` | object | — | Optional bounded read-only health observation; it never creates or repairs provider resources. | | `.probe.kind` | `driver-health` | `driver-health` | Read-only observation kind: driver-health. | -| `.probe.maximum_age` | string | `5m` | Maximum age of a probe observation bound into a plan. Expects a duration such as 30s, 5m, 1h30m or 14d. | +| `.probe.max_age` | string | `5m` | Maximum age of a probe observation bound into a plan. Expects a duration such as 30s, 5m, 1h30m or 14d. | | `.probe.timeout` | string | `5s` | Maximum duration of one read-only probe. Expects a duration such as 30s, 5m, 1h30m or 14d. | -| `.protection_owner` | string | — | Operator or provider responsible for backup, restore, upgrades, credentials, and durability. Expects a stable operator or provider identity of letters, digits, dots, @, colons, slashes, underscores and hyphens. | diff --git a/site/src/content/docs/reference/fields/observability.mdx b/site/src/content/docs/reference/fields/observability.mdx deleted file mode 100644 index e4f2fc9a..00000000 --- a/site/src/content/docs/reference/fields/observability.mdx +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: "observability" -description: "Declared logging, metric and alerting intent. Validated and planned, but the local engine runs nothing continuous for it." -summary: "Declared logging, metric and alerting intent. Validated and planned, but the local engine runs nothing continuous for it." -status: intent-only -generated: true -sidebar: - order: 110 -read_when: - - "Recording observability intent that another system will act on" ---- - -{/* Generated by cmd/ob-docgen. Do not edit by hand. */} - -:::note[Intent only] -These values are validated and carried into plans, but the local engine runs no -continuous collectors or alert managers. Status reports them as declared, not -managed. -::: - -This page is generated from the same Go declarations the loader enforces, so it -cannot drift from what `ob validate` accepts. - -## Fields on this page - -`alerts` · `enabled` · `logs` · `metrics` · `retention` · `unhealthy_after` - -## Reference - -| Field | Type | Default | What it does | -| --- | --- | --- | --- | -| `alerts` | object | — | Declared alerting intent. Continuous management is not currently provided. | -| `alerts.unhealthy_after` | string | — | Desired duration of unhealthy state before alerting. Expects a duration such as 30s, 5m, 1h30m or 14d. | -| `logs` | object | — | Declared log-retention intent. Continuous management is not currently provided. | -| `logs.enabled` | boolean | `false` | Declare that log collection is desired. | -| `logs.retention` | string | — | Desired log-retention period. Expects a duration such as 30s, 5m, 1h30m or 14d. | -| `metrics` | object | — | Declared metric-collection intent. Continuous management is not currently provided. | -| `metrics.enabled` | boolean | `false` | Declare that metric collection is desired. | diff --git a/site/src/content/docs/reference/fields/services.mdx b/site/src/content/docs/reference/fields/services.mdx index eea72167..15734d47 100644 --- a/site/src/content/docs/reference/fields/services.mdx +++ b/site/src/content/docs/reference/fields/services.mdx @@ -18,32 +18,31 @@ cannot drift from what `ob validate` accepts. ## Fields on this page -`allow_backup_interruption` · `cpus` · `cron` · `driver` · `maximum_data_loss` · `memory` · `minimum_generations` · `mode` · `persistence` · `proof_maximum_age` · `protection` · `recovery_kind` · `recovery_window` · `resources` · `restore_drill` · `retention` · `schedule` · `settings` · `staging_filesystem` · `target` · `timezone` · `version` · `volumes` +`allow_downtime` · `backup` · `cpus` · `cron` · `drill` · `driver` · `keep` · `max_age` · `max_data_loss` · `memory` · `mode` · `persistence` · `recovery_kind` · `resources` · `retention` · `schedule` · `settings` · `target` · `timezone` · `version` · `volumes` · `window` ## Reference | Field | Type | Default | What it does | | --- | --- | --- | --- | +| `.backup` | object | — | Recovery intent for this service. Onebox selects the qualified native implementation; declaring intent alone does not establish backup. | +| `.backup.allow_downtime` | boolean | `false` | Whether recurring backup operations may use the driver-declared stopped-service window. | +| `.backup.drill` | object | — | Exact isolated restore-test schedule, proof age, and optional staging filesystem. | +| `.backup.drill.max_age` | string | `7d` | Maximum age of the latest passing restore proof. Expects a duration such as 30s, 5m, 1h30m or 14d. | +| `.backup.drill.schedule` | object | — | Exact recurring isolated restore-test schedule. | +| `.backup.drill.schedule.cron` | string | — | Five-field cron schedule translated to a host timer. Expects five cron fields. | +| `.backup.drill.schedule.timezone` | string | `UTC` | IANA timezone used to interpret the cron schedule. Expects an IANA zone name such as UTC or Europe/Berlin. | +| `.backup.max_data_loss` | string | — | Maximum tolerable interval between the latest recoverable point and failure. Expects a duration such as 30s, 5m, 1h30m or 14d. | +| `.backup.recovery_kind` | `snapshot` · `pitr` · `cold` | — | Required recovery envelope: snapshot, pitr, or cold. | +| `.backup.retention` | object | — | Portable minimum recovery history that the selected native driver must be able to preserve. | +| `.backup.retention.keep` | integer | `7` | Minimum number of independently recoverable base generations to retain. | +| `.backup.retention.window` | string | `7d` | Minimum continuous recovery history the native retention mapping must preserve. Expects a duration such as 30s, 5m, 1h30m or 14d. | +| `.backup.schedule` | object | — | Exact recurring base-backup schedule. | +| `.backup.schedule.cron` | string | — | Five-field cron schedule translated to a host timer. Expects five cron fields. | +| `.backup.schedule.timezone` | string | `UTC` | IANA timezone used to interpret the cron schedule. Expects an IANA zone name such as UTC or Europe/Berlin. | +| `.backup.target` | string | — | Name of a project-level backup target. Expects lower-case letters, digits and hyphens, starting with a letter, at most 40 characters. | | `.driver` | string | — | Built-in service driver. Defaults to the service map key. Expects lower-case letters, digits and hyphens, starting with a letter, at most 40 characters. | | `.persistence` | object | — | Data-lifetime declaration for this supporting service. | | `.persistence.mode` | `durable` · `ephemeral` · `external` | `durable` | Data lifetime: durable, ephemeral, or external. | -| `.protection` | object | — | Recovery intent for this service. Onebox selects the qualified native implementation; declaring intent alone does not establish protection. | -| `.protection.allow_backup_interruption` | boolean | `false` | Whether recurring backup operations may use the driver-declared stopped-service window. | -| `.protection.maximum_data_loss` | string | — | Maximum tolerable interval between the latest recoverable point and failure. Expects a duration such as 30s, 5m, 1h30m or 14d. | -| `.protection.recovery_kind` | `snapshot` · `pitr` · `cold` | — | Required recovery envelope: snapshot, pitr, or cold. | -| `.protection.restore_drill` | object | — | Exact isolated restore-test schedule, proof age, and optional staging filesystem. | -| `.protection.restore_drill.proof_maximum_age` | string | `7d` | Maximum age of the latest passing restore proof. Expects a duration such as 30s, 5m, 1h30m or 14d. | -| `.protection.restore_drill.schedule` | object | — | Exact recurring isolated restore-test schedule. | -| `.protection.restore_drill.schedule.cron` | string | — | Five-field cron schedule translated to a host timer. Expects five cron fields. | -| `.protection.restore_drill.schedule.timezone` | string | `UTC` | IANA timezone used to interpret the cron schedule. Expects an IANA zone name such as UTC or Europe/Berlin. | -| `.protection.restore_drill.staging_filesystem` | string | — | Absolute filesystem path used for isolated restore materialization instead of the host default. Expects an absolute path with no control character or shell metacharacter. | -| `.protection.retention` | object | — | Portable minimum recovery history that the selected native driver must be able to preserve. | -| `.protection.retention.minimum_generations` | integer | `7` | Minimum number of independently recoverable base generations to retain. | -| `.protection.retention.recovery_window` | string | `7d` | Minimum continuous recovery history the native retention mapping must preserve. Expects a duration such as 30s, 5m, 1h30m or 14d. | -| `.protection.schedule` | object | — | Exact recurring base-backup schedule. | -| `.protection.schedule.cron` | string | — | Five-field cron schedule translated to a host timer. Expects five cron fields. | -| `.protection.schedule.timezone` | string | `UTC` | IANA timezone used to interpret the cron schedule. Expects an IANA zone name such as UTC or Europe/Berlin. | -| `.protection.target` | string | — | Name of a project-level backup target. Expects lower-case letters, digits and hyphens, starting with a letter, at most 40 characters. | | `.resources` | object | — | Memory and CPU limits for this supporting service. | | `.resources.cpus` | string | — | Container CPU limit expressed as a positive decimal count. Expects a number of CPUs such as 0.5 or 2. | | `.resources.memory` | string | — | Container memory limit. Expects a size such as 512MB or 1.5GB. | diff --git a/site/src/content/docs/reference/fields/top-level.mdx b/site/src/content/docs/reference/fields/top-level.mdx index 4ceabfb2..67f666b6 100644 --- a/site/src/content/docs/reference/fields/top-level.mdx +++ b/site/src/content/docs/reference/fields/top-level.mdx @@ -18,7 +18,7 @@ cannot drift from what `ob validate` accepts. ## Fields on this page -`api_version` · `app` · `args` · `base_path` · `build` · `compose` · `context` · `dockerfile` · `domain` · `entrypoint` · `exec` · `health` · `http` · `image` · `interval` · `middlewares` · `path` · `platform` · `port` · `protocol` · `pull` · `reference` · `registry` · `retries` · `routes` · `scheme` · `start_period` · `target` · `tcp` · `tls` · `within` +`api_version` · `app` · `args` · `base_path` · `build` · `compose` · `context` · `dockerfile` · `domain` · `entrypoint` · `exec` · `health` · `http` · `image` · `interval` · `middlewares` · `path` · `port` · `protocol` · `pull` · `reference` · `retries` · `routes` · `scheme` · `start_period` · `target` · `tcp` · `tls` · `within` ## Reference @@ -31,7 +31,6 @@ cannot drift from what `ob validate` accepts. | `build.args` | map | — | Build arguments supplied by the external build system. | | `build.context` | string | — | Repository-relative build context. Expects a path inside the repository, with no control character or shell metacharacter. | | `build.dockerfile` | string | — | Repository-relative Dockerfile path. Expects a path inside the repository, with no control character or shell metacharacter. | -| `build.platform` | string | — | Target image platform for the external build. | | `build.target` | string | — | Named Dockerfile stage to build. | | `compose` | string | — | Existing Compose service to adopt, as repository path#service. Expects a reference of the form path/to/compose.yaml#service. | | `domain` | string | — | Domain shorthand for one HTTPS route; requires port and cannot be combined with routes. | @@ -45,10 +44,8 @@ cannot drift from what `ob validate` accepts. | `health.tcp` | boolean | `false` | Probe the configured port by opening a TCP connection. | | `health.within` | string | — | Maximum time a rollout waits for readiness. Expects a duration such as 30s, 5m, 1h30m or 14d. | | `image` | object | — | Container image source, written as a reference string or an object. Also accepts an image reference. | -| `image.platform` | string | — | Platform selected when the image is multi-platform. | -| `image.pull` | `always` · `missing` · `never` | `missing` | Image pull policy: missing, always, or never. | +| `image.pull` | `always` · `missing` · `never` | `missing` | When to fetch the image from the registry: missing fetches only what the host does not already hold, always fetches every release, never fetches at all and fails on a missing image. | | `image.reference` | string | — | Complete container image reference, optionally tagged or digest-pinned. Expects a registry reference such as nginx:1.27 or ghcr.io/acme/app@sha256:…. | -| `image.registry` | string | — | Optional registry label retained in canonical configuration. Current authentication uses every top-level registries entry; this field does not select a login. | | `port` | integer | — | Container port used with domain shorthand and as the default HTTP health port. | | `routes` | list | — | Ingress routes exposed by this workload. | | `routes[].domain` | string | — | DNS name matched by the proxy. | diff --git a/site/src/content/docs/reference/fields/verifications.mdx b/site/src/content/docs/reference/fields/verifications.mdx deleted file mode 100644 index 6ce23812..00000000 --- a/site/src/content/docs/reference/fields/verifications.mdx +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: "verifications" -description: "What must be true before a release becomes current: external URLs, in-workload checks, or migration revision evidence." -summary: "What must be true before a release becomes current: external URLs, in-workload checks, or migration revision evidence." -status: shipped -generated: true -sidebar: - order: 60 -read_when: - - "Gating release activation on a health endpoint or a smoke test" ---- - -{/* Generated by cmd/ob-docgen. Do not edit by hand. */} - -This page is generated from the same Go declarations the loader enforces, so it -cannot drift from what `ob validate` accepts. - -## Fields on this page - -`advisory` · `applied_revisions` · `contains` · `equals` · `exec` · `http` · `job` · `json_assertions` · `migration_revisions` · `path` · `port` · `provider` · `required_headers` · `status_codes` · `url` · `workload` - -## Reference - -| Field | Type | Default | What it does | -| --- | --- | --- | --- | -| `[].advisory` | boolean | `false` | Report a failed check without blocking release activation. | -| `[].contains` | string | — | Text that the HTTP response body must contain. | -| `[].exec` | string | — | Shell command verified inside the named workload. | -| `[].http` | string | — | HTTP path verified inside the named workload. Expects a path beginning with /. | -| `[].json_assertions` | list | — | Scalar JSON response values that must match exactly. | -| `[].json_assertions[].equals` | — | — | Exact scalar value required at path. | -| `[].json_assertions[].path` | string | — | Dot-separated path to a scalar value in the JSON response. | -| `[].migration_revisions` | object | — | Expected migration provider and applied revisions, checked against captured job evidence. | -| `[].migration_revisions.applied_revisions` | list | — | Ordered migration revisions expected to be applied. | -| `[].migration_revisions.job` | string | — | Migration job whose result evidence is checked. | -| `[].migration_revisions.provider` | string | — | Migration provider expected in the job result. | -| `[].port` | integer | — | Container port used by an internal HTTP verification. | -| `[].required_headers` | map | — | Exact HTTP response headers required for success. | -| `[].status_codes` | list | — | Allowed HTTP response status codes. A successful 2xx response is expected when omitted. | -| `[].url` | string | — | External HTTP or HTTPS URL verified from the operator side. Expects an http or https URL. | -| `[].workload` | string | — | Workload in which an internal HTTP or exec verification runs. | diff --git a/site/src/content/docs/reference/fields/workloads.mdx b/site/src/content/docs/reference/fields/workloads.mdx index 3ac50ca5..57490b74 100644 --- a/site/src/content/docs/reference/fields/workloads.mdx +++ b/site/src/content/docs/reference/fields/workloads.mdx @@ -19,7 +19,7 @@ cannot drift from what `ob validate` accepts. ## Fields on this page -`args` · `bind` · `build` · `command` · `compose` · `condition` · `container` · `context` · `cpus` · `cron` · `data_effect` · `dockerfile` · `domain` · `drain` · `driver` · `entrypoint` · `env` · `env_files` · `exec` · `extra_hosts` · `file` · `grace` · `health` · `host` · `hostname` · `http` · `image` · `init` · `interval` · `labels` · `logging` · `memory` · `middlewares` · `mode` · `name` · `needs` · `options` · `path` · `persistence` · `platform` · `port` · `protocol` · `provider` · `published_ports` · `pull` · `reference` · `registry` · `replicas` · `resources` · `retries` · `role` · `routes` · `schedule` · `scheme` · `signal` · `source` · `start_period` · `stdin_open` · `strategy` · `target` · `tcp` · `timezone` · `tls` · `tty` · `user` · `volumes` · `wait` · `when` · `within` · `working_dir` +`args` · `bind` · `build` · `command` · `compose` · `condition` · `container` · `context` · `cpus` · `cron` · `data_effect` · `dockerfile` · `domain` · `drain` · `driver` · `entrypoint` · `env` · `env_files` · `exec` · `extra_hosts` · `file` · `grace` · `health` · `host` · `hostname` · `http` · `image` · `init` · `interval` · `labels` · `logging` · `memory` · `middlewares` · `mode` · `name` · `needs` · `options` · `path` · `persistence` · `port` · `protocol` · `provider` · `published_ports` · `pull` · `reference` · `replicas` · `resources` · `retries` · `role` · `routes` · `schedule` · `scheme` · `signal` · `source` · `start_period` · `stdin_open` · `strategy` · `target` · `tcp` · `timezone` · `tls` · `tty` · `user` · `volumes` · `wait` · `when` · `within` · `working_dir` ## Reference @@ -29,7 +29,6 @@ cannot drift from what `ob validate` accepts. | `.build.args` | map | — | Build arguments supplied by the external build system. | | `.build.context` | string | — | Repository-relative build context. Expects a path inside the repository, with no control character or shell metacharacter. | | `.build.dockerfile` | string | — | Repository-relative Dockerfile path. Expects a path inside the repository, with no control character or shell metacharacter. | -| `.build.platform` | string | — | Target image platform for the external build. | | `.build.target` | string | — | Named Dockerfile stage to build. | | `.command` | list | — | Container command as a shell string or argument list. Also accepts a command line or argument list. | | `.compose` | string | — | Existing Compose service to adopt, as repository path#service. Expects a reference of the form path/to/compose.yaml#service. | @@ -56,10 +55,8 @@ cannot drift from what `ob validate` accepts. | `.health.within` | string | — | Maximum time a rollout waits for readiness. Expects a duration such as 30s, 5m, 1h30m or 14d. | | `.hostname` | string | — | Hostname assigned inside the workload container. | | `.image` | object | — | Container image source, written as a reference string or an object. Also accepts an image reference. | -| `.image.platform` | string | — | Platform selected when the image is multi-platform. | -| `.image.pull` | `always` · `missing` · `never` | `missing` | Image pull policy: missing, always, or never. | +| `.image.pull` | `always` · `missing` · `never` | `missing` | When to fetch the image from the registry: missing fetches only what the host does not already hold, always fetches every release, never fetches at all and fails on a missing image. | | `.image.reference` | string | — | Complete container image reference, optionally tagged or digest-pinned. Expects a registry reference such as nginx:1.27 or ghcr.io/acme/app@sha256:…. | -| `.image.registry` | string | — | Optional registry label retained in canonical configuration. Current authentication uses every top-level registries entry; this field does not select a login. | | `.init` | boolean | — | Run a minimal init process as PID 1 inside the container. | | `.labels` | map | — | Additional container labels outside namespaces reserved by Onebox and the proxy. | | `.logging` | object | — | Container logging driver and driver-specific options. | diff --git a/site/src/content/docs/reference/naming.mdx b/site/src/content/docs/reference/naming.mdx index 3b07877a..005ce0ad 100644 --- a/site/src/content/docs/reference/naming.mdx +++ b/site/src/content/docs/reference/naming.mdx @@ -29,10 +29,10 @@ map named for the thing it configures. | Plural | Singular | | --- | --- | -| `workloads` `services` `registries` `environments` `hooks` `notifications` `backup_targets` `external_services` | `runtime` `deployment` `proxy` `observability` | +| `workloads` `services` `registries` `environments` `hooks` `notifications` `backup_targets` `external_services` | `runtime` `deployment` `proxy` | Arrays are plural too: `routes`, `env_files`, `volumes`, `published_ports`, -`needs`, `verifications`, `env_checks`. Four are not, because the singular reads +`needs`, `checks`, `env_checks`. Four are not, because the singular reads as the thing being stated rather than a list: `order`, `on`, `require`, `present`. @@ -142,7 +142,7 @@ New codes should be subject-first. ``` backup_target_unreachable protection_disable_pending -restore_drill_schedule_too_sparse +drill_schedule_too_sparse routing_incomplete ``` @@ -154,7 +154,7 @@ are pinned: | Word | Means | Never means | | --- | --- | --- | | **server** | the machine you deploy to, in the project file and in every artifact | a backup repository | -| **target** | a backup destination — `backup_targets`, `protection.target` | the machine you deploy to | +| **target** | a backup destination — `backup_targets`, `backup.target` | the machine you deploy to | | **workload** | a container Onebox runs from your declaration | a supporting service, a Compose service | `ob exec` and `ob logs` take a `` because those are two diff --git a/site/src/content/docs/reference/policies.mdx b/site/src/content/docs/reference/policies.mdx index c2f02c35..2ba7d01c 100644 --- a/site/src/content/docs/reference/policies.mdx +++ b/site/src/content/docs/reference/policies.mdx @@ -25,8 +25,8 @@ release. `ob version` reports which you have. environments: production: policy: - minimum_onebox_version: v2026.8.0 - minimum_plan_schema: onebox.run/executable-deploy-plan/v1alpha2 + min_onebox_version: v2026.8.0 + min_plan_schema: onebox.run/executable-deploy-plan/v1alpha2 ``` `ob doctor` reports whether the runner selected by `PATH` is compatible. @@ -96,8 +96,8 @@ redacted. | Class | JSON | NDJSON | Commands | | --- | --- | --- | --- | -| Finite envelope | yes | no | `ob approve` · `ob audit` · `ob canonical` · `ob doctor` · `ob eject` · `ob init` · `ob job plan` · `ob plan` · `ob preflight` · `ob preview` · `ob schema` · `ob secrets list` · `ob status` · `ob validate` · `ob version` | -| Finite operation stream | yes | yes | `ob abort` · `ob bootstrap` · `ob deploy` · `ob destroy` · `ob job run` · `ob proxy apply` · `ob resume` · `ob rollback` · `ob secrets push` · `ob service apply` | +| Finite envelope | yes | no | `ob approve` · `ob audit` · `ob backup status` · `ob canonical` · `ob doctor` · `ob eject` · `ob init` · `ob job plan` · `ob plan` · `ob preflight` · `ob preview` · `ob schema` · `ob secrets list` · `ob status` · `ob validate` · `ob version` | +| Finite operation stream | yes | yes | `ob abort` · `ob backup create` · `ob backup enable` · `ob backup disable` · `ob backup drill` · `ob backup prune` · `ob backup restore` · `ob backup verify` · `ob bootstrap` · `ob deploy` · `ob destroy` · `ob job run` · `ob proxy apply` · `ob resume` · `ob rollback` · `ob secrets push` · `ob service apply` | | Operator passthrough | finite only | yes | `ob logs` | | Operator passthrough | no | yes | `ob exec` | | Trusted editor | yes, after exit | no | `ob secrets edit` | diff --git a/site/src/content/docs/reference/project-file.mdx b/site/src/content/docs/reference/project-file.mdx index 0abdfa35..5acdd203 100644 --- a/site/src/content/docs/reference/project-file.mdx +++ b/site/src/content/docs/reference/project-file.mdx @@ -39,10 +39,10 @@ published reference on the first line of a scaffolded project. | `proxy` | Who runs the proxy and what routes. | [proxy](/reference/fields/proxy) | | `runtime` | Environment files and local environment-file checks. | [runtime](/reference/fields/runtime) | | `hooks` | Commands at lifecycle seams. | [hooks](/reference/fields/hooks) | -| `verifications` | What must be true for a release to activate. | [verifications](/reference/fields/verifications) | +| `checks` | What must be true for a release to activate, grouped by kind. | [checks](/reference/fields/checks) | | `external_services` | Dependencies operated outside Onebox, and how a workload reaches them. | [external_services](/reference/fields/external_services) | -| `backup_targets` | Off-host repositories the proposed protection layer would write to. | [backup_targets](/reference/fields/backup_targets) | -| `registries` `notifications` `observability` | Named maps. | [registries](/reference/fields/registries) · [notifications](/reference/fields/notifications) · [observability](/reference/fields/observability) | +| `backup_targets` | Off-host repositories a protected service writes its backups to. | [backup_targets](/reference/fields/backup_targets) | +| `registries` `notifications` | Named maps. | [registries](/reference/fields/registries) · [notifications](/reference/fields/notifications) | `external_services` and `backup_targets` are published in the JSON Schema and accepted by `ob validate`, but the lifecycle behind them is not shipped: diff --git a/site/src/content/docs/start/install.mdx b/site/src/content/docs/start/install.mdx index df1f7bb8..53a2bb17 100644 --- a/site/src/content/docs/start/install.mdx +++ b/site/src/content/docs/start/install.mdx @@ -38,7 +38,7 @@ import { Steps, Tabs, TabItem } from '@astrojs/starlight/components'; ``` `ob doctor` reports whether the runner selected by `PATH` satisfies the - environment's `minimum_onebox_version` and `minimum_plan_schema`, and names + environment's `min_onebox_version` and `min_plan_schema`, and names every workload and service holding durable data that has no backup. Run it from a directory that has an `ob.yml`. Outside a project it reports @@ -179,7 +179,7 @@ just build see the available build, test, formatting and check targets. :::note[Why a checkout build can be refused] -When an environment sets `minimum_onebox_version`, commit-derived and dirty +When an environment sets `min_onebox_version`, commit-derived and dirty checkout builds **fail closed** — they are not released runners. That is deliberate: an environment that pins a minimum is asking for a runner whose identity can be checked, and a dirty working tree has none. diff --git a/site/src/content/docs/start/reading-it-back.mdx b/site/src/content/docs/start/reading-it-back.mdx index 9fd13034..94e81ea5 100644 --- a/site/src/content/docs/start/reading-it-back.mdx +++ b/site/src/content/docs/start/reading-it-back.mdx @@ -49,7 +49,7 @@ checked against the conformance corpus, so what your editor tells you while you type is what `ob validate` tells you afterwards. :::caution[One place your editor is ahead of the engine] -The published schema includes `backup_targets`, `services..protection` and +The published schema includes `backup_targets`, `services..backup` and `external_services`. Your editor will complete them and `ob validate` accepts them, but the behaviour behind them is not yet executable — see [Shipped vs proposed](/status/capabilities). diff --git a/site/src/content/docs/status/capabilities.mdx b/site/src/content/docs/status/capabilities.mdx index d4cc7855..29ec1ff1 100644 --- a/site/src/content/docs/status/capabilities.mdx +++ b/site/src/content/docs/status/capabilities.mdx @@ -84,25 +84,22 @@ Nothing behind them executes. | Block | Reality today | | --- | --- | | [`backup_targets`](/reference/fields/backup_targets) | Declaring a target creates no repository and no schedule. | -| [`services..protection`](/reference/fields/services) | Declaring recovery intent establishes no protection. No backup is taken, no drill is run, and no service reports `Managed`. | -| [`external_services`](/reference/fields/external_services) | Connections are projected into the generated runtime, and two refusals are wired: an unknown `needs.env` part, and `condition: healthy` against a service with no probe. Health probes do not execute, and the remaining lifecycle refusals are not wired. | - -The [lifecycle failure codes](/reference/errors#lifecycle-failure-codes) are -defined and drift-tested in the binary, but most of the operations that raise -them are not yet executable. -## Intent only +| [`external_services`](/reference/fields/external_services) | Connections are projected into the generated runtime, and two refusals are wired: an unknown `needs.env` part, and `condition: healthy` against a service with no probe. Health probes do not execute, and the remaining lifecycle refusals are not wired. | -[`observability`](/reference/fields/observability) — `logs`, `metrics` and -`alerts` declare desired capability. `ob validate` reads the block. Nothing else -does: the local engine runs no collectors and no alert managers, and `ob status` -does not report the block at all — not even as declared. +Every [lifecycle failure code](/reference/errors#lifecycle-failure-codes) is +raised by a path in the shipped binary, checked against the source in both +directions. ## Not owned at all -**Backups.** Onebox does not take them. `ob doctor` reports the absence for every -workload and service holding durable data, because silence there would read as -approval. +**Workload volumes.** A workload's own durable data is not copied anywhere. +`ob doctor` reports it for every workload holding some, because silence there +would read as approval. Managed services are different: one declaring +`backup` is backed up continuously and recoverable to a point in time — see +the backup reference. Only the `postgres` driver has an executable contract +today; every other driver declares `policy_qualified: false` and its backup +policy is refused rather than accepted and ignored. **Restore proof.** No drill is run, and no service can currently prove a backup would restore.