fix(backup): retry the scheduled base backup before failing the night

[vishr]

What this changes

The scheduled base backup unit (onebox-backup-<service>-backup.service) ran wal-g backup-push as a single ExecStart under flock. One refused multipart part aborts a whole push, and nothing tried again, so a transient fault at the destination lost the night.

Observed on a production host using Hetzner Object Storage (wal-g 3.0.8 from the pinned PostgreSQL image, default upload settings): on three of six consecutive nights a ~14.6 GB push failed 4–9 minutes in with

MultipartUpload: upload multipart failed
        upload id: 2~…
caused by: BadRequest: N/A
        status code: 400, request id: , host id:
Unable to continue the backup process because of the loss of a part N.

(parts 4, 6 and 10 on different nights) while the same configuration and credentials succeeded on the other nights and on demand. WAL archiving was unaffected throughout.

Why nothing retried, from the sources:

• wal-g 3.0.8 treats a failed partition upload as fatal for the whole backup (internal/storage_tar_ball.go, Fatalf("Unable to continue the backup process because of the loss of a part %d")).
• wal-g 3.0.8 and 3.0.9 use aws-sdk-go v1 (1.55.5 / 1.55.8). The v1 default retryer retries 5xx, 429, throttling codes and connection errors; an HTTP 400 with code BadRequest is not retryable (aws/request/retryer.go). wal-g's own retryer (pkg/storages/s3/retryer.go) only adds "connection reset by peer" / "connection refused". WALG_S3_MAX_RETRIES (default 15) changes how many times a retryable error is retried, not which errors are.
• wal-g PR #2570 (merged to master 2026-09-17, after the aws-sdk-go-v2 migration) retries HTTP 400 responses whose body is not parseable XML. It is in no release, and this error parsed as a well-formed BadRequest, so it would not obviously cover it either.
• The response carried no request id and no host id, which Ceph RGW stamps on every response it generates; the 400 most likely comes from the edge in front of the cluster (inference). Hetzner's documented limits (750 req/s, 256 parallel sessions per source IP) are far above what wal-g's defaults produce here (at most 2 partitions × 16 part uploads in flight), so a rate limit is unlikely (inference). The provider documents 503, not 400, for overload.

So there is no documented wal-g or SDK knob in a released version that makes this part upload retry. This PR adds the layer Onebox owns: a bounded retry of the whole push.

The backup unit now runs a host-side script, installed beside the existing verify script (backup-<service>.sh), that attempts backup-push up to three times, a minute and then five minutes apart, and exits with the last attempt's status. Retention stays a separate ExecStart, so it never runs after a push that failed every attempt. The flock still wraps the whole script, retries included, so an interactive ob backup waits rather than interleaving.

It is a script rather than Restart= because systemd re-runs every ExecStart on a restart (retention included), and its start rate limiter bounds restarts by time rather than count, so a push slower than the limiter's window would restart forever.

No issue was filed first: this is a bug fix diagnosed from the host's journal.

Closes #

Why this is correct

• TestTheScheduledBaseBackupRetriesBeforeFailingTheUnit executes the rendered script under sh with a stand-in push that fails a chosen number of times: one run when the first attempt succeeds, two when the first fails, three runs and the push's own exit status (7) when every attempt fails, with each attempt's reason on stderr. It also checks the installed script runs the real docker exec … backup-push, carries the real waits (set -- 60 300, attempts=3), has no set -e (which would leave the loop on the first failure), and passes sh -n. It fails against the previous unit, which had no script.
• TestTheScheduledBackupUnitRetriesThePushAndNotTheRetention runs SyncBackupSchedules against a protected PostgreSQL fixture and asserts the unit's first ExecStart is the script under the same flock, that delete retain follows as its own ExecStart, that the unit itself contains no backup-push and no Restart=, and that the installed script pushes this service's container.
• Ran: just check (fmt, vet, full go test ./..., generated docs, site build). Not run: the Docker e2e suite; no change touches a path it covers differently.

Effect on the safety envelope

A scheduled push may now run up to three times per firing, and the backup flock can be held about six minutes longer between attempts, so an interactive ob backup started in that window waits longer. Nothing new is done to a running system; retention, the verify unit and the interactive commands are unchanged. Existing hosts get the new unit and script on their next ob service apply or ob backup enable; until then they keep the old unit. No /status/capabilities change.

Checklist

• just check passes locally.
• Tests cover the new behaviour, including the failure paths.
• Generated documentation is current (just check verifies this).
• I have accepted the CLA, or will when the bot asks on my first pull request.

https://claude.ai/code/session_01VubhEStkwQngp9TwWiH9T7

[Copilot]

Copilot review overview

🟢 Approval recommended

No unresolved review comments remain.

Review effort: Lite
Findings: None

What changed in this PR

Improves scheduled WAL-G base backups by retrying transient failures before the unit fails.

Changes:

• Adds a host-side retry script with staged delays.
• Runs retention only after a successful backup.
• Adds tests and documentation.

File	Description
site/​src/​content/​docs/​guides/​back-up-a-database.mdx	Documents scheduled backup retries.
internal/​engine/​schedule_test.go	Tests retry behavior and unit ordering.
internal/​engine/​backup_schedule.go	Generates retry scripts and updated units.
internal/​app/​names.go	Adds the retry script path helper.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.