Field-test backlog: open findings from three throwaway-host runs

[vishr]

Open items from deploying to three throwaway Hetzner hosts (gitea, a purpose-built feature project, and ghost on managed MySQL). Everything below was reproduced against a live server, not inferred from source. The fixed findings shipped in #30, #32, #33 and #34; these are what is left.

Ordered by what I would do first.

---

1. Network ownership — an app can silently join a stranger's network

Proven on a host. Create a network under a name a project will use, then run preflight:

$ docker network create foo_default        # labels: {}
$ ob preflight
  ok    name collisions   3 derived names, none held by anything else

$ docker compose -p foo up -d              # what ob does
stranger's network id: 3630dcad6cfb…
container joined id  : 3630dcad6cfb…       # joined it

Two causes: Spec.All() does not enumerate <app>_default, so preflight never asks about it; and the network carries no ob.app label, so ownedNames could not attribute it even if asked. Volumes and containers are labelled — this is the one resource class that is not.

Wider than one network. internal/engine/services.go:52 creates the service network with a bare docker network create and no label either. No network onebox makes carries ownership.

The naive fix is destructive — I tried it in #34 and reverted. Declaring networks.default in the generated runtime makes Compose manage it, and the e2e caught it immediately:

Network obe2e_default Removed
Error response from daemon: error while removing network: network
obe2e_default has active endpoints (name:"obe2e-traefik-1")

On a live host that removes the application's network mid-deploy.

What a coherent fix looks like: mark the app network external in the generated runtime, create it out of band the way the service network already is, label every network onebox creates with ob.app, and add it to Spec.All(). Needs care for hosts whose networks already exist unlabelled — treating an unlabelled network at a derived name as foreign would break every existing deployment at preflight.

---

2. ob bootstrap installs Docker outside the lock and the journal

internal/engine/bootstrap.go:44 runs, as the SSH user (root in the normal case):

curl -fsSL https://get.docker.com | sh && systemctl enable --now docker

It fires only when docker version fails, and it is announced in the output. But the block introducing it says "one regime for every mutation: bootstrap locks, fences, and journals like a deploy" — and this runs at line 44, while AcquireLock is line 61 and the first journal write is line 70. So the most privileged action onebox performs is the one that regime does not cover:

• not journaled — nothing records that a runtime was installed, or which version; ob audit shows only bootstrap … bootstrapped
• not locked — two concurrent ob bootstrap runs both reach the install
• no opt-out — no policy field or flag means "refuse if no runtime is present rather than install one"
• not documented — start/install.mdx says bootstrap "prepares the host"; that it may pipe a remote script into a root shell appears on no page

It is also the only path with no plan, no approval and no digest, while ob deploy refuses to run without a digest-bound plan because what was reviewed is what executes. get.docker.com is by definition not what was reviewed.

Suggested: move the install inside the lock, journal the resolved Docker version, document it on the page that introduces ob bootstrap, and add an opt-out for operators whose host provisioning is someone else's job.

---

3. Every deploy recreates every workload, databases included

Changing one environment variable on an application workload recreated the database container too:

deploy 1: waiting 735d15bb4bb1 → healthy
deploy 2: waiting 55a3b5f110cb → healthy   # different container

The plan diff showed the only change to db was its ob.release label. Because that label is stamped on every service, no container's config hash is ever unchanged between releases, so Compose recreates all of them. For a database that is a dropped-connection event on every unrelated deploy.

engine.OnlyReleaseLabelsChanged already exists and is used to decide whether a whole deploy is a no-op — the same idea per workload would fix this. Alternatively omit ob.release from workloads whose rendered definition is otherwise identical; the release is already recorded in the journal and the current symlink.

---

4. ob canonical no longer shows inferred durability

Accepted knowingly in #33 rather than papered over. The contract publishes persistence.mode defaulting to durable; a workload with a managed named volume is now treated as durable by ob doctor and the migration-backup requirement, but the document is not edited, so ob canonical does not show the inference.

Materialising it into the document was tried and reverted: the "this was inferred" exemption is in-memory state, and deepCopy round-trips through JSON, so Resolve refused projects that had loaded — blaming a replicas override nobody wrote. Making it visible and consistent needs either accepting a tightened constraint or teaching canonical to annotate a value that is not in the document.

---

5. The hook environment is a public contract documented nowhere

A local hook receives OB_APP, OB_SERVER, OB_HOST, OB_SSH_USER, OB_SSH_PORT, OB_RELEASE_DIR, OB_RELEASE_ID (internal/engine/recreate.go:127-134). People write scripts against these. No page in site/src/content/docs lists them.

---

6. The example corpus barely exercises the contract

Across all eleven apps in e2e/apps:

Feature	Apps using it
role: job, data_effect, schedule	0 / 11
verifications, notifications, hooks	0 / 11
env_files (secrets), registries	0 / 11
protection, backup_targets, external_services	0 / 11
replicas, strategy	0 / 11

So a rolling deploy was never exercised on a host until ghost, and migrations, schedules and the backup gate needed a project written by hand. Three hosts produced three different classes of finding precisely because each reached contract the previous could not; a fourth app declaring the same fields as gitea would find nothing.

Worth adding one or two corpus projects that use the contract, rather than more apps that use the same tenth of it.

---

Smaller

• LICENSE is absent. The blocker for going public, and the reason the onebox.run/v1 renames were time-sensitive.
• app.SchemaID points at raw.githubusercontent.com rather than https://onebox.run/onebox.run-v1.schema.json.
• The facts manifest has no published JSON Schema. ob backup-evidence template (fix: make durable data mean one thing, and the backup gate satisfiable #33) closes the authoring gap, and the manifest is strictly decoded, but a schema file would let editors and CI validate it. Blocked on reusing the schema generator across packages.
• ob exec free-text flag guard is partial. TestEveryFlagNamedInAnErrorStringExistsOnThatCommand matches the backticked `ob cmd --flag` form only; prose like "re-run with --allow-destructive-mounts" is correct but unguarded.
• 19 instances of "that is the point" / "the whole point" across code and docs — a stylistic tic, deliberately left alone.