Add production-realistic remote E2E on disposable Hetzner hosts

[vishr]

Summary

Add a production-realistic end-to-end test lane that runs the compiled ob CLI against a disposable Hetzner VM and verifies the application through a public onebox.run hostname.

This complements the existing local Docker E2E; it does not replace it. The current suite is a good deterministic PR gate for deployment choreography, but its primary path uses transport.NewLocal(). The existing e2e/apps/one-app-one-host.sh reaches a real Hetzner host, yet verifies the application by curling a container IP from inside that host. Neither path proves the complete user boundary:

released CLI -> SSH -> bare host -> bootstrap -> managed proxy
             -> public DNS -> HTTP redirect -> TLS -> route -> workload

This is the implementation counterpart to the corpus coverage finding in #35.

Goals

• Exercise the compiled CLI exactly as an operator or LLM would use it; do not call internal engine APIs from the remote test.
• Prove bootstrap and deployment from a bare supported Linux image over real SSH.
• Prove public DNS, HTTP-to-HTTPS redirect, ACME issuance, TLS termination, proxy routing and external reachability.
• Prove lifecycle behavior over the public edge: rolling deployment, health gating, no-op deployment, rollback and interrupted-deploy recovery.
• Exercise a small deterministic feature project plus at least one unrelated real OSS application.
• Produce sanitized, machine-readable evidence when a run fails.
• Bound cost and guarantee eventual cleanup after test cancellation or runner loss.

Non-goals

• Do not replace the required local Docker E2E with an internet-dependent gate.
• Do not make remote E2E required on every pull request initially.
• Do not claim schema-only or intent-only capabilities are operational. In particular, Onebox still does not take backups, run restore drills or install observability collectors.
• Do not put multiple applications on one host. Every scenario receives a fresh VM, preserving the one-application-per-host product boundary.
• Do not use Cloudflare's HTTP proxy in front of the host; that could mask origin TLS, cache responses and obscure downtime.

Infrastructure approach

Stable IP, disposable VM

Reserve one Hetzner Primary IPv4 in the selected location with auto_delete disabled. The test serially assigns that address to a new disposable VM and deletes only the VM afterward. The address remains for the next run.

Initial concurrency is deliberately one. If remote E2E later needs parallelism, add a small explicit pool of reserved addresses rather than making DNS and host allocation implicit.

Permanent wildcard DNS

Create one permanent DNS-only Cloudflare record:

*.e2e.onebox.run  A  <reserved Hetzner IPv4>
proxied: false

Each run then uses a unique, single-label hostname such as:

umami-31953433457.e2e.onebox.run

This gives disposable hostnames without per-run DNS mutations. It avoids placing a Cloudflare API credential in GitHub Actions and eliminates DNS-record cleanup as a failure mode. A Cloudflare token scoped to DNS writes on onebox.run is still authority over the whole zone, not only the e2e prefix; the permanent wildcard avoids that unnecessarily broad CI authority.

The harness must assert before provisioning that the chosen hostname resolves to the reserved address and that the record is DNS-only from the request path's point of view.

Test lanes

Lane	Trigger	Purpose
Existing local Docker E2E	Every PR	Fast, deterministic deploy/recovery regression gate
Remote lifecycle canary	workflow_dispatch and scheduled main	Real SSH, bare host, public DNS/TLS, proxy and CLI lifecycle
OSS compatibility matrix	Weekly/manual	Third-party application compatibility and upstream image drift

The remote workflow should use a repository/environment concurrency group so only one run owns the reserved IP at a time. It should not cancel a running owner to start a newer job.

Harness design

Add a small harness under e2e/remote/, preferably in Go so assertions, timeouts, structured output and cleanup are testable without adding a shell test framework. The harness may invoke hcloud, ssh and the locally built ob binary as subprocesses, but it must not import Onebox execution packages.

Expose one local entry point, for example:

just e2e-remote scenario=canary
just e2e-remote scenario=umami

The harness should use an explicit opt-in such as OB_REMOTE_E2E=1 and fail, rather than skip, when opted in but required tools or credentials are unavailable.

Per-run resource identity

Derive a unique run ID from GitHub run ID/attempt or a local timestamp plus random suffix. Apply it consistently to:

• server name;
• ephemeral SSH key name;
• test hostname;
• Hetzner labels;
• artifact directory and structured report.

Hetzner resources must carry at least:

purpose=onebox-e2e
run=<run-id>
expires=<bounded timestamp>

Never clean up by a broad name prefix alone. Capture exact resource IDs as they are created and delete those IDs.

Remote lifecycle scenario

One canary run should perform the following through the compiled CLI.

1. Provision

1. Build ob from the checked-out commit.
2. Generate an ephemeral SSH keypair on the runner.
3. Register only the public key in the Hetzner project.
4. Create a uniquely named Ubuntu 24.04 VM in the reserved IP's location, with the reserved Primary IP assigned.
5. Wait for SSH with a bounded timeout and record the new host key in an isolated known_hosts file.
6. Render a temporary project configuration containing root@<reserved-ip> and the run hostname.

The private SSH key must never leave the runner or appear in logs/artifacts.

2. Safe and planning surfaces

Exercise and assert the structured results of:

ob validate
ob canonical --output json
ob preview
ob preflight --output json
ob doctor --output json

These should run before mutation where applicable, making failures attributable to authoring, local runner capability or the remote host.

3. Bootstrap and deploy through the normal authority path

Use the real operator ceremony:

ob bootstrap
ob plan --out <plan>
ob approve --plan <plan> --out <approval>
ob deploy --plan <plan> --approval <approval>

Do not add a test-only bypass around planning or approval.

4. Public ingress assertions

Probe from the CI runner, not from the VM or a container:

• DNS resolves the hostname to the reserved origin IP.
• http://<hostname> redirects to HTTPS.
• HTTPS presents a certificate valid for the hostname.
• The response body identifies the expected release.
• The origin response is not served by Cloudflare's proxy/cache.
• A provider-qualified route middleware adds the expected header only on its declared route.
• A second route without that middleware remains unaffected, proving route isolation rather than only middleware existence.

5. Rolling v1 -> v2 deployment

Run a continuous external probe while deploying a changed release:

• count every request, transport error and non-success status;
• require zero request failures during the roll;
• observe the response transition from v1 to v2;
• require the old container to drain and disappear;
• assert ob status --output json reports the current release and no divergence.

Deploy identical v2 inputs again and assert the operation is a no-op rather than another replacement.

6. Operational surfaces

Exercise at least:

ob status --output json
ob audit --output ndjson
ob logs
ob exec --reason "remote e2e assertion" <workload> -- <safe command>

Assertions must cover exit status and structured terminal records, not merely search human output for a success word.

7. Rollback

Run ob rollback through its normal plan/approval path and assert externally that:

• the previous version serves again;
• health and route middleware still behave correctly;
• status and audit identify the rollback rather than a fresh unrelated deploy.

8. Interrupted deploy and resume

In a dedicated run or sub-scenario:

1. Start another changed deployment.
2. Detect that the newcomer has appeared on the host.
3. Terminate the local ob process to model runner loss.
4. Start a fresh CLI process and invoke ob resume.
5. Keep the public request probe running across interruption and recovery.
6. Require zero failed public requests, a terminal journal result and exactly the expected live workload count.

This is the remote/SSH equivalent of the existing local recovery test, not a replacement for it.

9. Ownership refusal

Attempt to preflight or bootstrap a differently named application against the already claimed host. Assert that Onebox refuses it before mutation and the original application continues serving.

10. Destroy

Exercise ob destroy once in the canary after other assertions. Verify the application stops while default volume-retention behavior remains honest. The VM is still deleted by infrastructure cleanup afterward.

Fixtures

Deterministic canary

Add a small purpose-built project whose output can change between v1 and v2. It should deliberately cover behavior the current OSS corpus does not combine:

• rolling application with at least two replicas;
• health check and external verification;
• managed Traefik proxy and terminating TLS;
• two routes, with a typed file-provider middleware on only one route;
• worker or daemon dependency where useful;
• a harmless manual job with a typed result;
• an environment override used for the release change.

The fixture exists to prove Onebox semantics deterministically, not to imitate a large application.

First unrelated OSS application: Umami

Start the real-app lane with Umami because the existing fixture is already known to run and covers a web application, Postgres, dependency ordering, health and persistent storage without the heaviest image set.

The remote test should:

• deploy it using a run-specific public hostname;
• reach /api/heartbeat over public HTTPS;
• write a small piece of application/database state when feasible;
• redeploy and confirm the state survives;
• report actual container, volume and health evidence.

Weekly matrix

After the canary and Umami lane are stable, execute the deployable corpus serially, one fresh host per application:

• Vaultwarden
• Uptime Kuma
• Gitea
• Umami
• n8n
• Paperless
• Ghost
• Authentik
• Penpot
• Immich

Rocket.Chat remains a validation/rendering compatibility fixture unless its MongoDB requirement is changed to a replica-set-capable daemon. Its known application failure must not be counted as a successful Onebox deployment.

The matrix must use fail-fast: false semantically: one upstream image failure should not erase evidence from the remaining applications.

TLS strategy

Repeated disposable hosts also create repeated ACME orders. Use two modes:

• Broad scheduled/manual scenarios: a checked-in test proxy configuration pointing Traefik at Let's Encrypt staging. Verify handshake, SNI, routing and certificate hostname, while explicitly trusting the staging test chain or treating trust separately.
• One production canary on a lower cadence: use Onebox's default production resolver and require normal public trust.

Do not issue a new production certificate for every application on every nightly run. Keep the production cadence comfortably below CA limits and make it separately observable.

GitHub Actions

Add a dedicated workflow rather than extending the required local CI job.

Suggested triggers:

on:
  workflow_dispatch:
    inputs:
      scenario: canary|umami|full
      production_acme: false|true
  schedule:
    - <remote canary cadence on main>

Requirements:

• protected remote-e2e GitHub environment;
• HCLOUD_TOKEN stored only in that environment;
• reserved Primary IP ID/address and location stored as environment variables, not secrets where secrecy is unnecessary;
• no Cloudflare token in the workflow;
• contents: read only and checkout with persisted Git credentials disabled;
• actions pinned to immutable commits, matching the existing workflow policy;
• concurrency group that queues rather than cancels the current IP owner;
• hard job timeout and per-operation timeouts;
• cleanup step guarded with if: always() in addition to process-level cleanup.

Cleanup and leak recovery

Normal cleanup must run from a trap/defer and the Actions always() step:

1. delete the exact VM by captured ID;
2. delete the exact temporary Hetzner SSH key by captured ID;
3. retain the reserved Primary IP;
4. remove local temporary keys, plans and approvals;
5. upload sanitized evidence.

Cancellation can prevent both cleanup paths. Add a small scheduled/manual janitor that:

• lists only resources with purpose=onebox-e2e;
• requires an expired expires label and a recognized run identity;
• refuses to delete ambiguous or unlabeled resources;
• deletes stale VMs and ephemeral SSH keys;
• never deletes the reserved Primary IP.

The janitor should support a dry-run mode and publish what it considered and removed.

Evidence and diagnostics

Every run should upload a bounded artifact even on failure:

• tested commit and ob version output;
• scenario, hostname, VM image/type/location and timings;
• sanitized CLI JSON/NDJSON terminal records;
• status snapshots before and after lifecycle operations;
• public probe counts and latency summary;
• container names/images/health states and release IDs;
• proxy logs around ACME/routing failures;
• cleanup result.

Never upload:

• private SSH keys;
• approvals or plan material that contains secrets;
• decrypted SOPS files;
• registry credentials;
• raw environment dumps;
• raw ACME storage (acme.json contains private keys).

Acceptance criteria

• Existing required local Docker E2E remains unchanged as the fast PR gate.
• just e2e-remote scenario=canary provisions a fresh VM and invokes the compiled CLI over SSH.
• A permanent DNS-only *.e2e.onebox.run record points at a retained Hetzner Primary IP; CI carries no Cloudflare credential.
• The canary proves public HTTP redirect, TLS, route middleware isolation and expected application content.
• Public probing observes zero failed requests during a v1 -> v2 rolling deployment.
• An identical redeploy is proven to be a no-op.
• Status, audit, logs and exec are exercised through their public CLI contracts.
• Rollback restores the prior externally visible version.
• Killing the runner mid-release and invoking resume from a new process completes without public downtime.
• A second application identity is refused on the claimed host before mutation.
• Umami deploys on its own fresh host and serves /api/heartbeat over public HTTPS.
• Production ACME is tested on a bounded lower cadence; broad testing does not exhaust production issuance limits.
• Normal cleanup and a conservative stale-resource janitor are both tested.
• Failure artifacts are useful and contain none of the forbidden secret material.
• Documentation distinguishes deterministic local E2E, remote lifecycle proof and OSS compatibility evidence.

Delivery order

Keep the first implementation slice small enough to review as one coherent change:

1. reserved Primary IP plus permanent wildcard DNS setup;
2. remote harness and deterministic canary;
3. manual/scheduled workflow, cleanup and evidence artifact;
4. Umami remote scenario;
5. only then expand the weekly OSS matrix and additional lifecycle scenarios.

The first slice is successful when a clean repository checkout can take a bare throwaway host to a publicly reachable TLS application, roll it without failed requests, recover from a killed runner, and leave no disposable infrastructure behind.