ops: backup and restore, and the readme for a first release

[cemhurturk]

This branch sits on #49, and the base moves to main once #49 merges.

This adds backup.sh and restore.sh, a clickmonk version command, a disk ClickHouse writes its own backups to, and a README pass that makes every claim true for a first release. It carries no version bump. The release is its own branch after this merges.

What a backup holds

Six files per run, in a directory named for the moment it started: clickhouse.zip (the clicks and rollups), postgres.dump (domains, links, cap counters, settings, the admin account, its sessions and API keys), spool.tar (clicks accepted and not yet shipped), caddy-data.tar (certificates, their keys and the certificate authority account), env (a copy of .env) and MANIFEST (versions, row counts and a SHA-256 of each file). Every file is written 0600 under umask 077. The README says why: a backup is a credential.

The spool and ClickHouse are copied at one instant. The worker is the only thing that moves clicks between them, so it is the one container stopped, and only for the ClickHouse copy and the spool copy. The redirect keeps answering and keeps writing to the spool. A batch the worker had shipped but not yet deleted from the spool is in both places. That is correct, not a double count: every count ClickMonk makes is of distinct clicks (uniqExact states in the rollups), so a click shipped twice after a restore adds nothing. The round-trip test holds clicks in both places and reads each once.

.env is in every backup, and restore.sh never writes it. That was Cem's decision. The restore warns before its confirmation if the running install's CLICKMONK_SECRET or admin host differ from the backup's, and the README says what each difference means.

It counts what exists. A checkout from before the reporting rollups has no rollup tables, so the manifest records which tables it counted, and a restore verifies exactly those. That keeps the pre-0.1.0 upgrade path working on any checkout, back to one from before TLS. That path was run end to end from a schema-4 checkout.

What a restore does

A restore is everything, and an outage. That was forced rather than preferred: the redirect cannot keep writing a spool that is being replaced. It stops the four services and replaces Postgres, ClickHouse, the spool and Caddy's data. It clears what was there first, so clicks recorded after the backup do not survive into it. Then it verifies every recorded count before starting anything. Every check runs before anything is stopped: the checksums, the schema version, the free space (about 2.5 times the archive, net of the database it replaces), and a typed confirmation with a two-minute timeout. Each refusal is tested to leave the running install exactly as it was.

Into a newer release. The guard reads the schema version from the image up will run, not from a container that may be older. It refuses a backup newer than that image. A backup from an older release restores, and the worker applies the missing migrations on start. The round trip undoes migrations 009 and 010 before its backup and proves both are applied afterwards.

Certificates come back as they were. Caddy's data is restored whole, so the round trip presents the same certificate and places no new order with the certificate authority.

Interrupted runs leave a true state. Both scripts share one lock per install, and a stale lock is refused rather than removed. A signal that arrives while services are still stopping lets the stop finish, then starts them again. A restore interrupted while a step is still writing waits for that step, says what it left, and leaves a marker. backup.sh refuses while the marker exists, because a backup of half-restored stores would be a picture of a moment that never existed. Rerunning the restore clears it. backup.sh also refuses while ClickHouse is still running a backup or restore that a killed run started.

Everything else

clickmonk version prints the release and the schema version, and answers before reading any environment. One VERSION is pinned equal across every package.json. ClickHouse gets a backups disk, mounted from clickhouse/backup-disk.xml.

Upgrading is written out: take a backup, pull, rebuild, and start the worker before the redirect. There is a heading for checkouts from before 0.1.0, with the extra steps a pre-TLS checkout needs. Latency is measured, not asserted: a median of 1.1 to 1.4 ms and a p99 of 2.3 to 4.4 ms, on an 8-core virtual machine with 6 GB of memory. A capped link is the exception: it makes one Postgres query per click, bounded at 150 ms. Security says what an install exposes and what it does not. It requires Docker Compose 2.23 or later, for COMPOSE_ENV_FILES.

No migration. SCHEMA_VERSION stays 10.

Verification

The gate ran in CI's order: pnpm lint, pnpm lint:sh, pnpm typecheck and pnpm build, then pnpm test with 1,565 service tests and 562 interface tests, the interface suite five runs in a row. The stack suites passed 170 tests across eight files. The backup and restore suite is the largest, at over 80 tests and about five minutes. The durability suite passed, and the history's secret scan is clean. Every script runs under bash 3.2 as well as 5.

Every guard was mutated individually and caught by exactly its own test. That includes signals that arrive while a stop is still draining, and while a restore step is still writing. A handful are protected by a comment rather than a test, where no test can observe them: the Caddy step's wait, a chown on the backups directory, the ClickHouse archive's lock file, and a worker that is crash-looping when the backup starts. The upgrade from the last merged main was run by hand, following the README word for word.