Repository navigation
Fingerprint a stopped PHP source database for backup proof - #523
Conversation
The takeover design (#517) proves a backup by comparing a fingerprint of the stopped source with the same fingerprint of the backup restored into a separate database. This adds the read-only upgrade-fingerprint command. The digest covers the qualified source catalog and every row of every table in primary-key order, each value rendered by the database itself in one fixed form: SQLite quote() with 17-digit reals, PostgreSQL quote_nullable() under fixed time zone, date, interval, float and bytea settings, and MySQL QUOTE() in UTC with binary columns as hex. Values are length-framed before hashing. It reads in one snapshot through the same read-only source checks as inspection, and reports only the digest, the per-table row counts and the elapsed time. Every compatibility, backup, quiescence and takeover flag stays false. The SQLite source checks are factored out of upgrade-inspect unchanged. Unit tests cover equality for a byte copy and for rows inserted in another order, a change for each altered value including NULL against 0 and text against blob, a return to the original digest when values are restored, an unchanged source file, and refusal of an owned or unknown catalog. The upgrade source cohort now fingerprints the published PHP source twice, restores its backup into a separate database (mysqldump, pg_dump and pg_restore, or a file copy for SQLite), requires an equal digest there, and requires one changed row in the restored copy to change it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A PHP 2.8.0 MySQL source restored from mysqldump prints each column as CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, where the source printed COLLATE utf8mb4_unicode_ci. The schema is the same, since a collation names its character set, but the catalog check compared the text, so inspection refused every installation restored from a dump as unqualified. The live catalog, the frozen native catalogs and the published PHP variant deltas now all drop a character set that the following collation names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SQLite, PostgreSQL and MySQL interruption tests wait for the child's receipt file to exist and then read it. std::fs::write creates the file before writing it, so a parent can read it empty: #508's MySQL job failed with left "" right "dirty-partial". The child now writes beside the receipt and renames it into place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upgrade inspection knew only the frozen layout and PHP 2.8.0, so a source on the current release, Server 2.9.7 with Core 2.7.2, was refused as an unknown catalog: Core added workflow_tasks.queue_delivery_reference. - Deltas are now chained: each recorded release layout is the previous one plus its delta. rust/upgrade/php-2.9.7 records the 2.9.7 delta for SQLite, MySQL, MariaDB and PostgreSQL, captured from real published-image databases. - On MariaDB, an installation using Laravel's separate mariadb connection stores the backup hold owner as uuid. mariadb-driver.json records that, and every release layout is also tried with it. - Inspection and fingerprint reports name the matched layout (source_layout), which takeover will use to choose its conversion. - The upgrade source cohort gains a labeled run against published 2.9.7 in the MySQL/MariaDB, PostgreSQL and SQLite jobs, with separate state, evidence, containers and databases, and checks the reported layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Published PHP stores encoded values in history payloads as bare strings
beside their codec and wraps them into {codec, blob} envelopes when it
reads them (ExternalPayloadEnvelopeService::historyPayload). It also lifts
the command's principal to each public history event and keeps only the
neutral fields of an activity failure. Native stores the envelope itself,
so history written by PHP read differently through native: command
payloads and activity arguments came back as strings, and principals were
missing.
Native now applies the same rules to every history event it returns, in
the public history and in worker task history. Only bare strings are
wrapped, so native's own envelopes are unchanged; an unsupported codec or a
PHP external reference is left as stored.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
upgrade-takeover implements the SQLite part of the takeover design (#517). In one BEGIN EXCLUSIVE transaction it: - runs the inventory, queue and payload inspections and requires their qualified results, refusing external payloads and service calls; - recomputes the fingerprint and refuses unless it equals the operator's verified backup (DW_UPGRADE_BACKUP_FINGERPRINT); - adds the native relations, converts the recorded PHP release layout (PHP's registration incarnations and tokens copied into native's, PHP-only columns and table dropped), records the native migration and runs native's own startup verification. A refusal or interruption leaves the PHP file unchanged; a committed takeover is a database native opens and PHP 2.5.14+ refuses. The SQLite upgrade source cohort now takes over a copy of the stopped published-PHP source, starts native on it and resumes the originals: the PHP-leased activity completes with its original attempt, the PHP-completed workflow task is refused as stale, the PHP-scheduled timer fires at its original deadline, both workflows complete on their original history, and PHP refuses the upgraded file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Control history now presents each event's command principal as PHP does. These tests rebuild control events from worker events, so the projection includes the principal, null for their unauthenticated commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MySQL inspection and fingerprint now open their source through one helper that detects the backend and bounds lock waits and statements. Each backend's fingerprint builds its row select and closes its connection through shared helpers, keeping only its own value rendering. The cohort script uses [[ ]] tests, names its tool argument and refuses an unknown backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to rust-325-upgrade-layouts
# Conflicts: # rust/src/runtime/upgrade/fingerprint.rs
upgrade-takeover now also accepts DB_CONNECTION=pgsql. One read-committed transaction takes native's bootstrap lock, locks every public table in ACCESS EXCLUSIVE mode (refusing with upgrade_source_in_use when a session still holds one after the 5 s lock wait), runs the inspections and the fingerprint against the operator's verified backup digest, adds the native relations, converts the recorded PHP release layout, records the native migration and runs native's startup verification before committing. The layout conversion and ledger record are now shared by SQLite and PostgreSQL through the Backend trait. Inspection and fingerprint reads can run inside the takeover's transaction, and PostgreSQL fingerprint reads are bounded by the client deadline instead of the 30 s statement limit. A takeover killed before its commit leaves nothing committed. On SQLite it leaves a rollback journal that read-only inspection refuses; a takeover rerun with the same digest lets SQLite roll it back, and the fingerprint check then proves the file is the backed-up one. Both backends have a test that SIGKILLs a real takeover process just before its commit and reruns it. The upgrade source cohort takes over a restored copy of the published-PHP PostgreSQL source too, resumes the originals under native and checks PHP's refusal, as it does for SQLite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MySQL and MariaDB commit DDL implicitly, and LOCK TABLES refuses to create or drop tables it does not hold, so their takeover cannot be one transaction. Under native's bootstrap lock, upgrade-takeover now: - refuses while any other session uses the database (which needs the PROCESS privilege to see), then verifies the inventory and fingerprint in one consistent snapshot against the backup digest; - writes native's marker with a rust-takeover:<layout> row holding the digest. PHP refuses the database from then on, and native startup refuses it with native_takeover_incomplete; - checks the sessions and the fingerprint again with the marker set aside, then runs each conversion step, recording its completion in the row. Row changes commit with their step; DDL steps check what is left to do, so repeating an interrupted one is harmless; - commits the migration ledger with native's final marker. A rerun with the same digest resumes after the last recorded step; a different digest is refused. Laravel's mariadb driver layout, whose backup hold owner is a uuid, is converted too. The upgrade source cohort now takes over a restored copy on every backend, and its steps are named for what they do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runner's own user copies the backup, and the cohort's containers run as uid 1000, so the copy is made writable like the source file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to rust-325-upgrade-layouts
…ust-325-sqlite-takeover
Like the restored backup, the runner's user makes the copy that the uid 1000 takeover and native containers write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ust-325-postgres-takeover # Conflicts: # scripts/conformance/server-parity/upgrade-source-profile.sh
… rust-325-mysql-takeover
The cohort's mysql backend runs on MariaDB too, where native reports the takeover backend as mariadb. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The source timer fires 60 s after it was authored. On a slow host the cohort reaches the resume after that, so the timer workflow's task is ready before the resumed activity workflow's and was claimed first. The resume now accepts each original workflow's task in either order and checks the same histories, deadline and stale attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ust-325-postgres-takeover
PHP Server keeps SQLite in WAL mode, and inspection, fingerprint and takeover need the rollback journal, so a default installation could not be upgraded. upgrade-prepare-sqlite, run once PHP is stopped, opens the file as its only connection and switches it to the rollback journal: SQLite checkpoints every committed WAL frame into the file first, so the stored rows are unchanged. It refuses with upgrade_source_in_use while another process has the file open, and confirms the header and that no WAL, shared-memory or journal file remains. A journal left by a crashed writer is rolled back the same way, and a source already in rollback mode is left byte-identical. The upgrade source cohort now runs its SQLite source in PHP's default WAL mode and prepares it after PHP stops, as an operator does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resume a PHP-accepted signal after takeover and refuse unsupported features
Prepare a PHP SQLite source in WAL mode for the upgrade commands
Take over a stopped PHP MySQL or MariaDB database in recorded steps
Take over a stopped PHP PostgreSQL database backup-first
Take over a stopped PHP SQLite database backup-first
Recognize the current PHP 2.9.7 layout and report the matched layout
|
The whole backup-first takeover stack is now on this branch. #539 merged into #538, then down through #536, #533, #532 and #529. The stack top's parity run 39466257207 passed all 13 jobs on
|
Sonar flagged a blocking process API and blocking file reads inside async tests. The SQLite takeover kill test now supervises its child on a blocking thread, the WAL copy and original read use Tokio's file API, and the SQLite upgrade cohort uses [[ for its backend test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Sonar's gate failed on three findings in the stack's new test code. Two were blocking APIs inside async tests: the SQLite takeover kill test's child process, and file reads and copies. The third was a |
|
|
|
I turned auto-merge off until the Shared Server fixtures run for this head (39508342128) passes. The stack's last full pass, 39466257207, predates the Sonar fixes in |
|
The Shared Server fixtures run for
I cancelled the duplicate parity run for the merged head; 39508342128 stays the stack's qualification. Auto-merge is back on. |
|
|
|
Updated with main after #557 merged (b7d631c; the merge only brings #557's monotonic identifiers and #560's CI change; Rust unit tests pass locally on the merge). I cancelled this head's Shared Server fixtures run 39534264085 to free runners for the required checks: run 39508342128 already passed all 13 jobs on 2656864, and #557 passed its own full run. Main's post-merge fixtures run covers the combination. |
|
2 participants
Part of #325: the backup-first in-place upgrade gate. First implementation slice of the takeover design in #517.
Change
upgrade-fingerprintis a new read-only development command (DW_RUST_EXPERIMENTAL=1, same database settings asupgrade-inspect). It prints a digest of a stopped PHP source database's complete logical contents, so the stopped source can be compared with its backup restored into a separate database. Takeover will check the same digest again before writing anything.quote(), with 17-digit reals;quote_nullable()under fixedTimeZone,DateStyle,IntervalStyle,extra_float_digitsandbytea_output;QUOTE()in UTC, with binary columns as hex.The SQLite source checks are factored out of
inspect_sqlite_cliunchanged, asreadonly_sourceandopen_sqlite_source.MySQL catalog fix
The MySQL restore check found a defect in the existing inspection. After a
mysqldumprestore, MySQL 8.0 prints each column asCHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, where PHP's migrations producedCOLLATE utf8mb4_unicode_ci. The schema is the same, since a collation names its character set. But the catalog check compared the text, soupgrade-inspectrefused every MySQL installation restored from a dump as unqualified. The live catalog, the frozen native catalogs and the PHP 2.8.0 variant deltas now all drop a character set that the following collation names (mysql::canonical, with a unit test).Test race fix
The interrupted-bootstrap tests on all three backends waited for the child's receipt file to exist and then read it.
std::fs::writecreates the file before writing, so the parent could read it empty, as #508's MySQL job did (left: "",right: "dirty-partial"). The child now writes beside the receipt and renames it into place. The SQLite test and, as CI runs them, the MySQL (10) and PostgreSQL (7) qualification tests pass locally.Tests
mysqldump --single-transactioninto a new MySQL database,pg_dump -Fcandpg_restoreinto a new PostgreSQL database, or a file copy for SQLite. The restored copy must fingerprint equal, and one changed row in it must change the digest. The existing before/after source snapshot still has to match. Cleanup drops the restored database.🤖 Generated with Claude Code