Skip to content

Take over a stopped PHP SQLite database backup-first - #532

Merged
rmcdaniel merged 102 commits into
rust-325-upgrade-layoutsfrom
rust-325-sqlite-takeover
Oct 20, 2026
Merged

rmcdaniel merged 102 commits into
rust-325-upgrade-layoutsfrom
rust-325-sqlite-takeover

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Part of #325: the backup-first in-place upgrade gate. Stacked on #529 (and #523); review those first. Implements the SQLite part of the design in #517.

Takeover

upgrade-takeover (development command; SQLite only for now) takes over a stopped PHP SQLite database in one BEGIN EXCLUSIVE transaction:

  1. Runs the inventory, queue and payload inspections and requires their qualified results. External payloads and service calls are refused, since native cannot represent them yet.
  2. Recomputes the fingerprint and refuses unless it equals DW_UPGRADE_BACKUP_FINGERPRINT, the digest the operator matched against the restored backup. Any write after the backup stops the takeover.
  3. Adds the native relations and converts the recorded PHP release layout (frozen, php-2.8.0, php-2.9.7):
    • copies PHP's registration incarnations and tokens into native's table, so a running SDK worker's fenced deregistration still matches, and creates one for any active registration without it;
    • drops the PHP-only incarnation table and columns.
  4. Records the native migration in the ledger and runs native's own startup verification (marker, ledger and full catalog) before committing.

A refusal or interruption leaves the PHP file byte-identical. A committed takeover is a database native opens and PHP 2.5.14+ refuses. rust/README.md documents the procedure.

History written by PHP

The resume check found that native presented PHP-written history differently from PHP. PHP stores encoded values as bare strings beside their codec and wraps them into {codec, blob} envelopes when reading. It also lifts the command's principal to each public event and keeps only neutral fields of an activity failure. Native stores envelopes, so command.payload and activity.arguments came back as strings and principal was missing.

history_presentation.rs ports those rules (ExternalPayloadEnvelopeService::historyPayload, HistoryController::eventPrincipal) into native's history rendering. It applies to public history and worker task history. Only bare strings are wrapped, so native's own envelopes are unchanged.

Checks

  • Unit tests (SQLite).
    • A 2.9.7-shaped database with PHP incarnations is taken over and opened by native: the PHP token is kept, retired incarnations keep their state, and a registration without one gets one.
    • The frozen layout is taken over.
    • A wrong or stale fingerprint, a malformed one, a service call and an external payload are each refused with the file unchanged.
    • Presentation tests cover envelopes, principals, failure redaction, and unsupported codecs and references.
  • Upgrade source cohort (SQLite; both the default 2.8.0 run and the labeled 2.9.7 run). Takes over a copy of the stopped published-PHP source and starts native on it. The PHP-written originals then resume:
    • the PHP-leased activity completes with its original attempt, and a replay records nothing;
    • the PHP-completed workflow task is refused with 409 task_not_leased;
    • the PHP-scheduled timer fires at its original deadline;
    • both workflows complete, and their PHP-written events read exactly as PHP returned them, payloads and principals included;
    • PHP then refuses the upgraded file with php_database_refused.
  • Local results. Both cohort runs passed (timer fired 0.012 s and 0.102 s after its deadline). Full native suite on SQLite, PostgreSQL and MySQL: unit and execution tests passed on all three, apart from two execution tests that rebuild control history from worker events and lacked the new principal field. With the projection corrected, both pass, and the upgrade tests (40) pass on all three after merging Fingerprint a stopped PHP source database for backup proof #523's latest head.

CI

This branch contains #523 and #529, so its Shared Server fixtures run qualifies the whole stack; #529's own run is cancelled to save runners.

Not covered yet

PostgreSQL and MySQL/MariaDB takeover (transaction and exclusion per the design), nested payload columns the inspection does not scan yet, takeover with external payloads or service calls, and multi-node qualification. The takeover report keeps stored_representation_compatibility_verified: false.

🤖 Generated with Claude Code

durable-workflow-ops and others added 30 commits October 20, 2026 03:22
Five transitions run Avro work while holding the global transition lock:
the rejected-signal command, signal arguments applied by a resumed task,
cooperative cancellation requests, task repair and fenced deregistration.
They shared the single codec permit with describe, query and listing
reads, so under read load a transition waited on the lock for unrelated
decodes. A per-call-site probe on the CPU deadline-health cell measured
0.58 to 0.80 s of such waiting in a cancellation request.

These transitions now take a separate transition_codec permit. Reads keep
signal_codec. Encodings, values and refusals are unchanged.

New tests hold the read permit and require a cancellation request, a
signal-resumed completion, a fenced deregistration and a range repair to
finish within one second. They time out with the old wiring.

Part of #325.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker protocol requires poll_status "leased" on a workflow task
claim, and published PHP (2.5.14 and 2.8) reports "leased" and "empty"
for workflow, activity and query polls alike. Native reported "task" and
"timeout". It now reports the same two values. The wake hint
task_queue_changed and every refusal status are unchanged.

The cross-kind poll contract, which accepted either claim status, now
requires "leased".

Part of #325.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each held native poll used to repeat its full claim every 50 ms: a
transaction under the global transition lock that read the registration,
deleted expired poll receipts and selected candidates. With no work, idle
cost grew by about 0.035 cores per waiting poll, two-thirds of it in MySQL,
and every probe queued behind, and delayed, real transitions.

A held poll now claims under the lock on its first probe. Afterwards it
claims only when a lock-free read finds something it could claim or must
refuse: an eligible task or query, a receipt for its own poll request, or a
removed, moved or replaced registration. The read runs at once when a commit
on the same node wakes the poll, and otherwise every 50 ms, which is how a
poll still observes other server processes. A false positive costs one empty
claim, and a commit racing a read is seen on the next read.

New tests hold the transition lock from outside the runtime and check that
idle workflow, activity and query polls still end at their deadline (each
fails with the previous probing), and that a held poll still claims a task
or query, and refuses a removed registration, committed by another node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The published PHP SQLite reference shares its database file with its
maintenance loop and sometimes refuses a request with its documented HTTP 503
backend_lock_pressure reply: retryable, nothing committed, worker mutations
fenced by task and attempt. Probe runs failed on these refusals for polls,
worker operations and a Rust SDK actor's deregistration.

- Parity probes now build their SDK clients through parityClient(), whose
  default transport is the SDK's own Psr18Transport over Guzzle with an
  innermost retry middleware. The transport keeps every capability the SDK
  checks (bounded requests and payload transfer), unlike the earlier
  wrapper. Observed-transport stacks add the same middleware innermost, so
  recording middleware sees only the final reply.
- lockPressureOperationRetryDelay retries any non-poll request only on
  status 503, reason backend_lock_pressure and retryable true, within 15
  retries, 30 s of waiting and the request's own timeout. Polls keep the
  stricter rule that no task was handed out. Bodies that cannot be replayed
  are never retried. Retries are reported once on stderr.
- The Rust SDK actor retries its deregistration on the same refusal, under
  the same bounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The storage audit lists the takeover gates, and the inspection commands keep
their backup, quiescence, compatibility and takeover flags false. This note
proposes what turns each one true:

- a fingerprint of the database's complete logical contents, compared
  between the stopped source and a restored backup, then checked again
  inside the takeover's own exclusive transaction before the marker is
  written;
- per-backend writer exclusion and atomicity, with a marker step journal
  where DDL commits implicitly;
- resumption of an interrupted takeover from the marker, refusal of a
  different fingerprint, and the per-backend upgrade fixtures and checks.

It records what is not yet known, such as MySQL's exclusion method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the October 20 profile host, lock-free reads every 50 ms cut MySQL's
idle cost at 24 waiting polls only from about 0.53 to 0.46 cores, and the
native process stayed near 0.28 cores. The cost is the read rate, 20 per
second per poll, not the lock.

A held poll now reads 50 ms after a probe and then at doubling intervals up
to 250 ms. A same-node wake still reads at once and starts again from 50 ms,
so same-node work is unaffected, and work committed by another server
process is observed within 250 ms. No wait runs past the poll's deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The task read now sits next to the task claim in store.rs and the query
read next to the query claim in queries.rs, inside their existing Store
impls. poll_readiness.rs keeps only the read pacing. No behavior changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Published PHP stores encoded values in history payloads as bare strings
beside their codec and wraps them into {codec, blob} envelopes when it
reads them (ExternalPayloadEnvelopeService::historyPayload). It also lifts
the command's principal to each public history event and keeps only the
neutral fields of an activity failure. Native stores the envelope itself,
so history written by PHP read differently through native: command
payloads and activity arguments came back as strings, and principals were
missing.

Native now applies the same rules to every history event it returns, in
the public history and in worker task history. Only bare strings are
wrapped, so native's own envelopes are unchanged; an unsupported codec or a
PHP external reference is left as stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
upgrade-takeover implements the SQLite part of the takeover design (#517).
In one BEGIN EXCLUSIVE transaction it:

- runs the inventory, queue and payload inspections and requires their
  qualified results, refusing external payloads and service calls;
- recomputes the fingerprint and refuses unless it equals the operator's
  verified backup (DW_UPGRADE_BACKUP_FINGERPRINT);
- adds the native relations, converts the recorded PHP release layout
  (PHP's registration incarnations and tokens copied into native's, PHP-only
  columns and table dropped), records the native migration and runs
  native's own startup verification.

A refusal or interruption leaves the PHP file unchanged; a committed
takeover is a database native opens and PHP 2.5.14+ refuses.

The SQLite upgrade source cohort now takes over a copy of the stopped
published-PHP source, starts native on it and resumes the originals: the
PHP-leased activity completes with its original attempt, the PHP-completed
workflow task is refused as stale, the PHP-scheduled timer fires at its
original deadline, both workflows complete on their original history, and
PHP refuses the upgraded file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
durable-workflow-ops and others added 23 commits October 20, 2026 13:44
With no lock wait, a writer that had just closed could still hold its lock
when the preparation opened the file, and the preparation refused it as in
use. A two-second wait covers that; a file another process keeps open is
still refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atures

The upgrade source author now also writes a third original: a workflow that
opens a signal wait, then receives its declared signal under published
PHP, whose resume task stays unclaimed. After the takeover, native must
deliver that signal from PHP's rows: the resumed task keeps the original
history and the signal's PHP-written arguments, and the workflow
completes. The resume accepts the three originals in any order.

The payload inspection now also checks the codec-tagged payloads of
workflow commands, signal records and updates.

Takeover refuses a source holding rows of a published PHP feature native
does not implement yet (runtime credentials, message streams, service
definitions, update validation tasks, worker sessions, build ID rollouts
and search attribute definitions), rather than leaving them unread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Give codec work under the transition lock its own permit
The upgrade source author adds a fourth original whose worker stops after
claiming its workflow task. PHP's lease on that task expires after the
takeover. Native must then redeliver the same task as its next attempt,
refuse the expired PHP attempt's completion, and complete the workflow on
its original history.

Part of #325.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…volution-518

Allow replay-compatible workflow definition upgrades and explain routing
Two alternating PHP/native pairs on a second dedicated host repeat the
morning's matched comparison on main after #516. Native meets or beats PHP
in every window, with no drift within any run. PHP's own capacity at 1/s
differs between the two hosts, which the observations record.

Part of #325.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssure

Retry the PHP reference's lock-pressure refusals in parity probes
…try-safety

Prepare Server 2.9.9 entry-safety release
Record two repeated matched PHP/native pairs after #516
…e 2.9.7 cohort

Public history now lifts each command's principal onto its event, and worker
task history has none, so the recovered terminal resumption compares the
fields both views share. The SQLite PHP 2.9.7 upgrade cohort steps set the
job's resource scope like their neighbours; the cleanup step failed without
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resume a PHP-accepted signal after takeover and refuse unsupported features
Prepare a PHP SQLite source in WAL mode for the upgrade commands
Take over a stopped PHP MySQL or MariaDB database in recorded steps
Take over a stopped PHP PostgreSQL database backup-first
@rmcdaniel
rmcdaniel merged commit 74538b9 into rust-325-upgrade-layouts Oct 20, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants