Skip to content

Resume a PHP-accepted signal after takeover and refuse unsupported features - #539

Merged
rmcdaniel merged 82 commits into
rust-325-sqlite-wal-sourcefrom
rust-325-upgrade-signal
Oct 20, 2026
Merged

rmcdaniel merged 82 commits into
rust-325-sqlite-wal-sourcefrom
rust-325-upgrade-signal

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Part of #325: the backup-first in-place upgrade gate. Stacked on #538, the top of the takeover stack; review those first.

A PHP-accepted signal resumed under native

The design asks for an upgrade fixture with an acknowledged signal. The upgrade source author now writes a third original through the raw protocol:

  1. It registers its worker with a declared go signal contract, as an SDK worker would.
  2. It starts a workflow and completes its first task with open_signal_wait.
  3. It sends the signal through the control plane.

Published PHP accepts the signal (SignalWaitOpened, SignalReceived) and makes the resume task, which stays unclaimed.

After the takeover, the resume script accepts the three originals' tasks in any order. For the signal original it checks four things:

  • native delivers that task from PHP's rows;
  • the claimed history keeps PHP's events;
  • its SignalReceived carries PHP's arguments, compared without key order;
  • the workflow completes on its original history.

Native's history then reads StartAccepted, WorkflowStarted, SignalWaitOpened, SignalReceived, MessageCursorAdvanced, SignalApplied, WorkflowCompleted, as the embedded host records it.

A PHP lease that expires after the takeover

A fourth original claims its first workflow task under PHP, and its worker then stops. PHP's 60-second workflow task lease is still live at the takeover and expires afterwards. The resume script then checks three things:

  • native redelivers the same task as attempt 2 only after PHP's recorded expiry, with PHP's history;
  • the expired PHP attempt's completion is refused (409 workflow_task_attempt_mismatch) while attempt 2 is live;
  • the workflow completes on its original history.

The author claims it before the signal original, so its poll can't pick up the signal's ready resume task.

Payload inspection

upgrade-payload-inspect also checks the codec-tagged payloads of:

  • workflow_commands.payload;
  • workflow_signal_records.arguments;
  • workflow_updates.arguments and workflow_updates.result.

Each must be an Avro inline value or a qualified reference. The remaining codec-tagged tables (stream items and update validation tasks) are listed as unscanned, and takeover refuses any source that has rows in them (below).

Unsupported features are refused

Some published PHP features have no native implementation yet. Takeover now refuses a source holding rows of any of them, before any write, rather than leaving those rows unread. Each feature has its own reason:

PHP feature Refusal reason
Runtime credentials upgrade_source_runtime_credentials_unsupported
Message streams upgrade_source_message_streams_unsupported
Service definitions upgrade_source_services_unsupported
Update validation tasks upgrade_source_update_validation_unsupported
Worker sessions upgrade_source_worker_sessions_unsupported
Build ID rollouts upgrade_source_build_id_rollouts_unsupported
Search attribute definitions upgrade_source_search_attribute_definitions_unsupported

Service calls and external payloads were already refused.

Some PHP tables stay as they are: derived projections (timeline, timer and lineage entries) and PHP's own queue and poll bookkeeping. Native does not read them.

The published source fills none of the refused tables by default. This was checked on a fresh published PHP installation with the three originals.

Checks

  • Unit tests:
    • The takeover refusal test now also refuses a runtime credential and a message stream row, each with the file unchanged.
    • The payload scan test expects the new unscanned list.
    • All 44 upgrade tests pass.
  • Upgrade source cohort:
    • The cohort now pins the new counts: 4 runs, 7 tasks and 13 history events, plus 8 payload columns holding 11 validated Avro values.
    • The SQLite representations reader accepts the four originals.
    • Locally, the cohort passed with the default source (PHP 2.8.0 layout) on SQLite, PostgreSQL and MySQL, and with the PHP 2.9.7 source on SQLite. In each run the signal workflow completed under native with PHP's arguments, and native redelivered the expired lease as attempt 2 between 0.02 and 2.5 seconds after PHP's recorded expiry. It refused the PHP attempt, and all four workflows completed.

CI

This branch contains the whole takeover stack, so its Shared Server fixtures run qualifies the stack. That run is deferred until #515's lock-pressure retries are on main.

Not covered yet

  • Takeover with external payloads, service calls or the refused features.
  • Multi-node qualification.

🤖 Generated with Claude Code

durable-workflow-ops and others added 30 commits October 20, 2026 03:22
Five transitions run Avro work while holding the global transition lock:
the rejected-signal command, signal arguments applied by a resumed task,
cooperative cancellation requests, task repair and fenced deregistration.
They shared the single codec permit with describe, query and listing
reads, so under read load a transition waited on the lock for unrelated
decodes. A per-call-site probe on the CPU deadline-health cell measured
0.58 to 0.80 s of such waiting in a cancellation request.

These transitions now take a separate transition_codec permit. Reads keep
signal_codec. Encodings, values and refusals are unchanged.

New tests hold the read permit and require a cancellation request, a
signal-resumed completion, a fenced deregistration and a range repair to
finish within one second. They time out with the old wiring.

Part of #325.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker protocol requires poll_status "leased" on a workflow task
claim, and published PHP (2.5.14 and 2.8) reports "leased" and "empty"
for workflow, activity and query polls alike. Native reported "task" and
"timeout". It now reports the same two values. The wake hint
task_queue_changed and every refusal status are unchanged.

The cross-kind poll contract, which accepted either claim status, now
requires "leased".

Part of #325.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each held native poll used to repeat its full claim every 50 ms: a
transaction under the global transition lock that read the registration,
deleted expired poll receipts and selected candidates. With no work, idle
cost grew by about 0.035 cores per waiting poll, two-thirds of it in MySQL,
and every probe queued behind, and delayed, real transitions.

A held poll now claims under the lock on its first probe. Afterwards it
claims only when a lock-free read finds something it could claim or must
refuse: an eligible task or query, a receipt for its own poll request, or a
removed, moved or replaced registration. The read runs at once when a commit
on the same node wakes the poll, and otherwise every 50 ms, which is how a
poll still observes other server processes. A false positive costs one empty
claim, and a commit racing a read is seen on the next read.

New tests hold the transition lock from outside the runtime and check that
idle workflow, activity and query polls still end at their deadline (each
fails with the previous probing), and that a held poll still claims a task
or query, and refuses a removed registration, committed by another node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The published PHP SQLite reference shares its database file with its
maintenance loop and sometimes refuses a request with its documented HTTP 503
backend_lock_pressure reply: retryable, nothing committed, worker mutations
fenced by task and attempt. Probe runs failed on these refusals for polls,
worker operations and a Rust SDK actor's deregistration.

- Parity probes now build their SDK clients through parityClient(), whose
  default transport is the SDK's own Psr18Transport over Guzzle with an
  innermost retry middleware. The transport keeps every capability the SDK
  checks (bounded requests and payload transfer), unlike the earlier
  wrapper. Observed-transport stacks add the same middleware innermost, so
  recording middleware sees only the final reply.
- lockPressureOperationRetryDelay retries any non-poll request only on
  status 503, reason backend_lock_pressure and retryable true, within 15
  retries, 30 s of waiting and the request's own timeout. Polls keep the
  stricter rule that no task was handed out. Bodies that cannot be replayed
  are never retried. Retries are reported once on stderr.
- The Rust SDK actor retries its deregistration on the same refusal, under
  the same bounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The storage audit lists the takeover gates, and the inspection commands keep
their backup, quiescence, compatibility and takeover flags false. This note
proposes what turns each one true:

- a fingerprint of the database's complete logical contents, compared
  between the stopped source and a restored backup, then checked again
  inside the takeover's own exclusive transaction before the marker is
  written;
- per-backend writer exclusion and atomicity, with a marker step journal
  where DDL commits implicitly;
- resumption of an interrupted takeover from the marker, refusal of a
  different fingerprint, and the per-backend upgrade fixtures and checks.

It records what is not yet known, such as MySQL's exclusion method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the October 20 profile host, lock-free reads every 50 ms cut MySQL's
idle cost at 24 waiting polls only from about 0.53 to 0.46 cores, and the
native process stayed near 0.28 cores. The cost is the read rate, 20 per
second per poll, not the lock.

A held poll now reads 50 ms after a probe and then at doubling intervals up
to 250 ms. A same-node wake still reads at once and starts again from 50 ms,
so same-node work is unaffected, and work committed by another server
process is observed within 250 ms. No wait runs past the poll's deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The task read now sits next to the task claim in store.rs and the query
read next to the query claim in queries.rs, inside their existing Store
impls. poll_readiness.rs keeps only the read pacing. No behavior changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parity jobs write Rust test output to evidence files, so a failed
cargo test left only its exit code in the step log. An ERR trap now prints
the panics and the failed test names from those files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
durable-workflow-ops and others added 8 commits October 20, 2026 13:33
…atures

The upgrade source author now also writes a third original: a workflow that
opens a signal wait, then receives its declared signal under published
PHP, whose resume task stays unclaimed. After the takeover, native must
deliver that signal from PHP's rows: the resumed task keeps the original
history and the signal's PHP-written arguments, and the workflow
completes. The resume accepts the three originals in any order.

The payload inspection now also checks the codec-tagged payloads of
workflow commands, signal records and updates.

Takeover refuses a source holding rows of a published PHP feature native
does not implement yet (runtime credentials, message streams, service
definitions, update validation tasks, worker sessions, build ID rollouts
and search attribute definitions), rather than leaving them unread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Give codec work under the transition lock its own permit
The upgrade source author adds a fourth original whose worker stops after
claiming its workflow task. PHP's lease on that task expires after the
takeover. Native must then redeliver the same task as its next attempt,
refuse the expired PHP attempt's completion, and complete the workflow on
its original history.

Part of #325.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Pushed 6978d60b1: the upgrade fixture now adds a PHP workflow-task lease that expires after the takeover (described above). I cancelled its parity and PHPUnit runs. This branch runs parity once #515 is on main and merged here, so one run qualifies the whole stack.

durable-workflow-ops and others added 10 commits October 20, 2026 15:03
…volution-518

Allow replay-compatible workflow definition upgrades and explain routing
Two alternating PHP/native pairs on a second dedicated host repeat the
morning's matched comparison on main after #516. Native meets or beats PHP
in every window, with no drift within any run. PHP's own capacity at 1/s
differs between the two hosts, which the observations record.

Part of #325.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssure

Retry the PHP reference's lock-pressure refusals in parity probes
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged main (with #515's probe retries) as 3dab83ea7. The SQLite library (136) and execution (282) suites pass locally on it. This branch's parity run now qualifies the whole takeover stack, #523 through #539.

…try-safety

Prepare Server 2.9.9 entry-safety release
@rmcdaniel

Copy link
Copy Markdown
Member Author

Parity run failures so far are infrastructure, not code. Around 16:39–16:40 UTC, mirror.gcr.io refused image pulls with toomanyrequests: Quota exceeded for the pinned Rust, MariaDB and MySQL images. That failed "Fetch locked Rust dependencies" and the codec-deadline-health binary checks before any test ran. I'll rerun the failed jobs once the run completes.

rmcdaniel and others added 4 commits October 20, 2026 11:53
Record two repeated matched PHP/native pairs after #516
…e 2.9.7 cohort

Public history now lifts each command's principal onto its event, and worker
task history has none, so the recovered terminal resumption compares the
fields both views share. The SQLite PHP 2.9.7 upgrade cohort steps set the
job's resource scope like their neighbours; the cleanup step failed without
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Run 39454020872 failed in three jobs (execution-and-codec, postgres-storage existing-php-matrix and pg17) for one reason. restart.php compares the recovered terminal resumption's worker task history with the public history recorded before the kill. Since def78ff7e (#532), public history events carry principal and worker task history events do not, so the comparison failed on the field count. The probe now compares the fields both views share (70f610bef).

The SQLite Retire PHP 2.9.7 upgrade source cohort step also failed: it and its profile step lacked RESOURCE_SCOPE, which the other SQLite cohort steps set per step. Both now set it. The profile step had been skipped after the earlier failure, so it would have failed on an otherwise passing run.

The two codec-deadline-health failures (mariadb-10.11, mysql-8.4) were mirror.gcr.io toomanyrequests quota errors at image pull.

I cancelled the rest of the run and pushed ffeccc9a3, which merges main (ef7283006; no Rust changes). The new run qualifies the stack top.

@rmcdaniel
rmcdaniel merged commit 7670598 into rust-325-sqlite-wal-source Oct 20, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants