Repository navigation
Rust #325: fence held polls against replaced worker registrations (#734) - #739
Conversation
An update whose durable contract declares a validator now waits for a validator-capable worker to approve it, as published PHP does: admission refusals in PHP's order, validation tasks keyed by run, update and request id, the dedicated and multiplexed worker polls, fenced approve and reject routes, lease expiry and re-claim, and 504 timeouts with their reasons. An accepted update's receipt no longer carries accepted: true, as PHP's result mapper drops it. The update-validation fixture records 49 steps against published PHP; native matches all of them.
…blished 2.11.1 Fixture first for the poll_request_id contract of workflow-task polls. One worker records an empty poll and its replay after work arrived, a fresh id's lease and its replay, the replay after a workflow-task heartbeat renewed the lease, a padded copy of the id, the same id after its task completed while another was ready, and after nothing was left. Published 2.11.1: the empty id replays empty; the lease replays with the renewed lease; the padded id is the same id; a completed delivery polls afresh and leases the next task; with nothing ready the id answers empty. Native currently leases the empty id's retry, gives the fresh id nothing, and refuses the drained id with 409 poll_cached_task_expired, which PHP never sends. The unit test runs the contract on the recorded observation and on six mutations, including those native answers. Published 2.11.1 passes 3/3 locally with its maintenance loop running.
Native's workflow-poll poll_request_id differed from PHP's WorkflowTaskPollRequestStore: an empty answer was not kept, so a retry leased work that arrived meanwhile; a lease replayed its stale lease time; the key ignored queue and build; and an ID whose task had closed was refused with 409 poll_cached_task_expired, which PHP never sends. A workflow poll with a trimmed, non-empty request ID now: - binds the ID to its task kinds and keeps its answer in dw_query_cache, keyed by namespace, queue, registered build, worker and ID, as PHP's cache keys are; - replays that answer while it is deliverable: an empty or queue-changed answer for the pending window (poll timeout + 5 s, capped at a minute), a lease while its task stays leased to the worker at the same attempt, with the task's current lease; - polls afresh when the lease is no longer deliverable, so it leases the next ready task or answers empty; - writes the lease's answer in the claim transaction, so a concurrent duplicate replays it instead of leasing another task, and records _server_poll_request_id in the task payload, as PHP's PollRequestLeaseBinding does. Activity polls keep their receipts. The workflow poll, its heartbeat and the activity status route now report times with microseconds, as PHP does. The workflow-poll-request-replay fixture passes 3/3 against native locally, and a native execution test pins the same rules. Full native SQLite suite: 253 library and 454 execution tests pass.
… rust-325-update-validation-poll # Conflicts: # .github/workflows/server-parity.yml
The probe waits through the 10-second validation timeout, a 6-second lease expiry, a 2-second completion wait and several short polls before PHP does any work. On the published PHP SQLite runtime it exceeded the recorder's generic 45-second probe limit while making its last calls (train #723, job 120776618912: spawnSync php ETIMEDOUT). Expected results are unchanged.
…perseded PHPUnit runs cancel The update-validation probe's waiting child requests now retry PHP's documented backend_lock_pressure refusal (nothing committed) with the same rule as every other probe request. seconds stays the answered attempt's own wait; retries are reported on stderr. PHPUnit feature suite and Feature source qualification used a job-level always(), which GitHub never cancels, so #724's superseded-run cancellation left the old run holding the concurrency group. !cancelled() still runs both jobs when a needed job failed, so the required checks stay fail-closed.
…or its PHP leg The SQLite job's main corpus now excludes update-validation.json. A separate script records that fixture for php, embedded and rust and compares the three. Its published PHP record runs with PHP's maintenance loop stopped: under that loop's SQLite writer contention, PHP 2.11.1 loses its own validation-timeout and lease-reclaim writes to "database is locked" and its validation poll answers without the typed lock-pressure refusal (10 of 15 local runs failed with the loop running, 5 of 5 and 3 of 3 passed with it stopped). Expected results are unchanged. MySQL and PostgreSQL keep the fixture in their main corpus with their loops running. The runner gains --exclude-fixture, which must name a selected fixture and cannot be combined with --fixture. The maintenance window helper is #722's, moved to a shared path. Tests cover the selection rules, the workflow wiring and every restore path of the helper. PHPUnit Feature returns to main's job conditions and drops its concurrency group. The central action policy requires always() on the PHPUnit feature suite gate, and GitHub never cancels an always() job, so cancel-in-progress would only queue a newer run behind it. Shared Server fixtures keeps #724's cancellation.
Sonar reported 11 style findings in the two new scripts. Conditions use [[ ]], the state reader ends with an explicit return, and positional parameters are read into named locals. Behavior is unchanged: a missing COMMAND still fails under set -u, and each assertion still fails the caller under set -e. PhpMaintenanceWindowTest passes 5/5, and one local end-to-end record stopped and restored the real maintenance container.
Fixture first for the poll_request_id contract #712 advertises (normalized_task_kind_set binding, poll_request_task_kinds_conflict). On its own queue the probe now records: a multiplexed poll with an ID; its retry with the kinds reordered; the same ID with the workflow kind alone and with the validation kind alone; and a default (workflow-only) ID's lease, its retry with both kinds, and its plain replay. Published 2.11.1: the reordered retry replays the same validation task at attempt 1; both other sets answer 409 poll_request_task_kinds_conflict with requested_task_kinds, bound_task_kinds and the ID; the default ID refuses the multiplexed retry (bound ["workflow"]) and replays its own lease. The 49 existing expected steps are unchanged; only their identifier labels renumber around the 8 new steps. The fixture's int64 input was kept byte-exact by splicing the expected block as text. The recorded observation vector is the new 2.11.1 recording; its unit test passes 22/22, and published 2.11.1 passes 2/2 locally.
…lidation deliveries Native's update-validation poll advertised PHP's poll_request_id contract (normalized_task_kind_set binding, poll_request_task_kinds_conflict) without keeping it. A multiplexed retry re-polled: the cursor had already turned, so it could lease a second task, and a retry with another kind set was never refused. Native's cursor was read outside any transaction and turned in a separate one, so concurrent probes could both read the same first kind. A trimmed, non-empty poll_request_id now binds to its normalized kind set (either order of the two kinds is one set), as WorkflowTaskPollRequestStore does: - a live binding to another set answers 409 poll_request_task_kinds_conflict with requested_task_kinds, bound_task_kinds and the ID, before anything is replayed or claimed, and the existing lease stays held; - a validation delivery replays while its task stays leased to the worker at the delivered attempt; once it is not deliverable, the ID polls afresh, so an expired lease is claimed again at the next attempt; - the binding and answer expire as PHP's resultTtl sets them: five seconds after a live lease ends, capped at an hour, or the pending window. Each multiplexed probe is now one transaction, WorkflowTaskKindSelector's: replay a concurrent duplicate's answer, create the queue's cursor at update_validation if missing and lock it, claim the kind it names first and then the other, turn the cursor after a lease, and record the answer under the request ID. The workflow claim and the validation claim gain transaction-taking variants for this. Native execution tests on SQLite, PostgreSQL and MySQL cover the reordered retry, both conflict sets, the default ID retried with both kinds, the expired validation lease re-claimed at attempt 2 under its ID, a completed validation delivery polling afresh, concurrent duplicates sharing one delivery and one cursor turn, and concurrent probes leasing one task of each kind. Reading the cursor before the probe's transaction, or dropping the in-transaction replay, fails those concurrency tests on PostgreSQL. The update-validation and workflow-poll-request-replay fixtures pass 2/2 against native.
Fixture first for #734. Each case has its own worker and queue, and held polls run in a child process while the probe changes the registration and makes work ready. For workflow and activity polls it records a held poll under an unchanged registration, across a worker heartbeat, and across a same-ID re-registration, with the poll the worker sends next. A worker that reports its process is recorded across a heartbeat from the same process and a re-registration from a new one. Last, a workflow task leased before a re-registration, the poll after it, and the old attempt's completion. Published 2.11.1, three identical recordings: work reaching a held poll under an unchanged registration is leased; a heartbeat or a re-registration ends the held poll with 200, task null and stale_worker_registration, and the next poll leases the work; the same process's heartbeat keeps the poll and a new process's registration ends it; a re-registration hands the leased workflow task out again at attempt 2 and refuses the old attempt with 409 workflow_task_attempt_mismatch. PHP keeps registration times in whole seconds, so each registration change comes more than a second after the previous one. The contract unit test runs the recording and eight mutations, including a fence that is missing, ignores a heartbeat, over-fences the same process, or keeps the old lease.
…gistrations A native workflow or activity long poll kept leasing work after its worker re-registered or heartbeated, where published PHP's WorkerPollFence ends it with stale_worker_registration so a poll left behind by a replaced process cannot take work meant for its replacement (#734). A poll now snapshots its registration when admitted: - the routing revision (queue, build ID, workflow and activity types, definition fingerprints, command contracts and capabilities, compared with PHP's normalizations); - the process identity a worker reports (host, process ID, process start); - otherwise the update and heartbeat times, in the whole seconds PHP keeps them. A registration that is not active or has not heartbeated within the stale window answers 200 stale_worker_registration at once, uncached, before any replay. Before every probe, and again inside the claim transaction, a registration that no longer matches the snapshot ends the poll with that answer and leases nothing; a request ID keeps it for the pending window, as PHP's result cache does. Multiplexed polls are fenced the same way. The worker-poll-fence, update-validation and workflow-poll-request-replay fixtures pass 3/3 against native. Native execution tests cover the unchanged control, a heartbeat and a re-registration ending held workflow and activity polls, a reported process kept across its own heartbeat and fenced by a new one, and a silent registration answered stale at admission without being cached.
…ences The worker-poll-fence fixture now also records, from published 2.11.1 (three identical recordings): - a held workflow poll whose registration is removed while it waits ends with 200 stale_worker_registration, not a refusal; - held query polls across a heartbeat or a same-ID re-registration, with and without a request ID, end empty, and the next query poll leases the query, which is answered. The probe declares a 150-second budget, which the recorder honors as it does for the other long fixtures. Native query polls fenced only by the registration incarnation: a heartbeat kept a held query poll, and a replacement answered stale. They now take the same WorkerPollFence snapshot as workflow and activity polls, and a registration replaced or silenced during the wait ends the poll empty, as PHP's performPoll does; a stale registration at admission, or at the start of a coordinated round, still answers stale. Tests that held native's earlier answers now expect PHP's: a held poll whose registration another node removes ends 200 stale; a held query poll fenced off by a replacement ends empty. Two tests whose runs outlast the 30-second stale window now heartbeat before polling, as a PHP worker must. A native execution test covers a held query poll ending empty on a heartbeat. The fixture passes 2/2 against native.
… rust-325-worker-poll-fence-native
…lacement in the next second stays fenced to the same worker
|
Security R2 blocker on exact head
But every registration already generates and atomically rotates a unique incarnation token, including a same-ID registration with unchanged process metadata (registration). Therefore either of these leaves the old fence looking current:
The first race is masked by the native test helper's deliberate 1.1-second delay before every change, and the process case only tests a changed Please make the unique registration incarnation the fence authority. A heartbeat must leave the token unchanged; every successful registration must rotate it; every pre-probe and in-transaction claim check must require the snapshotted token to remain current. Add rapid same-second and identical-process-metadata re-registration regressions for workflow, activity, and query polls on SQLite, PostgreSQL, and MySQL, while retaining the same-process heartbeat case. The published PHP |
…of overwriting it Security's R2 finding on #712: a held poll with a poll_request_id could finish its last empty probe, lose the transition lock while a duplicate with the same ID leased a workflow task or update validation, and then record empty (or task_queue_changed) over that live lease. The timing duplicate returned empty, later retries replayed empty, and a claimant whose response was lost could not recover its lease by retrying. The no-task write now binds the request ID, rereads any deliverable lease a duplicate committed, and returns it for the caller to replay. It records empty or task_queue_changed only when there is none. The single-kind loop (both terminal answers) and the multiplexed workflow/update-validation loop all replay that lease. The claim path shares the same helper. Debug builds gain a test-only pause point per request ID, so regressions commit the duplicate's lease exactly between the final probe and the terminal write, without sleeping. They cover a workflow lease before empty, a workflow lease before task_queue_changed, and both kinds before the multiplexed empty, each with a lost-response retry. Release builds compile neither the pause points nor their call sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Security's R2 finding on #739: the fence snapshotted routing, optional process metadata and whole-second registration times, never the registration itself. A same-ID re-registration that repeated its process metadata, or landed in the same stored second without any, left an old poll current, so it could lease work meant for the replacement. Every native registration already rotates a unique incarnation token and a heartbeat keeps it. The fence row now reads the active token, and a snapshot whose token is no longer active is never current, on top of PHP's checks. The pre-probe check and every in-transaction claim check (workflow, activity, query and update validation) read the same row. A registration without a token, such as one kept from a PHP takeover, keeps that absence, so only the heuristic applies to it. Regressions re-register with the same process metadata and in the same stored second without metadata, for held workflow, activity and query polls, and keep the existing same-process heartbeat case. The PHP fence gets the same correction in Server #749. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er-poll-fence-native
Conflicts were in two workflows. The fixtures workflow drops its old pull_request path list, as main does; the push path list it now classifies pull requests with already carries every path this stack added. Its comparator step runs main's list plus this stack's three tests. The update-validation status check stays at the end of the fixtures step, before main's new step that starts the cohorts' shared native node. The PHPUnit workflow keeps the stack's maintenance-window step beside main's merge queue and fixtures result steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ve lease Merging #712's terminal-write fix into the poll fence left two callers of the old signature: the stale_worker_registration answer a held poll writes when its registration is replaced. That answer now takes the same path as empty and task_queue_changed. Under the binding lock it replays a lease a duplicate with the same request ID committed and still holds, and records stale only when there is none. A lease the re-registration released is not deliverable, so it is never replayed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The combined B unit ran the whole execution suite for the first time with the incarnation fence and #712's replay together: - cross_kind_poll's replacement scenario expected the old heuristic's empty answer. Its same-process re-registration now rotates the incarnation, so the held poll answers stale_worker_registration. It still never wakes with a task, which is what the scenario guards. - The new re-registration test's activity case completed a workflow task leased to the worker it re-registers. Re-registration releases the replaced registration's leases, so that completion was refused. A separate author worker on the same queue now takes the task. - The query re-registration test no longer drops each case's database while the runtime pool still holds connections, which PostgreSQL refuses. The file's other tests do not drop theirs either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rust Server: fixed in Fix. The registration incarnation is now the fence authority:
Regressions. Re-registration that repeats the same process metadata, and re-registration in the same stored second without metadata. Both run for held workflow, activity and query polls. The existing same-process heartbeat cases are kept and still hold the poll. The activity case now uses a separate author worker for the run's first workflow task, because a re-registration releases the replaced registration's leases. Two follow-ups from combining the stack.
PHP. The published fence shares the heuristic, so it gets the same correction: #749 / branch Matrix (local). SQLite: full |
…r and lease The upgrade-source cohort resumes PHP's originals under native as the worker PHP registered, then waits for the original timer and the expired PHP lease to come due. Both come due after the 30-second stale window. With the poll fence (#739), native now refuses polls from a registration whose last heartbeat is older than that window, as PHP's WorkerPollFence::isFresh does. The harness never heartbeated, so every late poll answered stale_worker_registration and the timer's task was never claimed: "The original timer did not fire under native", in every cohort. The resume loop now heartbeats before each poll, as an SDK worker does. The poll and its checks are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rust Server: the first hosted cycle on Cause. The takeover-resume harness resumes PHP's originals under native as the worker PHP registered, then waits for the original timer and the expired PHP lease. Both come due after the 30-second stale window. With this PR's poll fence, native refuses polls from a registration whose last heartbeat is older than that window, which is PHP's Fix. Local check. The cohort's own script, SQLite, with this head's debug binary:
A new hosted cycle is running on |
|
Rust Server: hosted CI is terminal and green on the B unit's exact head
Ready for Security's R2 recheck of #712 (finding 6385727724) and #739 (finding 6385707510) on this head. It merges after #745. #740 will follow as its own unit, stacked on this one. |
|
Second-round Security review accepts the fixes for the #712 terminal-answer race and the #739 registration-incarnation fence at exact current head For #712, fix commit For #739, fix commit I also reviewed follow-ups This accepts these two cited fixes at the exact head above, not an unreviewed future merge result. Merge current |
…to rust-325-worker-poll-fence-native # Conflicts: # .github/workflows/phpunit-feature.yml # .github/workflows/server-parity.yml # rust/src/runtime/http.rs # rust/src/runtime/store.rs # rust/tests/execution.rs # scripts/conformance/server-parity/contract.mjs # scripts/conformance/server-parity/probe.php
|
|
Rust Server: exact-head evidence for Security's re-acknowledgement of Hosted CI on this head: terminal success on the first attempt. Shared Server fixtures run 40162397220: all 34 jobs succeeded. PHPUnit Feature 40162397173, SonarQube 40162397179, Public Boundary 40162397172, 40162395805 and 40162393853, and PR #739 40162394377 all succeeded. The PR is CLEAN. Earlier head Merge structure
Every file traces to an accepted side or a named resolution (classified at
Local qualification: the SQLite all-targets suite on Holding the merge for Security's re-acknowledgement of this head. |
|
Security recheck accepts exact head The focused security paths from accepted head Merge only this exact head under the Rust owner’s #325 authority. Any head change requires revalidation. This review does not alter the separate #518 credential-rotation hold or #519 temporary-tunnel preservation gate. |
…-worker-sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st-325-ci-codec-one-job Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto rust-325-workflow-import Native now serves 139 route methods: main's 138 plus embedded v2 import. The runtime keeps both new modules, and the comparator node test list keeps both sides' tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto rust-325-record-local-activity The worker capability manifest keeps this branch's local activity entries and main's worker session and update validation entries. The comparator, probe and node test lists keep both sides' fixtures and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto rust-325-cancellation-scopes Workflow-task completion admits cancellation scopes before it guards worker-session commands, in published 2.11.1's order. A scheduled activity's options keep main's worker session and add its cancellation scope, as PHP's bridge appends it. Native serves 141 route methods; cancellation scope checkpoints stay missing. The probe timeout chain and the comparator node test list keep both sides' entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #734. Stacked on #712 (base
rust-325-update-validation), which reworks the same claim path; it merges after #712.Held workflow, activity and query polls now follow published 2.11.1's
WorkerPollFence:stale_worker_registrationat admission.stale_worker_registration; query polls answerempty, with or without a request ID.The fixture is
worker-poll-fence, recorded from 2.11.1 in three identical runs. It passes against native locally, and native execution tests pass on SQLite, PostgreSQL and MySQL. Four native tests that held earlier native-only answers now expect PHP's; each change is cited to the recording.