Skip to content

Rust #325: fence held workflow and activity polls against replaced worker registrations #734

Closed

Description

@rmcdaniel

Goal

A native workflow or activity long poll that started under one worker registration must not lease work after that registration is replaced, as published PHP's WorkflowTaskPoller and ActivityTaskPoller ensure with WorkerPollFence.

Published PHP behavior

  • At poll admission, a registration that is not fresh answers 200 {"task":null,"poll_status":"stale_worker_registration"}, uncached.
  • The poll snapshots the registration: routing revision (queue, build ID, workflow and activity types, definition fingerprints, command contracts, capabilities), process identity, or else updated_at and last_heartbeat_at.
  • Before every probe, and inside the claim transaction under a row lock, a snapshot that no longer matches the current row ends the poll with stale_worker_registration and leases nothing. That covers a same-ID re-registration and a changed routing.

Native fences query polls this way (query_worker_proof), but workflow and activity polls don't. A held poll whose worker re-registered can lease work intended for the replacement.

Acceptance

  • A shared fixture recorded against published 2.11.1 (fixture first):
    • a held workflow poll and a held activity poll;
    • the same worker ID re-registered while each waits, then work made ready;
    • the old poll's answer, and the replacement's lease;
    • an unchanged-registration control.
  • Native passes it on SQLite, MySQL-family and PostgreSQL, with execution-test counterfactuals.
  • The dedicated update-validation poll follows Fence dedicated update-validation polls against replaced worker registrations #732's accepted PHP behavior once it is released.

Activity

  1. rmcdaniel commented on Oct 24, 2026

    @rmcdaniel
    MemberAuthor

    Progress (Rust Server owner):

    • Fixture recorded against published 2.11.1 (worker-poll-fence, three identical recordings). Each case has its own worker and queue.
      • Held workflow and activity polls end with 200 stale_worker_registration after a worker heartbeat or a same-ID re-registration, and the next poll leases the work.
      • A worker that reports its process keeps its held poll across a heartbeat from the same process, and loses it to a new process.
      • A re-registration hands a leased workflow task out again at attempt 2 and refuses the old attempt with 409 workflow_task_attempt_mismatch. Native already matched this part.
      • PHP keeps registration times in whole seconds, so the fixture spaces registration changes more than a second apart.
    • Native fence implemented locally.
      • It snapshots routing, process identity, or the update and heartbeat times in whole seconds.
      • A registration that isn't fresh is answered stale at admission.
      • The snapshot is checked again before every probe and inside the claim transaction. Multiplexed polls are fenced too.
      • The fixture passes 3/3 against native. Execution tests cover each case.
    • Landing: it is stacked on Serve synchronous update validation like Server 2.11.1 #712's poll-request work, which reworks the same claim path, and lands after Serve synchronous update validation like Server 2.11.1 #712.
    • Follow-up in scope: native query polls fence by registration incarnation token, so a heartbeat does not end a held query poll. PHP's query broker uses the same WorkerPollFence, so a heartbeat does end it. That divergence gets its own fixture step.
  2. added a commit that references this issue on Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions