Skip to content

Refuse a lost workflow task claim with PHP's ownership guard fields - #548

Merged
rmcdaniel merged 7 commits into
mainfrom
rust-325-stale-completion
Oct 21, 2026
Merged

rmcdaniel merged 7 commits into
mainfrom
rust-325-stale-completion

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Part of #325.

Why

In the multi-node runs (#549), a paused SDK worker resumed holding workflow tasks the server had already given to the other worker. Native refused its late lease renewals and completion with only a reason (task_not_leased, conflicting_completion). The worker could not tell that its claim was lost, so it exited; this happened 6 times. Published PHP refuses the same requests through its ownership guard. Its body carries the task status and, once the run has closed, can_continue: false, and current workers read those fields to drop the claim and keep running.

What changes

  • Workflow task completion, failure, lease renewal and task history now share one guard that follows PHP's WorkflowTaskOwnership::guard order:
    • Task no longer leased. Refused with run_closed (stop reason, run status, closure, task status, owner) when its run has closed. Otherwise refused with task_not_leased and the task status.
    • Task still leased. Checked for expiry (lease_expired), then owner (lease_owner_mismatch), then attempt (workflow_task_attempt_mismatch with current_attempt).
  • Each refusal has PHP's body: error, the reason's fields, protocol_version and server_capabilities.
  • A completion that differs from the recorded one now gets the guard's refusal instead of the native-only conflicting_completion. An identical replay still returns the recorded already_completed receipt.
  • A completed workflow task now clears its lease expiry and keeps its owner, as PHP's does.

Evidence

  • New fixture. late-workflow-completion re-registers a worker that holds a claim. It then sends the replaced claim's completion, renewal and failure three times: while the new claim holds the lease, after the new claim completes, and after the run closes. The contract checks PHP's exact bodies apart from timestamps.
  • Local parity. In a local copy of the SQLite parity step, published PHP 2.5.14, embedded PHP and native all passed, and the comparison passed. The pre-change native binary failed the same fixture.
  • Native tests. a_replaced_workflow_claim_is_refused_with_the_published_ownership_fields is new. Two existing tests asserted conflicting_completion after a run-closing completion. They now expect PHP's run_closed; after a failed run, the task status is failed, as PHP stores it.
  • Local runs. The execution suite passed on SQLite, PostgreSQL and MySQL (283 tests each).
  • Local full parity. On the same SQLite copy, native passed every default fixture except visibility-current-runs, which needs the published CLI artifact that CI supplies.
  • Contract unit tests. tests/Unit/ServerParityLateWorkflowCompletionContractTest.mjs (19 tests) is added to the CI list.

Not covered

PHP also refuses an otherwise valid claim whose worker registration is no longer fresh (stale_worker_registration). The local activity and cancellation scope checkpoints still use the older fence. Both are recorded as known gaps and need their own fixtures.

🤖 Generated with Claude Code

A paused worker that resumed with workflow tasks already given to another
worker got only a reason from native (task_not_leased,
conflicting_completion) and exited. Published PHP refuses those requests
through WorkflowTaskOwnership::guard, whose body carries the task status and,
once the run has closed, can_continue false; current workers read those
fields to drop a lost claim and keep running.

Completion, failure, lease renewal and task history now share one guard in
PHP's order, with PHP's refusal bodies. A completion that differs from the
recorded one gets the guard's refusal instead of conflicting_completion. A
completed workflow task clears its lease expiry and keeps its owner, as PHP's
does. The late-workflow-completion fixture covers the replaced claim's
requests while the new claim holds the lease, after it completes and after
the run closes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged main (9c10bd0eb, with #499 and #510) as 74ac2675f; the merged tree passed the SQLite execution suite (287 tests). I cancelled the superseded Shared Server fixtures run 39471824937 for f43e6fae2. The run for 74ac2675f qualifies this PR.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Sonar's quality gate failed on new-code duplication (3.0%, limit 3): workflow_task_guard.rs repeated the impl Store where-clause that every store module carries. I'm moving the guard into store.rs beside the fence it replaces; the module keeps only the refusal-body builder. I cancelled the in-flight Shared Server fixtures run 39473539144, which the fix supersedes. The push follows local verification.

The guard needed its own copy of the store's where-clause in a separate
module, which Sonar counted as duplicated new code. It now lives in the
store's existing impl next to the fence; the module keeps only the refusal
body builder. The two task_not_leased branches share one refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel pushed a commit that referenced this pull request Oct 20, 2026
The multi-node schedule also passed on MariaDB 10.11. A confirmation run
with the #548 binary shows every late request refused with PHP's ownership
guard body; SDK 2.0.10's remaining exits are its own lack of lost-claim
handling. With PHP SDK 2.5.3, all 1,080 workflows completed once on the four
databases and no worker exited. Summaries now record the SDK version, and
the record describes the concurrent bootstrap race found on the way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion

# Conflicts:
#	docs/rust-server-port.md
@rmcdaniel
rmcdaniel enabled auto-merge October 20, 2026 22:20
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged main (49a56f5c0: #551 namespace update, #545 database TLS, #556 certifi) into this branch as 24285ccc1. The only conflict was in the port-log checkpoint list, and I kept both entries.

Parity. I cancelled the Shared Server fixtures run for the merged head (39507398656) as a duplicate:

  • the branch's own change is unchanged since 39483535410, which passed all 13 jobs on 2fcefc4ee;
  • main's additions run their own parity after each merge.

Local check of the merged head. On SQLite, the native lib suite (120) and the execution suite (288, including late_workflow_claims) pass.

Auto-merge is on for the required checks.

…tion

# Conflicts:
#	.github/workflows/server-parity.yml
#	docs/rust-server-port.md
#	scripts/conformance/server-parity/contract.mjs
#	scripts/conformance/server-parity/probe.php
@rmcdaniel
rmcdaniel disabled auto-merge October 20, 2026 22:53
@rmcdaniel
rmcdaniel enabled auto-merge October 20, 2026 22:55
@rmcdaniel

Copy link
Copy Markdown
Member Author

772cc5849 merges main after #543 (terminate and worker listing). Both branches add fixtures to the same registries, so I resolved additive conflicts in:

  • contract.mjs and probe.php (fixture wiring);
  • server-parity.yml (contract unit test lists);
  • the port log (known gaps and checkpoints).

Local checks of the merged head:

  • Native suites on SQLite. The lib suite (122) and the execution suite (292, including late_workflow_claims and the terminate tests) pass.
  • Contract unit tests. The late completion, worker listing and worker replacement tests pass (50).
  • SQLite parity replica. PHP 2.5.14, embedded PHP and native pass these fixtures, and the comparison passes:
    • late-workflow-completion;
    • worker-listing;
    • worker-replacement-activity-lease;
    • terminate-before-claim, terminate-pending-timer and terminate-leased-activity.

The Shared Server fixtures run for this head (39511283057) is left running. Auto-merge is back on for the required checks.

@rmcdaniel

Copy link
Copy Markdown
Member Author

36e013e1e merges main again (987b07e10, #552 Workflow 748 core consumer), which changes no native code, parity script or parity workflow. I cancelled its duplicate parity run 39516312432; the merged-head run 39511283057 stays.

…tion

# Conflicts:
#	.github/workflows/server-parity.yml
#	docs/rust-server-port.md
#	rust/tests/execution.rs
#	scripts/conformance/server-parity/contract.mjs
#	scripts/conformance/server-parity/probe.php
@rmcdaniel

Copy link
Copy Markdown
Member Author

b306127ce merges main after #550 (search attribute registry), resolving the same additive registry conflicts. The resolution also restored a closing brace between the late-completion and search-attribute embedded blocks in probe.php. Locally on the merged head:

  • the native lib suite (124) and SQLite execution suite (295) pass;
  • the SQLite parity replica passes late-workflow-completion, search-attributes and worker-listing on all three targets, and the comparison passes.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 3fc05da into main Oct 21, 2026
10 checks passed
@rmcdaniel
rmcdaniel deleted the rust-325-stale-completion branch October 21, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants