Repository navigation
Cancelling a select member is refused and blocks the run in service mode #583
Description
Problem
In service mode, cancelling a select member blocks the run, and the operation the workflow tried to cancel still runs. The Python guide's first-completion example does exactly this when the deadline wins.
Reproduced on published durableworkflow/server:2.10.0 with durable-workflow==2.7.0 from PyPI. The workflow selects between an activity on a queue with no worker yet and a 2-second timer. The timer wins, the code yields selected.handles["act"].cancel(), and then returns.
-
The worker sends a well-formed command:
{"type":"cancel_selection_operation","selection_group_id":"select-calls:1:2","member_key":"act","member_index":0,"member_base_sequence":1,"member_size":1,"operation_kind":"activity","operation_identity":"<activity id>"} -
The Server refuses the completion with "Selection cancellation must identify one authored member within a durable selection group".
-
The workflow task is marked failed and the run stays
waitingwith livenessworkflow_replay_blocked. -
When a worker for the activity's queue starts, the activity it was meant to cancel runs and completes.
-
The same workflow without the cancel completes normally.
Cause: workflow-task completion keeps only command fields that have validation rules. Commands go through validated(), and the rules don't include selection_group_id, member_key, member_index, member_base_sequence, member_size, operation_kind or operation_identity. The stripped command then fails the normalizer's check. The PHP SDK and Rust SDK send the same command (PHP, Rust).
Acceptance criteria
- A service-mode workflow task that cancels a
selectmember through the Server HTTP route is accepted. The member is cancelled,SelectionOperationCancelledis recorded, an activity that hadn't started never runs, and the run completes. - A Server feature test sends
cancel_selection_operationthrough workflow-task completion. A test fails if any command type in the normalizer's grammar loses fields to request validation. - Published-artifact acceptance with the Python, PHP and Rust SDKs, each cancelling a losing activity member.
- Runs already blocked by this refusal can make progress again after upgrading, or the release notes give the recovery step.
Activity
rmcdaniel commentedon Oct 21, 2026
Repair select-member cancellation through the PHP worker HTTP route
Accepted as a P1 product defect. Published SDKs already send the command and the runtime already implements it. Server validation discards the seven member-identity fields before normalization, preventing cancellation and blocking the run.
I am implementing the narrow PHP request-validation repair and real HTTP regressions on current main
8eeaa11f37d56811773e30d42265bb028064db77, retaining published Workflow 2.9.0. The tests will verify cancellation, original identity and namespace fences, no claim of the cancelled queued activity, and recovery of a run blocked by the prior refusal. A grammar coverage check will guard other supported commands against field loss.Native/shared/runner/frozen/schema/OpenAPI and dependency bytes stay outside this correction. The unpublished lease-maintenance stack remains separate. Exact candidate CI and Sonar must qualify before merge; compatible publication and installed Python/PHP/Rust acceptance are part of delivery.
rmcdaniel commentedon Oct 21, 2026
Problem
Fixes the PHP HTTP validation defect in #583. A normal SDK cancellation loses all seven selection-member identity fields before the existing package normalizer sees it. The valid command returns 422, the SDK can mark replay blocked, and the losing activity remains claimable.
Changes
WorkflowTaskCompletionValidationso a contract test can exercise the exact transport rules against every package-advertised command type. No duplicated validation rule list is added.Workflow stays pinned to published 2.9.0. Native/shared/runner/frozen paths, schemas, protocol versions and package dependencies are unchanged. Server 2.10.1 / Helm 0.2.3 metadata is generated by the existing release synchronizer; Composer changes only its product-train and lock content hash, with no resolution or package/reference change.
Evidence
The tests-first commit
f2504601breaches the actual completion route on unchanged production source and fails with the reported 422 and exact selection-cancellation message. Earlier fixture setup mistakes are not product evidence.The final focused suite passes 16 tests / 152 assertions, including actual cancellation, missing/foreign identity, namespace isolation, original lease expiry and real blocked-run repair. Its grammar contract covers every one of the 17 package command types and keeps unknown fields discarded and forged principal fields prohibited. The affected selection also runs the existing worker protocol suites: all 160 existing cases pass; the first two expanded runs retain one fixture-only failure in the new expiry test (the existing lease-recovery response does not carry
recorded, and legitimately appends recovery history). The corrected final test asserts the actuallease_expiredresponse and absence of member cancellation, without changing production behavior.Pint, PHP lint and release-metadata synchronization pass. Local generator-test attempts were runner-blocked by missing Git/PHP in their respective Node images; no full generator-test pass is claimed. Required repository CI runs its normal supported tool environment.
Exact final candidate
d055dc55dcd15c59a64dfc6bc728d154a3197368includes current main0335f3bawithout editing native/shared/frozen files. Its PHP feature run39590225989, Sonar39590226016, Helm and boundary gates are fresh and pending. Superseded queued preparatory runs are cancelled. The current optional performance, memo-rolling and query-topology runs are also cancelled because this PHP validation patch changes none of those runtimes or contracts. Required PHP features, Sonar, Helm and boundary gates remain normal. Publication and installed Python/PHP/Rust acceptance remain pending.Issue #583 remains open until the published artifact and installed acceptance are verified.
rmcdaniel commentedon Oct 21, 2026
The candidate PHP correction passes actual normal-install Python 2.7.0 and PHP SDK 3.1.0 workflows in default and named namespaces, including exact typed output after timer-winner cancellation.
The additional actual Rust 4.1.0 original is failed, not a pass: its 30-second observation times out. Read-only SQL proves the original losing activity is cancelled with zero activity claims, both workflow tasks complete on their first attempt, and the run remains waiting. The Rust cancellation future emits the request and returns Pending, although this nonterminal command creates no further workflow task. It never produces the final workflow-completion command. This is distinct from the repaired PHP field loss.
I am fixing the SDK request behavior under this same acceptance requirement.
cancel()returns unit, so completing the request must not falsely report that cancellation beat an already committed result. The existing recorded cancellation/terminal outcome validation must continue to governawait_result()and replay. No weakened identity check, artificial extra wake or wait-budget extension is proposed. The first Rust original is retained unchanged; new controls and any publication remain separate.Server #585 stays draft and #583 stays open. Server 2.10.1 source, publication and final installed-image acceptance remain separately qualified. The native/shared/frozen stack is unchanged.
rmcdaniel commentedon Oct 21, 2026
Problem
Completes the Rust SDK part of #583. Actual published Rust 4.1.0 against the corrected candidate Server records cancellation of the queued activity, then stalls until its original 30-second observation expires.
CancelDurableOperationCallemits its nonterminal command and returns Pending, but that command alone creates no further workflow task. The workflow never emits its final completion.Change
Complete the unit-returning cancellation request locally so the following workflow commands can commit atomically in the same task. This does not report a cancellation outcome:
await_result()remains Pending without recorded terminal/cancellation history, and existing marker identity/replay validation is preserved. Repeated polling still emits only one request.The tests-first commit
d56b8f7fails on the preceding runtime with the exact Pending-versus-Ready assertion. Final source858ac8e34e23e113a119c16bd68453ceef0b74a8/ tree99f1aa7e724504ac6bca62c9bc94d133deaeb13fpasses 621 library tests on Rust 1.86.0. Actual source-SDK consumers against the corrected candidate PHP Server pass default and named originals inside their original 30-second budgets, preserving int64/null/Unicode output and recording cancellation before any activity claim.Two earlier complete source runs remain 619 PASS / one existing cooperative mock failure each. The mock handled a closed TCP connection (including its shutdown wakeup) as an HTTP request and passed an empty body to the JSON response handler. A constructed closed-connection control fails on that helper. The corrected helper ignores incomplete requests before capture/dispatch, retaining its 200 ms read bound and every client assertion. The final complete 621-test pass includes that control. No application timeout, resource bound or cancellation assertion is relaxed.
Formatting passes with the already available Rustfmt 1.99 using edition/style 2021; pinned hosted tooling remains required. Local Rust 1.86's image lacks cargo-fmt. Git diff checks pass. All compiler/dependency/application/database/raw resources are retired separately in the private owning record.
No dependency, wire, schema or version change. Fresh CI, analyzer, publication and published-image/SDK acceptance remain pending. Earlier #878 lease-maintenance source remains independently owned and is not called delivered by this correction.
rmcdaniel commentedon Oct 21, 2026
Published Server 2.10.1 acceptance
Immutable source
7a02bbcec4d0f2c470d7556632da9f4d5d15b522, Core 2.9.0 /2bc1ba0b57df34b74de02427d2225bd8e4dc6322. Docker Hub and GHCR agree on indexsha256:4a1819ec6fed7d7bbbadfbc302251f37a226622c99f8950887f2bed5895f83df:184bc1ddf2f2776d8025a0fe1818a977d341d5eb3019148d15d0ede1538a0d46;6b0e948ad72af39bbe9d5cb008a6004622004ffe58fed50c84c724c52f3812ca.The protected image job in Release 39654639711 passes exact source/package/provenance, readiness, Compose, database TLS/certificate refusals and alias verification. Its separate chart 0.2.3 publication remains queued. The original cold-build run 39637101549 remains failed; no tag was moved. Actual independent runtime checks below are amd64, not an ARM runtime claim.
Independent installed-image results
af34359a1e77a159a86ab2c2db943ea24887692f5039553d76c8c0e815feb9e2. JSON over 2 MiB and declared binary over 64 MiB receive their typed 413 refusals. Only the input-generating CLI uses a larger memory allowance.dfaae75a, Python 2.7.0, and Rust 4.2.0 each complete a genuine timer-winner / cancel-losing-activity workflow in default and named namespaces. All six originals return the exact int64/null/Unicode result within their original 30-second run/execution/observation budgets. Separate real late activity workers in both namespaces invoke zero callbacks. All consumer commands exit 0.SelectionResolved,SelectionOperationCancelled,ActivityCancelledandWorkflowCompletedper run. The retained binary registry matches the original size/hash. The audit preserves the database file fingerprint.Both 2.10.1 release notes and 2.9.12 release notes now name the critical PHP CVEs, upgrade instructions and immutable digests. For runs already blocked by #583, upgrade first and follow the selected-run repair procedure. Upgrading does not undo an activity that already executed.
This completes the published PHP-image and three-SDK acceptance for #583 and both security-image lines for #591. It does not qualify a native release/cutover, the queued chart, or the separate activity-lease release. The task-owned current-line image, database, dependencies and raw output are retiring; the borrowed 2.9.12 image and other owners' resources are preserved.