Repository navigation
Rust #325: serve workflow-task cancellation scopes (open, prepare, deliver) - #771
Conversation
….11.1 A raw-HTTP worker on protocol 1.20, registered as a cooperative Python runtime in its own namespace, opens a scope tree (a shielded scope among them), schedules a scoped activity and timers on completion and requests the run's cancellation. The fixture records the scope requests that request leaves, the scope-request claim, S1's preparation at its activity and its delivery, which fences the activity and a descendant scope's timer, the activity worker's view and its acknowledgement with the scope's request, and completion admission by scope before and after preparation. A second run prepares a descendant before its parent, so the parent's delivery waits and a root command at its reserved boundary is refused; the delivered boundary consumes its sequence. Each route is refused for its protocol, validation, ownership, request and boundary faults, and for claims without the capability or polled on 1.19. The probe normalizes the observation: identities and digests become labels, the run requests' instants named anchors and other instants their format, and lease relations measured facts. Three recordings against published 2.11.1 on SQLite, with CI's maintenance loop running, normalize identically. Published 2.11.1 answers prepare and deliver with scope_id "root" with a bare 500: Workflow throws root_scope_requires_run_cancellation and the portable delivery does not convert it. The fixture bounds that defect and expects the route's 409 refusal.
… order
The probe labeled each ULID `{ulid:N}` by first appearance while walking
the observation in each response's own key order. Published PHP answers
in insertion order and native in sorted order, so two identities in one
delivery response (the delivery's event and a fenced timer's event)
traded labels between the servers although both were recorded in the
same order. The walk now visits object keys in sorted order, so labels no
longer depend on the order a server writes its keys.
The fixture was recorded again against published 2.11.1 with the new
labels, three times, identically. Against the previous expectation, 39
labels renumber one to one and nothing else changes. The two root-scope
defect steps keep their published answers from the new recording, and
their intended refusals take the same relabeling. The recorded
observation the contract test reads is replaced by the new pr1 recording.
The contract test passes 22/22.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Native serves the open, prepare and deliver routes for workflow-task cancellation scopes, as published Server 2.11.1 does over Workflow 2.10's portable scope opening and delivery: - A scope is a node of a per-run tree rooted at the implicit root scope, kept only in history. Completion commands join it by cancellation_scope_id, and a run cancellation request records CancellationScopeRequested for every unshielded descendant of root. - A worker holding the claim prepares a delivery, freezing its subtree inventory, and then delivers it: every activity and timer member is fenced first, then the barrier is recorded. Scope delivery never closes the run, releases the claim or creates a task. The activity worker acknowledges a fenced attempt with the scope's request. - Scoped waits and children stay refused at completion. A scope holding one (possible only in history PHP wrote before a takeover) is refused with cancellation_scope_delivery_unavailable, as are parallel and selection boundaries below the next sequence. Native passes the cancellation-scopes fixture recorded from published 2.11.1. The three routes leave the unserved route inventory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cancellation-scopes The port log's route count is corrected to this branch's state: 133 of 175 served, 42 unserved as the route list records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto rust-325-cancellation-scopes Workflow-task completion admits cancellation scopes before it guards worker-session commands, in published 2.11.1's order. A scheduled activity's options keep main's worker session and add its cancellation scope, as PHP's bridge appends it. Native serves 141 route methods; cancellation scope checkpoints stay missing. The probe timeout chain and the comparator node test list keep both sides' entries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st-325-cancellation-scopes # Conflicts: # .github/workflows/server-parity.yml # docs/rust-server-port.md
…llation-scopes # Conflicts: # .github/workflows/server-parity.yml # rust/src/runtime/store.rs
…fence Preparing or delivering a cancellation scope, and a run cancellation request, reread the run's whole history (up to 10000 rows) for every descendant scope and every fenced member, all under the transition lock. A 40-scope tree with a timer in each read 4183 history rows to request, 5413 to prepare and 12181 to deliver, against a final history of 165. The transaction now keeps what scope_events read for one run and later reads fetch only the rows past its last sequence. A delta that does not continue that sequence is never spliced in; the history is read again. The in-place payload update and the history deletes clear the cache, and it ends with the transaction. Payloads are shared and immutable, so a snapshot already handed out never changes. The same tree now reads 122, 124 and 165 rows. Behavior is unchanged and the 10000-row bound holds. Tests cover append-after-read, a gapped delta, the in-place update, rollback and retry, another run, and the wide tree end to end, whose row bounds fail with the cache disabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Security review package: #771 at
|
| Area | File:lines |
|---|---|
| Route registration | rust/src/runtime/http.rs 265–276 (open, prepare, deliver); handlers 2250–2279 |
| Body bound before parse | http.rs 340 (payload_limits::enforce layer); payload_limits.rs 24–64 |
| Field validation | cancellation_scopes.rs 1095–1115 (OPEN_RULES, DELIVERY_RULES); 727–743 (boundary_syntax, spans ≤ 1000) |
| Refusal mapping | cancellation_scopes.rs 1071–1089 (delivery_reason, open_reason) |
| Authority gate | cancellation_scopes.rs 1245–1285 (scope_route_gate); 1222–1240 (scope_claim); cooperative.rs 212 (claim_supports_cancellation) |
| Open | cancellation_scopes.rs 1288–1378 |
| Prepare/deliver | cancellation_scopes.rs 1397–1451 (route), 1458–1558 (engine), 1580–1684 (write_boundary), 1688–1700 (renew_hosting) |
| Fences | cancellation_scopes.rs 1824–1887 (timer), 1891–2040 (activity, with its recorder) |
| Descendant requests | cancellation_scopes.rs 2042–2111 |
| Scope tree from history | cancellation_scopes.rs 258–285 (scopes), 434–448 (descendants), 663–677 (admission_refusal), 806–838 (matches_membership) |
| Transition lock | backend.rs 292 (SQLite BEGIN IMMEDIATE), 407 (PG pg_advisory_xact_lock), 607 (MySQL FOR UPDATE on dw_server_schema) |
| History cache | cancellation_scopes.rs 218–227 (HistoryCache), 1163–1205 (scope_events); transaction.rs 16, 29 |
| Cache invalidation | store.rs 4077 (payload UPDATE); operator_metrics.rs 1363 (retention delete); namespace_cleanup.rs 210 (namespace delete) |
| Budget | store.rs 2838 (completion meter) |
| Checkpoint unserved | rust/tests/fixtures/unserved-php-routes.v2.11.1.txt 23 |
| Cache tests | cancellation_scopes.rs 2263 onward (history_cache) |
Security gates
- Authority comes from the task, not from scope IDs. The gate loads the task by its path ID and the authenticated namespace. It loads the run from the task's
workflow_run_idin that namespace, then checks the instance's namespace.lease_ownerandworkflow_task_attemptmust equal the task's. The issued claim capability (cooperative.rs212) binds task, run, namespace, worker, attempt and protocol. The worker's registration in the namespace must be fresh. Under the lock,scope_claimre-checks: leased, a workflow task, lease unexpired, same namespace as the run. A caller-suppliedscope_id,parent_scope_idorrequest_idis only looked up in that run's own history.scopes()rejects any opened event whose schema orworkflow_run_iddiffers, whose parent was not recorded earlier, or whose ID or sequence repeats. The request must be the scope's accepted request in this run. - Foreign, cross-run and stale inputs fail closed, before any write. The fixture records each case:
- unknown task, and a known task in another namespace: 404
task_not_found; - foreign owner: 409
lease_owner_mismatch; - wrong attempt: 409
workflow_task_attempt_mismatch; - a completed claim: claim mismatch;
- unknown scope, unknown parent, wrong request: refused;
- protocol below 1.20: refused.
A stale registration answersstale_worker_registration. Engine refusals before any effect roll back.
- unknown task, and a known task in another namespace: 404
- Idempotent under duplicate and concurrent calls. Each route is one transaction from
begin(), which takes the server-wide transition lock first, so concurrent calls serialize.- A replayed open returns the recorded scope (
duplicate: true); a different parent or shield answerscancellation_scope_replay_mismatch. write_boundaryreturns the recorded marker for the same boundary and refuses any other (cancellation_scope_delivery_mismatch).- A repeated delivery answers from history and writes nothing.
The fixture coversopen_s1_again,open_s1_explicit_root,open_s1_shielded_replay,prepare_s1_again,prepare_s1_other_boundary,deliver_s1_againanddeliver_s1_other_boundary. The wide-tree lib test asserts the history row count is unchanged after a redelivery.
- A replayed open returns the recorded scope (
- Cancellation stays inside its persisted scope.
- Membership is read only from history, and descendants exclude shielded subtrees.
- A boundary must touch the scope's unshielded subtree, and the scope must be recorded before it.
- A shielded scope refuses a parent request (
cancellation_scope_parent_shielded). - A preparation freezes the inventory, and
admission_refusalblocks new members under a prepared scope. - Root needs run cancellation (
root_scope_requires_run_cancellation). Wait and child members are refused (cancellation_scope_delivery_unavailable).
The fixture covers the shielded scope, the root scope, and the shielded S3 timer staying unfenced.
- Bounded before allocation.
- The body is refused from
Content-Length, and at mostmax + 1bytes are read before JSON parsing. - Strings are capped at 255. Integers must be at least 1. Spans are capped at 1000 before any span loop.
- Each history read is
LIMIT 10001, and a cumulative count above 10000 answerscancellation_inventory_exceeds_budget. - The scope tree is built iteratively, so its depth and width are bounded by the history.
- With the cache, one request reads O(N + new rows) history rows instead of O((D+M)·N); figures are below.
- The body is refused from
checkpointstays unserved. It is not registered, it is listed in the unserved inventory, and the route-coverage execution test asserts its 501.- Metering matches Workflow main. A workflow-task completion sets its meter (
store.rs2838) before recording commands, including commands that join a scope. The three scope routes open their own transactions without a meter, likeDefaultWorkflowTaskBridge::openCancellationScope,prepareCancellationScopeDeliveryanddeliverCancellationScopeon Workflow main (4eb49a20), which do not usemeteredTransaction. Workflow main meters the scope checkpoint (checkpointCommands); native leaves that route unserved, and serving it later must meter it.
History cache, against your requirements
- Transaction-local and run-keyed: the cache is
Transaction.scope_history, keyed by run. Reading another run replaces it with a fresh full read. - Created after the lock and authority:
begin()takes the lock when the transaction is created. In the routes, the firstscope_eventscall runs afterscope_route_gateandscope_claim. - Never shared: it lives in the per-request transaction, which commit and rollback consume. There is no static or global state, and a retry is a new transaction.
- Strict deltas: a reread fetches
sequence > last. The rows must continuelast + 1, last + 2, …; any gap discards the cache and rereads from the start. - Invalidation: every in-place mutation of
workflow_history_eventsclears it: the payload UPDATE (store.rs4077), the retention delete (operator_metrics.rs1363) and the namespace delete (namespace_cleanup.rs210). Those are the only UPDATE and DELETE statements on that table. Inserts go throughrecord(last + 1) or the two imports, whose transactions never read through the cache. - Bound kept: the cumulative 10000-row refusal is unchanged.
- No mutable aliasing:
Event.payloadisArc<Value>and is never mutated throughget_mutormake_mut. The event list is copy-on-write (Arc::make_mut), so a snapshot already handed out keeps what it held; a test asserts this.
Measured on a 40-scope tree with a timer in each scope (final history 165 rows), in history rows read:
| Step | Cache disabled | Cache |
|---|---|---|
| Run cancellation request | 4183 | 122 |
| Prepare the parent scope | 5413 | 124 |
| Deliver (40 timer fences) | 12181 | 165 |
| Deliver again | 13860 | 165 |
Tests (history_cache):
a_reread_fetches_only_new_rows_and_sees_every_writecovers append-after-read, an unchanged reread returning the same snapshot with no rows fetched, another run, the in-place update, a gapped delta forcing a full reread, and rollback and retry.a_wide_scope_tree_is_requested_and_delivered_reading_history_about_oncedrives the router end to end: 40 scopes, run request, prepare, deliver with 40 timer fences, then an idempotent redelivery. Its row bounds fail with the cache disabled (the run request then reads 4183 rows).
Verification at this head
Local, at bef921707 with a clean tree:
- SQLite, all targets: lib 308, including both
history_cachetests; execution 519 (1 ignored); all other targets pass. - Native shared fixtures (recorded over HTTP against the debug server built from this head):
cancellation-scopes,activity-cancellation-acknowledgement,cooperative-before-claim-cleanup,cooperative-cleanup-deadline,cooperative-pending-timer-cleanup,cancel-leased-activityandcancel-pending-timerpass on SQLite, PostgreSQL 17, MySQL 8 and MariaDB 10.11. - PostgreSQL 17 and MySQL 8: execution
completion_budget,parallel_cancellation,cooperative*,activity_cancellation_ack,history_retention,namespaces,local_activitiesandchild_cascadepass, 81/81 on each. These include the retention and namespace delete paths that clear the cache. - Counterfactual: with the cache disabled, the wide-tree test fails on its first row bound (the run request read 4183 rows).
Hosted CI on this exact head is running. I'll post its terminal state here. On the previous head (5bc3dd03), the only failure was a published-PHP 2.11.1 500 on update submit while recording update-validation on MariaDB (job 121384886683), the #726 lock-pressure class; every native check passed.
|
Hosted CI on exact head |
|
Security source clearance received for exact head |
|
Merged as |
…-operations The port guide counts 145 served route methods: Nexus reads add two and cancellation scopes three to the base 140. The workflow test list carries both contract tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…low-stream-writes The port guide counts 145 served route methods: stream writes add two and cancellation scopes three to the base 140. The workflow test list carries both contract tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-adapters The port guide counts 144 served route methods: bridge webhooks add one and cancellation scopes three to the base 140. The workflow test list carries both contract tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>



Part of #325. Serves the open, prepare and deliver routes for workflow-task cancellation scopes, fixture first. Security class: R2 (worker protocol surface).
Fixture (recorded from published 2.11.1)
cancellation-scopes(9b07c76, with identity labels independent of key order in 2a6ce03) records scope opening, nesting and shielding over raw worker HTTP. It covers a run cancellation request reaching every unshielded descendant of the root, prepare and deliver with their fences and refusals, and the activity worker's acknowledgement under the scope's request.Native (e288d1d)
Native follows Workflow 2.10's portable scope opening and delivery, as published Server 2.11.1 does:
cancellation_scope_id. A run cancellation request recordsCancellationScopeRequestedfor every unshielded descendant of the root.cancellation_scope_delivery_unavailable, as are parallel and selection boundaries below the next sequence.DefaultWorkflowTaskBridge. Workflow main also meters the scopecheckpointroute; native leaves that route unserved, and serving it later must meter it.checkpointstays unserved. Native serves 143 of the 175 published route methods on this branch.History reads (bef9217)
Preparing or delivering a scope, and a run cancellation request, used to reread the run's whole history (up to 10000 rows) for every descendant scope and every fenced member, all under the transition lock. The transaction now keeps what
scope_eventsread for one run, and a later read fetches only the rows past its last sequence. A delta that does not continue that sequence is never spliced in; the history is read again from the start. The in-place payload update (attribute_worker_terminal) and the history deletes (retention pruning, namespace deletion) clear the cache, and it ends with the transaction. Payloads are shared behindArcand never mutated, so a snapshot already handed out does not change. Behavior and the 10000-row bound are unchanged.A 40-scope tree with a timer in each (final history 165 rows), in history rows read:
history_cachelib tests cover append-after-read, a gapped delta, the in-place update, rollback and retry, another run, and that tree end to end. The tree's row bounds fail with the cache disabled.Verification on this head
Head
bef921707(current main086ba7famerged, which includes #768 and #766):cancellation-scopes,activity-cancellation-acknowledgement, the threecooperative-*cleanup fixtures,cancel-leased-activityandcancel-pending-timerall pass on SQLite, PostgreSQL 17, MySQL 8 and MariaDB 10.11.completion_budget,parallel_cancellation,cooperative*,activity_cancellation_ack,history_retention,namespaces,local_activitiesandchild_cascadepass, 81/81 on each.app/source changes.🤖 Generated with Claude Code