Skip to content

Rust #325: serve cancellation-scope prefix checkpoints - #822

Merged
rmcdaniel merged 5 commits into
mainfrom
rust-325-scope-checkpoint
Oct 27, 2026
Merged

rmcdaniel merged 5 commits into
mainfrom
rust-325-scope-checkpoint

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Part of #325 (decision comment 6449500902). Native now serves POST /api/worker/workflow-tasks/{taskId}/cancellation-scopes/checkpoint, the last published PHP route it answered with 501. All 175 published API route methods are now served; unserved-php-routes.v2.11.1.txt is empty.

Why it matters. Published PHP SDK 3.6.1 calls this route when cooperative cancellation is on and a turn recorded commands (a side effect, marker, or memo/search-attribute upsert) before opening or delivering to a scope. Native advertises cooperative_cancellation: true, and the SDK fails the task on any refusal, so such workflows got stuck on native.

What it does (Server's checkpointCancellationScopePrefix with Core 2.15's checkpointCommands):

  • Applies the non-terminal prefix under the held claim, without closing the task or moving the lease. Run status changes only at completion, as in Core.
  • Keeps Core's portable_scope_checkpoint receipt in the task payload. An identical retry answers duplicate: true; a changed retry answers cancellation_scope_checkpoint_mismatch.
  • Answers Core's gates, refusals and refused command types in Core's order, all checkpoint-shaped with history_refresh_page_token.
  • complete_workflow's non-terminal command application is factored into a shared apply_non_terminal.

Completion corrections found in review, each now matching Core, with a regression test that fails on the earlier head:

  1. An already-applied resume signal is left as it is, instead of 409 signal_already_applied (Core recordAppliedSignalForSignalResume).
  2. SearchAttributesUpserted counts toward the next command sequence (Core nextDurableCommandSequence).
  3. complete_update/fail_update advance the batch's sequence, so a later command in the same batch no longer reuses it.

Recorded divergences (in docs/rust-server-port-checkpoints/2026-10-27-scope-checkpoints.md):

  • Fingerprint. The receipt's fingerprint is an opaque retry identity. Core hashes its parsed internal command arrays; native hashes the validated request commands. No client computes it, and the fixture compares only its reuse.
    • After a takeover, a worker retrying a checkpoint that PHP recorded mid-claim would get a mismatch. That fails one workflow task attempt, and the next attempt checkpoints afresh.
  • Wait rows. Native writes an activity's or timer's wait row when it schedules it; PHP writes it at completion. After a checkpoint, native lists the new wait while the claim is still held. They agree once the turn completes. Deferring native's write needs a rebuild path in its wait projection.

Shared fixture. cancellation-scopes gains a run under the new key scope_checkpoints. Its expectations are what published 2.11.5 answered, and no existing step or label changes.

Checked (by the implementer, on the Rust dev host):

  • New scope_checkpoints execution tests, route coverage, and the four regression tests.
  • Before the rebase: lib 338/338; execution 570/570 on SQLite, PostgreSQL 17 and MySQL 8.0 (CI's images).
  • After rebasing on Rust #325: move the Shared Server fixtures reference to published 2.11.5 #815:
    • lib 339/339; SQLite execution 570/570;
    • PostgreSQL and MySQL execution 569/570. The one failure is updates_have_no_lifetime_cap_and_preserve_accepted_work_after_wait_timeout, a one-second wait test. It failed while a release build saturated the host and passed 3/3 alone on each database.
  • Contract and selection unit tests: 33/33.
  • Full CI SQLite corpus via native-image-corpus.sh with main's php-2.11.5.json and a native image built from this branch: published 2.11.5, native and embedded each pass 73/73, and the records compare equal.

🤖 Generated with Claude Code

rmcdaniel and others added 5 commits October 27, 2026 19:54
POST /api/worker/workflow-tasks/{taskId}/cancellation-scopes/checkpoint was
the last published PHP route native answered 501. The published PHP SDK
(3.6.1) calls it under cooperative cancellation, which native advertises,
when a turn recorded commands before opening or delivering a scope, and it
fails the workflow task on any refusal.

Native now serves it as Server's checkpointCancellationScopePrefix over
Core's checkpointCancellationScopePrefix:
- protocol 1.20, then the checkpoint fields (checkpoint_id, start_sequence,
  prohibited stream cursors, waits and sticky cache) with Laravel's
  messages, then a completion's own validation and guards;
- the claim must be the cooperative claim (409
  active_claim_cancellation_not_supported, without the refresh token);
- Core's checks in Core's order: the checkpoint arguments, terminal and
  turn-closing commands refused, the claim, a closed run or passed
  deadline, the retained receipt (an identical retry answers it with
  duplicate: true, anything else under its id is a mismatch), a root
  cancellation request (allowed only after its delivery, past the
  delivered boundary, before the cleanup deadline), the authored sequence,
  scope membership and update commands;
- the commands apply as a completion applies them, metered, and the task
  stays leased with its lease unchanged; the task payload keeps the
  portable_scope_checkpoint receipt.
Answers are 200, 404 for an unknown task, or 409, in the worker envelope
with history_refresh_page_token.

complete_workflow's non-terminal command arms move unchanged into
apply_non_terminal, which the completion and the checkpoint share.

The receipt's fingerprint is native's SHA-256 over the validated commands;
Core hashes its own normalized command arrays, so the values differ. No
client computes or compares it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cancellation-scopes probe adds a fifth run, R5, whose cooperative claim
checkpoints the commands of its turn before opening a scope:
- protocol 1.19, missing and invalid fields, a foreign lease owner;
- Core's refusals: a terminal command, a signal wait;
- the prefix (a side effect and a version marker at 1 and 2), its
  identical retry, the same id with other commands, a new id at a used
  sequence;
- a second prefix (a timer at 3, with its task), and the first prefix
  refused once the second replaced its receipt;
- the scope opening at 4 on the unchanged claim, the completion, and the
  run's history.
R3's plain claim and R4's protocol 1.19 claim are refused with
active_claim_cancellation_not_supported.

Every step sits under scope_checkpoints, which sorts after every earlier
key, and every new identity and fingerprint is named, so no earlier label
in the normalized observation changes.

The expectations are published Server 2.11.5's answers (image
sha256:bcc67285…, SQLite), where every existing step still matched. The
contract unit test's stored observation gains the same steps, with checks
that a retry does not commit again, keeps its fingerprint and the claim,
and that refusals carry the refresh token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The port log's parity surface now counts all 175 published API route
methods, its missing-surface list marks cancellation scopes served with the
prefix checkpoint, and the registration-freshness note says the checkpoint
shares the completion's guard. The new checkpoint file records the
contract, why the release candidate needed it, the fingerprint divergence,
and what was verified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the scope checkpoint found divergences from Core 2.15.2, three of
them shared with the workflow task completion:

- A signal-resumed claim's signal is applied once
  (recordAppliedSignalForSignalResume): an already-applied signal is left
  as it is instead of refused with signal_already_applied, so a claim that
  checkpointed its prefix still completes (completion and checkpoint).
  On the checkpoint path, a signal Core would not apply (command
  unavailable, its wait not open, another wait open) is skipped as Core
  skips it; the completion keeps native's refusals for those.
- The authored sequence counts SearchAttributesUpserted, as
  nextDurableCommandSequence does: a turn or prefix ending in an upsert is
  followed at the next sequence (completion, checkpoint and scope open).
- complete_update and fail_update take the batch's next sequence and
  advance it (applyCompleteUpdate, applyFailUpdate): a later command in the
  same batch takes the following sequence (completion and checkpoint).
- A checkpoint changes no run status and no run summary: Core's
  markRunWaiting and projection run at completion. The completion keeps
  setting the run waiting after an activity, timer or wait, as before.
- Core's own checkpoint refusals, signal application's included, answer as
  a refused checkpoint. Server's exception answers (structural limits, memo
  validation) keep their shapes, as PHP's catch block gives them for both.

Also: the receipt's doc comment is back on COMPLETION_RECEIPT; the
unreachable command-count and cancel_selection_operation checks are
removed, with a note; one RFC 3339 parser serves the cancellation context
checks; the fingerprint hashes serde_json::to_vec(commands), the same
bytes.

Each of the four new execution tests fails on the previous head and
passes now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security defensive review finding at exact head 0e7e955244d7a30fb19694bb04c263b90c44e582:

A checkpoint whose final command is complete_update or fail_update cannot safely continue at the receipt's returned next_sequence.

  • apply_non_terminal advances the in-batch sequence for update completion/failure, and checkpoint_prefix stores and returns that advanced next_sequence.
  • A later checkpoint and open_cancellation_scope both rederive the boundary from authored_sequence(...)+1.
  • authored_sequence intentionally excludes UpdateCompleted / UpdateFailed, because update events do not consume the next turn's durable sequence.
  • Therefore, while the same claim is retained, a valid worker following the checkpoint receipt can receive cancellation_scope_checkpoint_sequence_mismatch or cancellation_scope_sequence_mismatch.

The existing regression covers complete_update followed by a version marker in the same batch. It does not cover a checkpoint ending with the update command, followed by a new checkpoint or scope open at the returned sequence.

Please add a focused regression for both update outcomes (or the shared path): accepted update -> checkpoint ending in update completion/failure -> open a scope or checkpoint again at the returned next_sequence. The retained checkpoint boundary needs to participate while that exact lease/attempt remains active, without making update history consume a sequence on a later task. This should be corrected before merge; no wider refactor is requested.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security follow-up for exact reviewed head 0e7e955244d7a30fb19694bb04c263b90c44e582:

The update-tail sequence concern in my earlier review is now confirmed as an existing shared PHP/Core defect, not a native-only regression introduced by this PR. General reproduced it against normally installed Core 2.15.2 and published Core 2.15.4 for both complete_update and fail_update; adding a later workflow command makes the controls pass. Workflow now owns the product correction and cross-runtime acceptance in durable-workflow/workflow#953.

This supersedes my earlier request to fix that behavior before merging #822. On this exact head, my complementary review of authorization/tenant binding, claim and lease fencing, replay/idempotency, sequence advancement, transaction rollback, bounded/malformed command handling, and refusal surfaces found no other defensive issue. Security acceptance remains conditional on the protected hosted checks; any head change needs delta review.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General review at exact head 0e7e955244d7a30fb19694bb04c263b90c44e582: no additional blocker in the reviewed completion refactor or prefix path, with the shared Core defect explicitly tracked in Workflow #953.

The already-applied signal behavior, SearchAttributesUpserted sequence reconstruction and same-batch update sequence advancement match the Core methods inspected. The relevant Core 2.15.1 and 2.15.2 bridge methods are identical. I reviewed the extracted non-terminal command application, checkpoint receipt/claim retention and refusal paths, and the documented wait-row projection difference.

The update-tail concern is confirmed in PHP too. Against both normal-installed Core 2.15.2 and current published Core 2.15.4, each eight-case/32-assertion run has four update-tail regression failures and four passing marker-after-update controls, with no test errors. Both successful and failed update tails return a next_sequence that a following checkpoint or scope-open refuses. General owns the PHP contract repair and published follow-through in #953; this does not establish a native-only divergence introduced here.

Independent executable review scope: 27/27 cancellation-scope contract tests pass against this exact source. These are the JavaScript fixture/validator controls. I did not repeat Rust execution, database qualification or the owner's recorded corpus comparison, and this review is not an installed-release acceptance claim. Preserve the explicit fingerprint and wait-row differences and link the eventual #953 native contract follow-through.

@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit aa2d7c8 Oct 27, 2026
139 of 143 checks passed
@rmcdaniel
rmcdaniel deleted the rust-325-scope-checkpoint branch October 27, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant