Skip to content

Workflow-task workflow_stream items store payload_reference without verification or claim #786

Closed

Description

@rmcdaniel

Problem

The workflow-task workflow_stream command accepts commands.*.workflow_stream.items.*.payload_reference as any string of up to 191 characters (WorkflowTaskCompletionValidation). WorkflowStreamCommandProcessor passes it to WorkflowStreamService::append, which stores it. Nothing checks it against the external payload registry and nothing claims it.

The HTTP append route behaves differently. The transport middleware resolves the opaque reference, the controller fetches and integrity-checks the bytes, and the reference is claimed. Since #785 it is claimed inside the append transaction, so a refused append claims nothing.

Consequences of an unverified command-path reference:

  • A stored item can name an object that was never registered, has expired, or belongs to another namespace's storage.
  • An upload such an item names is never claimed, so it can expire and be deleted while the item still points at it.

Effect on cleanup (why #785 does not widen it)

Run retention (ExternalPayloadRetentionCleanup::deleteForRun) and #785's stream retention pass both collect these strings and route them through the same established deletion path. That path is limited in two ways:

  • Ownership. RuntimeExternalPayloadRegistry::deleteUri deletes only an object this namespace owns, or one with no registered owner, through this namespace's own storage driver. An object registered to another namespace is left alone.
  • Retained references. An object any other retained row still references is kept (isReferencedByRetainedRun).

So the objects a forged command-path reference can cause to be deleted are the same with or without #785: objects in the same namespace that no retained row references. #785 changes when that can happen (at stream expiry instead of run pruning), not which objects are reachable.

Proposed fix

Verify and claim command-path references the way the HTTP route does, inside the completion transaction, and refuse the completion with the same typed external-payload errors when a reference doesn't resolve. That changes workflow-task completion behaviour, so it needs its own review and fixture.

Found while reworking #785 (Server #774) under the #325 line.

🤖 Generated with Claude Code

Activity

  1. rmcdaniel commented on Oct 26, 2026

    @rmcdaniel
    MemberAuthor

    Correction to this issue's problem statement, and the fix.

    What actually happens on the command path.

    • The runtime transport middleware lists commands.*.workflow_stream.items.*.payload_reference among its incoming reference paths (RuntimeExternalPayloadTransport::INCOMING_REFERENCE_PATHS).
    • A worker therefore has to send an opaque runtime reference object there. A plain string is refused with 422 external_payload_unsupported.
    • The middleware resolves the object to this namespace's registered URI and checks that the registration is ready, unexpired and matches its codec, size and hash (RuntimeExternalPayloadRegistry::rowForReference).

    So the stored value is never an arbitrary string, and it never names another namespace's object. The real gap was narrower: the reference was never claimed. Its upload could expire and be garbage-collected while the item still pointed at it. Retention could also race the unclaimed write, because the write took no object lock.

    Fix, in #785. WorkflowStreamCommandProcessor now passes the append a claim step, RuntimeExternalPayloadRegistry::claimRegistered:

    • it claims the references of the items the append will write, under each object's lock, inside the completion transaction, in canonical lock order;
    • it checks the registration again and refuses a missing or expired one with the same typed errors.

    Stream and run retention re-check references under that same lock before deleting an object. Either the command's reference commits first and the object stays, or the object goes first and the claim fails.

    I'll close this when #785 merges, with its receipt.

    🤖 Generated with Claude Code

  2. rmcdaniel commented on Oct 26, 2026

    @rmcdaniel
    MemberAuthor

    Fixed by #785, merged at a3e0bab0191f9fc004c297f98fa78eabc5136273. Workflow command stream appends now claim their transport-resolved references (claimRegistered) under each object's lock, inside the completion transaction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions