Nexus workflow-bound service handlers are unusable for SDK-worker workflows in 2.11.1

Current source: #813 merged through the normal queue as abfd6114a9728ffe88d5b5d43b7fbb64ee0cd15c, with normal published Core 2.15.4 at 8af7a0c8. All 43 combined Shared Server jobs, full PHP feature qualification and Public Boundary pass. The PR head passed 16 checks with seven conditional skips. Local related suites passed 56 tests / 627 assertions. The separate file/sync profile has the same 17 database-lock errors before and after integration; it is not counted as a pass. Prior real-worker repetitions retain their original source scope.

Next: complete the exact published Server/Core tuple and full affected Nexus/SQL-engine pressure acceptance through the approved publication path. Merged source and prior candidate repetitions do not complete that acceptance. This issue remains open; no new Server image release is claimed.

Original report and acceptance:

Published Server 2.11.1 (Workflow 2.10.2) answers Nexus operations bound to a workflow (handler_binding_kind start_workflow, signal_workflow, update_workflow or query_workflow) in ways that make them unusable for workflows run by SDK workers. These came out of the source-level spec for the native port (#325). Native will not copy them; it answers 501 for these kinds until this issue sets the intended contract.

Citations: CP is src/V2/Support/DefaultServiceControlPlane.php in Workflow 2.10.2, WS is src/V2/WorkflowStub.php in the same package, and C is app/Http/Controllers/Api/ServiceCatalogController.php at 2.11.1.

Common setup for the steps: POST /api/service-endpoints {"endpoint_name":"pay"}, POST /api/service-endpoints/pay/services {"service_name":"billing"}, then register the operation named in each item under /api/service-endpoints/pay/services/billing/operations and POST .../operations/{name}/execute.

Defects that block the feature

An update call is never accepted for an SDK-worker workflow.
- ServerWorkflowControlPlane::update goes straight to DefaultWorkflowControlPlane::update. It skips ExternalWorkflowUpdateAdmission, which the regular POST /api/workflows/{id}/update/{name} uses.
- The stub then rejects with unknown_update (no durable contract lists the update; always the case for a service-started run) or workflow_definition_unavailable (the class can't load in the server).
- Steps: start a workflow through the regular route on a queue a worker polls, so it declares update approve. Register {"operation_name":"upd","operation_mode":"async","handler_binding_kind":"update_workflow","handler_target_reference":"approve"}. Execute with {"target_workflow_instance_id":"<that id>"}.
- Actual: 409 handler failure unknown_update or workflow_definition_unavailable. Expected: admitted as the regular route admits it.
- The bridge adapter path had the same gap (Bridge adapter update_workflow skips external update admission, so SDK-worker updates are always rejected #747).
A started workflow lands on a NULL task queue.
- workflowStartOptions (CP:1927-1973) copies queue and connection from the binding only.
- The regular route resolves a queue, default at least. Workflow-task polls match workflow_tasks.queue = <queue>, so as far as the code shows no worker ever claims these tasks.
- Steps: register {"operation_name":"go","operation_mode":"async","handler_binding_kind":"start_workflow","handler_target_reference":"OrderWorkflow"}, execute {}, then GET /api/workflows/{linked_workflow_instance_id}.
- Actual: the run and its ready task have queue NULL. Expected: the namespace's default queue, as the regular start.
A query's or update's value is discarded.
- publicResult (CP:2006-2017) strips result and result_envelope before the handler object and metadata.control_plane are built.
- No nexus_result is stored, so a caller can never read the answer.
- Steps: a query_workflow operation against a running workflow whose worker answers the query.
- Actual: call completed; no value in the body or the row. Expected: the value in result_envelope, as a completed start_workflow call carries its output.
Arguments are dropped or mangled by kind.
- start_workflow drops JSON-list arguments; only an Avro string reaches the run (CP:1960-1966). The run's arguments column is then NULL.
- update_workflow drops Avro arguments: DefaultWorkflowControlPlane::update reads only the array form.
- signal_workflow with an Avro blob records arguments: [] in the command payload while the signal record gets the blob (WS:3148-3160).

Other defects

Cross-namespace type disclosure on start.
- resolveOrCreateInstance checks the durable type before the namespace (DefaultWorkflowControlPlane.php:1212-1226).
- A binding workflow_instance_id held by another namespace answers Workflow instance [<id>] is reserved for durable type [<other namespace's type>] .... The regular route refuses with workflow_id_reserved_in_namespace without the type.
- Low severity, but it is a cross-namespace leak; flagging it for Security too.
Linked calls stay open after some run endings. LinkedServiceCallCloser acts only on WorkflowCompleted, WorkflowCancelled, WorkflowFailed and WorkflowTerminated. A run ending in WorkflowTimedOut, or continuing as new, leaves its workflow_run call started forever.
Cancel propagation.
- With cancellation_policy.propagate_to_linked_workflow: true, a start_workflow call loses metadata.linked_cancel: the WorkflowCancelled write closes the call first, then cancelCall finds it terminal and writes nothing (CP:323-352).
- An exception from the linked cancel is not caught, so the answer is a 500 (CP:314-318).
wait_timeout_seconds: 0 on an update waiting for completion passes execute's min:0 rule, then throws ValidationException inside the control plane. The answer is a 409 workflow_update_exception handler failure instead of a 422.

Not defects, recorded for the contract

A signal_workflow call stays started forever; nothing completes it.
A query_workflow call to a live run with no worker blocks about 45 s (server.query_tasks.timeout), then fails 409 query_worker_unavailable.

Native's position

Native answers 501 rust_capability_not_implemented for the four workflow-bound kinds, before anything is written (#793). Once this issue settles the intended contract, native will implement it fixture-first against the fixed PHP release. The activity and invocable kinds, admission and limits, show, cancel and the catalog reads are served now.

General accepts this PHP handoff under the operator's direction. Native remains with the Rust owner and can retain its explicit501 for these four kinds until the corrected PHP contract and fixture receipt are available. This work is separate from the already-qualified #791.

The first reproductions confirm that the defects persist with normally installed published Workflow2.14.2 (d60ce5f60e70b834c5d4ef142c3d5dc415cd39fe), using Server source at #791 head 9d7cd8fcb. The regression branch is fix/794-workflow-bound-service-handlers, head 0e3afdca8; only tests have changed so far.

Four expected failures, 4 tests /45 assertions, on PHP8.3.36 and SQLite. The three new binding cases use real top-level commits; the strengthened query case retains its existing transaction harness:

The public regular start resolves queue default; the equivalent service-bound start stores NULL. The test also requires a public worker poll to lease the Nexus-created task once the queue is corrected.
A service start with an explicit queue drops the JSON list [7, "order Ω", null]; its arguments column is NULL.
An update binding with wait_for=completed and wait_timeout_seconds=0 answers409 instead of refusing input with422 before admitting a call.
A query operation uses a scripted worker polling and completing through the public query routes and reaches completed, including retry, but its response has no result_envelope.codec; the worker's value is discarded.

The new NexusWorkflowBindingContractTest and strengthened existing NexusWorkflowQueryServiceExecutionTest preserve these reproductions. The dependencies are mounted read-only, with no package overrides or edits. Tests run in an isolated initialized container under UID/GID1000 with networking disabled. Pint and diff checks pass. This is failed baseline evidence, not a claim that a fix passes.

Current2.14.2 source also still strips results in DefaultServiceControlPlane::publicResult, omits a queue fallback in workflowStartOptions, and limits LinkedServiceCallCloser to the four originally reported terminal events. The missing external-update admission, encoded update/signal arguments, namespace refusal, timeout/continue-as-new lifecycle and cancellation-result/exception paths remain to be reproduced and corrected. I have not treated source inspection alone as runtime proof of those cases.

Next: finish the current-package counterfactuals, define the four-kind input/result and linked-lifecycle contract, repair the Core/Server boundaries with regressions, qualify the new package and Server integration, and verify the published artifact. The existing explicit authority, namespace and protocol boundaries remain acceptance requirements. General has an isolated Core branch from current main; Cloud's independent #896 release continues without a hold.

Security reachability and severity review across two distinct evidence tuples (source review only, no authentication attempt): published Server 2.11.1 contains Core/Workflow 2.10.2; the normally installed Core/Workflow 2.14.2 evidence belongs to the current unpublished Server successor. These are not one published tuple.

executeOperation is behind authentication plus the operator,admin role gate. Tenant-bound principals must match the requested X-Namespace; if caller_namespace is supplied, it must also match the principal tenant. Worker credentials are denied. Legacy/full-access and unscoped operator credentials are already intentionally broader than one namespace, so this finding does not grant them new authority.
Inside an authorized namespace A, an A-owned workflow-start operation may accept target_workflow_instance_id from the request when the binding does not fix an ID. resolveOrCreateInstance performs a global ID lookup and compares workflow_type before comparing namespace. If the occupied ID belongs to namespace B and its type differs, the thrown message includes B's workflow type. If the type happens to match, the later namespace check returns the generic different-namespace message.
The service layer persists and returns that exception text through message, outcome_message, failure_message, typed-error metadata, and the serialized call. I found no path in this failure that reads B's arguments, result, history, payload references, or other row contents, and the foreign instance is locked but not modified.

Classification: low-severity authenticated metadata disclosure, specifically a workflow-instance collision/type oracle. Exploitation requires an operator/admin principal authorized for another namespace, an accessible workflow-bound operation, and a known or guessed foreign instance ID. Current evidence does not support a payload/customer-data disclosure, privilege escalation, incident declaration, or operator page.

Security acceptance for the #794 successor:

After the locked instance lookup, reject namespace mismatch before comparing or returning durable type/class metadata.
Use one stable generic namespace-reservation reason/message for both same-type and different-type foreign instances. Do not include the foreign workflow type/class in the HTTP response, persisted service-call failure fields, attempt metadata, or logs.
Regress a tenant-bound operator in namespace A against namespace B instances with matching and differing types; both must produce the same sanitized refusal, and the foreign row must remain unchanged.
Preserve the intended audit fact for the A-owned attempted call without copying foreign-instance metadata into it.
Re-run this boundary case through the normally installed fixed Workflow package and integrated Server artifact before closing Nexus workflow-bound service handlers are unusable for SDK-worker workflows in 2.11.1 #794.

This review adds no Rust #325 gate and does not broaden #794 beyond its existing released-surface fix.

Route workflow-bound Nexus through worker admission

General's first repair checkpoints are pushed on fix/794-workflow-bound-service-handlers:

Workflow #947, source head 2dd3c5ecd3b5e27ebc6b0bd047672b36113f1c9d, adds retained query/update results through the existing nexus_result/output-reference path, preserves generic control-plane payload stripping, accepts JSON-list workflow-start arguments, and checks namespace before durable type on instance collisions. 51 focused tests /1,490 assertions pass on PHP8.5.12 and SQLite. Twelve new result cases cover null/false/zero/empty/nested-int64/external values and immutable replay; same/different foreign types give the same sanitized refusal and preserve the foreign row. Prior result, argument and namespace failures are retained as counterfactual evidence. ECS, targeted PHPStan and diff checks pass.
Server Nexus workflow-bound service handlers are unusable for SDK-worker workflows in 2.11.1 #794, source head 6df08b77857a214a835bd070dd1bb36d77fe2c5b, routes service starts through ordinary queue/cohort/fingerprint and durable worker-command capture, and routes updates through the existing external-worker admission gate while preserving service-effect identity. 84 tests /701 assertions pass on PHP8.3.36 and SQLite, including regular/service starts, worker polls, version pinning, external request identity and wait bounds. The ordinary update admitted while the old service path returned409; both now admit the declared remote update. Pint and diff checks pass. Server qualification still uses normally installed published Core2.14.2 read-only; it does not yet consume the Core candidate above.

A commit-level regression also confirms the Nexus start's enclosing transaction already keeps the captured command contract durable before commit. The initial suspected extra transaction race was not reproduced; no extra transaction was added on that basis.

These are partial source checkpoints. Remaining: encoded update/signal arguments, complete per-kind result/lifecycle semantics, timeout/continue-as-new and linked cancellation/exception handling, zero completion-wait validation, integrated candidate qualification, full package quality/coverage/features and actual published installed acceptance. The two known package-dependent Server baseline assertions remain expected failures until the normally installed fixed Core package is adopted. Neither issue is complete and no release contains these repairs yet. Native can keep its four explicit501 responses until the corrected contract and fixed-PHP fixture receipt are available.

Separately, #791 is merged as qualified group8009904 and its independent main qualification is green. PHPServer2.11.3 publication is running in Release40414327142 from that exact source with Core2.14.2; #794 is not claimed fixed by that version.

Security source follow-up on exact Server checkpoint 92083828b542e413181392c83c16070e4a3f59d8: the zero-completion-wait correction aligns with #794's existing acceptance and adds no new security gate.

The route derives the effective wait from explicit wait_for, then from the requested/catalog operation mode; only workflow-update bindings that will wait for completed invoke Workflow's existing positive-timeout validator. That validation occurs after the authorized catalog operation is resolved but before payload decoding, idempotency lookup, service-call admission, or update creation, so a zero completion wait produces the typed 422 refusal without leaving a durable attempted call. The included regression covers explicit wait_for=completed, implicit completion through mode_override=sync, and the preserved accepted-only zero-second path with one admitted service call/update.

This is source-level acceptance of this 16-line delta only. Final #794 security acceptance still requires the already recorded tenant-bound namespace cases through a normally installed fixed Workflow package and integrated Server artifact. The remaining argument/result/lifecycle work is unchanged; this checkpoint does not authorize release, close #794, or gate Rust #325.

The next source checkpoint is pushed, with normal published dependencies still unchanged:

Workflow f7ebfa30cb58dec40f2675e272b51dfca2284523 preserves JSON/Avro argument lists through workflow update/signal admission and their durable target/command records. Opaque input references resolve in the service request's namespace, independently of the caller namespace used for policy. Actual external-argument regressions cover both command kinds. Linked workflow timeouts now close the service call with the target's typed handler failure; the original service-call deadline still wins when expired.
Result metadata is replaced atomically across inline/external representations, addressing Security's focused finding. The seeded stale-result controls failed in all 12 cases before the fix; all null/false/zero/empty/nested and external query/update values now reconstruct the actual result, retain unrelated metadata and replay immutably.
The combined Core focused suites pass 111 tests / 2,021 assertions on PHP 8.5.12 / SQLite. ECS and targeted PHPStan pass. This is source qualification, not the full repository quality cycle or published-package acceptance.
Server 92083828b542e413181392c83c16070e4a3f59d8 now refuses a zero-second update completion wait with HTTP 422 before admitting a call, including implicit synchronous mode. Explicit accepted-only zero waits and non-waiting workflow-task admission remain valid. The focused HTTP/wait checks pass 4 tests / 38 assertions, plus the earlier routing/admission checkpoint's 84 / 701; changed files pass Pint.

Next: retain linked cancellation receipts across the terminal-event race, report propagation exceptions without HTTP 500, follow canonical continuation, settle/document all four binding lifecycles, then run full package qualification and integrate the normally published successor. No new release or completion claim is made for this checkpoint. Server 2.11.3 publication belongs to #791; its separate installed consumer run is 40418210934.

Preserve workflow-bound Nexus payloads and terminal outcomes (#948)

Installed-artifact acceptance checkpoint, 2026-10-27 UTC: immutable Server 2.11.5 / Core 2.15.1 (index sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20, source 721b2831975cc898cc39d7528c5ba7e8441fe744) and normally installed PyPI SDK 2.12.0 executed six real worker-bound cases in a disposable SQLite namespace, without an application or SDK source overlay:

Nexus start preserved JSON-list arguments and selected the namespace's default queue, which the ordinary Python Worker claimed.
Query returned the actual worker result in result_envelope.
Signal preserved Avro arguments.
Update was admitted through the external worker contract, preserved Avro arguments, and retained the exact completed value. Its initial async response was started; the call was then read through terminal reconciliation.
A distinct fresh worker queried the original run after cold replay and recovered its initial, signal and update state.
Finishing the same workflow closed its original linked start call with the exact typed result.

The result retained signed 64-bit values (including values above JavaScript's safe-integer range), nulls and Unicode. The maintenance process was stopped for this passing six-case run.

Remaining confirmed qualification failure: with the one-second SQLite repair/schedule loop active, a worker-bound query repeatedly returned 503 backend_lock_pressure while storing its successful result. Six bounded same-key attempts retained the durable call identity but exhausted recovery. The API trace reaches DefaultServiceControlPlane::markCompleted after the worker answered successfully; its update of workflow_service_calls fails with SQLite database is locked. Core's explicit-call transaction currently spans handler dispatch and the external query wait. The maintenance-free pass is not pressure qualification.

General is preparing a focused regression and correction for that transaction boundary under this issue / Workflow #947. Keep #794 open for this repair and the remaining linked lifecycle/cancellation and cross-SDK installed scope. The separate admission/rate/committed-outcome results are recorded in #796; the bridge fix is #777, not included in 2.11.5.

General's query-pressure correction now has a reviewable Core candidate: Workflow PR #950, exact babea3ad48d863879309131aeef67da2e2376027. The two earlier transaction/observation prototypes are unaccepted and their broad QA was stopped.

The replacement reuses the existing durable service-call dispatcher with a sealed authenticated host-query capability. Admission and claim commit before the query worker wait; only the current unexpired generation may publish its result. Database write retries reuse the returned value, while configured business retries contact the worker separately. Concurrent replays share the immutable accepted intent.

Server companion source is pushed at 31b74a3354aa7db3579d49c1e4a02de53a5af1ac. It uses the existing long-poll helper outside SQL, returns the durable call on an accepted-only/zero/expired completion wait, and sizes the trusted dispatch lease to cover route discovery plus query response waiting. It still needs the normally published new Core dependency before its ordinary PR/CI delivery.

Actual candidate checks:

SQLite: 41 focused Core tests / 364 assertions, including advancing-clock independent writes, true two-process replay and forced same-snapshot preparation, origin forgery refusal, recovery, immutable intent, policy/cancellation/deadline and business retries.
Full Core ECS and PHPStan pass. Full unit coverage and PostgreSQL feature qualification are running.
Server: six public query-worker HTTP cases / 165 assertions on real committed databases; related catalog, payload, binding and query suite 78 tests / 678 assertions passes under the normal test configuration. Pint passes.
Normal PyPI SDK 2.12.0 with a real worker passes three fresh-namespace repetitions of seven candidate cases while one-second repair/schedule maintenance remains active: start JSON-list arguments, query result, Avro signal, update result, cold-worker query, three concurrent same-key query replays and linked terminal result. The replays share one call ID and one worker query; total query callbacks are exactly five including the workflow's final snapshot. Int64/null/Unicode are preserved.

This is explicit candidate-source qualification on the published base image, not new published-image acceptance. Next: finish exact Core qualification/review, normal Core publication, pin that package in the Server companion, then normal source CI/merge and publication through the authorized publisher path before exact installed-image pressure acceptance. Keep #794 open.

Navigation Menu

Repository navigation

Nexus workflow-bound service handlers are unusable for SDK-worker workflows in 2.11.1 #794

Defects that block the feature

Other defects

Not defects, recorded for the contract

Native's position

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Nexus workflow-bound service handlers are unusable for SDK-worker workflows in 2.11.1 #794

Description

Defects that block the feature

Other defects

Not defects, recorded for the contract

Native's position

Activity

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions