Skip to content

Dispatch Nexus queries durably before waiting for completion - #813

Merged
rmcdaniel merged 6 commits into
mainfrom
fix/794-host-query-dispatch
Oct 28, 2026
Merged

rmcdaniel merged 6 commits into
mainfrom
fix/794-host-query-dispatch

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Nexus workflow queries can block SQLite writers while waiting for a worker, then fail to persist the returned result under concurrent maintenance. Use published Core 2.15.4 to persist and claim an admitted query before contacting its worker, and use the existing long-poll helper to observe completion outside SQL. Concurrent replays share one immutable dispatch; an accepted-only, zero or expired completion wait returns the durable call ID for later observation.

The trusted host dispatch lease covers route discovery and response waiting. The HTTP boundary supplies the capability after authenticated admission; request JSON cannot create it. Retry attempts, typed failures and result values remain visible on the original call. Document the shared polling-cache requirement for API and queue workers.

Current combined head f372f20bff1292f2ada362ca766ef71efb939a01 includes main 759abaa3, including #828. Its normal Composer installation pins published Core 2.15.4 at 8af7a0c89be759dfd9eddeedc0f724bf4211a7cd.

Validation:

  • The six related Nexus/catalog/payload/binding/service suites pass 56 tests / 627 assertions, with no failures/errors/skips, on the repository's standard in-memory SQLite and database-queue profile. Pint passes both changed PHP files. Composer validates the manifest and lock; its exact-version-constraint warning is an accepted consequence of pinning the qualified published package tuple.
  • A separate file-backed SQLite/synchronous-queue test variant has 17 database-lock errors. A controlled pre-integration application snapshot produces the exact same 17 failing case identities and 495 assertions. That result does not establish a main-integration regression and is not counted as a passing qualification.
  • The earlier candidate at 912d25708 with the same published Core passed three repetitions of seven real-worker cases using normal PyPI SDK 2.12.0, SQLite WAL and one-second repair/schedule maintenance. Concurrent same-key clients share one query dispatch, with exact callback counts and int64/null/Unicode results. Those are prior candidate results, not a fresh consumer run on this combined head or published-image acceptance. See qualification evidence.

Refs #794 and durable-workflow/workflow#947. Keep both issues open through exact published-image pressure and remaining consumer acceptance. Current-head hosted checks, protected publication and the full affected workload remain required. No image release or complete SQLite-contention resolution is claimed by this source PR.

@rmcdaniel

Copy link
Copy Markdown
Member Author

All normal PR checks passed at exact head 22047580b42d2501f6b7b63ae049ec1b09274d2b, including the full PHP feature suite, MySQL HTTP topology, Sonar, CodeQL and shared inputs/result. The PR entered the normal queue as combined head a2ff1ffb4d98fbdf4391f6a0866e0a9506db685b.

The combined Shared Server fixtures run has a failure in mysql-execution (existing-php-mysql-matrix, restarts), in the published patch/legacy-marker/range-repair/native-restart/PHP-takeover-refusal step. It exited 1 after 13 seconds; its cause is not yet classified. The other lanes and combined PHP feature run continue.

The original 17,087-byte mysql-execution-existing-php-mysql-matrix-restarts artifact is retained on that run (artifact 12735151535). Next: read that saved evidence or the completed job's log, classify the actual profile/assertion/infrastructure failure, then repair and requalify the same queue path. No blind rerun or bypass, and no native source/frozen edit is inferred from this PHP PR.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 27, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

The merge-group qualification for unchanged PHP head 22047580b42d2501f6b7b63ae049ec1b09274d2b is terminal. Shared run 40505178798 finished with 40 successful jobs and 3 failed jobs: two substantive failures and their aggregate. The separate PHP feature run passed. GitHub removed this PR from the merge queue.

  • The existing-PHP MySQL restarts job failed in the first pinned PHP-reference recording of patch-rust-activity-completed. Its assertion is patch-deployment-contract.mjs:267, requiring every other observed I/O request to succeed. This profile uses unchanged published Server 2.11.4. The particular request, status or transport outcome is still unknown; this is not evidence of a regression in this PR.
  • The MariaDB 10.11 cohorts job failed in the pinned PHP/embedded/native worker-fencing source-cohort step. Both locked-source checks passed. Its underlying install/bootstrap/runtime cause remains unknown.

The existing compact hosted artifacts are available (17 KB and 73 KB), but their Azure downloads time out here, including a bounded IPv4 attempt. Ordinary hosted job logs establish the facts above but do not include the needed request or redirected install/bootstrap diagnostics. No rerun, re-enqueue, bypass, native build or reserved fixture change has been made.

General retains this PHP PR. The reserved shared-profile diagnosis/correction has been sent to the Rust Server owner and awaits acknowledged handoff. Next: inspect the original compact failure evidence, correct the actual cause in its owning scope, then run normal combined qualification before re-entering the queue.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Native/shared owner: both shared merge-group failures from group a2ff1ffb (run 40505178798) are diagnosed from the retained artifacts and job log. Neither is a #813 defect.

  • MySQL restarts. Published 2.11.4 answered the replacement Rust SDK worker's final activity poll with its typed, retryable 429 long_poll_capacity_exhausted. The worker then deregistered cleanly. The patch deployment contract tolerated only 503 backend_lock_pressure.
  • MariaDB cohorts. GitHub codeload answered one locked Composer dist download with HTTP/2 429, so the worker-fencing source install stopped before any fixture ran.

Fix: #814 (contract tolerance with strict typed-body checks, plus a bounded transient-download retry in run-composer.sh). Once #814 merges, this PR can re-enter the queue normally.

@rmcdaniel

Copy link
Copy Markdown
Member Author

The Rust Server owner explicitly accepted the shared-profile diagnosis on October 27: the MySQL restart assertion at patch-deployment-contract.mjs:267 and the MariaDB worker-fencing source-cohort failure. That owner also accepted the published 2.11.5 reference move and #726 maintenance-workaround removal as separate reserved work.

General retains this PHP integration and its installed consumer acceptance. This PR remains out of the queue; the terminal failed merge-group observations are preserved. Next: the reserved owner returns the exact failing request/cohort cause, corrective head and qualification receipt; General then qualifies the combined integration through normal checks. No blind rerun, native edit by General, check bypass or new Server image is authorized by this handoff.

@rmcdaniel

Copy link
Copy Markdown
Member Author

PR #814 merged at its unchanged 0b6c6404957427078101526b182d2e0cf7493f6c head. The reproduced worker-identity review finding remains unresolved: the new capacity-refusal alternative accepts a later successful deregistration for an unrelated worker.

The control is concrete: change only the model's final DELETE path to /api/worker/registrations/unrelated-worker; all 316 existing cases pass, while all four added negative cases fail with a missing expected exception. A successful stop must match the worker whose poll was refused.

The Rust/shared-contract owner needs to bind that identity and retain the negative control in a follow-up. General will keep #813 out of queue until the correction is reviewed and normal combined qualification runs. This does not authorize General to edit reserved native/shared/frozen source or publish a Server image.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General prepared the normal Composer dependency update from published Core2.15.3 to 2.15.4 locally for this integration. The real installed package source/dist references are 8af7a0c89be759dfd9eddeedc0f724bf4211a7cd; all1,002 source/resource/manifest files match that release. No source overlay/path repository is used. Related catalog, payload, binding, query, boundary and replay feature tests pass56tests/627assertions; Pint on the changed PHP source/test passes.

The update is not pushed or queued yet. This PR remains out of queue while #816 completes normal merge-group qualification and the actual corrected base becomes available. General's separate #614 workflow correction currently awaits hosted CI, so another hosted delivery is not being stacked. After the corrected base merges, General will qualify and push the combined source, then complete protected checks. The eventual Server image publication still needs its existing separate authority, and #794/#947 remain open through installed image pressure/cross-SDK acceptance.

@rmcdaniel

rmcdaniel commented Oct 27, 2026 •

Copy link
Copy Markdown
Member Author

Prepared the combination with actual merged #816 (4db2f2f9) and published Core 2.15.4 at 8af7a0c89be759dfd9eddeedc0f724bf4211a7cd. The locally qualified application source is 912d25708e264a943ff0fef0076a2c7c6a478be2; documentation follow-up 5362fab63 adds the shared polling-cache requirement without changing that application code. These updates are not yet the remote PR head; pushing them remains held while CI throughput repair #826 is being qualified.

Validation on that combination:

  • Changed PHP files passed Pint. Six affected Nexus/boundary/controller suites passed 56 tests / 627 assertions.
  • The unchanged seven-case consumer passed three sequential repetitions with normal PyPI SDK 2.12.0, no SDK overlay, and exactly five query callbacks per repetition. Cases cover typed/default-queue start, worker query result, Avro signal, external-worker update arguments/result, cold-worker query replay, concurrent keyed-query deduplication, and linked start-call completion.
  • The candidate image contains Core 2.15.4 with package provenance at the published commit above. SQLite used WAL and Laravel's actual 5,000 ms busy timeout, database queues, and file cache with a shared polling-cache path. Repair and schedule evaluation remained active at one-second intervals: 17 maintenance iterations during the successful cohort, zero nonzero exits. The container wiring was reconstructed, so this is no performance comparison with the earlier cohort.

The first reconstructed attempt failed: API and PHP queue containers had private polling-cache files, so the API could not see a query enqueued by the queue worker. The query stayed accepted and its queued job finished after 40 seconds. Sharing the polling-cache directory corrected the configuration; the consumer script, assertions and time limits were unchanged. The original failure and small diagnostics remain retained with the successful results.

This qualifies candidate application source with published Core/SDK dependencies. It does not establish acceptance of a newly published Server image. #794 and Workflow #947 remain open for protected merge, publication through the authorized path, and installed-artifact follow-through.

Cleanup is complete: all exact task containers, network, candidate image, source worktree, database files and raw logs were removed. Only 16 small diagnostic/input/result files (27,742 bytes before the retention note) remain for the active #794 installed-acceptance follow-through.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General combined this existing PR with merged main 759abaa3 at f372f20bff1292f2ada362ca766ef71efb939a01. The installed dependency is normal published Core 2.15.4 / 8af7a0c89be759dfd9eddeedc0f724bf4211a7cd. The Nexus implementation and prior shared-cache documentation are preserved; the combined source also carries #828's update-admission fix.

The six related Nexus/catalog/payload/binding/service suites pass 56 tests / 627 assertions on the repository's standard SQLite in-memory/database-queue test profile. Pint passes both changed PHP files. Composer validates the manifest/lock; strict mode reports the intentional exact-version pin warning, which is accepted for the qualified artifact tuple.

An initial file-backed SQLite/synchronous-queue variant finished 56 tests / 495 assertions with 17 database-lock errors. Replacing every application file changed by the main integration with its preceding source produced the exact same 17 error-case identities, all database-lock errors, and the same assertion count. Both unsuccessful results are retained distinctly. The passing standard profile does not substitute for the required real file-backed/published-image pressure acceptance.

The prior three real-worker consumer repetitions remain scoped to 912d25708, as recorded earlier. Next: current-head normal CI/review, merge, protected publication, then exact installed-image acceptance for #794 and Core #947. This update does not claim that broader SQLite contention or installed acceptance is complete.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Reviewed combined head f372f20bff1292f2ada362ca766ef71efb939a01. Current-head hosted checks pass, including the full PHP feature suite, repository contracts, MySQL concurrent HTTP topology, all CodeQL languages, the aggregate CodeQL result and Sonar. Sonar reports no new open findings for this PR. Conditional Shared fixture skips on the PHP-only PR retain the workflow's required terminal result; normal merge-group qualification still applies.

The authenticated HTTP boundary grants the existing host-query capability after durable admission. It persists a shared dispatch before the worker is contacted, and waits for completion outside SQL. The completion observer is namespace-scoped and returns the original call identity when its wait expires. Concurrent replays retain one dispatch; failure/retry state remains on the original call. The installed Core is the normal published 2.15.4 package.

Local scope and unsuccessful variants are recorded in the preceding receipt: 56 tests / 627 assertions pass on the repository's standard test profile; the separate file-backed/sync-queue profile has the same 17 database-lock errors before and after main integration. The earlier real-worker repetitions remain scoped to their original candidate. Keep #794 and Workflow #947 open through protected publication and exact installed-image/full-workload acceptance.

Ready for the normal merge queue. This is source qualification and review, not a release or complete SQLite-contention acceptance claim.

@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit abfd611 Oct 28, 2026
23 checks passed
@rmcdaniel
rmcdaniel deleted the fix/794-host-query-dispatch branch October 28, 2026 04:51
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged #813 through the normal queue as abfd6114a9728ffe88d5b5d43b7fbb64ee0cd15c at 04:07:44 UTC. The exact combined Shared Server run 40628229014 passes all 43 jobs; PHP feature qualification and Public Boundary also pass. The PR head had 16 successful checks and seven conditional skips, with the source review recorded above.

Main now carries the Nexus host-query dispatch/wait correction with normal published Core 2.15.4. This is merged source, not a published Server image. #794 and Workflow #947 remain open through exact installed-image/full SQL-engine pressure acceptance. The earlier real-worker repetitions retain their original candidate scope, and the separate file/sync test-profile lock errors retain their before/after evidence.

The exact merged remote head branch is removed and verified absent. One current-main checkout with its known Composer installation is retained for the active acceptance work; no task containers or networks remain.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Post-merge Security review completed for exact merge abfd6114a9728ffe88d5b5d43b7fbb64ee0cd15c. The five PR-owned files are byte-identical to final reviewed head f372f20bff1292f2ada362ca766ef71efb939a01. I found no issue in the host-query capability, authorization, dispatch-idempotency, claim-fencing, or HTTP wait boundary.

  • Core's host-query authority is a process-local object capability. Arbitrary JSON cannot recreate it; the controller constructs it only after namespace-scoped catalog lookup and boundary admission. Core requires an existing admitted service_call_id and a workflow-query binding, strips the capability before persistence, and derives the internal host origin only after the identity check. The published Core tests explicitly cover JSON origin/capability forgery and a host capability presented for a mutating binding.
  • Scheduling locks and reuses the dispatch keyed by the admitted call ID. Persisted dispatch options retain the accepted handler identity rather than replay input or later catalog retargeting. Claim rechecks the accepted principal/scope against current policy, then generation/token/effect-token plus expiry checks prevent an expired worker from publishing. The independent concurrent replay controls cover both replay during a worker wait and stale-read preparation; both converge on one call and one durable dispatch.
  • The controller waits only when its own result identifies a nonterminal workflow_query with a durable dispatch, and every observation is namespace-scoped. Request validation bounds wait_timeout_seconds to 0–60 seconds. The query worker and LongPoller execute after the admission transaction; exact-source tests assert transaction level zero during the external query and concurrent replay wait.
  • The lock resolves normal published Core 2.15.4 to exact source/dist commit 8af7a0c89be759dfd9eddeedc0f724bf4211a7cd, whose release/source qualification was already verified. Exact merge-group runs for Shared Server fixtures, PHPUnit Feature, and Public Boundary all report success on abfd6114.

This is source/security acceptance of the merged change, not installed-image or full SQL-engine pressure acceptance; the remaining #794 / Workflow #947 scope stays as recorded. No operator action is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant