Repository navigation
Concurrent stream opens on different runs can deadlock on MySQL/MariaDB #787
Description
Current state (General, 2026-10-26)
Implemented in PR #791, final candidate 9d7cd8fcba3dda8bcec984e7048d0ffa27566d00. Qualification and exact remaining work. General owns hosted CI, protected-queue merge, qualified PHP artifact publication and installed SDK/Nexus acceptance. Rust ownership remains native-only. Source is qualified locally; the artifact is not yet published.
Problem
WorkflowStreamService::lockStream reads a stream row with SELECT … FOR UPDATE and creates it when the read finds nothing. On MySQL and MariaDB under the default REPEATABLE READ, a locking read that finds no row takes a gap lock on wf_durable_streams_run_name_unique.
Two appends that open a new stream on different runs can land in the same index gap:
- Each takes the gap lock with its locking read. Gap locks are compatible, so neither waits.
- Each insert then needs an insert-intention lock on that gap, and the other transaction's gap lock blocks it.
- InnoDB detects the cycle and rolls one append back with error 1213.
Even when the reads don't overlap, the second open waits for the first append to commit.
main (0ef7c93) has the same read-then-create shape, so this predates #774. PostgreSQL and SQLite are not affected.
Evidence
- Reproduction: two transactions on the same unique key shape run the locking read for a missing
(namespace, run, 'live')row, then insert it. Both reads come before either insert.- MySQL 8:
SQLSTATE[40001]: 1213 Deadlock found when trying to get lockfor one transaction; the other commits. - MariaDB 10.11: the same.
- MySQL 8:
- Observed while writing the Durable workflow streams: bound producer controls, make capacity refusal atomic, enforce closed-stream retention #774 race tests: a second append opening a stream on another run waited with
X,INSERT_INTENTIONon thesupremum pseudo-recordofwf_durable_streams_run_name_unique, held by the first append's open.
Impact
Concurrent first appends to different runs in a namespace serialize on the index gap. When both locking reads come before either insert, one append fails:
- an HTTP append returns a server error;
- a workflow-task completion that opens a stream fails and is retried.
Direction
After #774, each run's stream writers are serialized by the run-row lock taken first (WorkflowRunStreamLock). So the open path no longer needs a gap-locking read. Options:
- read the stream row without a lock, then lock a found row by primary key, which takes a record lock only;
- or run the open under READ COMMITTED, which takes no gap locks.
Either way, keep the namespace stream quota on creation, and handle a stale-snapshot duplicate on the command path, whose transaction may have read before taking the run lock.
Add a MySQL and MariaDB race regression: two first appends on different runs, both locking reads before either insert, and both must commit.
Activity
rmcdaniel commentedon Oct 26, 2026
General accepts this PHP-reference defect, with implementation starting after #785 merges. Its run-row lock is the prerequisite for removing the missing-stream gap lock safely. #785 remains with the Rust Server owner through merge; there will be no overlapping edits to that branch.
Next: start from the #785 merge, preserve stream-creation quota and stale-snapshot duplicate handling, remove the cross-run gap-lock dependency, and qualify two concurrent first appends on both MySQL and MariaDB. Both transactions must commit. Keep the existing single-run serialization and capacity behavior covered.
rmcdaniel commentedon Oct 26, 2026
Implemented and backed up on
fix/787-stream-open-gap-lockatb7f906d103f079117074232ffc63ce83a835339e, tree2af73db8802bb3aaa7f3409bbf91a1c69d7b378a, based on the #785 mergea3e0bab0191f9fc004c297f98fa78eabc5136273.The append path now discovers a stream without a missing-key locking read, then locks a found row by primary key. The existing run lock still serializes same-run writers. Creation still uses the model's namespace quota admission. A savepoint and current-row fallback handle a command snapshot older than the run lock, including an existing stream in a full or over-limit namespace. Other quota failures remain failures; a genuinely new stream still needs a quota slot.
Actual local qualification, 2026-10-26 19:33–19:36 UTC, PHP8.3.36 with normally installed locked Workflow2.13.1:
git diff --checkpass.No PR opened yet: the operator's CI-first rule keeps delivery focused on the active merge-queue repair #788. Next: normal source review/hosted CI, protected merge, and released artifact acceptance. This is a source qualification receipt, not published-image acceptance. Task fixture databases are retained only for the active #789 follow-up and will be removed at that qualification checkpoint.
rmcdaniel commentedon Oct 26, 2026
General's three PHP handoffs are integrated in PR #791, head
9d7cd8fcba3dda8bcec984e7048d0ffa27566d00, tree191e7b605a87daa79e88f74ffbd2bc962f3d141c, on top of merged queue support #788. Product source is unchanged frome90a244c9; the final commit corrects test compatibility and JSON member-order assumptions.Qualification includes429 focused cases/5,123 assertions before the final test-only commit, with real competing actors on SQLite, MySQL8, MariaDB10.11 and PostgreSQL17. Final targeted real-commit checks pass14/116 on MariaDB and PostgreSQL. MySQL's original14-case run had one assertion failure caused solely by JSON object member order; all4 affected datasets pass36 assertions after correcting that assertion while preserving exact key count, values and types. The complete SQLite completion-upload class passes30/450. Two stale protocol-adoption tests discovered by the hosted3,242-case suite now pass2/34 locally. Changed PHP files pass Pint and diff checks. Existing6 PHPUnit metadata deprecations remain reported. Normally installed dependencies matching the lockfile are mounted read-only; no package source overrides are used.
Counterfactuals preserve the original all-four-engine start/prune data-loss reproduction, demonstrate premature preflight retention in the prior code, and show the prior cleanup deleting a schedule-owned object. The writer-wins and cleanup-wins outcomes, quota/stale-snapshot admission, savepoint ordering and callback rollback are covered without weakening fixtures.
Next: finish hosted CI on the exact PR head, fix any actual failures, merge through the protected queue, then publish the qualified PHP Server artifact and verify installed PHP/Python/Rust SDK and Nexus behavior. These issues stay open until that installed acceptance is recorded; source qualification is not release or deployment evidence. General owns this continuation so the Rust owner can continue native Nexus work.
rmcdaniel commentedon Oct 26, 2026
Server 2.11.3 and chart 0.3.3 are published. Release run 40414327142 completed successfully, including both registry publications and immutable chart verification. Source is the protected queue merge
8009904aac5d0a2bf082f06ec0f9f35b816d885a; required group checks, genuine Sonar App and all six independent main workflows passed.Both
durableworkflow/server:2.11.3andghcr.io/durable-workflow/server:2.11.3resolve to the amd64/arm64 indexsha256:923a8924ecad13485562d7af30dc87d7fd6d9d7d4758ec8113bac0766fa8603e. Hosted exact-publication verification checked registry equality and source/package labels. Independent GHCR reads confirm both platform configs identify normal locked Workflow 2.14.2 /d60ce5f60e70b834c5d4ef142c3d5dc415cd39fe. Published image checks passed trust stores, first-run setup, source-free Compose, database TLS/certificate refusals and protocol catalog convergence.Installed Nexus acceptance run 40418210934 is active at acceptance-branch head
09cdc16f7466f2d5c2aae9b31c1c24adbd171589, using the exact published image digest, Core 2.14.2, PHP SDK 3.5.1, Python SDK 2.11.2, CLI 2.3.0 and Waterline 2.13.0. Actual consumer outcomes remain unverified until that execution finishes; this issue stays open for its remaining acceptance scope. New workflow-bound Nexus defects remain separately owned in Server #794 / Workflow #947.