Repository navigation
Add a multi-node kill and restart qualification with results on four databases - #549
Merged
Merged
Conversation
Three native nodes share one MySQL, PostgreSQL or SQLite database behind HAProxy. Published-SDK workers and an offered one-activity workload reach them only through the balancer while nodes are killed and restarted and a worker is paused past its workflow task lease. Every workflow must complete once with its original output and an ordered history, and a database audit must find no duplicate committed outcome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All 810 offered workflows completed exactly once while nodes were killed and restarted and a worker was paused past its lease. The audits found no duplicate outcome or history sequence and no node panicked. The paused worker exited 7 times: 6 on native's lost-claim refusals, which lacked the fields PHP's ownership guard returns, and once on SDK 2.0.10's poll handling of a proxy 503. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The compose file no longer holds fixed database passwords: the driver generates both secrets for each run. The workload throws a dedicated verification exception, loads the capacity adapter once, and the driver's case statements and helpers handle unexpected values and name their arguments. MySQL and SQLite smoke runs passed after the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared fixtures qualify MariaDB 10.11 separately from MySQL, so the multi-node harness runs on it too, with the same image digest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…stories A current SDK worker that dies holding a workflow task lease leaves the task to the server's expired-lease recovery, which records RepairRequested, as published PHP's does. The workload accepted only the happy-path events and flagged that recovered workflow. It now accepts RepairRequested and reports the recoveries per workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The multi-node schedule also passed on MariaDB 10.11. A confirmation run with the #548 binary shows every late request refused with PHP's ownership guard body; SDK 2.0.10's remaining exits are its own lack of lost-claim handling. With PHP SDK 2.5.3, all 1,080 workflows completed once on the four databases and no worker exited. Summaries now record the SDK version, and the record describes the concurrent bootstrap race found on the way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel
changed the title
rmcdaniel
enabled auto-merge
October 20, 2026 21:35
|
This was referenced Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #325: the multi-node kill and restart gate.
What this adds
A harness for the multi-node qualification:
Each workflow must complete exactly once, with its original output, an ordered history, and an activity completed by the last attempt that started. A database audit must then find no duplicate workflow completion, activity completion or history sequence.
scripts/perf/run-server-port-multinode.sh: the driver, chaos loops, per-run database secrets, balancer summary, audit and panic check.scripts/perf/server-port-multinode.compose.ymlandmultinode-haproxy.cfg: the topology. The balancer retries only connection failures; a request a node accepted and then dropped fails back to its client. Workers restart on exit, and each restart is recorded.scripts/perf/multinode-workload.php: the offered workload and its per-workflow verification. The server's expired-lease recovery (RepairRequested) is accepted and counted.benchmarks/server-port/multinode/: the procedure, results and per-run summaries.Results
results-2026-10-20.md: the full schedule passed on all four databases in every run.
ce18934e1No native node panicked in any run. All binaries are unoptimized dev-profile builds; this checks correctness, not speed.
Worker exits. Before #548, native refused a paused worker's lost claims with only a reason. With #548, the remaining exits are SDK 2.0.10's own: it has no lost-claim handling, and it exits on a poll that gets a proxy's HTML 503. With SDK 2.5.3, no worker exited: the paused worker dropped 7 lost claims.
Found on the way.
🤖 Generated with Claude Code