Add a multi-node kill and restart qualification with results on four databases

benchmarks/server-port/multinode/README.md

@@ -0,0 +1,84 @@
+# Multi-node kill and restart qualification
+
+This qualifies native multi-node behavior: no lost durable work, no duplicate
+committed outcome, and no stale worker attempt accepted. It runs on each
+required database family while server nodes are killed and restarted, and
+while a worker loses its lease. It is a correctness check, not a capacity
+measurement.
+
+## Topology
+
+[`server-port-multinode.compose.yml`](../../../scripts/perf/server-port-multinode.compose.yml)
+runs these services:
+
+- **Database.** One of MySQL 8.4, MariaDB 10.11, PostgreSQL 17 or SQLite.
+  With SQLite, the three nodes share one database file on one host.
+- **Native nodes.** Three nodes with the same binary and configuration, one
+  CPU and 512 MiB each. All three start together against the empty
+  database, so bootstrap races too.
+- **Balancer.** HAProxy round-robins over the nodes. Health checks every
+  second take a dead node out after two failures. Like a production
+  balancer, it retries a request on another node only when it could not
+  connect, so the request never reached a server. A request a node accepted
+  and then dropped fails back to its client.
+- **Clients.** Two SDK workers and the offered workload, from the
+  [capacity bindings](../../capacity/v1/bindings/php) with the published PHP
+  SDK. They reach the nodes only through the balancer. A worker that exits is
+  restarted, as a supervisor would; `container-states.txt` records each
+  restart. A restart is an SDK finding, not a server failure, but it is
+  reported with the run.
+
+## Procedure
+
+[`run-server-port-multinode.sh`](../../../scripts/perf/run-server-port-multinode.sh)
+`OUTPUT_DIR mysql|mariadb|postgres|sqlite` offers one-activity workflows at
+0.5/s in three 180-second windows. Each window drains for up to 420 s, longer
+than the five-minute activity lease: an activity whose completion a killed node
+dropped is redelivered only when that lease expires. Two
+kinds of chaos run during the windows:
+
+- **Node kills.** Every 15 s, one node in turn is killed with SIGKILL and
+  restarted 4 s later.
+- **Worker pauses.** Every 90 s, one SDK worker is frozen with `docker pause`
+  for the workflow task lease plus 10 s, then resumed. By then the server has
+  given its workflow tasks to the other worker, so its completions are stale.
+
+[`multinode-workload.php`](../../../scripts/perf/multinode-workload.php)
+starts each workflow under its own ID. It retries a start, describe or history
+read that a dying node never answered, and a retried start finds its own run.
+Each window passes only if every planned workflow completed, with:
+
+- its original output;
+- contiguous history sequences;
+- exactly one `WorkflowStarted`, `ActivityScheduled`, `ActivityCompleted` and
+  `WorkflowCompleted`, in that order;
+- an `ActivityCompleted` from the last attempt that started;
+- no other events except `ActivityStarted` and `RepairRequested`. The server
+  records `RepairRequested` when it recovers the expired lease of a worker
+  that died holding a workflow task; the workload counts these recoveries.
+
+Afterwards the database audit must find:
+
+- every run completed;
+- no run with two `WorkflowCompleted` events;
+- no activity with two completed attempts;
+- no duplicate history sequence.
+
+The balancer log counts the stale completions the servers refused (`409`)
+and the requests a killed node never answered (`5xx`). Node logs are checked
+for panics.
+
+```sh
+# The locked SDK and an optimized binary, as the v2 README describes.
+export DW_PORT_SOURCE_ROOT=/abs/source DW_PORT_SDK_ROOT=/abs/sdk DW_PORT_NATIVE_BINARY=/abs/native/server
+for db in mysql mariadb postgres sqlite; do
+  bash scripts/perf/run-server-port-multinode.sh "results/$db" "$db"
+done
+```
+
+`--smoke` runs one 20-second window to check the setup. It is never a
+qualification.
+
+## Results
+
+- [2026-10-20](results-2026-10-20.md): MySQL, MariaDB, PostgreSQL and SQLite passed.

benchmarks/server-port/multinode/results-2026-10-20.md

@@ -0,0 +1,121 @@
+# Multi-node results, 2026-10-20
+
+The full schedule passed on MySQL 8.4, MariaDB 10.11, PostgreSQL 17 and
+SQLite. Every one of the 1,080 offered workflows completed exactly once, with
+its original output and an ordered history, while server nodes were killed
+and restarted and a worker was paused past its lease. The database audits
+found no duplicate committed outcome and no duplicate history sequence. No
+native node panicked.
+
+| Database | Workflows verified | Node kills | Worker pauses | Duplicate outcomes | Stale workflow completions refused | Worker exits |
+| --- | --- | --- | --- | --- | --- | --- |
+| MySQL 8.4 | 270 of 270 | 26 | 3 | 0 | 1 | 2 |
+| MariaDB 10.11 | 270 of 270 | 25 | 3 | 0 | 1 | 3 |
+| PostgreSQL 17 | 270 of 270 | 26 | 3 | 0 | 0 | 2 |
+| SQLite | 270 of 270 | 27 | 3 | 0 | 0 | 3 |
+
+Each workflow's activity ran once. No activity completion was lost to a
+killed node, so none waited for the five-minute activity lease.
+
+## Identity
+
+- Harness: `68d78be19` (MariaDB: `eeb510a3a`, which adds its profile), each
+  run from a clean detached checkout.
+- Native binary: an unoptimized dev-profile build of `rust-325-workflow-lease`
+  `ce18934e1`, SHA-256
+  `53d634e8fb7d96cab64d768e01d47db6e8c2f5a634cf0d6e67e6047b95ac03a9`. This
+  checks correctness, not speed.
+- Clients: the capacity bindings' locked PHP SDK 2.0.10. A later run used SDK 2.5.3 (below).
+- Machine: one Intel i5-6500 host with 4 logical CPUs and no other workload.
+- Raw node, balancer and worker logs were inspected and not kept. Each run's
+  summary is in `summary-<database>-2026-10-20.json`.
+
+## Worker exits
+
+Workers exited and were restarted 10 times in all:
+
+- **Native refusal shape (8 exits, all by the paused worker).** After a pause,
+  the worker sent requests for workflow tasks the server had given to the
+  other worker. Native refused the late lease renewals (`task_not_leased`) and
+  completions (`conflicting_completion`) with only a reason and the task
+  fence. Published PHP refuses the same requests with its ownership guard's
+  fields: the task status, and whether the run is closed. A current worker
+  reads those fields to drop a lost claim.
+  [#548](https://github.com/durable-workflow/server/pull/548) makes native
+  return PHP's bodies, with a parity fixture.
+- **SDK 2.0.10 poll (2 exits).** A poll got HAProxy's HTML 503 while a killed
+  node was still marked up. SDK 2.0.10 treats that body as invalid JSON and
+  exits. The SDK has retried polls on that failure since
+  [sdk-php#69](https://github.com/durable-workflow/sdk-php/issues/69).
+
+Current SDKs still exit when the same proxy failure hits a request for a
+task that is already claimed:
+[sdk-php#197](https://github.com/durable-workflow/sdk-php/issues/197).
+
+## Confirmation with the ownership guard
+
+The same schedule ran again with native built from
+[#548](https://github.com/durable-workflow/server/pull/548) (`f43e6fae2`,
+unoptimized dev profile, SHA-256
+`4c19c325cbd175d58d0b2710bb0450241f3760730514788eedf5185bab312e8f`) and the
+harness at `1a8add148` (MariaDB: `eeb510a3a`). Summaries are in
+`summary-guard-<database>-2026-10-20.json`.
+
+| Database | Workflows verified | Node kills | Duplicate outcomes | Worker exits |
+| --- | --- | --- | --- | --- |
+| MySQL 8.4 | 270 of 270 | 27 | 0 | 4 |
+| MariaDB 10.11 | 270 of 270 | 25 | 0 | 3 |
+| PostgreSQL 17 | 270 of 270 | 27 | 0 | 3 |
+| SQLite | 270 of 270 | 27 | 0 | 2 |
+
+No native node panicked. Every late request from the paused worker was now
+refused with PHP's ownership guard body. Eleven exits followed a late lease
+renewal after the run had closed, refused with `run_closed` and
+`can_continue: false`, the body published PHP returns. SDK 2.0.10 has no
+lost-claim handling (the SDK added it in 2.2.6), so it exits on that refusal
+from either server. The twelfth exit was the SDK 2.0.10 poll case above.
+
+## Current SDK
+
+The schedule ran a third time with the published PHP SDK 2.5.3 installed in
+place of the bindings' locked 2.0.10, and native from #548 as above. Summaries
+are in `summary-sdk253-<database>-2026-10-20.json`.
+
+| Database | Workflows verified | Node kills | Duplicate outcomes | Worker exits |
+| --- | --- | --- | --- | --- |
+| MySQL 8.4 | 270 of 270 | 25 | 0 | 0 |
+| MariaDB 10.11 | 270 of 270 | 26 | 0 | 0 |
+| PostgreSQL 17 | 270 of 270 | 24 | 0 | 0 |
+| SQLite | 270 of 270 | 25 | 0 | 0 |
+
+With a current SDK, the workers stayed up through every node kill and pause.
+Across the four runs the paused worker got 7 lost-claim refusals from native
+(`409` on a late lease renewal or completion) and dropped each claim without
+exiting.
+
+An earlier PostgreSQL run with the same SDK had one worker exit. A node was
+killed while answering that worker's lease renewal, and the balancer returned
+502, which the SDK does not retry while a task is claimed
+([sdk-php#197](https://github.com/durable-workflow/sdk-php/issues/197)). The
+exited worker still held its lease. Native recovered the expired lease as PHP
+does, recording `RepairRequested`, and the workflow completed once. The
+workload then accepted only the happy-path events and flagged that history.
+It now accepts and counts the recovery, and the PostgreSQL row above is the
+rerun.
+
+## Concurrent bootstrap
+
+All three nodes of each run start together against the empty database. In a
+later smoke run on MySQL, one node exited with `unsupported_rust_schema`: it
+started while another node was between creating the native marker table and
+recording its marker. MySQL and MariaDB commit DDL on their own, so that state
+is visible. The correction lets the node wait for the bootstrap lock, and its
+kill qualification covers the window.
+
+## Earlier attempt
+
+The first full MySQL run had no balancer retry and no worker restarts. Both
+workers exited on HAProxy 503 pages while killed nodes were still marked up,
+and window 2 completed nothing. That run measured the clients rather than the
+server. It led to the connect-failure retry and the restart policy in the
+current harness.

benchmarks/server-port/multinode/summary-guard-mariadb-2026-10-20.json

@@ -0,0 +1,132 @@
+{
+  "identity": {
+    "database": "mariadb",
+    "compose_project": "dw-mn-mariadb-20261020t190725",
+    "source_revision": "eeb510a3a59b18e0ad530e0a837325ac1d84d4fe",
+    "source_modified_files": 0,
+    "native_binary_sha256": "4c19c325cbd175d58d0b2710bb0450241f3760730514788eedf5185bab312e8f",
+    "smoke": false,
+    "schedule": {
+      "rate": 0.5,
+      "offer_seconds": 180,
+      "drain_seconds": 420,
+      "windows": 3,
+      "node_kill_every_seconds": 15,
+      "node_down_seconds": 4,
+      "worker_pause_every_seconds": 90,
+      "worker_pause_seconds": 70,
+      "workflow_task_lease_seconds": 60
+    },
+    "machine": {
+      "kernel": "5.15.0-198-generic",
+      "logical_cpus": 4,
+      "cpu_model": "Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz"
+    }
+  },
+  "windows": [
+    {
+      "sdk_php": "2.0.10",
+      "outcome": "pass",
+      "planned_starts": 90,
+      "accepted_starts": 90,
+      "completed": 90,
+      "verified": 90,
+      "activity_attempts_per_workflow": {
+        "1": 90
+      },
+      "repairs_by_workflow": null,
+      "retries": [],
+      "errors": []
+    },
+    {
+      "sdk_php": "2.0.10",
+      "outcome": "pass",
+      "planned_starts": 90,
+      "accepted_starts": 90,
+      "completed": 90,
+      "verified": 90,
+      "activity_attempts_per_workflow": {
+        "1": 90
+      },
+      "repairs_by_workflow": null,
+      "retries": [],
+      "errors": []
+    },
+    {
+      "sdk_php": "2.0.10",
+      "outcome": "pass",
+      "planned_starts": 90,
+      "accepted_starts": 90,
+      "completed": 90,
+      "verified": 90,
+      "activity_attempts_per_workflow": {
+        "1": 90
+      },
+      "repairs_by_workflow": null,
+      "retries": [],
+      "errors": []
+    }
+  ],
+  "chaos": {
+    "kill": 25,
+    "pause": 3,
+    "start": 25,
+    "unpause": 3
+  },
+  "balancer": {
+    "requests": 3978,
+    "workflow_task_completions_409": 0,
+    "activity_task_completions_409": 0,
+    "responses_5xx": 0
+  },
+  "database_audit": {
+    "runs completed": 270,
+    "tasks activity completed": 270,
+    "tasks workflow completed": 540,
+    "attempts completed": 270,
+    "duplicate_workflow_completions": 0,
+    "duplicate_activity_completions": 0,
+    "duplicate_history_sequences": 0
+  },
+  "containers": {
+    "lb": {
+      "restarts": 0,
+      "oom_killed": false,
+      "exit_code": 0
+    },
+    "mariadb": {
+      "restarts": 0,
+      "oom_killed": false,
+      "exit_code": 0
+    },
+    "native-1": {
+      "restarts": 0,
+      "oom_killed": false,
+      "exit_code": 0
+    },
+    "native-2": {
+      "restarts": 0,
+      "oom_killed": false,
+      "exit_code": 0
+    },
+    "native-3": {
+      "restarts": 0,
+      "oom_killed": false,
+      "exit_code": 0
+    },
+    "sdk-worker-1": {
+      "restarts": 3,
+      "oom_killed": false,
+      "exit_code": 0
+    },
+    "sdk-worker-2": {
+      "restarts": 0,
+      "oom_killed": false,
+      "exit_code": 0
+    }
+  },
+  "native_panics": 0,
+  "sdk_worker_exits": {
+    "sdk-worker-1: Uncaught DurableWorkflow\\Exception\\TransportException: ": 3
+  }
+}