Rust #325: take over published 2.11.1 databases, with service calls, worker sessions and search attribute definitions

.github/workflows/server-parity.yml

@@ -49,6 +49,8 @@ on:
       - tests/Unit/ServerParityActivityCancellationAckContractTest.mjs
       - tests/Unit/ServerParityCancellationScopesContractTest.mjs
       - tests/Unit/ServerParityServiceCatalogContractTest.mjs
+      - tests/Unit/ServerParityServiceEndpointsContractTest.mjs
+      - tests/Unit/ServerParityServiceCallsContractTest.mjs
       - tests/Unit/ServerParityWorkflowImportContractTest.mjs
       - tests/Unit/ServerParityWaterlineImportContractTest.mjs
       - tests/Unit/ServerParitySystemMetricsContractTest.mjs
@@ -849,6 +851,22 @@ jobs:
           DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.9.7
           DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:30c34c5f3256a812fbccbe505a068309b58b0c14d01ffff0465d88604cb36d50
         run: bash scripts/conformance/server-parity/upgrade-source-cleanup.sh mysql
+      - name: Back up, take over and resume a published PHP 2.11.1 upgrade source
+        if: ${{ matrix.part == 'cohorts' }}
+        timeout-minutes: 8
+        env:
+          DW_PARITY_UPGRADE_LABEL: php2111
+          DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.11.1
+          DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102
+        run: bash scripts/conformance/server-parity/upgrade-source-profile.sh mysql
+      - name: Retire PHP 2.11.1 upgrade source cohort
+        if: ${{ always() && matrix.part == 'cohorts' }}
+        timeout-minutes: 3
+        env:
+          DW_PARITY_UPGRADE_LABEL: php2111
+          DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.11.1
+          DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102
+        run: bash scripts/conformance/server-parity/upgrade-source-cleanup.sh mysql
       - name: Start the native node the corrected marker cohort uses
         id: prepare_corrected
         if: ${{ matrix.part == 'corrected-markers' }}
@@ -1483,6 +1501,22 @@ jobs:
           DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.9.7
           DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:30c34c5f3256a812fbccbe505a068309b58b0c14d01ffff0465d88604cb36d50
         run: bash scripts/conformance/server-parity/upgrade-source-cleanup.sh pgsql
+      - name: Back up, take over and resume a published PHP 2.11.1 upgrade source
+        if: ${{ matrix.part == 'cohorts' }}
+        timeout-minutes: 8
+        env:
+          DW_PARITY_UPGRADE_LABEL: php2111
+          DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.11.1
+          DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102
+        run: bash scripts/conformance/server-parity/upgrade-source-profile.sh pgsql
+      - name: Retire PHP 2.11.1 upgrade source cohort
+        if: ${{ always() && matrix.part == 'cohorts' }}
+        timeout-minutes: 3
+        env:
+          DW_PARITY_UPGRADE_LABEL: php2111
+          DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.11.1
+          DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102
+        run: bash scripts/conformance/server-parity/upgrade-source-cleanup.sh pgsql
       - name: Start the native node the corrected marker cohort uses
         id: prepare_corrected
         if: ${{ matrix.part == 'corrected-markers' }}
@@ -1807,7 +1841,7 @@ jobs:
           bash scripts/ci/run-composer.sh --rm -v "$PWD:/source" -w /source -- \
             install --working-dir=scripts/conformance/server-parity --no-interaction --no-progress
           docker run --rm --user=1000:1000 --entrypoint node -v "$PWD:/source:ro" -w /source \
-            "$image" --test tests/Unit/ServerParityRunnerTest.mjs tests/Unit/ServerParitySdkDirectionLanesTest.mjs tests/Unit/ServerParityCodecDeadlineCellsTest.mjs tests/Unit/ServerParityCooperativeContractTest.mjs tests/Unit/ServerParityChildCancellationContractTest.mjs tests/Unit/ServerParityScheduleContractTest.mjs tests/Unit/ServerParityVisibilityContractTest.mjs tests/Unit/ServerParityAdmissionContractTest.mjs tests/Unit/ServerParityRoleTokenContractTest.mjs tests/Unit/ServerParityWorkerDeregistrationContractTest.mjs tests/Unit/ServerParitySearchAttributeContractTest.mjs tests/Unit/ServerParityWorkerListingContractTest.mjs tests/Unit/ServerParityWorkerReplacementContractTest.mjs tests/Unit/ServerParityPatchDeploymentContractTest.mjs tests/Unit/ServerParityWaitingHistoryContractTest.mjs tests/Unit/ServerParityTaskRetryContractTest.mjs tests/Unit/ServerParityPublishedPatchContractTest.mjs tests/Unit/ServerParityPublishedPatchTransportTest.mjs tests/Unit/ServerParityRecordFileTest.mjs tests/Unit/ServerParityFailureLocationTest.mjs tests/Unit/ServerParityFailureExcerptTest.mjs tests/Unit/ServerParityLateWorkflowCompletionContractTest.mjs tests/Unit/ServerParityStorageTestContractTest.mjs tests/Unit/ServerParitySchedulePoliciesContractTest.mjs tests/Unit/ServerParitySignalsBeforeWorkerContractTest.mjs tests/Unit/ServerParityScheduleCatalogContractTest.mjs tests/Unit/ServerParityOperatorMetricsContractTest.mjs tests/Unit/ServerParityActivityLeaseRenewalContractTest.mjs tests/Unit/ServerParityRecordLocalActivityContractTest.mjs tests/Unit/ServerParityActivityCancellationAckContractTest.mjs tests/Unit/ServerParityServiceCatalogContractTest.mjs tests/Unit/ServerParityWorkflowImportContractTest.mjs tests/Unit/ServerParityWaterlineImportContractTest.mjs tests/Unit/ServerParitySystemMetricsContractTest.mjs tests/Unit/ServerParityOperatorDashboardTypesContractTest.mjs tests/Unit/ServerParityPrometheusMetricsContractTest.mjs tests/Unit/ServerParityPayloadCleanupContractTest.mjs tests/Unit/ServerParityRepairPassContractTest.mjs tests/Unit/ServerParityClusterInfoContractTest.mjs tests/Unit/ServerParityNamespaceDeleteContractTest.mjs tests/Unit/ServerParityMessageStreamsContractTest.mjs tests/Unit/ServerParityCancellationScopesContractTest.mjs tests/Unit/ServerParityWorkflowPollRequestContractTest.mjs tests/Unit/ServerParityUpdateValidationContractTest.mjs tests/Unit/ServerParityFixtureSelectionTest.mjs tests/Unit/ServerParityUpdateValidationRetryTest.mjs tests/Unit/ServerParityKilledAuthorPollTest.mjs tests/Unit/ServerParityImagePullTest.mjs tests/Unit/ServerParityWorkerSessionsContractTest.mjs tests/Unit/ServerParityWorkerSessionActivitiesContractTest.mjs tests/Unit/ServerParityNexusOperationsContractTest.mjs tests/Unit/ServerParityBridgeAdaptersContractTest.mjs tests/Unit/ServerParityWorkflowStreamWritesContractTest.mjs > parity-evidence/comparator-tests.txt
+            "$image" --test tests/Unit/ServerParityRunnerTest.mjs tests/Unit/ServerParitySdkDirectionLanesTest.mjs tests/Unit/ServerParityCodecDeadlineCellsTest.mjs tests/Unit/ServerParityCooperativeContractTest.mjs tests/Unit/ServerParityChildCancellationContractTest.mjs tests/Unit/ServerParityScheduleContractTest.mjs tests/Unit/ServerParityVisibilityContractTest.mjs tests/Unit/ServerParityAdmissionContractTest.mjs tests/Unit/ServerParityRoleTokenContractTest.mjs tests/Unit/ServerParityWorkerDeregistrationContractTest.mjs tests/Unit/ServerParitySearchAttributeContractTest.mjs tests/Unit/ServerParityWorkerListingContractTest.mjs tests/Unit/ServerParityWorkerReplacementContractTest.mjs tests/Unit/ServerParityPatchDeploymentContractTest.mjs tests/Unit/ServerParityWaitingHistoryContractTest.mjs tests/Unit/ServerParityTaskRetryContractTest.mjs tests/Unit/ServerParityPublishedPatchContractTest.mjs tests/Unit/ServerParityPublishedPatchTransportTest.mjs tests/Unit/ServerParityRecordFileTest.mjs tests/Unit/ServerParityFailureLocationTest.mjs tests/Unit/ServerParityFailureExcerptTest.mjs tests/Unit/ServerParityLateWorkflowCompletionContractTest.mjs tests/Unit/ServerParityStorageTestContractTest.mjs tests/Unit/ServerParitySchedulePoliciesContractTest.mjs tests/Unit/ServerParitySignalsBeforeWorkerContractTest.mjs tests/Unit/ServerParityScheduleCatalogContractTest.mjs tests/Unit/ServerParityOperatorMetricsContractTest.mjs tests/Unit/ServerParityActivityLeaseRenewalContractTest.mjs tests/Unit/ServerParityRecordLocalActivityContractTest.mjs tests/Unit/ServerParityActivityCancellationAckContractTest.mjs tests/Unit/ServerParityServiceCatalogContractTest.mjs tests/Unit/ServerParityServiceEndpointsContractTest.mjs tests/Unit/ServerParityServiceCallsContractTest.mjs tests/Unit/ServerParityWorkflowImportContractTest.mjs tests/Unit/ServerParityWaterlineImportContractTest.mjs tests/Unit/ServerParitySystemMetricsContractTest.mjs tests/Unit/ServerParityOperatorDashboardTypesContractTest.mjs tests/Unit/ServerParityPrometheusMetricsContractTest.mjs tests/Unit/ServerParityPayloadCleanupContractTest.mjs tests/Unit/ServerParityRepairPassContractTest.mjs tests/Unit/ServerParityClusterInfoContractTest.mjs tests/Unit/ServerParityNamespaceDeleteContractTest.mjs tests/Unit/ServerParityMessageStreamsContractTest.mjs tests/Unit/ServerParityCancellationScopesContractTest.mjs tests/Unit/ServerParityWorkflowPollRequestContractTest.mjs tests/Unit/ServerParityUpdateValidationContractTest.mjs tests/Unit/ServerParityFixtureSelectionTest.mjs tests/Unit/ServerParityUpdateValidationRetryTest.mjs tests/Unit/ServerParityKilledAuthorPollTest.mjs tests/Unit/ServerParityImagePullTest.mjs tests/Unit/ServerParityWorkerSessionsContractTest.mjs tests/Unit/ServerParityWorkerSessionActivitiesContractTest.mjs tests/Unit/ServerParityNexusOperationsContractTest.mjs tests/Unit/ServerParityBridgeAdaptersContractTest.mjs tests/Unit/ServerParityWorkflowStreamWritesContractTest.mjs > parity-evidence/comparator-tests.txt
           docker run --rm --user=1000:1000 --cpus=1 --memory=256m --memory-swap=256m \
             --network=none --entrypoint node -v "$PWD:/source:ro" -w /source \
             "$image" --test tests/Unit/ServerParityCodecDeadlineLoadTest.mjs > parity-evidence/codec-load-tests.txt
@@ -2130,6 +2164,24 @@ jobs:
           DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.9.7
           DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:30c34c5f3256a812fbccbe505a068309b58b0c14d01ffff0465d88604cb36d50
         run: bash scripts/conformance/server-parity/upgrade-source-cleanup.sh sqlite
+      - name: Back up, take over and resume a published PHP 2.11.1 upgrade source
+        if: ${{ matrix.part == 'cohorts' }}
+        timeout-minutes: 8
+        env:
+          RESOURCE_SCOPE: dw-parity-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}
+          DW_PARITY_UPGRADE_LABEL: php2111
+          DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.11.1
+          DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102
+        run: bash scripts/conformance/server-parity/upgrade-source-profile.sh sqlite
+      - name: Retire PHP 2.11.1 upgrade source cohort
+        if: ${{ always() && matrix.part == 'cohorts' }}
+        timeout-minutes: 3
+        env:
+          RESOURCE_SCOPE: dw-parity-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}
+          DW_PARITY_UPGRADE_LABEL: php2111
+          DW_PARITY_UPGRADE_EXPECTED_LAYOUT: php-2.11.1
+          DW_PARITY_UPGRADE_PHP_IMAGE: ghcr.io/durable-workflow/server@sha256:bf8bed13f5f809ee3425233180a581353eac4683adbee5c4ecbf63bb18bbd102
+        run: bash scripts/conformance/server-parity/upgrade-source-cleanup.sh sqlite
       - name: Start the native node the corrected marker cohort uses
         id: prepare_corrected
         if: ${{ matrix.part == 'corrected-markers' }}

docs/rust-server-port-checkpoints/2026-10-26-php-2111-takeover.md

@@ -0,0 +1,86 @@
+# October 26, takeover of published 2.11.1 databases
+
+- **Found by.** Native upgrade inspection of a database that the published
+  Server 2.11.1 image bootstrapped answered `upgrade_source_catalog_unqualified`.
+  Native recognized only the frozen, 2.8.0 and 2.9.7 layouts, so no
+  installation on the current release could take over in place.
+- **Layout.** Core 2.10.2 (in Server 2.11.1) adds a nullable
+  `observability_context` column, Core's trace carrier, to `workflow_runs`,
+  `workflow_commands` and `workflow_tasks`. Server's own migrations are
+  unchanged from 2.9.7. The deltas were read from databases the published
+  image bootstrapped on SQLite, PostgreSQL 17, MySQL 8.0 and MariaDB 10.11
+  ([`rust/upgrade/php-2.11.1`](../../rust/upgrade/php-2.11.1/README.md)).
+- **Two physical variants.** Core's migrations run in file order.
+  - A fresh 2.11.1 database puts the task carrier before 2.9.7's
+    `queue_delivery_reference`.
+  - A database 2.9.7 created and 2.11.1 then migrated has the opposite
+    order. It is recorded separately, read from real 2.9.7 databases that the
+    2.11.1 image migrated
+    ([`php-2.11.1-from-2.9.7`](../../rust/upgrade/php-2.11.1-from-2.9.7/README.md)).
+  - Inspection names whichever matches. MariaDB through Laravel's `mariadb`
+    connection adds `+mariadb-driver` to either.
+- **Conversion.** As decided on
+  [#325](https://github.com/durable-workflow/server/issues/325#issuecomment-6306545520),
+  takeover counts the carried values, reports them as
+  `observability_contexts_removed`, and drops the three columns. It also drops
+  2.9.7's delivery reference and adopts PHP's worker incarnations, as for 2.9.7.
+  MySQL and MariaDB take this as journal step 9, which counts every carrier
+  before dropping any column, so an interrupted takeover repeats it safely.
+- **Tests.**
+  - `takeover_converts_php_2111_databases_and_drops_their_trace_carriers` on
+    SQLite.
+  - `qualification_takeover_converts_php_2111_and_native_opens_it` on
+    PostgreSQL 17, MySQL 8.0 and MariaDB 10.11.
+  - Each builds both variants from the frozen layout and seeds a run and a task
+    that carry a trace context. It then checks the layout name, the count of 2,
+    that the columns are gone, and that native opens the database.
+  - Every other takeover qualification test still passes on the three
+    databases.
+- **Real databases.** Native prepared, fingerprinted and took over a SQLite
+  database bootstrapped by the published 2.11.1 image (`php-2.11.1`) and one
+  bootstrapped by 2.9.7 and migrated by 2.11.1 (`php-2.11.1-from-2.9.7`).
+- **CI.** Each database job's upgrade cohorts gain a labeled run against the
+  published 2.11.1 image, as for 2.9.7.
+- **Search attribute definitions.** Takeover no longer refuses a source
+  holding custom search attribute definitions. Native reads PHP's rows as
+  they are, to list them, type `upsert_search_attributes` values and filter
+  visibility. `takeover_keeps_php_search_attribute_definitions_that_native_then_lists`
+  checks that native lists a 2.11.1 database's definitions after takeover.
+- **Worker sessions.** Takeover no longer refuses a source holding worker
+  sessions.
+  - Native records which registration incarnation holds each session and
+    orphans a session whose holder is no longer the worker's current
+    registration.
+  - So takeover records, for each active PHP session, the lease owner's
+    current incarnation (adopted from PHP's own) as its holder, and reports
+    `session_holders_recorded`. MySQL and MariaDB do it in journal step 3,
+    with the incarnations.
+  - `takeover_keeps_php_worker_sessions_held_by_their_current_registration`
+    checks that native still reads such a session as active. Without the
+    holder, native reads it as orphaned with `worker_registration_replaced`.
+  - The 2.11.1 qualification tests check the recorded holder on PostgreSQL,
+    MySQL and MariaDB.
+- **Service calls.** Takeover no longer refuses a source holding Nexus
+  service calls.
+  - Native serves them from PHP's own rows. It keeps each call's deadline in
+    `dw_service_call_deadlines`, so takeover copies 2.8.0+'s
+    `call_deadline_at` there before dropping the column, and reports
+    `service_call_deadlines_copied`. MySQL and MariaDB copy in journal step 6,
+    and only what is not copied yet.
+  - PHP's shared admission counters live in its cache and do not carry over.
+  - `takeover_keeps_php_service_calls_and_their_deadlines` checks the copied
+    deadline and that the repair pass then times the due call out. The 2.11.1
+    qualification tests check the copied deadline on PostgreSQL, MySQL and
+    MariaDB.
+- **Repair pass.** The pass now times out due service calls as PHP's
+  `TaskWatchdog::runPass` does:
+  - up to 25 admitted open calls past their deadline, earliest first, within
+    the pass's runs and instance;
+  - a requested connection or queue matches the caller's or linked run
+    exactly; a call with neither run matches any scope.
+
+  Before, it reported the three service-call counters as zeros.
+  `the_repair_pass_times_out_calls_past_their_deadline` covers it.
+- **Still refused.** Takeover still refuses a source holding rows native
+  cannot carry over yet: runtime credentials, message streams, update
+  validation tasks and external payloads.