Skip to content

Rust #325: take over published 2.11.1 databases, with service calls, worker sessions and search attribute definitions - #795

Merged
rmcdaniel merged 15 commits into
mainfrom
rust-325-takeover-features
Oct 27, 2026
Merged

rmcdaniel merged 15 commits into
mainfrom
rust-325-takeover-features

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Part of #325: migration and cutover.

Stacked on #793, which is in the merge queue. The first seven commits are #793's; once it merges, this PR's diff is only its own commits. This PR's CI tests its head merged into current main.

Native can now take over databases created by the current published release. Until now its inspection refused every Server 2.11.1 database with upgrade_source_catalog_unqualified, because it recognized only the frozen, 2.8.0 and 2.9.7 layouts. I confirmed this against a SQLite database the published 2.11.1 image bootstrapped.

Review tier: advisory (pre-RC native). It changes native takeover, the recorded source layouts, their tests and the parity cohorts. It touches no published PHP surface, credentials, production data or CI trust. Policy: #325 comment 6423232503.

What changed between 2.9.7 and 2.11.1

Core 2.10.2 (in Server 2.11.1) adds one nullable column, observability_context (Core's trace carrier), to workflow_runs, workflow_commands and workflow_tasks. Server's own migrations are unchanged.

Core's migrations run in file order, which gives two real physical layouts:

  • Fresh 2.11.1 (or created at 2.8.x or earlier, then migrated): the task carrier comes before 2.9.7's queue_delivery_reference.
  • 2.9.x migrated to 2.11.1: 2.9.7 had already added queue_delivery_reference, so the carrier follows it.

The change

  • Recorded layouts. rust/upgrade/php-2.11.1 and rust/upgrade/php-2.11.1-from-2.9.7 record the deltas for SQLite, PostgreSQL 17, MySQL 8.0 and MariaDB 10.11. They were read from databases the published images created: fresh 2.11.1, and 2.9.7 then migrated by 2.11.1, where the 2.11.1 bootstrap ran exactly Core's three carrier migrations. The chain grows to four layouts. MariaDB through Laravel's mariadb connection adds +mariadb-driver to either name.

  • Conversion, as decided on #325:

    • Takeover counts the carried values, reports them as observability_contexts_removed, and drops the three columns.
    • The rest converts as for 2.9.7: incarnations adopted, queue_delivery_reference dropped.
    • MySQL and MariaDB do this in journal step 9. It counts every carrier before dropping any column, so an interrupted takeover repeats it safely.
    • Native has no carrier, carriers are not business state or history, and the verified backup keeps them.
  • Search attribute definitions. Takeover no longer refuses a source holding custom definitions, a guard from before native served them. Native reads PHP's rows as they are, to list them, type upsert_search_attributes values and filter visibility.

  • Worker sessions. Takeover no longer refuses a source holding worker sessions.

    • Native serves sessions from PHP's own rows, but it also records which registration incarnation holds each one (dw_worker_session_holders). A session whose holder is no longer the worker's current registration is orphaned.
    • Takeover therefore records, for each active PHP session, the lease owner's current (adopted) incarnation as its holder, and reports session_holders_recorded. MySQL and MariaDB do this inside journal step 3, with the incarnations.
    • Without it, native reads the session as orphaned / worker_registration_replaced on first read. I checked this by removing the insert and watching the test fail.
  • CodeQL. The variable rename promised on Rust #325: Nexus catalog administration, service-call execution, admission limits and call reads #793 clears CodeQL's two false-positive cleartext-logging alerts in rust/tests/execution/service_endpoints.rs. A test variable named billing becomes first_endpoint; no test data changes.

  • CI. Each database job's upgrade cohorts gain a labeled run against the published 2.11.1 image, as 2.9.7 has: back up, inspect, fingerprint, take over and resume.

  • Worker-sessions contract. Rust #325: Nexus catalog administration, service-call execution, admission limits and call reads #793's first merge group failed on reacquire_terminal with ttl_span 1799 against the recorded 1800. That is a race in the published PHP reference: it computes ttl_expires_at from its request clock, then Eloquent stamps created_at at insert. The contract now reads a span exactly one second short as the recorded one, and nothing else; its tests check that two seconds short and one second long still fail. Native stamps both from one clock reading.

  • Docs. A checkpoint (docs/rust-server-port-checkpoints/2026-10-26-php-2111-takeover.md) and a corrected port-log paragraph. The earlier "refuse rows that carry trace context" text predated the Replace PHP Server with a compatible Rust runtime and safe in-place upgrade #325 decision.

  • Service calls. Takeover no longer refuses a source holding Nexus service calls.

    • Native serves them from PHP's own rows, but it keeps each deadline in dw_service_call_deadlines. Takeover copies 2.8.0+'s call_deadline_at there before dropping the column, and reports service_call_deadlines_copied. MySQL and MariaDB copy in journal step 6, and only what is not copied yet.
    • PHP's shared admission counters live in its cache and don't carry over.
  • Repair-pass deadline sweep. The pass reported its service-call timeout counters as zeros. It now enforces call deadlines as PHP's TaskWatchdog::runPass does:

    • up to 25 admitted open calls past their deadline, earliest first, within the pass's runs and instance;
    • a requested connection or queue matches the caller's or linked run exactly; a call with neither run matches any scope;
    • each call expires under the transition lock.

    Native still expires a call lazily when it is read; the sweep covers reads that don't, such as the catalog call list.

Still refused at takeover, each a separate conversion to come: runtime credentials, message streams, update validation tasks (pending ones need a decision once their waiting caller is gone) and external payloads.

Qualification

  • Real published databases: native prepared, fingerprinted and took over a SQLite database bootstrapped by the published 2.11.1 image (php-2.11.1) and one bootstrapped by 2.9.7 and migrated by 2.11.1 (php-2.11.1-from-2.9.7), each taken_over.
  • SQLite: all 53 upgrade tests pass, including:
    • takeover_converts_php_2111_databases_and_drops_their_trace_carriers, which builds both variants, seeds a run and a task carrying a trace context, and checks the layout name, observability_contexts_removed: 2, the dropped columns and that native opens the database;
    • takeover_keeps_php_search_attribute_definitions_that_native_then_lists;
    • takeover_keeps_php_worker_sessions_held_by_their_current_registration;
    • takeover_keeps_php_service_calls_and_their_deadlines, which also checks that the repair pass then times the due call out;
    • the extended chain unit test.
  • PostgreSQL 17, MySQL 8.0, MariaDB 10.11: qualification_takeover_converts_php_2111_and_native_opens_it passes for both variants, including a held worker session and its recorded holder, and a service call and its copied deadline. Every other takeover qualification test passes too: PostgreSQL 6, MySQL 7, MariaDB 7, including the MySQL journal kill-at-each-step test.
  • Execution: the_repair_pass_times_out_calls_past_their_deadline, with the service-call and repair-pass tests (23).
  • Full native suite on SQLite at c8b87bd16 (cargo test --all-targets): lib 332, execution 557, upgrade_inspect 2, all passing.
  • No PHP app/ changes.

🤖 Generated with Claude Code

rmcdaniel and others added 15 commits October 26, 2026 21:27
Native now answers the 15 admin routes under /api/service-endpoints as
published Server 2.11.1's ServiceCatalogController does: create, list, show,
update and delete endpoints, services and operations by lowercased name,
with PHP's validation, 404 and 409 bodies, Eloquent's dirty check and the
deletion refusals. The catalog reads under /api/service-catalog now render
the namespace's endpoint, service and operation rows with PHP's paging and
related limits; calls stay empty until native records service calls.

The shared rules gain Laravel's regex (strings and numbers) and prohibited,
and size a number without a numeric rule by its PHP string, as getSize does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service-endpoints fixture drives the 15 catalog administration routes
and the catalog reads over their rows in order, 72 requests in a fresh
namespace, and was recorded against published Server 2.11.1 on its SQLite
reference. Generated ids and whole-second times are placeholders that must
recur consistently; everything else is exact. The port log counts 161 of
175 served route methods and lists the three documented differences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Native now serves the endpoint, service and operation tables, so takeover
no longer refuses a source that holds their rows; service calls are still
refused. Tests write the rows as published PHP does (lowercase ULIDs,
microsecond times, json_encode documents with escaped slashes) on SQLite,
PostgreSQL, MySQL and MariaDB, take the database over, and check that
native renders, compares and extends them as its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST .../operations/{operation}/execute, GET .../service-calls/{call}
and POST .../service-calls/{call}/cancel now answer as published Server
2.11.1's ServiceCatalogController does over Workflow 2.10.2's
DefaultServiceControlPlane, for the handler kinds that stay accepted
(activity_execution and invocable_http): Laravel validation, the payload
codec and Avro arguments, admission at the service boundary with an
audit row for every decision (required roles, caller allow and deny
lists, the circuit breaker, and the 503 for an invalid shared limit),
the stamped handler, handler failures retried in the request, deadlines
expired lazily on every read, idempotent replays that re-check
authorization, and cancellation.

A workflow-bound handler kind, an operation with a valid shared limit,
an external_payload argument and cancelling into a linked workflow
answer 501 before anything is written.

Schema version 7 adds the native-owned dw_service_call_deadlines, so
the frozen PHP tables stay as they are and takeover layouts stay
recognisable. The service-calls fixture records 71 requests against
published 2.11.1, with shared placeholder matching for generated ids
and times.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sonar flags the closure as redundant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service boundary's per-minute rate and concurrency limits now admit
or refuse calls instead of answering 501. PHP keeps these counters in its
cache under a cache lock (SharedServiceBoundaryPolicy::admitWithLock);
native keeps them in the native-owned dw_service_admission_counters,
read and written under the transition lock in the call's transaction,
with PHP's keys and lives: a rate count per boundary and minute for
120 s, and a concurrency lease per boundary for a day. A sync call
returns its lease when the request ends, even after a failed dispatch;
an async call's lease stays until it expires, as PHP's does.

Schema version 7's migration is renamed 0007_service_calls.sql and adds
the counters table on SQLite, PostgreSQL, MySQL and MariaDB, with the
catalogs, takeover relations and parity schema scripts. The SQLite
parity schema check now also ignores the deadline index, which the
previous step missed.

The service-calls fixture gains 11 cases recorded against published
2.11.1: a rate refusal with its configured retry, another caller's own
window, an async lease kept, a sync lease returned, and the throttled
audit row. The probe opens the rate window at least 15 seconds before a
minute ends so it cannot roll over mid-case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /api/service-catalog/calls and GET /api/service-catalog/calls/{id}
now read the calls native records, as published Server 2.11.1's
ServiceCatalogReadController does: ServiceCatalog's scopes (by default
the calls the namespace made or that target it), the status and outcome
filters and buckets, newest accepted first as orderByDesc reads them on
each database, and ServiceCallView's list item and detail with the
caller and link references, the result envelope and the failure type.
Payload references and the Nexus result show only to the namespace that
owns the call. Before this, both answered PHP's empty page and
not-found even when calls existed.

The service-calls fixture gains nine catalog reads recorded against
published 2.11.1. Each list holds only accepted or only rejected calls:
a rejected call has no accepted_at, and where NULL sorts differs between
databases, for PHP as for native.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeQL's cleartext-logging heuristic treats a variable named billing as
sensitive financial data whenever an assertion message formats it. The
value is a test endpoint's JSON; the variable is now first_endpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Native inspection refused every database the current release created:
it recognized only the frozen, 2.8.0 and 2.9.7 layouts, and Core 2.10.2
(in Server 2.11.1) adds a nullable observability_context trace carrier
to workflow_runs, workflow_commands and workflow_tasks.

- rust/upgrade/php-2.11.1 records that delta for SQLite, PostgreSQL,
  MySQL and MariaDB, read from databases the published image
  bootstrapped.
- rust/upgrade/php-2.11.1-from-2.9.7 records the other real variant: a
  2.9.7 database migrated by 2.11.1 has the task carrier after 2.9.7's
  queue_delivery_reference, where a fresh one has it before.
- Takeover converts both as decided on #325: it counts the carried
  values (observability_contexts_removed), drops the three columns, and
  converts the rest as for 2.9.7. MySQL and MariaDB do it in journal
  step 9, which counts before it drops so a repeated step is safe.
- Tests build both variants on SQLite, PostgreSQL, MySQL and MariaDB,
  seed carried values and check the count, the dropped columns and
  that native then opens the database. The MySQL test builder now
  replays deltas with foreign key checks off, since 2.11.1 replaces
  workflow_runs, which other tables reference.
- The upgrade source cohorts gain a labeled run against the published
  2.11.1 image in the MySQL/MariaDB, PostgreSQL and SQLite jobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Takeover refused a source holding custom search attribute definitions,
a guard from before native served them. Native now reads PHP's
search_attribute_definitions rows as they are, to list them, type
upsert_search_attributes values and filter visibility, so takeover keeps
them. A test takes over a 2.11.1 database with definitions in two
namespaces and checks that native lists the default namespace's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Takeover refused a source holding worker sessions, a guard from before
native served them. Native now serves them from PHP's own
workflow_worker_sessions rows, but it also records which registration
incarnation holds each session (dw_worker_session_holders) and orphans
a session whose holder is no longer the worker's current registration.
A PHP session carried over with no holder row would read as held by a
replaced registration and be orphaned on the first read.

Takeover now records, for each active PHP session, the lease owner's
current incarnation (adopted from PHP's own) as its holder, and reports
session_holders_recorded. MySQL and MariaDB do it in journal step 3,
inside the transaction that adopts the incarnations.

A SQLite test takes over a 2.11.1 database with a held and a closed
session and checks that native still reads the held one as active;
without the holder, native reads it as orphaned. The 2.11.1
qualification tests check the recorded holder on PostgreSQL, MySQL and
MariaDB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#793's merge group failed one PostgreSQL corpus cell on worker-sessions
reacquire_terminal: ttl_span 1799 where the recording has 1800. PHP
computes a new session's ttl_expires_at from its request clock, then
Eloquent stamps created_at when it inserts the row; when the insert
lands in the next second, the probe's ttl_span (ttl_expires_at minus
created_at) reads one second short. Native stamps both from one clock
reading, so the race is the published reference's.

The contract now reads a ttl_span exactly one second short of the
recorded one as the recorded one, and nothing else. Its test checks
that one second short passes and that two seconds short or one second
long still fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The repair pass reported its service-call timeout counters as zeros.
PHP's TaskWatchdog::runPass also enforces call deadlines
(ServiceCallDeadline::expiredCallIds, then expire): up to 25 admitted
open calls past their deadline, earliest first, within the pass's runs
and instance. A requested connection or queue matches the caller's or
linked run exactly, as its whereHas does; a call with neither run
matches any scope.

Native now does the same from dw_service_call_deadlines, expiring each
call under the transition lock, and reports candidates, enforcements and
failures. Native still also expires a call lazily when it is read; the
sweep matters for reads that do not expire, such as the catalog call
list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Takeover refused a source holding any Nexus service call. Native now
serves calls from PHP's own workflow_service_calls rows, but it keeps a
call's deadline in the native-owned dw_service_call_deadlines, not in
2.8.0+'s call_deadline_at column, which takeover drops.

Takeover now copies each deadline into dw_service_call_deadlines before
dropping the column (reported as service_call_deadlines_copied). MySQL
and MariaDB do it in journal step 6, copying only those not copied yet,
so a repeated step is safe. The frozen layout has no deadlines.

A SQLite test takes over a 2.11.1 database with a call past its
deadline and one without: native shows the copied deadline, the repair
pass times the due call out, and the other stays open. The 2.11.1
qualification tests check the copied deadline on PostgreSQL, MySQL and
MariaDB. The takeover refusal tests no longer list service calls.

PHP's shared admission counters live in its cache and do not carry over,
so a concurrency lease an async call held before takeover is not
counted afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 26, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 26, 2026
@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 26, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Requeued pinned to c8b87bd16 at 23:41Z after the first merge-queue group failed.

The group's execution-and-codec (corpus) job (run 40418397301) failed on the published PHP 2.11.1 record, not the native one. The artifact's php.failure-excerpt.txt reads service-calls (product-fail): rated[0]. PHP answered 429 to the first call of the fixture's rate-limited operation, whose per-minute window should still have been empty.

This PR does not touch service calls' admission, the service-calls fixture or PHP. The same fixture passed on this head's own PR run and in #793's queue group. The retained PHP logs only cover the last two minutes of the job, so they show the job's later SQLite database is locked errors but not the 23:26 call. I'm handing the reference flake to General with this evidence.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Source diagnosis for the published PHP rated[0] failure: Server 2.11.1 contains a direct mechanism matching the observation; this is not a Rust #795 source regression.

SharedServiceBoundaryPolicy::admitWithLock() increments the per-minute cache counter before ServiceCallBoundary::admitFor() writes the durable service-call audit row. If that insert throws under SQLite lock pressure, the boundary catch calls release(), but the shared policy tracks and releases only concurrency keys. The consumed rate key remains for the minute bucket. The fixture's bounded lock-pressure middleware then correctly retries the original 503, and that retry sees the already-consumed one-call budget and becomes the observed 429. The same ordering and rollback gap remain at this PR head.

This source path explains the evidence but has not yet been replay-reproduced with an injected lock at the audit insert. Classification is admission-accounting/availability: the limiter fails closed and can overcount a request that never became durable. I found no authentication bypass, cross-tenant access, or reason to hold the native change.

The PHP repair should use a distinct failed-admission rollback, not ordinary completion release. Store the exact boundary/namespace rate and concurrency keys reserved during evaluation; on audit-record failure, decrement all of those keys under the existing lock. After a durable admission succeeds, normal completion must continue to release only concurrency and must not refund rate capacity. The exact stored keys matter if the rollback crosses a minute boundary.

Focused acceptance: force the audit insert to fail after a one-request rate reservation, verify a retry is accepted and the next fresh request is limited; preserve normal post-success rate consumption and concurrency release; cover rollback across a minute boundary; then rerun the published SQLite fixture with the one-second maintenance loop. The existing PR requeue can proceed independently.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 27, 2026
@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 27, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Requeued again pinned to c8b87bd16 at 00:05Z. Group run 40421521721 failed the same way: on the published PHP 2.11.1 record of service-calls, not native's. The PHP record's php.failure-excerpt.txt reads service-calls (product-fail): catalog handler failures[1].data: length (5 !== 4), one extra handler-failure call in PHP's own catalog list. General tracks the PHP defect as #796.

Merged via the queue into main with commit b4bdf82 Oct 27, 2026
57 checks passed
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged through the merge queue at 00:29Z as b4bdf82663af909cbc78a8787b0d9a1bac07356f (tree 1a96ef625504db786d934450e6cbf456f29973ef, parents eec4251a9eb57c8fdb249b8b7ae1e00781ddd3ca and the tested head c8b87bd16c2badd0d7e7ac7d2f743414cfec6a22). The third group run passed. The two earlier groups failed only on the published PHP 2.11.1 record of service-calls, which General tracks as #796. Follow-ups (maintenance loop, runtime credentials and update validation kept at takeover, 2.11.3 takeover, CI adapter cache) go up next as one PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant