Skip to content

CI: corrected markers as their own fixture part; fix and locate the #754 flakes - #756

Merged
rmcdaniel merged 9 commits into
mainfrom
rust-325-ci-corrected-part
Oct 25, 2026
Merged

rmcdaniel merged 9 commits into
mainfrom
rust-325-ci-corrected-part

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Cuts the Shared Server fixtures' long pole and removes or locates the flakes that have been costing reruns (#754).

Wall time

  • Corrected markers get their own part (959504f). After CI: run the published-SDK direction lanes as their own fixture part #752 the cohorts part is the long pole at 18 to 22 minutes. Its slowest cohort (corrected markers, ranges, clocks and activity expiry) takes 8.8 minutes by itself and needs only a native node. It now runs as a fifth matrix part, corrected-markers, in the MySQL, PostgreSQL and SQLite jobs. That part starts its own native node, runs the cohort, and retires its resources. Every cohort still runs exactly once per database. ServerParityPartsTest.mjs requires the five parts and checks that the cohort, its node and its cleanup live in the new part.

Flakes (#754)

  • native-b readiness race (7723ae5). The corpus step started native-a and native-b together, waited up to 30 s for native-a, then checked native-b only once. Job 121059713123 failed with curl: (7) Failed to connect to parity-native-b port 8080 after 0 ms before any fixture was recorded. native-b now gets the same bounded wait.
  • query-sdk-python-rust status 0 against native (87b52c8). The probe stopped the query author as soon as the caller returned. Native wakes the waiting caller right after the completion commits, so the fast Rust client could finish before the author received its own completion response. Published Python 2.5.4 completes query tasks inline on a poller thread, and Worker.stop() cancels that thread, so the in-flight completion was cancelled and recorded as client_cancelled with status 0. The probe now writes the author's stop trigger only after the author's journal shows a successful, uncancelled completion of the query task it claimed, within the existing deadline. Full mechanism: Server parity: legacy-marker 503 recovery check fails intermittently on published PHP (SQLite) #754 comment 6393052357.
  • Request outline for failed fixtures (67d6dc4). The legacy-marker flake (same original poll recovers) has failed twice in the published PHP record, and the request order around the suspected 503 was lost both times. When fixtures differ, the recorder now writes <record>.failure-excerpt.txt. For each failed case it lists, in order, every request's method, path, status, transport flags, retry-after, and worker, poll, queue and lease identity, never bodies. It is bounded per case and in total, and the artifact summary keeps it at top priority.
  • Locatable failures (4ff1f2c, 7246971). A wrong SDK query status now names the phase, method, path and the observer's transport flags. Any fixture refusal now prints the contract file and line that refused it (at scripts/conformance/server-parity/<contract>.mjs:<line>), so a bare false !== true such as python-clock-genuine-one can be traced from the retained log. Neither change prints payload values.

Verification

  • New node tests: ServerParityFailureLocationTest.mjs (3) and two new cases in ServerParitySdkQueryDirectionContractTest.mjs. Both new contract cases fail against the previous contract.
  • New PHPUnit cases in ServerParitySdkQueryTransportTest.php: the author is released after a successful completion and stays held for status 0, client_cancelled, transport_error, an empty response object, a foreign receipt, the wrong order, or a foreign workflow or run.
  • Locally on this head: 106 node tests across the parts, lanes, failure-location, failure-excerpt, artifact-summary, query-contract, record-file and fixture-selection suites; 18 PHPUnit transport tests; actionlint; the Docker isolation policy; and the public boundary check over the branch range.

The query-author race can't be reproduced deterministically in a unit test. It counts as fixed once the sdk-directions lane stays green across cycles.

🤖 Generated with Claude Code

rmcdaniel and others added 9 commits October 25, 2026 02:37
After the SDK direction lanes moved to their own part (#752), the cohorts
part is the Shared Server fixtures' long pole at 18 to 22 minutes. Its
slowest cohort, corrected markers, ranges, clocks and activity expiry,
takes 8.8 minutes on its own (run 40136659746, mysql-8.0). It needs only a
native node, which it already had to itself among the cohorts.

It now runs in a fifth matrix part, corrected-markers, in the MySQL,
PostgreSQL and SQLite jobs. That part starts its own native node exactly as
the cohorts part starts its node, runs the cohort, and retires its
resources. The cohorts part keeps every other cohort. Every cohort still
runs once per database.

The parts test now requires five parts and the corrected cohort, its node
and its cleanup in the corrected-markers part.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MySQL and PostgreSQL corpus steps start native-a and native-b together,
wait up to 30 seconds for native-a, then check native-b once. On run
40156433545 (#745, mysql-execution mariadb-10.11 corpus), native-b was
still binding its port: that single check exited 7 ("Failed to connect to
parity-native-b port 8080"), and the step failed before recording anything.

native-b now gets the same bounded readiness loop. The survivor check after
native-a is killed stays a single check, since native-b has been serving
for the whole corpus by then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
query-sdk-python-rust has twice recorded HTTP status 0 against native
(#754). The comparator printed only "0 !== 200", and the artifact keeps
no journal, so we could not tell which request it was or whether the
client closed first or the upstream connection failed. Each status check
now reports the phase, method, path and the observer's transport flags,
never payload bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
query-sdk-python-rust twice recorded the author's query completion as
HTTP status 0 against native (#754). The probe stopped the author as soon
as the caller returned. Native wakes the waiting caller right after the
completion commits, so a fast caller (the Rust client) can finish before
the author's own completion response arrives. Published Python 2.5.4 runs
query tasks inline on a poller thread, and Worker.stop() cancels that
thread's task, so the in-flight completion was cancelled and the observer
recorded it as client-cancelled with status 0. Native had already applied
it, which is why the caller saw the right answer.

The probe now waits, within the existing fixture deadline, until the
author's journal shows its own successful, uncancelled completion of the
query task it claimed, and only then writes the stop trigger. PHP also
wakes its waiting caller through a signal channel, so the same race is
possible there in principle; it has only been observed against native,
and the wait covers both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed fixture logged only its assertion message, such as "false !==
true" for python-clock-genuine-one (#754). The record that would show
which check refused is not retained, so the failure could not be located.
The recorder now adds the first scripts/conformance stack frame
("at scripts/conformance/server-parity/<contract>.mjs:<line>") after the
message. Frames carry source paths and line numbers, never payload
values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-part

# Conflicts:
#	.github/workflows/server-parity.yml
The legacy-marker flake (#754) has failed twice in the published PHP
record, and the request order around the suspected 503 was not retained
either time, so the hypothesis could not be confirmed. When fixtures
differ, the recorder now writes <record>.failure-excerpt.txt next to the
record. For each failed case it lists, in observation order, every wire
request's method, path, status, transport flags, retry-after, and worker,
poll, queue and lease identity. It never reads request or response
bodies. The excerpt is bounded per case and in total, and the artifact
summary keeps it at the same priority as the recorder's own output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rrected-part

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Timing note, measured before this PR's run finishes. #739's last cycle (run 40162397220, same workflow without this PR) ran 25 min wall. The longest jobs:

  • corpus parts: 17 to 21 min. One step dominates: "Compare three runtimes, recover killed native nodes and refuse published PHP takeover", 1031 s on mysql-8.0.
  • cohorts parts: 18 to 20 min. Corrected markers was the largest step, 478 s on postgres-17.

So the corrected-markers split removes cohorts from the critical path, but corpus is now the long pole by itself, and the wall-time gain from this PR alone will be small. I'll post the measured per-part times when this run is terminal. The next CI change is partitioning the corpus comparison step across two parts with the fixture selection #739 introduced. The flake fixes stand on their own: each flake has cost a full-job rerun of 17 to 21 minutes.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Attempt 1 of run 40164318863 stopped before the fixture matrix started. published-sdk-adapters (job 121082165652) exited 125 because docker run couldn't pull the pinned Python image from mirror.gcr.io: docker: unknown blob (failed-sdk-setup-diagnostics, python-install.txt). That's a registry fault in setup, not this PR's code; the Rust adapter build in the same job finished. Attempt 2 reruns the failed jobs and the parts that depend on them.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Cloud's read-only container-wrapper review of exact head 058861cf97eb33fcc702de191ce8fb50d73ad97b is complete. Scope was .github/workflows/server-parity.yml and the existing corrected-marker cleanup entry point, with no native/shared/frozen implementation or stopped inspector evidence inspected or run.

The new fifth part keeps the corrected profile in its own runner, starts the matching scoped native node with non-root identity and CPU/memory bounds, gates comparison on preparation success, retains always-run profile/global cleanup and includes the part in artifact names. Database/node resource names agree with the existing cleanup, and no production or host-global prune is added.

One small reliability fix is recommended before the next CI cycle: the five added/expanded readiness loops call curl -fsS without --connect-timeout or --max-time (for example lines1378–1379 and1988–1989). A listener that accepts but never replies can hold one iteration until the60-minute job timeout;30attempts is not a wall-clock bound. Add a short per-request timeout to these probes, preserving the retry count/readiness assertion and cleanup. This is an execution-wrapper finding only; I did not diagnose the separate PHPUnit/source-qualification failures, run fixtures or accept native behavior.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Measured on this head (run 40164318863 attempt 2, all 40 jobs green) against #739's run 40162397220, which used the same workflow without this PR:

Part Before This PR
cohorts 18 to 20 min 13 min or less
corrected-markers (new) — 13 min or less
corpus 17 to 21 min 18 to 20 min
sdk-directions 11 to 16 min up to 16 min
native-tests up to 17 min up to 15 min

As expected, cohorts is off the critical path, but corpus now sets the wall time by itself, so the end-to-end gain is small. The follow-up is ready on rust-325-ci-corpus-restarts and will be the next CI PR after this one merges. It moves the corpus step's profiles and restart cases (about 9 of its 17 minutes) into their own part, and bounds every readiness probe's request per Cloud's review (--connect-timeout 2 --max-time 5). Expected result: the slowest part drops from about 20 to about 16 min, with sdk-directions and native-tests next.

CI note: PHPUnit attempts 1 and 2 on this head failed before running any test. Pulling the floating mirror.gcr.io/library/composer:2 tag hit unknown blob at about 05:20 and 05:37 UTC; main's 05:12 run had passed. The same pull succeeded from another host at 06:30, so attempt 3 is running.

@rmcdaniel
rmcdaniel merged commit ac28bdb into main Oct 25, 2026
67 of 73 checks passed
@rmcdaniel
rmcdaniel deleted the rust-325-ci-corrected-part branch October 25, 2026 05:57
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…-worker-sessions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…st-325-ci-codec-one-job

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…nto rust-325-workflow-import

Native now serves 139 route methods: main's 138 plus embedded v2 import.
The runtime keeps both new modules, and the comparator node test list
keeps both sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…nto rust-325-record-local-activity

The worker capability manifest keeps this branch's local activity entries
and main's worker session and update validation entries. The comparator,
probe and node test lists keep both sides' fixtures and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…nto rust-325-cancellation-scopes

Workflow-task completion admits cancellation scopes before it guards
worker-session commands, in published 2.11.1's order. A scheduled
activity's options keep main's worker session and add its cancellation
scope, as PHP's bridge appends it. Native serves 141 route methods;
cancellation scope checkpoints stay missing. The probe timeout chain and
the comparator node test list keep both sides' entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
) into rust-325-nexus-operations

Native now serves 140 route methods: main's 138 plus the two
caller-indexed Nexus operation reads. The comparator node test list keeps
both sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
) into rust-325-workflow-stream-writes

Native now serves 140 route methods: main's 138 plus durable workflow
stream appends and closes. The comparator node test list keeps both
sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant