Repository navigation
CI: run each fixture database job in three parallel parts - #746
Conversation
PHPUnit Feature, Shared Server fixtures, SonarQube and Public Boundary now also run on merge_group, so a merge queue can replace the require-up-to-date rerun loop. Base-revision checks use the queue group's base_sha, the boundary scan covers the group's exact base..head range, and the push trigger skips gh-readonly-queue/** branches that the merge_group run already scans. Shared Server fixtures has no path filter for merge_group, so every queued group gets the full profile.
tests/Unit/MergeGroupTriggersTest.mjs checks that PHPUnit Feature, Shared Server fixtures, SonarQube and Public Boundary run on merge_group, that base-revision checks and the boundary range use the queue group's base and head, that queue branches are not scanned twice, and that superseded-run cancellation never applies to a queue group. It runs in Repository contracts. It fails 6 of 7 checks on the train head without merge_group support.
The OpenAPI evolution and regression corpus contracts pin the exact base revision expression; include the merge_group base so a queued group diffs against the revision it was built on.
The Sonar job receives SONAR_TOKEN and its fork guard is always true for a merge group, so it no longer runs on merge_group; the contract test now requires that. Public Boundary scans every push again, including queue branches, and the test requires the unfiltered push trigger alongside the exact merge-group range.
…y group #723 removed PHPUnit Feature's concurrency group, because its required gate runs with always() under the central action policy. The test now requires that any cancel-in-progress in a workflow the queue waits for cancels only pull request runs, and that Shared Server fixtures keeps its pull-request-only cancellation.
In each database job of Shared Server fixtures, the nine published-SDK direction profiles ran one after another (two pairs already overlapped), about 22 of the job's 64-72 minutes (train #729, PostgreSQL 17 job 120866907013: 12:19:50 to 12:41:36). Each profile already has its own containers, PHP and embedded databases, evidence and state directory, and its own workflow-ID prefix on the shared native server. sdk-direction-lanes.sh now runs them in three lanes balanced by those durations, in order within a lane. Every profile runs even after another fails; the step fails if any did, after printing each profile's log in its own group. The job's existing cleanup steps still stop and remove every profile's containers. Expected saving: about 14 minutes per database job.
heartbeat_renews_only_progress_deadline_and_cannot_resurrect_elapsed_authority compared two lease_expires_at strings. Native times are whole milliseconds and serialize without a fraction on a whole second, so a claim lease of ...:26Z sorted after a renewal of ...:26.045Z and the assertion failed about one run in a thousand (train #729, PostgreSQL 17 job 120852521096). The test now parses both and reports the values.
A required check and the merge queue both need a result that always reports. The pull_request path filter meant a docs-only pull request got no Shared Server fixtures result at all, so the workflow could not be required. - Pull requests now run the workflow without a path filter. A short `changes` job applies the push trigger's path list to the pull request's merge commit (renames count on both sides), so one list governs both triggers. Push, merge_group and manual runs run every fixture job, as does any pull request that cannot be classified. - `published-sdk-adapters` runs only when the classifier says so; the other fixture jobs need it, so they skip with it. - A terminal `Shared Server fixtures result` job runs with always(). It passes only when every fixture job passed, or when no input changed and every fixture job was skipped. A failed, cancelled or unexpectedly skipped job, a failed classifier, or an unreadable needs context fails it. - ServerParityAggregateTest covers the workflow structure, the result decisions and the classifier against scratch pull request merge commits. Repository contracts runs it. No permissions, secrets, actions or artifact paths change. Making the result a required check, and enabling the merge queue, stay separate repository-setting steps.
The Docker workflow isolation policy accepts only literal .sh and .py repository helpers in workflow steps, so the new jobs' direct node calls failed Public Boundary. changes.sh and aggregate.sh now run the Node scripts from the repository root, as the workflow does, and the classifier tests go through changes.sh.
Every database job waited for published-sdk-adapters (about 15 minutes), but 10 of those minutes built the optimized native binary only the codec-deadline-health cells use (train #729: "Build one bounded optimized native artifact for deadline health", 10.0 minutes). The build, its upload and its container cleanup now run in a codec-native-build job beside the adapters; codec-deadline-health needs both, and the database jobs still need only the adapters, so they start about 10 minutes sooner. The build script and its bounded resources are unchanged.
…est the lane wrapper Each lane stops its profile's PHP server as soon as the profile ends, so at most three run at once, and a failed stop only warns. The profile's status is kept either way. A shell-level test runs the wrapper with stub profiles and docker: all nine profiles run, every server is stopped after its profile, a failed profile does not stop its lane, and the wrapper fails only after printing every status and log. Also resolves the wrapper's Sonar findings.
… fixture when patterns or names cannot be classified The file list is read as bytes with --name-only -z and split only on NUL, so a quoted name can no longer slip past a required prefix; wildcards match newlines; a name that is not UTF-8 or an unsupported pattern runs every fixture job instead of failing the classifier.
…and wait for it codec-native-build is a second root fixture job. It runs only through the change classifier, and the result needs it like every other fixture job.
Each MySQL, PostgreSQL and SQLite job becomes three matrix parts on separate runners: native-tests (the native database suites, or the codec suite, and the TLS checks), corpus (the shared corpus, its profiles and the restart and takeover cases), and cohorts (role tokens, upgrade sources, corrected markers, external consumers, query authority, worker fencing and pressure). Every part bootstraps the same database and native build, and none depends on another; the cohorts start their own native node for the corrected markers. Every check still runs exactly once, which a structure test pins.
|
Independent CI review of exact head Source review found no blocking regression in the split itself. All original workflow steps are retained; the four additions are the three independent cohort-node preparations and SQLite's build-only preparation. The native suites/TLS/codec remain in Independent qualification: 46/46 One performance-evidence correction is needed: the earlier “no queue” observation is no longer representative. At 14:33:09 UTC, run40101730551 had 18 queued jobs and six running jobs. Queued split parts were created at 14:22:42 UTC, so some had already waited more than ten minutes. The run remains nonterminal; no measured end-to-end speedup is established yet. GitHub's queued job records also populate Please retain the expected40-minute result as an estimate until this exact run finishes, and report queue delay, bootstrap/execution, total runner minutes and end-to-end required-gate duration separately. The split preserves coverage, but additional concurrent bootstraps consume shared runner capacity; current queue delay must be included in judging its throughput benefit. This review changed no repository settings, runs or merge authority. |
|
Security/release-path disposition for the proposed stacked landing at exact head
This permits one combined source landing without discarding any security, required-check, authorship, or queue-activation boundary. |
… the native server The namespace and selected-run signal profiles ran beside the plain signal profile, and on PostgreSQL 17 one's worker showed in another's routing diagnostics (server-parity-sdk-signals). The old serial workflow never overlapped them. A test now reads each profile's queues from its fixtures and refuses any two profiles that share one in different lanes.
…-325-merge-group-v2
…st-325-ci-db-parts
|
Follow-up independent source review at The delta from the earlier reviewed The new queue-isolation counterfactual is effective: mounting only the previous The earlier run40101730551 is now canceled, not performance acceptance. Its original queued-job evidence remains a historical observation. Owner cancellation of duplicate feature/lower-stack runs and the direct-main stack qualification are acknowledged. Current main-base exact-head hosted checks and terminal fixture evidence still govern landing; no measured end-to-end speedup or queue activation is claimed here. No source, settings, runs or provider resources were changed by this review. |
…ing it world-writable SonarCloud refused the new chmod 0777 (githubactions:S2612). Only the published PHP containers' user 1000 needs to write there; acl is on the hosted runner.
|
|
Security follow-up on exact head The only change replaces world-writable This is delta/source acceptance only. Merge still waits for every exact-head main-base check and every fixture cell to finish terminal and green under the previously recorded landing conditions; several native, corpus, cohort, PHPUnit, and Rust CodeQL jobs are currently running. I changed no source and started/cancelled no CI. |
|
Merged as The head contains #741 ( Gates at the head (41 success, 1 path-skipped):
No merge-queue or settings change. Measured timing (run 40104569657, end to end)The run took 36m11s from first job created to last job completed. The previous serial layout took 63–72 min per database job (#729's run). Queue wait was 1–3 s for every job.
The roots took 4m37s (published-sdk-adapters) and 7m20s (codec-native-build). The six deadline-health cells took 1–1.5 min each. The critical path is now the adapters plus the longest corpus part (MySQL 8.0, 31 min). Next cut, ready as a follow-up: run the six deadline-health cells in turn on one runner, which frees five runners per run at no wall-time cost. |
Review class: R0 (CI efficiency; every check keeps its coverage). Stacked on #742, which already contains #741; this PR's own change is the last commit.
Why
Measured on #729's final run, every fixture database job ran one long serial chain:
#741 takes about 14 minutes out of the corpus comparison. This PR removes the serial chain itself.
What
Each database job becomes three matrix parts on separate runners:
runtime::mysql::/qualification_plus--test execution), or the SQLite codec and--all-targetssuite, and the TLS trust checks.Every part bootstraps the same database and builds the same native binary; no part depends on another. The corpus and cohorts parts build without running the suites. The corrected-marker cohort used native-b (or SQLite's recovered node) left over from the corpus step. The cohorts part now starts that node itself (step
prepare_cohorts), on the same bootstrapped database. Each part retains its own summary artifact. The job timeout drops from 105 to 60 minutes.Expected: each part takes about 20–35 minutes, so the critical path is about 40 minutes, down from about 85 (about 60 with #741 alone). Runner use rises by the two extra bootstraps per database. There is no queueing today: no runs were queued while 20 were in progress, against the plan's 60 concurrent jobs.
Evidence
tests/Unit/ServerParityPartsTest.mjs, run in PHPUnit Feature beside the aggregate test. For all three jobs it pins:cohorts, the corpus comparison incorpus, and TLS/codec innative-tests;Retire …cleanups asalways() && cohorts;prepare_cohorts;native-testsin both database bootstraps.ServerParityAggregateTestandMergeGroupTriggersTest. Against the unsplit workflow, 0/4 of the new tests pass.