Repository navigation
Fix CI reliability: test compile OOM, separate dependency fetch, failure log tails - #501
Merged
Merged
Conversation
Two hosted runs failed execution-and-codec step 8 with exit 101, while the same suite passed four local runs under the CI limits, four under a one-CPU limit, six on a disposable host and three verbatim step-8 runs. Each hosted job starts with an empty Cargo home and downloads every locked crate inside the build-and-test step, so a registry or network failure and a failing test end in the same step with the same exit code. Each job that builds the native crate now fetches its locked dependencies in a separate step with network retries, and its builds and tests run offline. A test-step failure is then a real build or test failure, and a registry failure is visible as its own step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
First run at the fetch change: step 8 passed (fetch then offline build and tests). The SQLite three-runtime step failed separately at 95 s. Its failed-step log (read under the operator's new failed-step-log approval) shows the recorder receiving HTTP 500 from the published PHP 2.5.14 reference during fixture recording. The cause would be in the PHP server log, which the step did not print. Added a bounded |
At 50ca1f5 the SQLite three-runtime step failed while recording the published PHP reference: the recorder reported HTTP 500 from the PHP server and exited 1. The cause was in the PHP server's own log, which the step log did not include. Each three-runtime step now prints the last 40 lines of every container in its resource scope when it fails, inside a collapsed group, and keeps the original exit status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel
force-pushed
the
rust-325-ci-cargo-fetch
branch
from
October 20, 2026 00:59
cdcf09d to
ec403a4
Compare
The Docker workflow isolation policy refuses a trap whose argument runs Docker commands or expands variables. The three bootstrap steps now trap a plain function that runs container-log-tails.sh, which prints bounded tails of the job's own containers. The policy check and its unit tests pass locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
12 tasks
The intermittent exit 101 in the native test steps was rustc being OOM-killed, not a registry failure. Reproduced locally at #507's head under the CI limits (2 CPUs, 2 GB, two build jobs, empty target): compiling the library tests fails with exit 101 and the container cgroup records an oom_kill. Line-tables-only debug info still hits the limit. With 6 GB the same compile passes and peaks near 2.4 GB in sampled usage. The three containers that compile the library tests (MySQL, PostgreSQL and execution-and-codec) now get 4 GB. The others keep their limits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel
changed the title
#508's shared run failed while recording against the published PHP SQLite reference: an SDK poll in a probe received the documented 503 backend_lock_pressure refusal (task null, retryable) and the probe treated it as fatal. SQLite refuses a deferred writer immediately, without its busy timeout, while the maintenance loop writes. #478 already honored that refusal in restart.php. parityClient() now builds every SDK client in probe.php, the probe files it loads and cooperative-restart.php, keeping any transport the caller passes and wrapping it with the same bounded retry. Only that refusal on a POST poll is retried. Retries are reported on stderr at exit, so observations stay comparable across targets. The failure-evidence reader keeps its plain client and short timeouts. A local check with the installed parity SDK: a poll refused twice succeeds on the third call through the caller's transport, and both retries are reported. Part of #325. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel
changed the title
…ty probe" This reverts commit 361c903. The wrapper transport implements only the SDK's plain Transport interface, so it hid the bounded-request and bounded payload-transfer capabilities of the transport it wrapped. Cooperative workers refuse such a client ("Cooperative workers require a transport that honors bounded requests"), which failed every runtime-comparison step in shared run 39373612809. The probe retry needs to live below the SDK transport instead, and will come separately after a full local cohort run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel
changed the title
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #325.
Several hosted runs failed the native test step with exit 101: main after #478, #499, and now #507. The cause is memory. rustc is OOM-killed while compiling the library tests under the step's 2 GB container limit.
Reproduced locally at #507's head under the CI limits (2 CPUs, 2 GB, two build jobs, empty target):
could not compile durable-workflow-server (lib test)and exit 101;oom_kill 1;Main sits just under the line, so the failure was intermittent, and each added library test makes it more likely. My earlier local repetitions passed only because they disabled debug info.
This PR:
Memory. The three containers that compile the library tests (
mysql-execution,postgres-storage,execution-and-codec) now get 4 GB. Hosted runners have 16 GB. Other containers keep their limits.Separate dependency fetch. A
Fetch locked Rust dependenciesstep (cargo fetch --locked,CARGO_NET_RETRY=10) runs before each job's first build, and builds and tests run with--offline. A registry failure then shows as its own step instead of looking like a test failure.Failure log tails. The three runtime-comparison steps print bounded log tails of the job's own containers when they fail (
container-log-tails.sh, called from anERRtrap), so a failure such as the HTTP 500 seen during step-10 recording is diagnosable from that step's log. The Docker workflow isolation policy and its unit tests pass with this form.Probe lock-pressure retry: reverted. A version folded in from Retry the documented poll lock-pressure refusal in every parity probe #511 wrapped the SDK transport and hid its bounded-request and payload-transfer capabilities, so cooperative workers refused it and every runtime-comparison step failed in run 39373612809. It is reverted here. The retry will return separately, below the SDK transport, after a full local cohort run.
🤖 Generated with Claude Code