Skip to content

CI: pull each fixture job's pinned images with bounded retries first - #762

Merged
rmcdaniel merged 3 commits into
mainfrom
rust-325-ci-image-pull-retry
Oct 25, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
rust-325-ci-image-pull-retry

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Main's push run 40164159608 (Oct 25, 05:08Z) lost 24 fixture jobs at once in "Fetch locked Rust dependencies". Their first docker run could not pull the pinned Rust image: mirror.gcr.io/library/rust@sha256:ba81bc3e…: … docker: unknown blob (exit 125). Later runs pulled the same digest fine. The implicit pull inside docker run doesn't retry, so one brief registry inconsistency failed the whole run.

Change

  • scripts/ci/pull-images.sh IMAGE...: pulls each image up to DW_PULL_ATTEMPTS (4) times. The wait starts at DW_PULL_BACKOFF_SECONDS (15) and doubles after each failure, about 1 min 45 s worst case per image before the step fails. Empty arguments are skipped, and malformed bounds are refused with exit 2. It runs on the host, not inside a container command, so the Docker isolation policy is unaffected.
  • server-parity.yml: published-sdk-adapters, mysql-execution, postgres-storage and execution-and-codec run "Pull the pinned images, retrying registry errors" right after checkout. The list is each job's Rust image, its matrix database image ($MYSQL_IMAGE, $POSTGRES_IMAGE), the published 2.11.1 Server image and, where bootstrapped, the 2.5.14 layout image. Images that only one part uses (the upgrade profile's) still pull on first use. The script is added to the push path filter.
  • ServerParityImagePullTest.mjs (new; in the path filter and the comparator list) checks:
    • each of the four jobs pulls first, right after checkout;
    • every Rust digest a job runs appears in its pull list, so a digest bump can't leave the pull behind;
    • the database jobs pull their matrix image;
    • the script retries until success, stops at the attempt bound, skips empty arguments and refuses malformed bounds (fake docker).
  • ServerParityPartsTest.mjs lists the pull step among the steps every part runs.

Qualification

  • New test 5/5. Against main's workflow, the two structural tests fail, as expected.
  • Real smoke on Docker 27: a real image pulled on attempt 1, with an empty argument skipped. A nonexistent digest failed twice with manifest unknown, then could not pull … after 2 attempts, exit 1.
  • shellcheck clean; actionlint shows only main's existing 20 shellcheck warnings; Docker workflow isolation policy passes; comparator node tests 2239/2239.
  • composer_trust_test.py, patched_pip_test.py and brotli_trust_test.py (Start every Server pip install from the patched, bundle-only pip (GHSA-6q48-6c7w-9gw2) #673's repository inventories) pass on the merged tree. The real-pip bootstrap needs network and runs in CI.

No native, shared or frozen implementation changes and no fixture changes.

🤖 Generated with Claude Code

rmcdaniel and others added 3 commits October 25, 2026 12:56
Main's push run 40164159608 lost 24 fixture jobs at their first container:
mirror.gcr.io answered "unknown blob" for the pinned Rust image, and the
implicit pull inside `docker run` does not retry. scripts/ci/pull-images.sh
pulls each named image up to DW_PULL_ATTEMPTS (4) times, doubling a
DW_PULL_BACKOFF_SECONDS (15) wait, and fails the step only after the last
attempt.

published-sdk-adapters, mysql-execution, postgres-storage and
execution-and-codec run it right after checkout for the images every part
uses: their Rust image, the matrix database image, the published 2.11.1
Server image and, where bootstrapped, the 2.5.14 layout image. The new
test pins that order, that every Rust digest a job runs is in its pull
list, and the script's retry, bound, skip and refusal behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…age-pull-retry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…age-pull-retry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Fixtures run 40206210357 attempt 1 had one failure, mysql-execution (existing-php-mysql-matrix, corrected-markers). It's unrelated to this PR: the published PHP reference's clock record hit a known harness timing flake. The original Python worker was killed (-9) before the observing proxy forwarded its activity poll, and the contract requires every cancelled poll to have reached upstream. Classified on #754 (comment 6399755214) as a fixture-contract defect, with a narrow contract fix planned separately. The pull step itself ran in all 39 jobs. I've rerun only the failed job.

@rmcdaniel
rmcdaniel merged commit cce0a8c into main Oct 25, 2026
108 of 110 checks passed
@rmcdaniel
rmcdaniel deleted the rust-325-ci-image-pull-retry branch October 25, 2026 16:05
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…ock-unsent-poll

The path filter and the comparator node test list keep both sides' new
tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
…dec-one-job

The comparator node test list keeps main's image pull test and this
branch's codec deadline cell test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant