Make the multi-arch image build tolerate transient failures - #185
Merged
Merged
Conversation
Connection resets to raw.githubusercontent.com occasionally fail the scheduled multi-arch build even though a second attempt would succeed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every build and deploy job looked up the master HEAD on its own, and the deploy jobs pulled a whole image only to read RUBY_REVISION back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docker manifest create fetched both per-arch manifests and configs again for every tag, and was the most frequent cause of failed runs. The per-arch tags it needed were never deleted and have piled up to about 12,000 per architecture on Docker Hub. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The development stage starts from the ruby stage, so building both targets with one builder compiles Ruby once instead of twice. The dev images now also share the Ruby layers of their base images. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ubuntu mirrors in the middle of a sync, and registries resetting connections or rejecting pushes with "unknown blob", fail builds that pass when retried. The builder keeps its cache, so a retry resumes from the step that failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One failed build used to skip every deploy job in the run. A variant missing a digest still fails on its own, so no single-arch manifest is published. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The scheduled multi-arch build failed in 18 of 88 runs over the last month. All 22 failed jobs hit transient errors, mostly connection resets in
docker manifest create,unknown blobon ghcr.io pushes, and Ubuntu mirrors in the middle of a sync. Sincedeploy_multiarchneeded every build job, one flake also skipped all manifests in that run.The master revision is now resolved once, and each job builds
rubyanddevelopmenttogether and pushes them by digest to both registries.deploy_multiarchmerges the digests withdocker buildx imagetools create, which reads each image once for all tags. Builds and manifest pushes are retried, and a failed build blocks only its own variant.Published manifest lists keep the same tags and Docker v2 media types. I checked the tags against the old
Rakefilefor 144 combinations and ran the push and merge against local registries. Per-arch-<arch>-<run_id>tags are no longer pushed.Generated with Claude Code