Repository navigation
ci: fall back off a slow Azure apt mirror in smoke tests - #15203
Merged
Merged
Conversation
A degraded node in the azure.archive.ubuntu.com pool serves package indexes at full speed but archives at tens of kB/s. The runner images set Acquire::Retries 1 and Acquire::http::Timeout 15, which only catch a total stall, so apt never moves to the next entry in apt-mirrors.txt. The Playwright install-deps step then runs into its 15 minute limit and setup-r takes about 20 minutes (actions/runner-images#14823). The Azure line is dropped from apt-mirrors.txt only when the mirror is measured slow, so healthy runs keep using it and load moves to archive.ubuntu.com only when needed. Two checks decide: - Before setup-r, whose apt calls cannot be wrapped, curl samples one multi-megabyte .deb from the Azure mirror and drops it when the speed stays under 1 MB/s for 10 seconds. With the mirror throttled to 50 kB/s, setup-r then downloaded everything from archive.ubuntu.com. - The later apt steps run through a helper that gives each attempt three minutes (healthy runs take 20 to 45 seconds) and drops the mirror before retrying, which covers a node that degrades partway through a download. Probe runs caught such a node live; the retry finished in about 50 seconds on archive.ubuntu.com. The logic lives inline in test-smokes.yml instead of a local composite action: release mode checks out older tags, which would not contain the action directory.
The per-attempt timeout is not tied to the download phase. A slow download that finishes just before the deadline lets the kill land while dpkg unpacks, and every retry then stops on "dpkg was interrupted". dpkg --configure -a finishes that transaction and does nothing on a clean system. Precaution, untested: the probes only killed attempts during the download.
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On ubuntu runners, apt installs in the smoke tests sometimes crawl on a degraded
azure.archive.ubuntu.comnode. In this job from #15186,Set up Rtook 20 minutes andInstall Playwright system dependenciesfailed at its 15 minute limit.Root cause
The degraded node serves package indexes at full speed but archives at tens of kB/s. The runner images set
Acquire::Retries 1andAcquire::http::Timeout 15, which only catch a total stall, so apt never moves toarchive.ubuntu.com, the next entry in/etc/apt/apt-mirrors.txt. This is actions/runner-images#14823, still open (a fix is proposed in actions/runner-images#14596).Fix
The Azure line is dropped from
apt-mirrors.txtonly when the mirror is measured slow, so healthy runs keep using it and the load does not move to Canonical's servers.setup-r, whose apt calls can't be wrapped, a new step downloads one multi-megabyte.debfrom the Azure mirror withcurl --speed-limit, and drops the mirror if the speed stays under 1 MB/s for 10 seconds.$APT_WITH_FALLBACK, used by the Playwright deps step and the twoapt-getsteps. Each attempt gets 3 minutes (healthy runs take 20 to 45 seconds). After a failed attempt, the mirror is dropped,dpkg --configure -afinishes an install the timeout interrupted, and the command is retried.Both checks are needed: the speed sample is one point in time, and a node can start fast and slow down in the middle of a download.
The logic is inline in
test-smokes.ymlrather than a local composite action, because release mode intest-smokes-built.ymlchecks out older tags that would not contain the action directory.Testing
Tested with throwaway probe workflows on a fork, running the step text from
test-smokes.yml:archive.ubuntu.com.setup-rdownloaded everything fromarchive.ubuntu.com.The
dpkg --configure -arecovery was not exercised, as the probes only killed attempts during the download.Checklist
I have (if applicable):
AI-assisted PR
Note: autonomous AI agents submitting PRs without human oversight are not permitted — see the Code of Conduct.