Skip to content

ci: fall back off a slow Azure apt mirror in smoke tests - #15203

Merged
cderv merged 2 commits into
mainfrom
ci/apt-mirror-fallback
Oct 27, 2026
Merged

cderv merged 2 commits into
mainfrom
ci/apt-mirror-fallback

Conversation

@cderv

@cderv cderv commented Oct 27, 2026

Copy link
Copy Markdown
Member

On ubuntu runners, apt installs in the smoke tests sometimes crawl on a degraded azure.archive.ubuntu.com node. In this job from #15186, Set up R took 20 minutes and Install Playwright system dependencies failed at its 15 minute limit.

Root cause

The degraded node serves package indexes at full speed but archives at tens of kB/s. The runner images set Acquire::Retries 1 and Acquire::http::Timeout 15, which only catch a total stall, so apt never moves to archive.ubuntu.com, the next entry in /etc/apt/apt-mirrors.txt. This is actions/runner-images#14823, still open (a fix is proposed in actions/runner-images#14596).

Fix

The Azure line is dropped from apt-mirrors.txt only when the mirror is measured slow, so healthy runs keep using it and the load does not move to Canonical's servers.

  • Before setup-r, whose apt calls can't be wrapped, a new step downloads one multi-megabyte .deb from the Azure mirror with curl --speed-limit, and drops the mirror if the speed stays under 1 MB/s for 10 seconds.
  • The same step writes a small helper, $APT_WITH_FALLBACK, used by the Playwright deps step and the two apt-get steps. Each attempt gets 3 minutes (healthy runs take 20 to 45 seconds). After a failed attempt, the mirror is dropped, dpkg --configure -a finishes an install the timeout interrupted, and the command is retried.

Both checks are needed: the speed sample is one point in time, and a node can start fast and slow down in the middle of a download.

The logic is inline in test-smokes.yml rather than a local composite action, because release mode in test-smokes-built.yml checks out older tags that would not contain the action directory.

Testing

Tested with throwaway probe workflows on a fork, running the step text from test-smokes.yml:

  • The degraded node showed up live in some runs. Attempts were killed at 3 minutes and the retry finished in about 50 seconds on archive.ubuntu.com.
  • With traffic from the Azure mirror throttled to 50 kB/s, the speed check dropped it and setup-r downloaded everything from archive.ubuntu.com.
  • Healthy runs kept using the Azure mirror.

The dpkg --configure -a recovery was not exercised, as the probes only killed attempts during the download.

Checklist

I have (if applicable):

  • referenced the GitHub issue this PR closes
  • updated the appropriate changelog in the PR
  • ensured the present test suite passes
  • added new tests
  • created a separate documentation PR in Quarto's website repo and linked it to this PR
AI-assisted PR
  • AI tool used: Claude Code
  • Codebase grounding: local clone, actions/runner-images source via GitHub CLI
  • Human review: I have reviewed, tested, and verified the AI-generated content before submitting.

Note: autonomous AI agents submitting PRs without human oversight are not permitted — see the Code of Conduct.

cderv added 2 commits October 27, 2026 11:54
A degraded node in the azure.archive.ubuntu.com pool serves package
indexes at full speed but archives at tens of kB/s. The runner images
set Acquire::Retries 1 and Acquire::http::Timeout 15, which only catch a
total stall, so apt never moves to the next entry in apt-mirrors.txt.
The Playwright install-deps step then runs into its 15 minute limit and
setup-r takes about 20 minutes (actions/runner-images#14823).

The Azure line is dropped from apt-mirrors.txt only when the mirror is
measured slow, so healthy runs keep using it and load moves to
archive.ubuntu.com only when needed. Two checks decide:

- Before setup-r, whose apt calls cannot be wrapped, curl samples one
  multi-megabyte .deb from the Azure mirror and drops it when the speed
  stays under 1 MB/s for 10 seconds. With the mirror throttled to
  50 kB/s, setup-r then downloaded everything from archive.ubuntu.com.
- The later apt steps run through a helper that gives each attempt
  three minutes (healthy runs take 20 to 45 seconds) and drops the
  mirror before retrying, which covers a node that degrades partway
  through a download. Probe runs caught such a node live; the retry
  finished in about 50 seconds on archive.ubuntu.com.

The logic lives inline in test-smokes.yml instead of a local composite
action: release mode checks out older tags, which would not contain
the action directory.
The per-attempt timeout is not tied to the download phase. A slow
download that finishes just before the deadline lets the kill land
while dpkg unpacks, and every retry then stops on "dpkg was
interrupted". dpkg --configure -a finishes that transaction and does
nothing on a clean system. Precaution, untested: the probes only
killed attempts during the download.
@posit-snyk-bot

posit-snyk-bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cderv
cderv merged commit 1d82b12 into main Oct 27, 2026
51 checks passed
@cderv
cderv deleted the ci/apt-mirror-fallback branch October 27, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants