Repository navigation
ci: bound every job with a timeout and make apt survive a slow mirror - #148
Merged
Merged
Conversation
simulator-checks has hung on two consecutive pull requests, both times in the "Install simulator build dependencies" step rather than the build or the LVGL fetch. An unreachable archive mirror leaves apt-get waiting instead of failing. No job set timeout-minutes, so each inherited GitHub's six-hour default. That is what turned a transient network problem into a blocked pull request: the run sat occupying a slot until somebody noticed and cancelled it by hand. Both merges today needed that. Every job now has a ceiling sized against its observed runtime. Retrying apt would not have helped on its own, which is the part worth noting. The failure is a hang, not an error, so a command that never returns never reaches the retry. Each attempt therefore gets its own timeout, generous against the few seconds this normally takes but finite, and only then does the retry mean anything. Worst case is three attempts of 60 plus 90 seconds with backoffs, a little under eight minutes, inside the ten-minute step ceiling and the fifteen-minute job ceiling. The control flow was checked against stub executables covering a permanent hang, a transient failure that recovers on the third attempt, a healthy runner, and a hang in the install rather than the update. The first attempt at that harness passed a hang for the wrong reason, because timeout cannot run a shell function. Closes #147 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #147.
The failure
simulator-checkshung on two consecutive pull requests. Querying the job's steps while itwas stuck put it in the dependency install, not the build or the LVGL fetch:
An unreachable or slow archive mirror leaves
apt-getwaiting rather than failing. GitHubreported Actions operational at the time, so this is the Ubuntu archive, not a platform
incident.
The part that made it expensive
No job set
timeout-minutes, so every one inherited GitHub's six-hour default. That iswhat turned a transient network problem into a blocked pull request:
Both of today's merges needed manual cancel-and-rerun. Unattended, either would have sat
there for the rest of the day.
Why retrying alone would not have fixed it
Worth stating plainly, because it is the trap here: the failure is a hang, not an error.
A retry loop around a command that never returns never runs its second iteration. So each
attempt gets its own
timeout— generous against the few seconds this normally takes, butfinite — and only then does retrying mean anything.
Acquire::Retriesis set too, whichcovers the different case of individual mirror fetches failing outright.
Worst case is 3 x (60 + 90) seconds plus backoffs, a little under eight minutes, inside the
ten-minute step ceiling and the fifteen-minute job ceiling.
Verification
The control flow was exercised against stub executables, since the real condition cannot be
summoned on demand:
apt-get updatehangs foreverapt-get installhangs after update succeedsThe first version of that harness reported a pass for the wrong reason —
timeoutcannotrun a shell function, so the "hang" was really a command-not-found. Redone with real
executables.
CI on this PR exercises the new configuration directly, since pull requests run the
workflow from the branch.
Timeouts chosen
host-checkssimulator-checksfirmware-buildSized to a few times the observed runtime — loose enough not to fire on a slow-but-working
run, tight enough that a hang is caught in minutes rather than hours.