Skip to content

ci: bound every job with a timeout and make apt survive a slow mirror - #148

Merged
PurpleSentinel merged 1 commit into
mainfrom
fix/ci-timeouts-and-apt-retry
Aug 19, 2026
Merged

PurpleSentinel merged 1 commit into
mainfrom
fix/ci-timeouts-and-apt-retry

Conversation

@PurpleSentinel

Copy link
Copy Markdown
Contributor

Closes #147.

The failure

simulator-checks hung on two consecutive pull requests. Querying the job's steps while it
was stuck put it in the dependency install, not the build or the LVGL fetch:

2 Run actions/checkout@v7                 -> completed success
3 Install simulator build dependencies    -> in_progress     <-- here
4 Build and test the ... simulator        -> pending

An unreachable or slow archive mirror leaves apt-get waiting rather than failing. GitHub
reported Actions operational at the time, so this is the Ubuntu archive, not a platform
incident.

The part that made it expensive

No job set timeout-minutes, so every one inherited GitHub's six-hour default. That is
what turned a transient network problem into a blocked pull request:

Run Normal Hung for
#141 2m15s 30 min, cancelled by hand
#146 1m50s 11 min, then a re-run hung 18+ min

Both of today's merges needed manual cancel-and-rerun. Unattended, either would have sat
there for the rest of the day.

Why retrying alone would not have fixed it

Worth stating plainly, because it is the trap here: the failure is a hang, not an error.
A retry loop around a command that never returns never runs its second iteration. So each
attempt gets its own timeout — generous against the few seconds this normally takes, but
finite — and only then does retrying mean anything. Acquire::Retries is set too, which
covers the different case of individual mirror fetches failing outright.

Worst case is 3 x (60 + 90) seconds plus backoffs, a little under eight minutes, inside the
ten-minute step ceiling and the fifteen-minute job ceiling.

Verification

The control flow was exercised against stub executables, since the real condition cannot be
summoned on demand:

Case Result
apt-get update hangs forever bounded, 3 attempts, fails in 7s (scaled)
transient failure, recovers on 3rd attempt exit 0
healthy runner exit 0 on first attempt, no overhead
apt-get install hangs after update succeeds bounded, fails

The first version of that harness reported a pass for the wrong reason — timeout cannot
run a shell function, so the "hang" was really a command-not-found. Redone with real
executables.

CI on this PR exercises the new configuration directly, since pull requests run the
workflow from the branch.

Timeouts chosen

Job Observed Ceiling
host-checks 46s 15 min
simulator-checks ~2 min 15 min
firmware-build 6-7 min 30 min

Sized to a few times the observed runtime — loose enough not to fire on a slow-but-working
run, tight enough that a hang is caught in minutes rather than hours.

simulator-checks has hung on two consecutive pull requests, both times in the
"Install simulator build dependencies" step rather than the build or the LVGL fetch.
An unreachable archive mirror leaves apt-get waiting instead of failing.

No job set timeout-minutes, so each inherited GitHub's six-hour default. That is what
turned a transient network problem into a blocked pull request: the run sat occupying
a slot until somebody noticed and cancelled it by hand. Both merges today needed that.
Every job now has a ceiling sized against its observed runtime.

Retrying apt would not have helped on its own, which is the part worth noting. The
failure is a hang, not an error, so a command that never returns never reaches the
retry. Each attempt therefore gets its own timeout, generous against the few seconds
this normally takes but finite, and only then does the retry mean anything. Worst case
is three attempts of 60 plus 90 seconds with backoffs, a little under eight minutes,
inside the ten-minute step ceiling and the fifteen-minute job ceiling.

The control flow was checked against stub executables covering a permanent hang, a
transient failure that recovers on the third attempt, a healthy runner, and a hang in
the install rather than the update. The first attempt at that harness passed a hang
for the wrong reason, because timeout cannot run a shell function.

Closes #147

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@PurpleSentinel
PurpleSentinel merged commit 753c979 into main Aug 19, 2026
3 checks passed
@PurpleSentinel
PurpleSentinel deleted the fix/ci-timeouts-and-apt-retry branch August 19, 2026 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: simulator job hangs indefinitely on apt-get with no timeout

2 participants