From fd327fd3a869d0b8c109d0c7899bb6f598b2f433 Mon Sep 17 00:00:00 2001 From: mrsqr Date: Wed, 19 Aug 2026 21:33:10 +0100 Subject: [PATCH] ci: bound every job with a timeout and make apt survive a slow mirror simulator-checks has hung on two consecutive pull requests, both times in the "Install simulator build dependencies" step rather than the build or the LVGL fetch. An unreachable archive mirror leaves apt-get waiting instead of failing. No job set timeout-minutes, so each inherited GitHub's six-hour default. That is what turned a transient network problem into a blocked pull request: the run sat occupying a slot until somebody noticed and cancelled it by hand. Both merges today needed that. Every job now has a ceiling sized against its observed runtime. Retrying apt would not have helped on its own, which is the part worth noting. The failure is a hang, not an error, so a command that never returns never reaches the retry. Each attempt therefore gets its own timeout, generous against the few seconds this normally takes but finite, and only then does the retry mean anything. Worst case is three attempts of 60 plus 90 seconds with backoffs, a little under eight minutes, inside the ten-minute step ceiling and the fifteen-minute job ceiling. The control flow was checked against stub executables covering a permanent hang, a transient failure that recovers on the third attempt, a healthy runner, and a hang in the install rather than the update. The first attempt at that harness passed a hang for the wrong reason, because timeout cannot run a shell function. Closes #147 Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/README.md | 6 ++++ .../workflows/host-and-firmware-checks.yml | 33 ++++++++++++++++--- 2 files changed, 35 insertions(+), 4 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 37fc75c..f74e93a 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -10,5 +10,11 @@ - a headless 600 x 450 LVGL/SDL simulator build and smoke test - ESP32-S3 firmware compilation with ESP-IDF v6.0.2 +Every job sets `timeout-minutes`. Without it a job inherits GitHub's six-hour default, so a +network stall - `apt-get` waiting on an unreachable mirror, in the case that prompted this - +blocks a pull request for the rest of the day instead of failing visibly. Network-dependent +steps additionally time out and retry individually, because a retry alone does nothing for a +command that never returns. + The workflow pins release families rather than using moving `latest` tags. Update the Python, LVGL, ESP-IDF, dependency, and local documentation pins together in one reviewed PR. diff --git a/.github/workflows/host-and-firmware-checks.yml b/.github/workflows/host-and-firmware-checks.yml index 3019924..7a1c36c 100644 --- a/.github/workflows/host-and-firmware-checks.yml +++ b/.github/workflows/host-and-firmware-checks.yml @@ -15,6 +15,9 @@ concurrency: jobs: host-checks: runs-on: ubuntu-24.04 + # Runs in well under a minute. The ceiling exists so a hang fails visibly rather than + # occupying a runner for the six-hour default. + timeout-minutes: 15 steps: - uses: actions/checkout@v7 - uses: actions/setup-python@v7 @@ -23,24 +26,46 @@ jobs: cache: pip cache-dependency-path: requirements-dev.txt - name: Install pinned host dependencies + timeout-minutes: 10 run: python -m pip install --disable-pip-version-check -r requirements-dev.txt - name: Run host checks run: make check simulator-checks: runs-on: ubuntu-24.04 + timeout-minutes: 15 steps: - uses: actions/checkout@v7 - name: Install simulator build dependencies - run: >- - sudo apt-get update && - sudo apt-get install --yes --no-install-recommends - cmake libsdl2-dev ninja-build + timeout-minutes: 10 + # A slow archive mirror leaves apt waiting rather than failing, which has blocked + # pull requests here twice. Retrying alone would not help: a hung apt never + # returns, so the retry never fires. Each attempt therefore gets its own timeout, + # generous against the few seconds this normally takes, and the step and job + # timeouts above bound the whole thing. Worst case is 3 x (60 + 90) seconds plus + # the backoffs, a little under eight minutes, which fits inside the step timeout. + run: | + echo 'Acquire::Retries "3";' | sudo tee /etc/apt/apt.conf.d/99-ci-retries > /dev/null + for attempt in 1 2 3; do + if timeout 60 sudo apt-get update && + timeout 90 sudo apt-get install --yes --no-install-recommends \ + cmake libsdl2-dev ninja-build; then + exit 0 + fi + echo "::warning::apt attempt ${attempt} failed or timed out; retrying" + sudo rm -rf /var/lib/apt/lists/partial + sleep 10 + done + echo "::error::could not install simulator build dependencies after 3 attempts" + exit 1 - name: Build and test the 600 x 450 device simulator run: make simulator-test firmware-build: runs-on: ubuntu-24.04 + # Pulls a large pinned image before a full ESP-IDF build, so this is the slow job at + # six to seven minutes. Bounded for the same reason as the others. + timeout-minutes: 30 steps: - uses: actions/checkout@v7 - name: Build with pinned ESP-IDF image