From a2c387129009c116e7af4901b5d8b1b9328cd7cb Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Fri, 28 Aug 2026 19:39:36 -0700 Subject: [PATCH] fix(ci): stop the layer-cache prune from being able to fail a build MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The settle loop added in 4f651038f3 reads `cur="$(total)"`, and composite steps run under `bash -e -o pipefail`, where an assignment takes its command substitution's exit status. So any failing `buildctl du` aborts the step and fails the build. `echo "$(total)"` survives the same failure, which is why this was invisible in the paths that ran first. That is not hypothetical. Deleting a sticky disk out from under a running job makes buildkitd panic inside cache.(*cacheManager).DiskUsage, and it took a Build AMD64 job down that way. `grep` also exits 1 whenever du prints no Total line, so an empty cache would have done it too. A cache-hygiene step must never be able to fail a deploy, so `total()` now always returns 0. The prune's own failure was already handled — it is an `if` condition, and a failing condition does not trip `-e`. --- .github/actions/docker-build/action.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/actions/docker-build/action.yml b/.github/actions/docker-build/action.yml index 382562760e6..4aaf9959553 100644 --- a/.github/actions/docker-build/action.yml +++ b/.github/actions/docker-build/action.yml @@ -119,7 +119,16 @@ runs: # Print the whole Total line rather than picking a column: buildctl's du # table is whitespace-aligned and its layout is not a stable contract. - total() { sudo buildctl --addr "$addr" du 2>/dev/null | grep -iE '^total:' | tr -s ' \t' ' '; } + # + # The trailing `|| true` is load-bearing. Composite steps run under + # `bash -e -o pipefail`, where `cur="$(total)"` takes the substitution's + # exit status, so a failing du would abort the step and fail the build -- + # `echo "$(total)"` survives but the assignment in the settle loop does + # not. buildctl exiting non-zero here is entirely plausible: deleting a + # sticky disk out from under a running job makes buildkitd panic inside + # DiskUsage, and grep also exits 1 whenever the table has no Total line. + # Cache hygiene must never be able to fail a deploy. + total() { sudo buildctl --addr "$addr" du 2>/dev/null | grep -iE '^total:' | tr -s ' \t' ' ' || true; } echo "before prune -> $(total)" if sudo buildctl --addr "$addr" prune --all --keep-storage "$KEEP_MB"; then