Skip to content

Commit 0638f6b

Browse files
committed
fix(ci): actually prune the BuildKit layer cache
The max-cache-size-mb input added in 8fa7f0c never did anything. setup-docker-builder v1 accepted it and pruned in its own post step, but the v2 rewrite dropped the input, and the repo pins v2.1.0. GitHub only WARNS on an unknown composite input, so every build since has logged Unexpected input(s) 'max-cache-size-mb', valid inputs are ['cache-key', ...] and pruned nothing. Scanned every tag to confirm: the input exists in v1.8.0 through v1.12.0 and in none of v2.0.0, v2.0.1, v2.1.0. Rather than downgrade a builder rewrite to reach a config knob, run the prune ourselves. This is v1's command verbatim (its dist/index.js pruneBuildkitCache), against the fixed address v2 itself uses for `buildctl du` and `debug workers`: sudo buildctl --addr tcp://127.0.0.1:1234 prune --all --keep-storage <MB> --all is load-bearing. A plain prune reclaims only layer records and leaves `RUN --mount=type=cache` dirs untouched, and those mounts are most of what these disks hold: realtime's image is under 300 MB but its disk reached 249 GB. Being a blocking command is the other reason to own it here. Turborepo's equivalent eviction runs on a detached thread that is never joined, and on a 206 GB cache it makes no measurable progress inside a 150s job; a foreground buildctl prune completes or reports why it did not. Warn rather than fail, since an oversized cache is not worth failing a deploy over — but print `buildctl du` either side, because a silent no-op is exactly the failure mode that hid this regression for a day.
1 parent 9b76f6e commit 0638f6b

1 file changed

Lines changed: 44 additions & 15 deletions

File tree

.github/actions/docker-build/action.yml

Lines changed: 44 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,13 @@ inputs:
2121
required: true
2222
max-cache-size-mb:
2323
description: >-
24-
Layer cache to retain after the post-job prune, in MB. Must stay above one
24+
Layer cache to retain after this action prunes, in MB. Must stay above one
2525
build's working set (base + dependency layers + RUN --mount=type=cache
2626
dirs) or every build evicts what the next one needs. Falls back to the
27-
small-image default below when empty.
27+
small-image default in the prune step when empty — the fallback lives there
28+
rather than here because callers pass this from a matrix field, and an unset
29+
matrix key arrives as the empty string, which counts as "provided" and would
30+
bypass an input `default:` entirely.
2831
required: false
2932

3033
# Registry logins must precede this action. provenance/sbom stay off: attestation
@@ -49,24 +52,16 @@ runs:
4952
PLATFORMS: ${{ inputs.platforms }}
5053
run: echo "value=${GITHUB_REPOSITORY##*/}/${FILE#./}/${PLATFORMS//\//-}" >> "$GITHUB_OUTPUT"
5154

52-
# max-cache-size-mb is what bounds the disk: BuildKit's default GC is
53-
# time-based only (layers unused for 8 days), and setup-docker-builder skips
54-
# pruning altogether when the value is empty. On a repo that builds this
55-
# often nothing ever ages out, so the disks grew without limit —
56-
# app.Dockerfile/linux-amd64 reached 351 GB inside a day, and realtime, whose
57-
# image is under 300 MB, sat at 249 GB. Sticky disks bill at ~$0.51/GB-month,
58-
# so that was real money for layers no build would ever read again.
59-
#
60-
# The fallback is here rather than an input `default:` because callers pass
61-
# this from a matrix field, and an unset matrix key arrives as the empty
62-
# string — which counts as "provided", so a `default:` would never apply and
63-
# a row that forgot the field would silently go back to unbounded growth.
55+
# This action does NOT bound the disk — see the prune step below. BuildKit's
56+
# own GC is time-based only (layers unused for 8 days), and these disks are
57+
# mounted many times a day, so nothing ever ages out: app.Dockerfile/linux-amd64
58+
# reached 351 GB inside a day of being created, and realtime, whose image is
59+
# under 300 MB, sat at 249 GB. Sticky disks bill at ~$0.51/GB-month.
6460
- name: Set up Blacksmith builder
6561
if: inputs.provider == '' || inputs.provider == 'blacksmith'
6662
uses: useblacksmith/setup-docker-builder@a5256a73e30f09e37e3eceb8ca36043d17621d24 # v2
6763
with:
6864
cache-key: ${{ steps.cache-key.outputs.value }}
69-
max-cache-size-mb: ${{ inputs.max-cache-size-mb || '25600' }}
7065

7166
- name: Build and push (Blacksmith)
7267
if: inputs.provider == '' || inputs.provider == 'blacksmith'
@@ -80,6 +75,40 @@ runs:
8075
provenance: false
8176
sbom: false
8277

78+
# Bound the layer cache ourselves. setup-docker-builder v1 took a
79+
# max-cache-size-mb input and pruned in its own post step, but the v2 rewrite
80+
# dropped it — and GitHub only WARNS on an unknown composite input, so passing
81+
# it to v2 silently did nothing for a day while the app disk sat at 200+ GB.
82+
#
83+
# This is v1's command verbatim (its dist/index.js pruneBuildkitCache), against
84+
# the fixed address v2 itself uses for `buildctl du` and `debug workers`:
85+
# sudo buildctl --addr tcp://127.0.0.1:1234 prune --all --keep-storage <MB>
86+
#
87+
# --all is load-bearing. A plain prune only reclaims layer records and leaves
88+
# `RUN --mount=type=cache` dirs untouched, and those mounts are most of what
89+
# these disks hold (realtime's image is under 300 MB but its disk reached
90+
# 249 GB). Runs before the builder's post step, which is what commits the disk.
91+
#
92+
# Warn rather than fail: a cache that is too large is not worth failing a
93+
# deploy over. The du either side is what makes a silent no-op visible — the
94+
# failure mode that hid the v2 input regression in the first place.
95+
- name: Prune the layer cache
96+
if: (inputs.provider == '' || inputs.provider == 'blacksmith') && !cancelled()
97+
shell: bash
98+
env:
99+
KEEP_MB: ${{ inputs.max-cache-size-mb || '25600' }}
100+
run: |
101+
addr='tcp://127.0.0.1:1234'
102+
# Print the whole Total line rather than picking a column: buildctl's du
103+
# table is whitespace-aligned and its layout is not a stable contract.
104+
total() { sudo buildctl --addr "$addr" du 2>/dev/null | grep -iE '^total:' | tr -s ' \t' ' '; }
105+
echo "before prune -> $(total)"
106+
if sudo buildctl --addr "$addr" prune --all --keep-storage "$KEEP_MB"; then
107+
echo "after prune -> $(total) (keep-storage ${KEEP_MB} MB)"
108+
else
109+
echo "::warning::Layer cache prune failed; this sticky disk is unbounded for this run"
110+
fi
111+
83112
- name: Set up Docker Buildx
84113
if: inputs.provider != '' && inputs.provider != 'blacksmith'
85114
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4

0 commit comments

Comments
 (0)