Skip to content

improvement(ci): bound docker layer caches and right-size ten runners #7210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
waleedlatif1 merged 1 commit into from
Aug 28, 2026
Merged

improvement(ci): bound docker layer caches and right-size ten runners #7210

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/actions/docker-build/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,13 @@ inputs:
tags:
description: Comma-separated list of tags to push.
required: true
max-cache-size-mb:
description: >-
Layer cache to retain after the post-job prune, in MB. Must stay above one
build's working set (base + dependency layers + RUN --mount=type=cache
dirs) or every build evicts what the next one needs. Falls back to the
small-image default below when empty.
required: false

# Registry logins must precede this action. provenance/sbom stay off: attestation
# manifests break `imagetools create` retagging in promote-images.
Expand All @@ -42,11 +49,24 @@ runs:
PLATFORMS: ${{ inputs.platforms }}
run: echo "value=${GITHUB_REPOSITORY##*/}/${FILE#./}/${PLATFORMS//\//-}" >> "$GITHUB_OUTPUT"

# max-cache-size-mb is what bounds the disk: BuildKit's default GC is
# time-based only (layers unused for 8 days), and setup-docker-builder skips
# pruning altogether when the value is empty. On a repo that builds this
# often nothing ever ages out, so the disks grew without limit —
# app.Dockerfile/linux-amd64 reached 351 GB inside a day, and realtime, whose
# image is under 300 MB, sat at 249 GB. Sticky disks bill at ~$0.51/GB-month,
# so that was real money for layers no build would ever read again.
#
# The fallback is here rather than an input `default:` because callers pass
# this from a matrix field, and an unset matrix key arrives as the empty
# string — which counts as "provided", so a `default:` would never apply and
# a row that forgot the field would silently go back to unbounded growth.
- name: Set up Blacksmith builder
if: inputs.provider == '' || inputs.provider == 'blacksmith'
uses: useblacksmith/setup-docker-builder@a5256a73e30f09e37e3eceb8ca36043d17621d24 # v2
with:
cache-key: ${{ steps.cache-key.outputs.value }}
max-cache-size-mb: ${{ inputs.max-cache-size-mb || '25600' }}

@cubic-dev-ai cubic-dev-ai Bot Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: With the pinned builder revision, this with key is not a supported or read input, so the matrix value and 25600 fallback are ignored. Pin a builder revision that implements max-cache-size-mb, or perform the prune in this workflow, before relying on the cache cap.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At .github/actions/docker-build/action.yml, line 69:

<comment>With the pinned builder revision, this `with` key is not a supported or read input, so the matrix value and `25600` fallback are ignored. Pin a builder revision that implements `max-cache-size-mb`, or perform the prune in this workflow, before relying on the cache cap.</comment>

<file context>
@@ -42,11 +49,24 @@ runs:
       uses: useblacksmith/setup-docker-builder@a5256a73e30f09e37e3eceb8ca36043d17621d24 # v2
       with:
         cache-key: ${{ steps.cache-key.outputs.value }}
+        max-cache-size-mb: ${{ inputs.max-cache-size-mb || '25600' }}
 
     - name: Build and push (Blacksmith)
</file context>
Fix with cubic


- name: Build and push (Blacksmith)
if: inputs.provider == '' || inputs.provider == 'blacksmith'
Expand Down
31 changes: 22 additions & 9 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ jobs:
# (/api/desktop/update) starts offering automatically.
detect-desktop-changes:
name: Detect Desktop Changes
runs-on: blacksmith-4vcpu-ubuntu-2404
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 5
if: github.event_name == 'push' && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/staging')
outputs:
Expand Down Expand Up @@ -165,7 +165,15 @@ jobs:
# build` ~260s). The same `next build` runs on 16 vCPU in the separate
# Build App verification job, which does not gate anything; this one
# was doing comparable work on half the cores.
#
# cache_mb is the layer cache the post-job prune retains, and it is the
# only reason the sticky disks stay bounded — see docker-build's
# action.yml. Rows that omit it take the small-image default there. The
# app image overrides because it carries ~34 layers plus apt and bun
# cache mounts for the whole monorepo; 100 GB is several builds' worth
# of headroom over that working set.
- dockerfile: ./docker/app.Dockerfile
cache_mb: '102400'
ecr_repo_secret: ECR_APP
gh_runner: linux-x64-8-core
bs_runner: blacksmith-16vcpu-ubuntu-2404
Expand All @@ -176,11 +184,11 @@ jobs:
- dockerfile: ./docker/realtime.Dockerfile
ecr_repo_secret: ECR_REALTIME
gh_runner: ubuntu-latest
bs_runner: blacksmith-4vcpu-ubuntu-2404
bs_runner: blacksmith-2vcpu-ubuntu-2404
- dockerfile: ./docker/pii.Dockerfile
ecr_repo_secret: ECR_PII
gh_runner: ubuntu-latest
bs_runner: blacksmith-4vcpu-ubuntu-2404
bs_runner: blacksmith-2vcpu-ubuntu-2404
steps:
- name: Checkout code
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
Expand Down Expand Up @@ -214,6 +222,7 @@ jobs:
file: ${{ matrix.dockerfile }}
platforms: linux/amd64
tags: ${{ steps.login-ecr.outputs.registry }}/${{ steps.ecr-repo.outputs.name }}:dev
max-cache-size-mb: ${{ matrix.cache_mb }}

# Dev: deploy Trigger.dev background tasks to the preview "dev-sim" branch.
# Gated after migrate-dev for the same reason as build-dev — the new task
Expand Down Expand Up @@ -280,6 +289,7 @@ jobs:
matrix:
include:
- dockerfile: ./docker/app.Dockerfile
cache_mb: '102400'
ghcr_image: ghcr.io/simstudioai/simstudio
ecr_repo_secret: ECR_APP
gh_runner: linux-x64-8-core
Expand All @@ -293,12 +303,12 @@ jobs:
ghcr_image: ghcr.io/simstudioai/realtime
ecr_repo_secret: ECR_REALTIME
gh_runner: ubuntu-latest
bs_runner: blacksmith-4vcpu-ubuntu-2404
bs_runner: blacksmith-2vcpu-ubuntu-2404
- dockerfile: ./docker/pii.Dockerfile
ghcr_image: ghcr.io/simstudioai/pii
ecr_repo_secret: ECR_PII
gh_runner: ubuntu-latest
bs_runner: blacksmith-4vcpu-ubuntu-2404
bs_runner: blacksmith-2vcpu-ubuntu-2404
# No ECR repo is provisioned for cron, so it publishes to GHCR only.
# The tag step below omits the ECR tag when the repo name is empty.
- dockerfile: ./docker/cron.Dockerfile
Expand Down Expand Up @@ -382,6 +392,7 @@ jobs:
file: ${{ matrix.dockerfile }}
platforms: linux/amd64
tags: ${{ steps.meta.outputs.tags }}
max-cache-size-mb: ${{ matrix.cache_mb }}

# Promote the sha-tagged ECR images to the deploy tags once tests and
# migrations pass. Pushing the ECR latest/staging tag is what triggers
Expand Down Expand Up @@ -484,6 +495,7 @@ jobs:
# hang a release in `queued` rather than fail a PR.
include:
- dockerfile: ./docker/app.Dockerfile
cache_mb: '102400'
image: ghcr.io/simstudioai/simstudio
gh_runner: linux-arm64-8-core
bs_runner: blacksmith-8vcpu-ubuntu-2404-arm
Expand Down Expand Up @@ -522,6 +534,7 @@ jobs:
file: ${{ matrix.dockerfile }}
platforms: linux/arm64
tags: ${{ matrix.image }}:${{ github.sha }}-arm64
max-cache-size-mb: ${{ matrix.cache_mb }}

# Publish all mutable GHCR tags (latest, latest-amd64/arm64, version tags)
# and the multi-arch manifests from the immutable sha tags — only on main,
Expand Down Expand Up @@ -675,7 +688,7 @@ jobs:
# Job-level `if:` cannot read the secrets context, hence the probe job.
check-desktop-signing:
name: Check Desktop Signing Secrets
runs-on: blacksmith-4vcpu-ubuntu-2404
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 2
needs: [detect-version, detect-desktop-changes]
# !cancelled(): detect-desktop-changes is skipped on main (and
Expand Down Expand Up @@ -724,7 +737,7 @@ jobs:
# remains testable end to end with a manual download.
create-desktop-prerelease:
name: Create Desktop Prerelease
runs-on: blacksmith-4vcpu-ubuntu-2404
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 5
needs: [detect-desktop-changes, check-desktop-signing]
# Requires the signing probe to have actually succeeded (not just "not
Expand Down Expand Up @@ -813,7 +826,7 @@ jobs:
# point of view.
publish-desktop-prerelease:
name: Publish Desktop Prerelease
runs-on: blacksmith-4vcpu-ubuntu-2404
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 5
needs: [create-desktop-prerelease, desktop-prerelease]
permissions:
Expand All @@ -837,7 +850,7 @@ jobs:
# are always garbage by this point — the current run's release is published.
prune-desktop-prereleases:
name: Prune Desktop Prereleases
runs-on: blacksmith-4vcpu-ubuntu-2404
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 5
needs: [publish-desktop-prerelease]
permissions:
Expand Down
13 changes: 11 additions & 2 deletions .github/workflows/codeql.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,12 @@ permissions:
jobs:
analyze:
name: Analyze ${{ matrix.language }}
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}
# Sized per language, not per workflow. The two analyses are nothing alike:
# javascript-typescript peaks at 19.5 GB (p95 over 3090 runs), so it needs
# the 8 vCPU tier's 30.4 GB and would OOM on the 4 vCPU tier's 15.2 GB; the
# actions analysis peaks at 1.3 GB and averages 22% CPU over a 39s median
# run, so 8 vCPU was 4x more machine than it ever used.
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && matrix.bs_runner || 'ubuntu-latest' }}
timeout-minutes: 60
if: github.event.pull_request.draft != true
permissions:
Expand All @@ -71,7 +76,11 @@ jobs:
# entries default setup listed were one analysis, not three.
# `javascript-typescript` is the documented spelling. Python dropped:
# 7 files in the tree.
language: [javascript-typescript, actions]
include:
- language: javascript-typescript
bs_runner: blacksmith-8vcpu-ubuntu-2404
- language: actions
bs_runner: blacksmith-4vcpu-ubuntu-2404

steps:
- name: Checkout repository
Expand Down
Loading