Skip to content

improvement(ci): decouple image builds from the test gate#5701

Merged
waleedlatif1 merged 2 commits into
stagingfrom
improvement/ci-decouple-image-builds
Jul 16, 2026
Merged

improvement(ci): decouple image builds from the test gate#5701
waleedlatif1 merged 2 commits into
stagingfrom
improvement/ci-decouple-image-builds

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Problem

Push-to-production-deploy takes ~13.5 minutes because the pipeline is fully serial: tests (5m22) → migrate (0m46) → AMD64 image builds (4m48–7m01) → ECR tag push → CodePipeline. The builds only start after the gate clears, though nothing about building requires it. (Timings from run 29459378914.)

Design

Standard "build once, promote by retag" artifact promotion: images build in parallel with tests under immutable sha tags; passing the gate promotes the exact tested artifact by moving a tag (docker buildx imagetools create, a seconds-long manifest copy) — never by rebuilding.

Invariant: no mutable/documented tag moves, and nothing deploys, until tests + migrations pass. Only immutable sha tags exist before the gate.

ci.yml

  • build-amd64 and build-ghcr-arm64 start immediately and push only sha tags (ECR :<sha>, GHCR :<sha>-amd64 / :<sha>-arm64). This also closes a pre-existing hole: latest-arm64 used to be pushed before test results existed.
  • New promote-images job (after test-build + migrate) retags sha → latest/staging in ECR — the tag CodePipeline's EventBridge rules trigger on. Single job, not a matrix: it verifies all four sha manifests exist before moving any tag, so a partial mixed-version deploy can't happen.
  • create-ghcr-manifests (after promote + arm64 build) now owns every mutable GHCR tag: latest-{amd64,arm64}, latest, version tags, and the multi-arch sha manifest — all built from the immutable sha-arch tags.
  • Stale-run guards on both promoting jobs: re-running an old run used to be able to retag latest back to old code and trigger an immediate prod deploy of it. Now a superseded commit skips mutable-tag moves with a warning (immutable sha/version tags still publish, so releases can't be blocked by a racing push).

test-build.yml

Tests (2m05) and next build (2m26) were serial in one job; now parallel jobs. They share the warm bun-cache/node_modules sticky disks (identical content; Blacksmith disks are clone-on-mount + last-writer-wins commit, so concurrent mounts are safe) — but the build job gets its own Turbo cache key, since with a shared key the two jobs' new entries would evict each other every run.

Verification

  • CodePipeline triggers confirmed tag-filtered in AWS (image-tag: ["latest"|"staging"|"dev"]) for all 9 pipelines — sha pushes trigger nothing
  • ECR lifecycle policy expires images after 7 days → sha tags self-clean (the promote verify step also fails loudly if a sha image has expired rather than half-promoting)
  • actionlint clean (except pre-existing runner-label/SC2086-style notes shared with the old file)
  • Independent adversarial review of the full diff; all real findings fixed in the second commit
  • No branch protection/rulesets pin the old job names
  • PR CI run validates the test/build split (first run: green, jobs in parallel)

Expected impact

Path Before After
PR feedback ~5m20 ~3m30
Push → prod deploy trigger ~13m30 ~7m (bounded by slowest image build)
Full main run ~15m ~8m

Accepted trade-off

Untested immutable sha-arch tags now exist on public GHCR for commits whose tests later fail (already true for arm64 today). They're plumbing tags nothing documents or deploys, but GHCR has no lifecycle expiry — if we care about accumulation, a scheduled ghcr untagged/sha cleanup job is a small follow-up.

Watch on merge

The push path (sha push + promote) first executes on the merge to staging — watch that run's promote-images before this reaches main.

- build-amd64 now starts immediately and pushes only sha tags (ECR :sha,
  GHCR :sha-amd64). The EventBridge deploy triggers filter on exactly the
  latest/staging/dev ECR tags, so nothing deploys from these pushes.
- New promote-images job retags sha -> latest/staging (and GHCR
  latest-amd64/version-amd64) via buildx imagetools once tests and
  migrations pass — a seconds-long manifest copy instead of rebuilding
  after the gate. Cuts push-to-deploy from ~13.5 to ~7 minutes.
- Split the Next.js production build out of test-build into a parallel
  Build App job with its own sticky-disk keys, cutting PR feedback from
  ~5.5 to ~3.5 minutes.
- create-ghcr-manifests and process-docs now gate on promote-images.
@vercel

vercel Bot commented Jul 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jul 16, 2026 2:39am

Request Review

@cursor

cursor Bot commented Jul 16, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Changes the production/staging deploy gate and when CodePipeline fires; mis-timed promotion or guard bypass could deploy untested or stale images, though SHA verification and branch-head guards mitigate that.

Overview
Reworks main/staging deploy CI so image builds overlap tests and deploy only happens after a retag, not when images are first pushed.

ci.yml: build-amd64 and build-ghcr-arm64 no longer wait on tests/migrations and push only commit-SHA tags (ECR :<sha>, GHCR :<sha>-amd64 / :<sha>-arm64). A new promote-images job runs after migrate + build-amd64, verifies all four ECR SHA manifests exist, then retags sha → latest or staging (what triggers CodePipeline). Stale-run guards skip moving deploy/latest tags if the branch head has moved. create-ghcr-manifests now runs after promote-images and owns mutable GHCR tags (latest, version, multi-arch) via docker buildx imagetools create, with the same head-SHA guard for latest. process-docs waits on promote-images instead of build-amd64.

test-build.yml: Splits the monolithic job into parallel Lint and Test vs Build App; moves the Next.js build cache to the build job and gives it a separate Turbo sticky-disk key so test and build caches do not evict each other.

Reviewed by Cursor Bugbot for commit f457ba0. Configure here.

@greptile-apps

greptile-apps Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR builds images in parallel with validation and promotes them only after the deployment gate passes. The main changes are:

  • Builds and publishes immutable SHA-tagged images before promotion.
  • Promotes ECR and GHCR tags after tests and migrations succeed.
  • Prevents stale workflow runs from moving mutable tags.
  • Runs lint/tests and the production application build in parallel.

Confidence Score: 5/5

This looks safe to merge.

  • The renamed check is not required by the configured branch rules.
  • The deployment path still waits for tests and migrations before moving mutable tags.
  • No blocking issue related to the earlier review thread remains.

Important Files Changed

Filename Overview
.github/workflows/ci.yml Parallelizes immutable image builds and adds gated ECR and GHCR tag promotion with stale-run guards.
.github/workflows/test-build.yml Runs lint/tests and the application build as separate parallel jobs with dedicated Turbo caches.

Reviews (2): Last reviewed commit: "improvement(ci): harden promotion — atom..." | Re-trigger Greptile

Comment thread .github/workflows/test-build.yml
…ate all mutable GHCR tags

Review follow-ups:
- promote-images is a single job (not a matrix): verifies all four :sha
  manifests exist before moving any deploy tag, so a missing image can't
  cause a partial mixed-version deploy
- stale-run guard on promote-images and create-ghcr-manifests: re-running
  an old run no longer retags latest/staging back to stale code (a
  one-click prod rollback); superseded runs skip mutable tags with a
  warning while immutable sha/version tags still publish
- ARM64 build now pushes only the immutable :sha-arm64 tag; latest-arm64
  and version tags moved behind the gate into create-ghcr-manifests
  (closes the pre-existing hole where a failing run moved latest-arm64)
- dropped dead detect-version needs from both build jobs
- sticky-disk comment corrected (clone + last-writer-wins, not exclusive
  mounts); build job shares warm bun/node_modules disks, keeps its own
  turbo cache key so test/build entries don't evict each other
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit f457ba0. Configure here.

@waleedlatif1
waleedlatif1 merged commit 4bb5600 into staging Jul 16, 2026
21 checks passed
@waleedlatif1
waleedlatif1 deleted the improvement/ci-decouple-image-builds branch July 16, 2026 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant