docs(design): CI build performance roadmap + implementer notes (#363)#364
Draft
scottschreckengaust wants to merge 2 commits into
Draft
docs(design): CI build performance roadmap + implementer notes (#363)#364scottschreckengaust wants to merge 2 commits into
scottschreckengaust wants to merge 2 commits into
Conversation
Records how the build workflow's cost is distributed (//cdk:test was the ~91% long pole), why it was expensive (redundant type-check, fixed in #357 / PR #359), and the remaining levers (shard, coverage-gate, runner size, path-filter) with sequence and implementer notes — including the merge-queue required-check constraints that shape #2 and #5. Companion to umbrella issue #363; keeps the optimization roadmap as a durable, reviewable design artifact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3 tasks
Expands the CI build-performance doc with a full design for item #2 (shard the CDK suite): the jest --shard mechanism, a shards-vs-wall-time table showing fixed overhead dominates past ~4 shards, the fan-out + aggregate-gate pattern needed to keep `build` a single required context on merge_group, cross-shard coverage merge before threshold enforcement, and open questions (synth placement, deploy artifact, shard balance). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jun 16, 2026
scottschreckengaust
added a commit
that referenced
this pull request
Jun 17, 2026
The doc lives in draft PR #364 and exists on neither this branch nor main, so the three links (disable-bundling.ts, AGENTS.md, review_pr.md) are dead. Drop them and keep the stable #366 reference, removing the merge-order dependency on #364 and the link-checker (#300) risk. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
isadeks
pushed a commit
to isadeks/sample-autonomous-cloud-coding-agents
that referenced
this pull request
Jun 18, 2026
… synth) (aws-samples#366) (aws-samples#371) * perf(test): disable Lambda bundling in CDK unit-test synths (aws-samples#366) Template.fromStack() triggers a full CDK synth that bundles every NodejsFunction via esbuild — ~28s for the 413-resource AgentStack. Unit tests assert on CloudFormation structure, not bundled Lambda code, so the bundling is pure overhead. A jest setupFiles hook sets aws:cdk:bundling-stacks: [] via CDK_CONTEXT_JSON, which a bare new App() picks up with no call-site changes. Measured: single AgentStack synth 28.7s -> 1.9s (~15x); github-tags.test.ts ~135s -> 9s; full //cdk:test 155s -> 25s (8-core local). All 2177 tests pass; coverage thresholds unchanged (lines 92.36 >= 91). Durable regression guards added: AGENTS.md "Common mistakes" entry and a .abca/commands/review_pr.md checklist item — do not re-enable bundling in tests unless asserting on a bundled asset, and minimize full-stack synths (synth once in beforeAll). Both link docs/design/CI_BUILD_PERFORMANCE.md (PR aws-samples#364). Refs aws-samples#366. Part of aws-samples#363. Likely supersedes the sharding approach (aws-samples#365) — to be re-evaluated once this lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(test): correct bundling opt-out to postCliContext (aws-samples#366) The documented per-test opt-out used constructor `context`, which CDK silently overwrites with CDK_CONTEXT_JSON in App.loadContext — so `new App({ context: { 'aws:cdk:bundling-stacks': ['**'] } })` does NOT re-enable bundling. Only `postCliContext` (applied last) overrides the global disable. Corrects the opt-out instruction and precedence comment in all three documented locations (disable-bundling.ts, AGENTS.md, review_pr.md). Verified empirically against aws-cdk-lib@2.257.0. Comment/doc-only; runtime logic unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(test): clarify bundling opt-out is for asset output, not synth (aws-samples#366) Disabling bundling does not stop tests from synthesizing — `Template.fromStack()` still runs a full synth; it only skips the esbuild asset-bundling step. The `postCliContext` opt-out exists solely for the rare test that needs real bundled-asset output (asset hash / S3 key), where an unbundled synth would silently yield a placeholder. Makes this explicit in disable-bundling.ts and AGENTS.md. Doc-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(test): drop dead CI_BUILD_PERFORMANCE.md links (aws-samples#366) The doc lives in draft PR aws-samples#364 and exists on neither this branch nor main, so the three links (disable-bundling.ts, AGENTS.md, review_pr.md) are dead. Drop them and keep the stable aws-samples#366 reference, removing the merge-order dependency on aws-samples#364 and the link-checker (aws-samples#300) risk. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(test): harden bundling-disable against ambient CDK_CONTEXT_JSON (aws-samples#366) Two robustness fixes to the global bundling-disable setup file, both from PR aws-samples#371 review (finding 1 hardening nits): - Spread BUNDLING_DISABLED_CONTEXT LAST when merging a pre-set CDK_CONTEXT_JSON, so an ambient `aws:cdk:bundling-stacks` value can no longer silently win and re-enable bundling. This is safe precisely because the documented per-test opt-out is `postCliContext` (applied after the env var regardless), so the escape hatch still works. - Guard `JSON.parse` of the pre-set var with try/catch. A malformed value previously threw inside setupFiles and failed every test in every file with an opaque parse error; now the blame is localized to the offending value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cdk): add executable guard that Lambda bundling is disabled (aws-samples#366) PR aws-samples#371 review finding 1: the ~15× synth speedup rested entirely on the `disable-bundling.ts` setupFiles entry, but the only regression guards were honor-system prose (AGENTS.md) and a review_pr.md checklist item. Dropping the setupFiles wiring, reordering Jest setup, or a CDK rename of the context key would silently revert the suite to full-bundling synths with no failing check. Adds two assertions: the synth context carries `aws:cdk:bundling-stacks: []`, and a bare-App stack reports `bundlingRequired === false`. Verified empirically against aws-cdk-lib@2.259.0 that without the disable a bare App yields `bundlingRequired === true`, so these discriminate a real regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cdk): cover the postCliContext bundling opt-out precedence (aws-samples#366) PR aws-samples#371 review finding 2: the documented per-test opt-out (`postCliContext`) and the `loadContext` precedence this PR corrected had zero test coverage, so a future CDK version reordering context application could silently break the one supported escape hatch. Locks the contract with two assertions verified against aws-cdk-lib@2.259.0: `postCliContext: { 'aws:cdk:bundling-stacks': ['**'] }` re-enables bundling (`bundlingRequired === true`), while constructor `context` does NOT (the env var set by the global disable clobbers it). Uses a bare `Stack` rather than a heavy AgentStack since the precedence is a property of CDK context loading, not of any particular stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(test): reframe bundling as overhead-plus-smoke-test, not "pure overhead" (aws-samples#366) PR aws-samples#371 review finding 3: "pure overhead" undersold the tradeoff. esbuild bundling also validates that every Lambda `entry` resolves and the handler compiles, so disabling it in unit tests means a broken entry path or handler TS error stops surfacing at `Template.fromStack()` and only fails at real synth/deploy or in the `agent/` build. The tradeoff is still acceptable — real synth/deploy bundles, and handler code has its own compile/test under `agent/` — but the wording now names what unit tests cede. Updated in both places the phrase lived: the disable-bundling.ts header and AGENTS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: bgagent <345885+scottschreckengaust@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
docs/design/CI_BUILD_PERFORMANCE.md— a design doc capturing the CI build-performance work: where the build's time goes, why//cdk:testwas the ~91% long pole, the fix already shipped (#357 / #359), and the remaining levers (#2 shard, #3 coverage-gate, #4 runner size, #5 path-filter) with the suggested sequence and implementer notes.The Starlight mirror (
docs/src/content/docs/architecture/Ci-build-performance.md) is regenerated vianode scripts/sync-starlight.mjsand committed alongside the source (per the docs-generation boundary).Why
This is the durable, reviewable companion to umbrella issue #363. It keeps the optimization roadmap and — importantly — the merge-queue required-check constraints (which shape how #2 and #5 must be implemented) in one referenced place, so future implementers don't rediscover them the hard way.
Highlights captured
buildcheck is required and must report onmerge_group(feat(ci): require secrets/deps/SAST on every PR — make the merge gate enforceable (incident #313 class) #327) — so shard results must be aggregated into the required check, and path-filtering must keep the job and gate the steps, never skip the job.merge_group.Relates to #363. Follow-up to #357 / #359.
Validation
node scripts/sync-starlight.mjsis idempotent (re-run produces no further diff) — CI's "Fail build on mutation" will pass.🤖 Generated with Claude Code