fix: resolve recurring Terraform state lock in CI - #72
Conversation
3d784e9 to
c5c6ddc
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Receiver-mock AWS resource names must be made unique or the job must be serialized to prevent concurrent deployment races.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This pull request improves Terraform CI state isolation and cleanup reliability for receiver-mock deployments.
Changes:
- Uses
github.run_idfor unique Terraform state keys. - Initializes Terraform during destruction without relying on cache.
- Runs Lambda cleanup even when tests fail.
File summaries
| File | Summary |
|---|---|
.github/workflows/tests.yml |
Updates Terraform state handling and cleanup, but receiver-mock AWS resource names remain shared across concurrent same-language runs. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Replace matrix.architecture (undefined in deploy-receiver-mock) with github.run_id in receiver-mock BUCKET_KEY, eliminating the shared receivermock..nodejs state path that caused concurrent-run lock collisions - Add terraform init to destroy-receiver-mock and remove the cache dependency, so destroy is reliable even when deploy fails (cache is never saved on apply failure) - Change Cleanup AWS environment from if: success() to if: always() so Lambda infra is destroyed on test failure, not just on success
c5c6ddc to
ec3123c
Compare
…nitization - Add fail-fast: false to run-tests matrix so arm64 cleanup always runs even when amd64 fails, preventing Lambda infra leaks on partial failures - Append github.run_attempt to all state keys and resource names so re-runs get a clean slate instead of colliding with a partially destroyed prior attempt - Replace github.head_ref with github.run_id in pr-build artifact names; branch names with forward slashes (e.g. fix/foo) are invalid artifact names and cause the build to fail immediately
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical artifact retry failures and moderate retry-isolation and cleanup risks remain.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (3)
.github/workflows/pr-build-nodejs.yml:20
- The retry isolation is incomplete for the next reusable job:
publish-dev-layer.ymlstill derivesLAYER_NAMEandBUCKET_NAMEsolely fromgithub.run_id. A canceled or partially failed attempt can leave that S3 bucket, and a retry of the same run reuses it, causingaws s3 mbto fail or appending to the previous layer. Include the attempt in those resource names and provide cleanup for an older attempt.
ARTIFACT_NAME: ${{ github.run_id }}
.github/workflows/tests.yml:59
- The attempt isolation is incomplete for this workflow: the called
publish-dev-layer.ymlstill derives its S3 bucket and Lambda layer names fromgithub.run_idonly. If a prior attempt is canceled beforeClean s3runs, a re-run can reuse or fail on those resources; includegithub.run_attemptin those names as well.
TF_VAR_app_name: ll-${{ inputs.LANGUAGE }}-${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/tests.yml:59
- The ALB name is built as
${var.app_name}-${var.app_environment}-albandapp_environmentdefaults toaws(utils/receiver-mock/deploy/loadbalancer.tf:2,variables.tf:19-22). Fornodejs, an 11-digit run ID plus a three-digitrun_attemptproduces a 33-character ALB name, exceeding AWS's 32-character limit, so sufficiently many retries fail during apply. Generate a bounded-length per-attempt identifier before passing it asTF_VAR_app_name.
TF_VAR_app_name: ll-${{ inputs.LANGUAGE }}-${{ github.run_id }}-${{ github.run_attempt }}
- Files reviewed: 4/4 changed files
- Comments generated: 4
- Review effort level: Lite
State keys and resource names use run_id only — Terraform's apply is idempotent so re-runs reconcile against existing state rather than creating orphaned resources from a new empty state. Artifact names use run_id+run_attempt — upload-artifact@v4 keeps names immutable within a run so re-runs need a fresh name to avoid the already-exists error.
There was a problem hiding this comment.
🟡 Changes recommended
Re-run state, artifact, and layer resource names remain insufficiently attempt-scoped.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (7)
.github/workflows/pr-build-java.yml:20
- The attempt suffix is not propagated to the sample-app artifact names in
build-artifacts.yml(the Java, NodeJS, and Python uploads still usesample-app-${LANGUAGE}-${architecture}-artifacts). Becauseupload-artifact@v4artifacts are immutable within a run, a re-run will try to upload an existing artifact and fail before tests start; make those upload names attempt-scoped and update the matching download name intests.yml.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-java.yml:27
- Although the caller now passes an attempt-scoped artifact name,
publish-dev-layer.ymlstill constructsLAYER_NAMEandBUCKET_NAMEfromgithub.run_idonly. If a matrix job is canceled or fails before its cleanup, a re-run reuses the old S3 bucket (and layer namespace), somake create-bucketcan fail and stale layer resources remain; includegithub.run_attemptin those resource names too.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-nodejs.yml:20
- The attempt suffix is not propagated to the sample-app artifact names in
build-artifacts.yml(the Java, NodeJS, and Python uploads still usesample-app-${LANGUAGE}-${architecture}-artifacts). Becauseupload-artifact@v4artifacts are immutable within a run, a re-run will try to upload an existing artifact and fail before tests start; make those upload names attempt-scoped and update the matching download name intests.yml.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-python.yml:20
- The attempt suffix is not propagated to the sample-app artifact names in
build-artifacts.yml(the Java, NodeJS, and Python uploads still usesample-app-${LANGUAGE}-${architecture}-artifacts). Becauseupload-artifact@v4artifacts are immutable within a run, a re-run will try to upload an existing artifact and fail before tests start; make those upload names attempt-scoped and update the matching download name intests.yml.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-python.yml:27
- Although the caller now passes an attempt-scoped artifact name,
publish-dev-layer.ymlstill constructsLAYER_NAMEandBUCKET_NAMEfromgithub.run_idonly. If a matrix job is canceled or fails before its cleanup, a re-run reuses the old S3 bucket (and layer namespace), somake create-bucketcan fail and stale layer resources remain; includegithub.run_attemptin those resource names too.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/tests.yml:82
- This Lambda test state key also omits
github.run_attempt, so a retry reuses the prior attempt's architecture-specific state and lock, which can contain a partial deployment. Include the attempt in this key to keep retries isolated.
BUCKET_KEY: ${{ matrix.architecture}}.${{ inputs.LANGUAGE }}.${{ github.run_id }}/terraform_state.tfstate
.github/workflows/tests.yml:59
- This run-scoped resource name still omits
github.run_attempt. Re-runs therefore reuse the previous attempt's ECS/ALB/IAM names, so a partial destroy can make the new apply race with or adopt leftover infrastructure. Include the attempt in the name and use the same value for destroy.
TF_VAR_app_name: ll-${{ inputs.LANGUAGE }}-${{ github.run_id }}
- Files reviewed: 4/4 changed files
- Comments generated: 2
- Review effort level: Lite
Ensures re-runs get fully isolated state files, AWS resource names, S3 buckets, and Lambda layer names. Without run_attempt, a re-run targeting the same run_id hits the previous attempt's DynamoDB lock, existing S3 bucket, and stale layer — failing or producing incorrect results.
Terraform 1.16 warns that the "dynamodb_table" backend parameter is deprecated. "use_lockfile" moves locking into S3 and removes the DynamoDB stuck-lock failure mode instead of routing around it. Drop run_attempt from the Terraform state keys. A re-run must reuse the state of the previous attempt. If it does not, it starts from an empty state and permanently orphans every resource the failed attempt created. Hash the run id into the receiver-mock app_name. The embedded run id left the load balancer name at 31 of its 32 character limit, so a 12-digit run id or a two-digit attempt number would break the apply. The hash is a fixed 8 characters, which holds the name at 26. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`actions/upload-artifact@v4` rejects a name that contains a forward slash. The PR build workflows built the artifact name from `github.head_ref`, so every branch with a slash in its name, for example `fix/ci-optimizations`, failed the build before it compiled any code. Use `github.run_id`-`github.run_attempt`, which is always plain digits. The attempt number gives each re-run a new namespace, because an artifact name cannot be reused. `ARTIFACT_NAME` only names and finds artifacts. Layer names and bucket names come from `github.run_id`, so they do not change. The release workflows keep `github.ref_name`, because a tag name has no slash. This matches the same fix in PR #72, so the two branches do not disagree. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Artifact naming still breaks concurrent runs and reruns because sample-app artifacts remain shared and prior-attempt artifacts are not preserved.
Review details
Suppressed comments (6)
.github/workflows/pr-build-java.yml:27
- On a "rerun failed jobs", GitHub does not rerun successful
build-artifacts, so the only available artifact still has the previous attempt's name (for example,<run_id>-1-...). This downstream input switchesdownload-artifactto<run_id>-2-..., causing the layer job to fail before deployment. Preserve/retrieve the artifact from the attempt that produced it, or otherwise make the rerun path explicitly handle the prior attempt rather than unconditionally using the current attempt.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-java.yml:20
- This run-scopes only the layer artifact name, but
build-artifacts.ymlstill uploads the sample app assample-app-${{ env.LANGUAGE }}-${{ matrix.architecture }}-artifacts(lines 102/121/140). Two concurrent PR runs for this language will therefore collide on the immutableupload-artifact@v4upload, and the tests still download that shared name. Include the run-scoped name in the sample-app upload and update its consumers as well.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-nodejs.yml:27
- On a "rerun failed jobs", GitHub does not rerun successful
build-artifacts, so the only available artifact still has the previous attempt's name (for example,<run_id>-1-...). This downstream input switchesdownload-artifactto<run_id>-2-..., causing the layer job to fail before deployment. Preserve/retrieve the artifact from the attempt that produced it, or otherwise make the rerun path explicitly handle the prior attempt rather than unconditionally using the current attempt.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-nodejs.yml:20
- This run-scopes only the layer artifact name, but
build-artifacts.ymlstill uploads the sample app assample-app-${{ env.LANGUAGE }}-${{ matrix.architecture }}-artifacts(lines 102/121/140). Two concurrent PR runs for this language will therefore collide on the immutableupload-artifact@v4upload, and the tests still download that shared name. Include the run-scoped name in the sample-app upload and update its consumers as well.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-python.yml:27
- On a "rerun failed jobs", GitHub does not rerun successful
build-artifacts, so the only available artifact still has the previous attempt's name (for example,<run_id>-1-...). This downstream input switchesdownload-artifactto<run_id>-2-..., causing the layer job to fail before deployment. Preserve/retrieve the artifact from the attempt that produced it, or otherwise make the rerun path explicitly handle the prior attempt rather than unconditionally using the current attempt.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
.github/workflows/pr-build-python.yml:20
- This run-scopes only the layer artifact name, but
build-artifacts.ymlstill uploads the sample app assample-app-${{ env.LANGUAGE }}-${{ matrix.architecture }}-artifacts(lines 102/121/140). Two concurrent PR runs for this language will therefore collide on the immutableupload-artifact@v4upload, and the tests still download that shared name. Include the run-scoped name in the sample-app upload and update its consumers as well.
ARTIFACT_NAME: ${{ github.run_id }}-${{ github.run_attempt }}
- Files reviewed: 5/5 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
build-artifacts uploads the layer artifact and publish-dev-layer downloads it. A partial re-run can re-execute one of the two jobs and carry the other over, so an attempt number in the name makes the download look for an artifact that the upload job never created. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The artifact name no longer holds run_attempt, so a full re-run uploads under the name that the first attempt used. upload-artifact fails on a name that exists unless overwrite is set. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Point 1 — the artifact name and re-runs. Correct.
"Re-run failed jobs" increases Fixed in d85b065. That fix opens the opposite case, so 8b9dad4 closes it. The name now holds no attempt number, so a full re-run uploads under the name that the first attempt used. Point 2 — the sample-app artifact name and parallel runs. Not correct. Artifact names are unique inside one workflow run. They are not unique across the repository. Two runs can each hold an artifact with the same name. The conflict message from Here is an example from this repository. Run 35069236697 and run 35203397591 each hold an artifact named An attempt number in this name would also cause the problem in point 1, because — written by an agent on behalf of Shubham Gupta |
* fix(ci): use RenovateBot app token to push release tag to main The default GITHUB_TOKEN cannot push to branches protected by branch-protection rules. Using the RenovateBot-SumoLogic GitHub App (already used in this org) to mint a short-lived token that has the required bypass permission on main. * fix(ci): ignore release tag links in markdown-link-check Docs PRs add a CHANGELOG entry with a link to the upcoming release tag before that tag exists. The link checker returns 404 and fails the PR. Release tags are immutable once created, so skipping live HTTP checks on them is safe. Any genuine typo in a tag URL would be caught by the release-tag workflow itself failing to push. * fix(ci): only run relevant checks per changed file type Three problems in pull-request-checks.yml: - markdown-link-check had no job-level if guard so it ran on every PR - yamllint referenced a non-existent 'chart-changed' output (typo), always silently skipping but still spinning up a runner - terraform-lint ran unconditionally regardless of whether any .tf files changed Fixes: - Add a terraform-changed job that detects .tf/.tfvars changes - Add if guard to markdown-link-check at the job level - Fix yamllint to reference docs-changed instead of chart-changed - Gate terraform-lint on terraform-changed * ci: fix yaml-lint * ci: make required status checks report on every PR Branch protection matches checks by name. Two gating styles gave different results: - A job that an `if` condition skips reports a `skipped` conclusion. Branch protection accepts this. - A workflow that `on.pull_request.paths` skips reports no check run at all. Branch protection waits for it forever. The three `pr-build-*` workflows used path filters, so a docs-only PR left six required contexts pending and blocked the merge. A job-level `if` cannot replace the path filter directly, because those jobs call reusable workflows and the nested check names do not exist when the call is skipped. Each `pr-build-*` workflow now runs on every PR, moves the file test into a `changes` job, and ends with a plain aggregator job that always reports. `pull-request-checks.yml` gets the same aggregator. Also move the four lint guards in `pull-request-checks.yml` from step level to job level, so those jobs no longer start a runner to do nothing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): give artifacts a name that has no slash `actions/upload-artifact@v4` rejects a name that contains a forward slash. The PR build workflows built the artifact name from `github.head_ref`, so every branch with a slash in its name, for example `fix/ci-optimizations`, failed the build before it compiled any code. Use `github.run_id`-`github.run_attempt`, which is always plain digits. The attempt number gives each re-run a new namespace, because an artifact name cannot be reused. `ARTIFACT_NAME` only names and finds artifacts. Layer names and bucket names come from `github.run_id`, so they do not change. The release workflows keep `github.ref_name`, because a tag name has no slash. This matches the same fix in PR #72, so the two branches do not disagree. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): pin tj-actions/changed-files to v47.0.6 SHA Mutable tags can be retargeted silently. tj-actions/changed-files had a supply-chain incident in 2025 where a tag was hijacked to exfiltrate secrets. Pinning to an immutable commit SHA prevents a future tag retarget from changing CI behaviour. Also upgrades from v44 to v47.0.6. No breaking changes to the inputs or outputs used (files, files_ignore, any_changed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * ci: comment workflow trigger from gh token, now its automatic based on branch name * ci: fix link --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Problem
The Terraform state lock for receiver-mock gets stuck. Each time, someone must delete the lock item from DynamoDB by hand. Concurrent CI runs also leave ECS and Lambda infrastructure behind when tests fail.
There are three causes:
Evidence for cause 1
The matrix of
deploy-receiver-mockdefines onlyaws_region, somatrix.architectureexpanded to an empty string. Every run got the same key, with a telltale double dot:On 2026-09-16 two runs started two minutes apart on one branch. Run 35069025256 took the lock at 07:36:16. Run 35069236697 failed at 07:38:22 while the first run still held it:
The
run-testsjob had the same class of bug. Its key wasamd64.nodejs/terraform_state.tfstate, which is static for each language and architecture.Changes
All changes are in
.github/workflows/.Give each run its own state and its own resource names
tests.ymlgithub.run_idin place of the undefinedmatrix.architecture.tests.ymlrun-testsstate key getsgithub.run_id.tests.ymlTF_VAR_app_namechanges from a fixedlambda-layers-<language>to a run-scoped name. See Resource names.Concurrent runs no longer compete for one lock, and they no longer race to create the same ECS cluster, load balancer, target group, and IAM roles.
Make cleanup run, and make it work
tests.ymlCleanup AWS environmentchanges fromif: success()toif: always(). Lambda infrastructure is now destroyed when tests fail.tests.ymldestroy-receiver-mockgets an explicitterraform init. It ran in an uninitialized directory before, so it failed exactly when it was needed.tests.ymlCache terraform working directorysteps are removed. The cache was keyed to the run and was never written when apply failed, so destroy always got a miss. Nothing reads the cache now.tests.ymlrun-testsmatrix getsfail-fast: false. A failure in one architecture no longer cancels the sibling job, so that job gets the full time thatterraform destroyneeds.Move the lock into S3
tests.ymlterraform initcalls replace--backend-config "dynamodb_table=..."with--backend-config "use_lockfile=true".Terraform 1.16 warns that
dynamodb_tableis deprecated. S3-native locking keeps the lock file next to the state, so it cannot survive as an orphan in a separate table. This removes the stuck-lock failure mode instead of routing around it.The OIDC role holds
AdministratorAccess, so it has thes3:DeleteObjectpermission that lock release needs.Fix artifact names that break the build
pr-build-{java,nodejs,python}.ymlARTIFACT_NAMEchanges fromgithub.head_reftogithub.run_id.publish-dev-layer.ymlLAYER_NAMEandBUCKET_NAMEgetgithub.run_attempt.build-artifacts.ymlactions/upload-artifact@v4steps getoverwrite: true.github.head_refholds the branch name. GitHub Actions rejects an artifact name that contains/, so a branch such asfix/terraform-state-lockfailed the build before it compiled any code.ARTIFACT_NAMEcarries no attempt number.LAYER_NAMEandBUCKET_NAMEdo. See Which names getrun_attempt.Because the artifact name holds no attempt number, a full re-run uploads under the name that the first attempt used.
upload-artifactfails on a name that exists, so the uploads setoverwrite: true.Resource names
TF_VAR_app_nameis a prefix on about 30 resources inutils/receiver-mock/deploy. Two of them cap at 32 characters: the load balancer (loadbalancer.tf:2) and the target group (target_group.tf:2). Both append-aws-and a suffix.A run-scoped name must therefore stay short. The workflow hashes the run id to a fixed width:
An embedded run id leaves only one character spare, and the width grows over time:
app_namell-nodejs-35083149032-1ll-nodejs-35083149032-1-aws-albll-nodejs-350831490321-10ll-nodejs-350831490321-10-aws-albll-nodejs-2d31e6ddll-nodejs-2d31e6dd-aws-albThe hash is always 8 characters, so a longer run id or a two-digit attempt number cannot break the apply.
The hash is also deterministic.
destroy-receiver-mockcomputes the same name from the same run id, so no job output is needed between the two jobs.Which names get
run_attemptThe rule: a name that one job writes and another job reads must not hold the attempt number. A name for a resource that a job creates for its own use must hold it.
The reason is that "Re-run failed jobs" increments
run_attemptfor the whole run, but it re-executes only the jobs that failed. A job that passed is carried over. The two sides of a cross-job name can therefore sit on different attempt numbers.run_attemptTF_VAR_app_namedestroy-receiver-mockmust derive the same name thatdeploy-receiver-mockapplied.ARTIFACT_NAMEpublish-dev-layerdownloads whatbuild-artifactsuploaded. A partial re-run can re-execute one of the two and carry the other over.LAYER_NAME,BUCKET_NAMEaws s3 mbfails if the bucket exists, so a re-attempt needs a fresh name.— written by an agent on behalf of Shubham Gupta