deploy: AWS Lambda + GCP Cloud Run deploy workflows failing on every main commit since ~Jul 3

Both the Deploy to AWS Lambda and Deploy to GCP Cloud Run workflows have concluded FAILURE on every main commit since approximately Jul 3, 2026 (verified pre-existing before the ladder-schema merge #1355 and still failing on 7adacd7). These are deploy pipelines, not PR gates, so they do not block merges, but continuous deploy is effectively broken for those two targets.

Needs: pull the failed run logs for each, identify root cause (likely infra/credentials/terraform drift rather than code, given they fail regardless of commit content), fix at root. Azure Container Apps deploy appears healthy by comparison.

Found during the merge-driver sweep (tracker context: LeanerCloud/cloud-commitments-go#27 work verified these failures are unrelated to the ladder merges).

Root cause (AWS deploy failure): RDS auto-minor-upgraded cudly-dev-426fc8af-postgres from 16.6 to 16.13. Because engine_version was pinned to the exact minor "16.6", Terraform detected a mismatch and attempted to downgrade -- which RDS rejects: InvalidParameterCombination: Cannot find upgrade path from 16.13 to 16.6.

Fix: All six engine_version pins changed from "16.6" to "16" (major-only). The AWS provider v5 stores the actual minor in engine_version_actual and suppresses diff when the configured value is a major-only prefix, so future auto-minor upgrades are transparent and no downgrade is ever attempted. auto_minor_version_upgrade = true is now explicit in the module.

PR: #1375

GCP deploy auth failure - operator runbook

Root cause confirmed. The WIF provider's `attribute_condition` gates on both repository AND ref. The live condition (verified via `gcloud`) is still pinned to the old branch:

assertion.repository == 'LeanerCloud/CUDly' && assertion.ref == 'refs/heads/feat/multicloud-web-frontend'

Every push to `main` is rejected because the ref does not match. The local `terraform.tfvars` (gitignored, per-env) was last applied with `deploy_ref = "refs/heads/feat/multicloud-web-frontend"` and was not updated after the branch merge.

No unexpected drift: the live condition exactly matches what Terraform produced from the stale tfvars. The code-side default in `variables.tf` is already `refs/heads/main`.

How to fix (privileged operator, ~2 min)

cd terraform/environments/gcp/ci-cd-permissions

# 1. Update your local terraform.tfvars
#    Change the deploy_ref line to:
#      deploy_ref = "refs/heads/main"

# 2. Init (use your existing backend.hcl)
terraform init -backend-config=backend.hcl

# 3. Plan - should show one update to the WIF provider attribute_condition
terraform plan

# 4. Apply
terraform apply

The plan output should show:

~ resource "google_iam_workload_identity_pool_provider" "github" {
    ~ attribute_condition = "... assertion.ref == 'refs/heads/feat/multicloud-web-frontend'"
                        -> "... assertion.ref == 'refs/heads/main'"
}

How to verify

After applying, verify the live condition:

gcloud iam workload-identity-pools providers describe github-actions \
  --workload-identity-pool=github-actions \
  --location=global \
  --project=serene-bazaar-666 \
  --format="value(attributeCondition)"

Expected output:

assertion.repository == 'LeanerCloud/CUDly' && assertion.ref == 'refs/heads/main'

Then re-run the Deploy to GCP Cloud Run workflow from the Actions tab (trigger a push to main or use workflow_dispatch). The auth step should succeed.

PR #1379 fixes the README (which incorrectly omitted the `deploy_ref` clause) and adds warnings to prevent this from recurring after future branch changes.

GCP half APPLIED and verified (owner-authorized): the stale local terraform.tfvars had deploy_ref pinned to the retired feat/multicloud-web-frontend branch; flipped to refs/heads/main and applied terraform in gcp/ci-cd-permissions. Plan contained exactly one in-place change (the WIF provider attribute_condition); the live condition now reads: assertion.repository == 'LeanerCloud/CUDly' && assertion.ref == 'refs/heads/main' (verified via gcloud describe). The Deploy to GCP Cloud Run workflow has been re-triggered on main to confirm end-to-end. Docs hardening for the pinning trap is in PR #1379. AWS half (RDS engine-version downgrade) remains in PR #1375.

GCP deploy CONFIRMED FIXED end-to-end: Deploy to GCP Cloud Run run 29514101154 (dispatched on main, env=dev) completed SUCCESS after the WIF attribute-condition apply. The AWS half (#1375, RDS engine-version pin) is the remaining item on this issue.

AWS half merged via #1375 (major-only RDS engine_version pin). The next Deploy to AWS Lambda run on main should pass the ModifyDBInstance step; will confirm on the next deploy. GCP half already confirmed fixed end-to-end. Once the AWS deploy is verified green this issue can close.

Navigation Menu

Repository navigation

deploy: AWS Lambda + GCP Cloud Run deploy workflows failing on every main commit since ~Jul 3 #1372

GCP deploy auth failure - operator runbook

How to fix (privileged operator, ~2 min)

How to verify

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

deploy: AWS Lambda + GCP Cloud Run deploy workflows failing on every main commit since ~Jul 3 #1372

Description

Activity

cristim commented on Jul 16, 2026

cristim commented on Jul 16, 2026

GCP deploy auth failure - operator runbook

How to fix (privileged operator, ~2 min)

How to verify

cristim commented on Jul 16, 2026

cristim commented on Jul 16, 2026

cristim commented on Jul 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions