Skip to content

sec(ci): GCP cleanup picks the Cloud SQL instance by substring and orphans it in state #1971

Closed

Description

@cristim

Summary

The GCP branch of cleanup-staging.yml selects the Cloud SQL instance to delete with gcloud sql instances list --filter="name:cudly-staging" | head -1. gcloud's : operator is a contains match, not equality, so any instance whose name contains cudly-staging is a candidate and head -1 picks whichever sorts first. The delete runs with || true, and the following terraform state rm google_sql_database_instance.main runs unconditionally outside the if [ -n "$INSTANCE" ] block, so a failed or wrong delete still leaves the real instance running, billing, and untracked. This is the exact over-match that scripts/select-owned-name.sh was written to remove for ECR (#1592/#1820) and RDS (#1821); the GCP path never got the guard, and the two regression sweeps (test-ecr-delete-selection.sh, test-rds-deletion-protection-scope.sh) key on aws ecr delete-repository and aws rds modify-db-instance, so they never look at gcloud sql instances delete. The step validates PROJECT carefully at line 396 precisely because a silent skip is unacceptable, but applies no equivalent check to INSTANCE.

Location

.github/workflows/cleanup-staging.yml:401 at 3c0f8ac (selection at 401-402, delete with || true at 405, unconditional terraform state rm at 419-421)

Failure scenario

The GCP project holds cudly-staging-prod-mirror and cudly-staging-5e4d3c2b. The cleanup run lists both, head -1 returns the mirror, and gcloud sql instances delete --quiet destroys it. The staging instance survives, terraform state rm drops it from state anyway, and the next destroy neither sees nor bills for it.

Evidence

INSTANCE=$(gcloud sql instances list --project="$PROJECT" \
  --filter="name:cudly-staging" --format="value(name)" 2>/dev/null | head -1)
if [ -n "$INSTANCE" ]; then
  echo "Deleting Cloud SQL instance $INSTANCE..."
  gcloud sql instances delete "$INSTANCE" --project="$PROJECT" --quiet || true
terraform state rm google_sql_database_instance.main 2>/dev/null || true

Suggested fix

Read the owned instance name from terraform output -json and pipe the gcloud sql instances list output through scripts/select-owned-name.sh, matching the ECR/RDS pattern; drop the || true on the delete and move the terraform state rm lines inside the branch that confirmed the deletion. Add a gcloud sql instances delete case to the existing selection sweep.


Found by the 2026-09-02 codebase audit, finding A14-004, reported by one reviewer and independently confirmed by a second. Full report: docs/audits/codebase-audit-2026-09-02.md.

Activity

  1. added a commit that references this issue on Sep 14, 2026
    a12f323
  2. added a commit that references this issue on Sep 14, 2026
    81f2fc3
  3. cristim commented on Sep 14, 2026

    @cristim
    MemberAuthor

    Merged in PR LeanerCloud/cloud-commitments-cli#2085 as 81f2fc3ac44ce47221682fa1c50dec26e4f24eed at 2026-09-14 20:41:09 UTC. The merged tree exactly matches the reviewed candidate.

    The actual pre-fix workflow block was tested against stubs. It selected the wrong first mirror, swallowed deletion failure, then attempted three unqualified state removals. Those old addresses do not match the actual module.database resources. The owned staging name is cudly-staging-postgres, without a random suffix.

    The final wrapper selects only the Terraform-owned name and stops on listing or deletion failure before state removal. Native macOS suites passed 62,48,53,26 checks. A separate 59-check wrapper probe and seven independent adversarial cases passed. Fresh-context Astra, full eight-file CodeRabbit, and all exact-head CI workflows passed before normal merge.

    No live cloud deletion, actual Terraform state operation, or cleanup workflow dispatch was used for verification. The manual cleanup workflow has not been executed. Post-merge workflows are being monitored. Existing peering cleanup work remains tracked by LeanerCloud/cloud-commitments-platform#330. The code fix is merged and this issue auto-closed; live cleanup behavior is not claimed as tested.

  4. cristim commented on Sep 14, 2026

    @cristim
    MemberAuthor

    Post-merge verification

    PR #2085 merged normally as 81f2fc3ac44ce47221682fa1c50dec26e4f24eed. The entire merged tree is identical to the locally verified and independently reviewed candidate 46027497a689400bcd9374cfced3f772ac5b4479.

    All seven workflows triggered on the merge passed, and all seven local watchers returned success:

    The changed cleanup path was verified natively on macOS against stubbed Terraform/cloud commands, including actual pre-fix workflow behavior and final exact-selection/failure handling. No cleanup workflow was dispatched, and no real cloud deletion or Terraform state removal was performed. Deployment smoke checks are not evidence of live cleanup behavior.

    Issue #1971 was closed by the merge. Related peering cleanup remains tracked by LeanerCloud/cloud-commitments-platform#330. No additional follow-up issue was required by this post-merge verification. Worktrees, branches and historical evidence are preserved; no cleanup was authorized.

  5. added a commit that references this issue on Sep 27, 2026
    62030d6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions