Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions .github/scripts/wait-for-ecs-cutover.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# Waits for the ECS blue/green deploy triggered by a specific app image push to
# reach its traffic cutover (CodeDeploy AllowTraffic == Succeeded), then exits 0.
#
# ECR app images use a floating tag (latest/staging) with no git SHA, so the
# only durable key linking this CI push to its ECS deploy is the image DIGEST.
# Correlation: image digest -> CodePipeline execution (AppEcrImage revision) ->
# Deploy action externalExecutionId (== CodeDeploy deployment id) -> AllowTraffic.
#
# Usage: wait-for-ecs-cutover.sh <pipeline-name> <image-digest>
# Requires: awscli v2, configured credentials with codedeploy + codepipeline read.
set -euo pipefail

PIPELINE="${1:?pipeline name required}"
DIGEST="${2:?image digest required}"

POLL_INTERVAL="${POLL_INTERVAL:-15}"
# 70 min covers a prod deploy whose Deploy stage is queued behind a prior
# deploy's ~50-min termination bake before its own traffic shift begins.
OVERALL_TIMEOUT="${OVERALL_TIMEOUT:-4200}"

deadline=$(( $(date +%s) + OVERALL_TIMEOUT ))
remaining() { echo $(( deadline - $(date +%s) )); }
log() { echo "[wait-for-ecs-cutover] $*"; }
fail_if_expired() {
if [ "$(remaining)" -le 0 ]; then
log "ERROR: timed out after ${OVERALL_TIMEOUT}s waiting for: $1"
exit 1
fi
}

log "Pipeline: $PIPELINE"
log "Target app image digest: $DIGEST"

# Phase A: find the pipeline execution whose ECR source revision matches our
# digest. --max-items bounds the fetch (the CLI otherwise auto-paginates the whole
# execution history); our push is the newest execution, so it's on the first page.
# The revisionId match is done server-side via JMESPath; grep isolates the UUID
# from any trailing pagination-token line in text output.
EXECUTION_ID=""
while [ -z "$EXECUTION_ID" ]; do
fail_if_expired "pipeline execution matching digest"
EXECUTION_ID=$(aws codepipeline list-pipeline-executions \
--pipeline-name "$PIPELINE" --max-items 30 \
--query "pipelineExecutionSummaries[?sourceRevisions[?actionName=='ECR_Source' && revisionId=='$DIGEST']].pipelineExecutionId" \
--output text 2>/dev/null | tr '\t ' '\n\n' | grep -Em1 '^[0-9a-f-]{36}$' || true)
Comment on lines +43 to +46

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Historical Deployment Satisfies Gate

When a previous deployment has the same image digest, this lookup can select that completed execution before the current tag push creates its execution. The old deployment already has a successful AllowTraffic, so the new Trigger.dev version is promoted before the current ECS cutover. Correlate the execution with this tag-promotion event, such as by requiring a start time after the retag, rather than accepting any historical digest match.

if [ -z "$EXECUTION_ID" ]; then
log "No matching pipeline execution yet; retry in ${POLL_INTERVAL}s (remaining $(remaining)s)"
sleep "$POLL_INTERVAL"
fi
done
log "Matched pipeline execution: $EXECUTION_ID"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale pipeline digest match

Medium Severity

Phase A picks any pipeline execution whose ECR source revisionId equals the digest, using the first UUID returned. A prior completed run for the same image digest can match before the new execution exists, so AllowTraffic may already be Succeeded and Trigger.dev promotion runs immediately instead of waiting for this push’s cutover.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 30e1270. Configure here.


# Phase B: resolve the CodeDeploy deployment id from the Deploy action. This may
# stay empty for a while if the Deploy stage is queued behind a prior deploy.
DEPLOYMENT_ID=""
while [ -z "$DEPLOYMENT_ID" ] || [ "$DEPLOYMENT_ID" = "None" ]; do
fail_if_expired "CodeDeploy deployment id (Deploy stage may be queued behind a prior deploy's bake)"
status=$(aws codepipeline get-pipeline-execution \
--pipeline-name "$PIPELINE" --pipeline-execution-id "$EXECUTION_ID" \
--query 'pipelineExecution.status' --output text 2>/dev/null || true)
case "$status" in
Failed|Stopped|Superseded)
log "ERROR: pipeline execution $EXECUTION_ID ended in status $status before deploy"
exit 1
;;
esac
DEPLOYMENT_ID=$(aws codepipeline list-action-executions \
--pipeline-name "$PIPELINE" \
--filter pipelineExecutionId="$EXECUTION_ID" \
--query "actionExecutionDetails[?stageName=='Deploy'].output.executionResult.externalExecutionId | [0]" \
--output text 2>/dev/null || true)
if [ -z "$DEPLOYMENT_ID" ] || [ "$DEPLOYMENT_ID" = "None" ]; then
log "Deploy stage not started yet (pipeline status: $status); retry in ${POLL_INTERVAL}s (remaining $(remaining)s)"
sleep "$POLL_INTERVAL"
fi
done
log "CodeDeploy deployment: $DEPLOYMENT_ID"

# Phase C: wait for the traffic cutover (AllowTraffic lifecycle event Succeeded).
while true; do
fail_if_expired "AllowTraffic (traffic cutover)"
dstatus=$(aws deploy get-deployment --deployment-id "$DEPLOYMENT_ID" \
--query 'deploymentInfo.status' --output text 2>/dev/null || true)
case "$dstatus" in
Failed|Stopped)
log "ERROR: CodeDeploy deployment $DEPLOYMENT_ID ended in status $dstatus; not promoting"
exit 1
;;
esac
target_id=$(aws deploy list-deployment-targets --deployment-id "$DEPLOYMENT_ID" \
--query 'targetIds[0]' --output text 2>/dev/null || true)
at_status=""
if [ -n "$target_id" ] && [ "$target_id" != "None" ]; then
at_status=$(aws deploy get-deployment-target --deployment-id "$DEPLOYMENT_ID" --target-id "$target_id" \
--query "deploymentTarget.ecsTarget.lifecycleEvents[?lifecycleEventName=='AllowTraffic'].status | [0]" \
--output text 2>/dev/null || true)
if [ "$at_status" = "Succeeded" ]; then
log "Traffic cutover complete (AllowTraffic Succeeded) for $DEPLOYMENT_ID"
Comment on lines +91 to +99

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Only First Target Gates Promotion

When a CodeDeploy deployment contains multiple ECS targets, targetIds[0] can finish AllowTraffic while another target is still pending or has failed. The script then promotes Trigger.dev while part of the application remains on the old version. Require successful AllowTraffic for every returned ECS target before exiting.

exit 0
fi
fi
log "Deployment $DEPLOYMENT_ID status=$dstatus AllowTraffic=${at_status:-pending}; wait ${POLL_INTERVAL}s (remaining $(remaining)s)"
sleep "$POLL_INTERVAL"
done
Loading
Loading