From a058f03882dccd43c862b84db0e6c501161986d4 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Tue, 19 May 2026 16:16:36 +0100 Subject: [PATCH 01/73] chore: only upload full artifacts when changes found --- .github/actions/terragrunt/README.md | 6 +++--- .github/docs/README.md | 2 +- infra/README.md | 1 + infra/scripts/handle-plan-artifact.sh | 14 ++++++++++---- 4 files changed, 15 insertions(+), 8 deletions(-) diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index fbe58fcc..22ed7683 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -38,7 +38,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - `apply` Runs `terragrunt apply -auto-approve` - `plan` - Runs `terragrunt plan -detailed-exitcode -out=terragrunt.tfplan`. The shared Terragrunt root `after_hook` then renders `terragrunt.plan.txt`, writes `terragrunt.plan.meta.json`, and uploads the per-stack plan bundle to the derived plan bucket when `TG_ENABLE_PLAN_ARTIFACTS=true` and `PLAN_ARTIFACT_RUN_ID` is set. + Runs `terragrunt plan -detailed-exitcode -out=terragrunt.tfplan`. The shared Terragrunt root `after_hook` renders `terragrunt.plan.txt`, writes `terragrunt.plan.meta.json`, always uploads the metadata, and only uploads `terragrunt.tfplan` plus `terragrunt.plan.txt` when the metadata says the stack has changes. - `apply_plan` Runs `terragrunt apply terragrunt.tfplan`. The shared Terragrunt root `before_hook` downloads the saved plan bundle into the Terragrunt working directory when `TG_ENABLE_PLAN_ARTIFACTS=true` and `PLAN_ARTIFACT_RUN_ID` is set, and fails early if the saved metadata reports mocked dependency outputs. - `destroy` @@ -52,9 +52,9 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - artifact name: `infra-plan-metadata` - file: `plan-metadata.json` - Each Terragrunt stack or module stores its own plan bundle at: - - `s3:///terragrunt_plan///terragrunt-plan-/terragrunt.tfplan` - - `s3:///terragrunt_plan///terragrunt-plan-/terragrunt.plan.txt` - `s3:///terragrunt_plan///terragrunt-plan-/terragrunt.plan.meta.json` + - `s3:///terragrunt_plan///terragrunt-plan-/terragrunt.tfplan` only when changes exist + - `s3:///terragrunt_plan///terragrunt-plan-/terragrunt.plan.txt` only when changes exist ## AWS Credentials diff --git a/.github/docs/README.md b/.github/docs/README.md index 952ffc81..9f82b5f9 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -143,7 +143,7 @@ Run these checks on every CI, workflow, or deploy-contract change. - compare expected outputs against actual `jobs..outputs.*` - verify optional inputs are intentionally omitted, not accidentally missing - the repo-local `./.github/actions/terragrunt` action supports `tg_action: plan` for producing the binary plan locally; the shared Terragrunt root `after_hook` then renders `terragrunt.plan.txt` and writes `terragrunt.plan.meta.json` -- shared Terragrunt root hooks now upload per-stack plan artifacts on `plan` and download them on `apply_plan` only when `TG_ENABLE_PLAN_ARTIFACTS=true`, using the caller-provided `PLAN_ARTIFACT_RUN_ID` plus the root-derived `plan_bucket`, so graph executors like `shared_infra.yml` do not need separate `./.github/actions/just` steps for those transfers +- shared Terragrunt root hooks now always upload per-stack `terragrunt.plan.meta.json` on saved `plan`, but only upload `terragrunt.tfplan` and `terragrunt.plan.txt` when the metadata reports real changes; `apply_plan` still downloads the normal plan bundle when one exists - both repo-local composite actions, `./.github/actions/just` and `./.github/actions/terragrunt`, now assume AWS credentials are already configured in the current job when they need AWS access. The repo pattern is to run `aws-actions/configure-aws-credentials` at the top of each AWS-using job and then call the local actions without extra auth inputs - `./.github/actions/just` installs the requested `just` version through `extractions/setup-crate@v2` in the same minimal composite-action shape as `extractions/setup-just`, rather than depending on `extractions/setup-just` itself - `./.github/actions/terragrunt` installs the requested Terragrunt version through `jdx/mise-action@v4`, while Terraform stays pinned separately through `hashicorp/setup-terraform` diff --git a/infra/README.md b/infra/README.md index 5c9264c0..1bb410ac 100644 --- a/infra/README.md +++ b/infra/README.md @@ -157,6 +157,7 @@ That `containers/lib` directory is helper code only and is not treated as a depl - `*_infra` wrappers need the inputs required to apply infra safely, such as directory-derived stack matrices and any artifact-derived bootstrap references - in `prod`, the `*_infra` wrappers read shared artifact resources from `ci` but only apply service and task stacks in `prod` - saved `plan` / `apply_plan` artifacts live in the dedicated plan bucket under `terragrunt_plan///...` +- each saved-plan stack always uploads `terragrunt.plan.meta.json`; the binary `terragrunt.tfplan` and rendered `terragrunt.plan.txt` are uploaded only when the plan contains real changes - deploy workflows: - publish Lambda versions and use Lambda CodeDeploy - optionally invoke the `migrations` Lambda when it is part of the Lambda deploy matrix diff --git a/infra/scripts/handle-plan-artifact.sh b/infra/scripts/handle-plan-artifact.sh index 4cee9200..7d5e35e9 100644 --- a/infra/scripts/handle-plan-artifact.sh +++ b/infra/scripts/handle-plan-artifact.sh @@ -68,11 +68,17 @@ case "$mode" in '{tg_directory: $tg_directory, has_changes: $has_changes, contains_mocked_outputs: $contains_mocked_outputs}' \ > "$plan_meta_path" - echo "Uploading plan artifacts for ${logical_tg_dir} to ${artifact_s3_prefix}" >&2 - aws s3 cp "$plan_path" "${artifact_s3_prefix}/terragrunt.tfplan" - aws s3 cp "$plan_text_path" "${artifact_s3_prefix}/terragrunt.plan.txt" + echo "Uploading plan metadata for ${logical_tg_dir} to ${artifact_s3_prefix}" >&2 aws s3 cp "$plan_meta_path" "${artifact_s3_prefix}/terragrunt.plan.meta.json" - echo "Uploaded plan artifacts for ${logical_tg_dir}" >&2 + + if [[ "$(jq -r '.has_changes' "$plan_meta_path")" == "true" ]]; then + aws s3 cp "$plan_path" "${artifact_s3_prefix}/terragrunt.tfplan" + aws s3 cp "$plan_text_path" "${artifact_s3_prefix}/terragrunt.plan.txt" + echo "Uploaded plan artifacts for ${logical_tg_dir}" >&2 + else + echo "Plan for ${logical_tg_dir} has no changes. Uploaded metadata only." >&2 + fi + rm -f "$plan_json_path" ;; *) From ea8efa036e1ffe9522fcb09b0363a18f09d902d5 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 10:08:02 +0100 Subject: [PATCH 02/73] docs: note on script ownerships --- REPO_INSTRUCTIONS.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/REPO_INSTRUCTIONS.md b/REPO_INSTRUCTIONS.md index aa002a83..514bc439 100644 --- a/REPO_INSTRUCTIONS.md +++ b/REPO_INSTRUCTIONS.md @@ -40,6 +40,12 @@ These instructions apply to the entire repository. - runtime behavior: `lambdas/**/README.md` and `containers/**/README.md` - before editing, read the relevant local contract docs for the files you plan to touch and follow those contracts +## Script Ownership + +- reserve `infra/scripts/**` for Terraform or Terragrunt owned helper behavior that is part of the infra runtime contract +- prefer implementing GitHub Actions or workflow-only helper logic directly in `justfile.ci` when practical +- when a workflow-only helper needs more than a small recipe body, keep its ownership in the CI/workflow layer rather than under `infra/scripts/**` + ## Context Loading Order - load context lazily and only as needed From d6102f1daa2f7d66e8e6454ca82c211c3fdc130d Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 10:34:46 +0100 Subject: [PATCH 03/73] chore: graph deps --- infra/live/dependencies/cluster.hcl | 2 +- infra/live/dependencies/database.hcl | 2 +- infra/live/dependencies/frontend.hcl | 4 ++-- infra/live/dependencies/messaging.hcl | 2 +- infra/live/dependencies/network.hcl | 2 +- infra/live/dependencies/security.hcl | 2 +- infra/live/dev/aws/network/terragrunt.hcl | 2 +- infra/live/prod/aws/network/terragrunt.hcl | 2 +- justfile | 7 +++++++ 9 files changed, 16 insertions(+), 9 deletions(-) diff --git a/infra/live/dependencies/cluster.hcl b/infra/live/dependencies/cluster.hcl index 2bbd9e09..6cefa9f4 100644 --- a/infra/live/dependencies/cluster.hcl +++ b/infra/live/dependencies/cluster.hcl @@ -6,7 +6,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } inputs = { diff --git a/infra/live/dependencies/database.hcl b/infra/live/dependencies/database.hcl index 02dd17c6..f1c4fadf 100644 --- a/infra/live/dependencies/database.hcl +++ b/infra/live/dependencies/database.hcl @@ -9,7 +9,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } inputs = { diff --git a/infra/live/dependencies/frontend.hcl b/infra/live/dependencies/frontend.hcl index 01dba858..93208f3f 100644 --- a/infra/live/dependencies/frontend.hcl +++ b/infra/live/dependencies/frontend.hcl @@ -5,7 +5,7 @@ dependency "network" { api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } dependency "cognito" { @@ -18,7 +18,7 @@ dependency "cognito" { auth_readonly_group_name = "readonly" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } inputs = { diff --git a/infra/live/dependencies/messaging.hcl b/infra/live/dependencies/messaging.hcl index 12399468..3b70891b 100644 --- a/infra/live/dependencies/messaging.hcl +++ b/infra/live/dependencies/messaging.hcl @@ -16,7 +16,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } inputs = dependency.messaging.outputs diff --git a/infra/live/dependencies/network.hcl b/infra/live/dependencies/network.hcl index de2a1de5..e00bd4ba 100644 --- a/infra/live/dependencies/network.hcl +++ b/infra/live/dependencies/network.hcl @@ -16,7 +16,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } inputs = { diff --git a/infra/live/dependencies/security.hcl b/infra/live/dependencies/security.hcl index 07ac0a9e..644040d8 100644 --- a/infra/live/dependencies/security.hcl +++ b/infra/live/dependencies/security.hcl @@ -10,5 +10,5 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/network/terragrunt.hcl b/infra/live/dev/aws/network/terragrunt.hcl index 93443bf3..d0db46e5 100644 --- a/infra/live/dev/aws/network/terragrunt.hcl +++ b/infra/live/dev/aws/network/terragrunt.hcl @@ -18,7 +18,7 @@ dependency "cognito" { auth_issuer_url = "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_mock" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } inputs = { diff --git a/infra/live/prod/aws/network/terragrunt.hcl b/infra/live/prod/aws/network/terragrunt.hcl index 93443bf3..d0db46e5 100644 --- a/infra/live/prod/aws/network/terragrunt.hcl +++ b/infra/live/prod/aws/network/terragrunt.hcl @@ -18,7 +18,7 @@ dependency "cognito" { auth_issuer_url = "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_mock" } - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show"] + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } inputs = { diff --git a/justfile b/justfile index f29abaa9..67d4cfdd 100644 --- a/justfile +++ b/justfile @@ -152,6 +152,13 @@ tg-all op: terragrunt run-all {{op}} +# Print the Terragrunt dependency graph for one environment/provider root. +tg-graph env provider='aws': + #!/usr/bin/env bash + cd {{justfile_directory()}} + terragrunt graph-dependencies --terragrunt-working-dir infra/live/{{env}}/{{provider}} + + # Open an ECS Exec shell in the worker debug container. worker-debug-shell env: #!/usr/bin/env bash From 3c837b3e7df4aebbd270fd01e857256bbf33739a Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 10:57:21 +0100 Subject: [PATCH 04/73] chore: re add duped deps --- README.md | 2 +- infra/live/dependencies/cluster.hcl | 15 --- infra/live/dependencies/database.hcl | 21 ----- infra/live/dependencies/frontend.hcl | 30 ------ infra/live/dependencies/messaging.hcl | 22 ----- infra/live/dependencies/network.hcl | 35 ------- infra/live/dependencies/security.hcl | 14 --- infra/live/dev/aws/database/terragrunt.hcl | 15 ++- infra/live/dev/aws/frontend/terragrunt.hcl | 31 ++++++- infra/live/dev/aws/lambda_api/terragrunt.hcl | 60 +++++++++++- .../live/dev/aws/lambda_worker/terragrunt.hcl | 23 ++++- infra/live/dev/aws/migrations/terragrunt.hcl | 37 +++++++- infra/live/dev/aws/network/terragrunt.hcl | 15 ++- .../dev/aws/rds_reader_tagger/terragrunt.hcl | 22 ++++- infra/live/dev/aws/service_api/terragrunt.hcl | 68 ++++++++++++-- .../dev/aws/service_worker/terragrunt.hcl | 92 +++++++++++++++++-- infra/live/dev/aws/task_worker/terragrunt.hcl | 47 +++++++++- infra/live/prod/aws/database/terragrunt.hcl | 15 ++- infra/live/prod/aws/frontend/terragrunt.hcl | 31 ++++++- infra/live/prod/aws/lambda_api/terragrunt.hcl | 60 +++++++++++- .../prod/aws/lambda_worker/terragrunt.hcl | 23 ++++- infra/live/prod/aws/migrations/terragrunt.hcl | 37 +++++++- infra/live/prod/aws/network/terragrunt.hcl | 15 ++- .../prod/aws/rds_reader_tagger/terragrunt.hcl | 22 ++++- .../live/prod/aws/service_api/terragrunt.hcl | 68 ++++++++++++-- .../prod/aws/service_worker/terragrunt.hcl | 92 +++++++++++++++++-- .../live/prod/aws/task_worker/terragrunt.hcl | 47 +++++++++- 27 files changed, 735 insertions(+), 224 deletions(-) delete mode 100644 infra/live/dependencies/cluster.hcl delete mode 100644 infra/live/dependencies/database.hcl delete mode 100644 infra/live/dependencies/frontend.hcl delete mode 100644 infra/live/dependencies/messaging.hcl delete mode 100644 infra/live/dependencies/network.hcl delete mode 100644 infra/live/dependencies/security.hcl diff --git a/README.md b/README.md index 5f08f9c1..655da606 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ Lambda + ECS with CodeDeploy rollouts, plus provisioned concurrency controls for ## Bootstrap-Friendly Plans -For cross-stack contracts that often block CI plans before upstream stacks exist, this repo prefers Terragrunt `dependency` wiring in the live stack plus `mock_outputs` for non-mutating commands such as `plan` and `validate`. The Terraform modules should consume explicit inputs rather than reaching back into sibling stack state directly when the contract needs bootstrap-friendly plan behavior. +For cross-stack contracts that often block CI plans before upstream stacks exist, this repo prefers Terragrunt `dependency` wiring in the live stack plus `mock_outputs` for non-mutating commands such as `plan` and `validate`. Keep those `dependency` blocks in the consuming stack instead of hiding them behind `read_terragrunt_config(...)` helper indirection, because Terragrunt graph commands only emit direct stack edges. The Terraform modules should consume explicit inputs rather than reaching back into sibling stack state directly when the contract needs bootstrap-friendly plan behavior. Use [CONTRIBUTING.md](CONTRIBUTING.md) for expectations when changing the repo itself. diff --git a/infra/live/dependencies/cluster.hcl b/infra/live/dependencies/cluster.hcl deleted file mode 100644 index 6cefa9f4..00000000 --- a/infra/live/dependencies/cluster.hcl +++ /dev/null @@ -1,15 +0,0 @@ -dependency "cluster" { - config_path = "${get_original_terragrunt_dir()}/../cluster" - - mock_outputs = { - cluster_id = "mock-cluster-id" - cluster_name = "mock-cluster" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - -inputs = { - cluster_id = dependency.cluster.outputs.cluster_id - cluster_name = dependency.cluster.outputs.cluster_name -} diff --git a/infra/live/dependencies/database.hcl b/infra/live/dependencies/database.hcl deleted file mode 100644 index f1c4fadf..00000000 --- a/infra/live/dependencies/database.hcl +++ /dev/null @@ -1,21 +0,0 @@ -dependency "database" { - config_path = "${get_original_terragrunt_dir()}/../database" - - mock_outputs = { - database_credentials_secret_arn = "arn:aws:secretsmanager:eu-west-2:111111111111:secret:mock-database-credentials" - database_readwrite_endpoint = "mock-database.cluster-abcdefghijkl.eu-west-2.rds.amazonaws.com" - database_name = "app" - database_port = 5432 - database_cluster_identifier = "mock-database-cluster" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - -inputs = { - database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn - database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint - database_name = dependency.database.outputs.database_name - database_port = dependency.database.outputs.database_port - database_cluster_identifier = dependency.database.outputs.database_cluster_identifier -} diff --git a/infra/live/dependencies/frontend.hcl b/infra/live/dependencies/frontend.hcl deleted file mode 100644 index 93208f3f..00000000 --- a/infra/live/dependencies/frontend.hcl +++ /dev/null @@ -1,30 +0,0 @@ -dependency "network" { - config_path = "${get_original_terragrunt_dir()}/../network" - - mock_outputs = { - api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - -dependency "cognito" { - config_path = "${get_original_terragrunt_dir()}/../cognito" - - mock_outputs = { - auth_user_pool_id = "eu-west-2_mock" - auth_user_pool_client_id = "mock-user-pool-client-id" - auth_hosted_ui_url = "https://mock-domain.auth.eu-west-2.amazoncognito.com" - auth_readonly_group_name = "readonly" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - -inputs = { - api_invoke_url = dependency.network.outputs.api_invoke_url - auth_user_pool_id = dependency.cognito.outputs.auth_user_pool_id - auth_user_pool_client_id = dependency.cognito.outputs.auth_user_pool_client_id - auth_hosted_ui_url = dependency.cognito.outputs.auth_hosted_ui_url - auth_readonly_group_name = dependency.cognito.outputs.auth_readonly_group_name -} diff --git a/infra/live/dependencies/messaging.hcl b/infra/live/dependencies/messaging.hcl deleted file mode 100644 index 3b70891b..00000000 --- a/infra/live/dependencies/messaging.hcl +++ /dev/null @@ -1,22 +0,0 @@ -dependency "messaging" { - config_path = "${get_original_terragrunt_dir()}/../messaging" - - mock_outputs = { - worker_topic_name = "mock-worker-events" - worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" - worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" - lambda_worker_queue_name = "mock-lambda-worker-queue" - lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" - lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" - lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" - lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" - lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" - ecs_worker_queue_name = "mock-ecs-worker-queue" - ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" - ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - -inputs = dependency.messaging.outputs diff --git a/infra/live/dependencies/network.hcl b/infra/live/dependencies/network.hcl deleted file mode 100644 index e00bd4ba..00000000 --- a/infra/live/dependencies/network.hcl +++ /dev/null @@ -1,35 +0,0 @@ -dependency "network" { - config_path = "${get_original_terragrunt_dir()}/../network" - - mock_outputs = { - default_target_group_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:targetgroup/mock-default/1234567890abcdef" - load_balancer_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:loadbalancer/app/mock-internal/1234567890abcdef" - default_http_listener_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:listener/app/mock-internal/1234567890abcdef/abcdef1234567890" - load_balancer_arn_suffix = "app/mock-internal/1234567890abcdef" - target_group_arn_suffix = "targetgroup/mock-default/1234567890abcdef" - internal_invoke_url = "http://mock-internal-123456.eu-west-2.elb.amazonaws.com" - api_id = "mockapi123" - api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" - api_execution_arn = "arn:aws:execute-api:eu-west-2:111111111111:mockapi123" - api_stage_name = "$default" - vpc_link_id = "vpclink-mock123" - http_api_authorizer_id = "auth-mock123" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - -inputs = { - network_default_target_group_arn = dependency.network.outputs.default_target_group_arn - network_load_balancer_arn = dependency.network.outputs.load_balancer_arn - network_default_http_listener_arn = dependency.network.outputs.default_http_listener_arn - network_load_balancer_arn_suffix = dependency.network.outputs.load_balancer_arn_suffix - network_target_group_arn_suffix = dependency.network.outputs.target_group_arn_suffix - network_internal_invoke_url = dependency.network.outputs.internal_invoke_url - network_api_id = dependency.network.outputs.api_id - network_api_invoke_url = dependency.network.outputs.api_invoke_url - network_api_execution_arn = dependency.network.outputs.api_execution_arn - network_api_stage_name = dependency.network.outputs.api_stage_name - network_vpc_link_id = dependency.network.outputs.vpc_link_id - network_http_api_authorizer_id = dependency.network.outputs.http_api_authorizer_id -} diff --git a/infra/live/dependencies/security.hcl b/infra/live/dependencies/security.hcl deleted file mode 100644 index 644040d8..00000000 --- a/infra/live/dependencies/security.hcl +++ /dev/null @@ -1,14 +0,0 @@ -dependency "security" { - config_path = "${get_original_terragrunt_dir()}/../security" - - mock_outputs = { - load_balancer_sg = "sg-00000000000000001" - api_vpc_link_sg = "sg-00000000000000002" - vpc_endpoint_sg = "sg-00000000000000003" - ecs_sg = "sg-00000000000000004" - runtime_sg = "sg-00000000000000005" - postgres_sg = "sg-00000000000000006" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} diff --git a/infra/live/dev/aws/database/terragrunt.hcl b/infra/live/dev/aws/database/terragrunt.hcl index c6ee9ddc..d11b514c 100644 --- a/infra/live/dev/aws/database/terragrunt.hcl +++ b/infra/live/dev/aws/database/terragrunt.hcl @@ -2,8 +2,19 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { diff --git a/infra/live/dev/aws/frontend/terragrunt.hcl b/infra/live/dev/aws/frontend/terragrunt.hcl index fa22a838..c7f13fce 100644 --- a/infra/live/dev/aws/frontend/terragrunt.hcl +++ b/infra/live/dev/aws/frontend/terragrunt.hcl @@ -2,12 +2,37 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - frontend = read_terragrunt_config(find_in_parent_folders("dependencies/frontend.hcl")) +dependency "network" { + config_path = "${get_original_terragrunt_dir()}/../network" + + mock_outputs = { + api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "cognito" { + config_path = "${get_original_terragrunt_dir()}/../cognito" + + mock_outputs = { + auth_user_pool_id = "eu-west-2_mock" + auth_user_pool_client_id = "mock-user-pool-client-id" + auth_hosted_ui_url = "https://mock-domain.auth.eu-west-2.amazoncognito.com" + auth_readonly_group_name = "readonly" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { source = "../../../../modules//aws//frontend" } -inputs = local.frontend.inputs +inputs = { + api_invoke_url = dependency.network.outputs.api_invoke_url + auth_user_pool_id = dependency.cognito.outputs.auth_user_pool_id + auth_user_pool_client_id = dependency.cognito.outputs.auth_user_pool_client_id + auth_hosted_ui_url = dependency.cognito.outputs.auth_hosted_ui_url + auth_readonly_group_name = dependency.cognito.outputs.auth_readonly_group_name +} diff --git a/infra/live/dev/aws/lambda_api/terragrunt.hcl b/infra/live/dev/aws/lambda_api/terragrunt.hcl index 957503d9..5ff0ef12 100644 --- a/infra/live/dev/aws/lambda_api/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_api/terragrunt.hcl @@ -2,9 +2,46 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - network = read_terragrunt_config(find_in_parent_folders("dependencies/network.hcl")) - messaging = read_terragrunt_config(find_in_parent_folders("dependencies/messaging.hcl")) +dependency "network" { + config_path = "${get_original_terragrunt_dir()}/../network" + + mock_outputs = { + default_target_group_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:targetgroup/mock-default/1234567890abcdef" + load_balancer_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:loadbalancer/app/mock-internal/1234567890abcdef" + default_http_listener_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:listener/app/mock-internal/1234567890abcdef/abcdef1234567890" + load_balancer_arn_suffix = "app/mock-internal/1234567890abcdef" + target_group_arn_suffix = "targetgroup/mock-default/1234567890abcdef" + internal_invoke_url = "http://mock-internal-123456.eu-west-2.elb.amazonaws.com" + api_id = "mockapi123" + api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" + api_execution_arn = "arn:aws:execute-api:eu-west-2:111111111111:mockapi123" + api_stage_name = "$default" + vpc_link_id = "vpclink-mock123" + http_api_authorizer_id = "auth-mock123" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "messaging" { + config_path = "${get_original_terragrunt_dir()}/../messaging" + + mock_outputs = { + worker_topic_name = "mock-worker-events" + worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" + worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" + lambda_worker_queue_name = "mock-lambda-worker-queue" + lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" + lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" + lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" + lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" + lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" + ecs_worker_queue_name = "mock-ecs-worker-queue" + ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" + ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -12,8 +49,21 @@ terraform { } inputs = merge( - local.network.inputs, - local.messaging.inputs, + { + network_default_target_group_arn = dependency.network.outputs.default_target_group_arn + network_load_balancer_arn = dependency.network.outputs.load_balancer_arn + network_default_http_listener_arn = dependency.network.outputs.default_http_listener_arn + network_load_balancer_arn_suffix = dependency.network.outputs.load_balancer_arn_suffix + network_target_group_arn_suffix = dependency.network.outputs.target_group_arn_suffix + network_internal_invoke_url = dependency.network.outputs.internal_invoke_url + network_api_id = dependency.network.outputs.api_id + network_api_invoke_url = dependency.network.outputs.api_invoke_url + network_api_execution_arn = dependency.network.outputs.api_execution_arn + network_api_stage_name = dependency.network.outputs.api_stage_name + network_vpc_link_id = dependency.network.outputs.vpc_link_id + network_http_api_authorizer_id = dependency.network.outputs.http_api_authorizer_id + }, + dependency.messaging.outputs, { api_5xx_alarm_threshold = 20.0 api_5xx_alarm_evaluation_periods = 1 diff --git a/infra/live/dev/aws/lambda_worker/terragrunt.hcl b/infra/live/dev/aws/lambda_worker/terragrunt.hcl index bdb94fdd..5d60f8d2 100644 --- a/infra/live/dev/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_worker/terragrunt.hcl @@ -2,8 +2,25 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - messaging = read_terragrunt_config(find_in_parent_folders("dependencies/messaging.hcl")) +dependency "messaging" { + config_path = "${get_original_terragrunt_dir()}/../messaging" + + mock_outputs = { + worker_topic_name = "mock-worker-events" + worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" + worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" + lambda_worker_queue_name = "mock-lambda-worker-queue" + lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" + lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" + lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" + lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" + lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" + ecs_worker_queue_name = "mock-ecs-worker-queue" + ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" + ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -11,7 +28,7 @@ terraform { } inputs = merge( - local.messaging.inputs, + dependency.messaging.outputs, { sqs_dlq_alarm_threshold = 1 # fail when any messages are in the DLQ (quick fail for testing) sqs_dlq_alarm_evaluation_periods = 1 diff --git a/infra/live/dev/aws/migrations/terragrunt.hcl b/infra/live/dev/aws/migrations/terragrunt.hcl index edc1b7f2..653e7061 100644 --- a/infra/live/dev/aws/migrations/terragrunt.hcl +++ b/infra/live/dev/aws/migrations/terragrunt.hcl @@ -2,12 +2,33 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -locals { - database = read_terragrunt_config(find_in_parent_folders("dependencies/database.hcl")) +dependency "database" { + config_path = "${get_original_terragrunt_dir()}/../database" + + mock_outputs = { + database_credentials_secret_arn = "arn:aws:secretsmanager:eu-west-2:111111111111:secret:mock-database-credentials" + database_readwrite_endpoint = "mock-database.cluster-abcdefghijkl.eu-west-2.rds.amazonaws.com" + database_name = "app" + database_port = 5432 + database_cluster_identifier = "mock-database-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -18,5 +39,11 @@ inputs = merge( { runtime_security_group_id = dependency.security.outputs.runtime_sg }, - local.database.inputs, + { + database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn + database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint + database_name = dependency.database.outputs.database_name + database_port = dependency.database.outputs.database_port + database_cluster_identifier = dependency.database.outputs.database_cluster_identifier + }, ) diff --git a/infra/live/dev/aws/network/terragrunt.hcl b/infra/live/dev/aws/network/terragrunt.hcl index d0db46e5..a3b77887 100644 --- a/infra/live/dev/aws/network/terragrunt.hcl +++ b/infra/live/dev/aws/network/terragrunt.hcl @@ -2,8 +2,19 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { diff --git a/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl index bc618a77..ec3ff4d8 100644 --- a/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl @@ -2,12 +2,28 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - database = read_terragrunt_config(find_in_parent_folders("dependencies/database.hcl")) +dependency "database" { + config_path = "${get_original_terragrunt_dir()}/../database" + + mock_outputs = { + database_credentials_secret_arn = "arn:aws:secretsmanager:eu-west-2:111111111111:secret:mock-database-credentials" + database_readwrite_endpoint = "mock-database.cluster-abcdefghijkl.eu-west-2.rds.amazonaws.com" + database_name = "app" + database_port = 5432 + database_cluster_identifier = "mock-database-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { source = "../../../../modules//aws//rds_reader_tagger" } -inputs = local.database.inputs +inputs = { + database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn + database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint + database_name = dependency.database.outputs.database_name + database_port = dependency.database.outputs.database_port + database_cluster_identifier = dependency.database.outputs.database_cluster_identifier +} diff --git a/infra/live/dev/aws/service_api/terragrunt.hcl b/infra/live/dev/aws/service_api/terragrunt.hcl index 501e7a44..4c136329 100644 --- a/infra/live/dev/aws/service_api/terragrunt.hcl +++ b/infra/live/dev/aws/service_api/terragrunt.hcl @@ -2,13 +2,51 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "cluster" { + config_path = "${get_original_terragrunt_dir()}/../cluster" + + mock_outputs = { + cluster_id = "mock-cluster-id" + cluster_name = "mock-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -locals { - cluster = read_terragrunt_config(find_in_parent_folders("dependencies/cluster.hcl")) - network = read_terragrunt_config(find_in_parent_folders("dependencies/network.hcl")) +dependency "network" { + config_path = "${get_original_terragrunt_dir()}/../network" + + mock_outputs = { + default_target_group_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:targetgroup/mock-default/1234567890abcdef" + load_balancer_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:loadbalancer/app/mock-internal/1234567890abcdef" + default_http_listener_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:listener/app/mock-internal/1234567890abcdef/abcdef1234567890" + load_balancer_arn_suffix = "app/mock-internal/1234567890abcdef" + target_group_arn_suffix = "targetgroup/mock-default/1234567890abcdef" + internal_invoke_url = "http://mock-internal-123456.eu-west-2.elb.amazonaws.com" + api_id = "mockapi123" + api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" + api_execution_arn = "arn:aws:execute-api:eu-west-2:111111111111:mockapi123" + api_stage_name = "$default" + vpc_link_id = "vpclink-mock123" + http_api_authorizer_id = "auth-mock123" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -19,6 +57,22 @@ inputs = merge( { ecs_security_group_id = dependency.security.outputs.ecs_sg }, - local.cluster.inputs, - local.network.inputs, + { + cluster_id = dependency.cluster.outputs.cluster_id + cluster_name = dependency.cluster.outputs.cluster_name + }, + { + network_default_target_group_arn = dependency.network.outputs.default_target_group_arn + network_load_balancer_arn = dependency.network.outputs.load_balancer_arn + network_default_http_listener_arn = dependency.network.outputs.default_http_listener_arn + network_load_balancer_arn_suffix = dependency.network.outputs.load_balancer_arn_suffix + network_target_group_arn_suffix = dependency.network.outputs.target_group_arn_suffix + network_internal_invoke_url = dependency.network.outputs.internal_invoke_url + network_api_id = dependency.network.outputs.api_id + network_api_invoke_url = dependency.network.outputs.api_invoke_url + network_api_execution_arn = dependency.network.outputs.api_execution_arn + network_api_stage_name = dependency.network.outputs.api_stage_name + network_vpc_link_id = dependency.network.outputs.vpc_link_id + network_http_api_authorizer_id = dependency.network.outputs.http_api_authorizer_id + }, ) diff --git a/infra/live/dev/aws/service_worker/terragrunt.hcl b/infra/live/dev/aws/service_worker/terragrunt.hcl index 7b23580e..6e0a23c5 100644 --- a/infra/live/dev/aws/service_worker/terragrunt.hcl +++ b/infra/live/dev/aws/service_worker/terragrunt.hcl @@ -2,14 +2,72 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "messaging" { + config_path = "${get_original_terragrunt_dir()}/../messaging" + + mock_outputs = { + worker_topic_name = "mock-worker-events" + worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" + worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" + lambda_worker_queue_name = "mock-lambda-worker-queue" + lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" + lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" + lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" + lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" + lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" + ecs_worker_queue_name = "mock-ecs-worker-queue" + ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" + ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "cluster" { + config_path = "${get_original_terragrunt_dir()}/../cluster" + + mock_outputs = { + cluster_id = "mock-cluster-id" + cluster_name = "mock-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -locals { - messaging = read_terragrunt_config(find_in_parent_folders("dependencies/messaging.hcl")) - cluster = read_terragrunt_config(find_in_parent_folders("dependencies/cluster.hcl")) - network = read_terragrunt_config(find_in_parent_folders("dependencies/network.hcl")) +dependency "network" { + config_path = "${get_original_terragrunt_dir()}/../network" + + mock_outputs = { + default_target_group_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:targetgroup/mock-default/1234567890abcdef" + load_balancer_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:loadbalancer/app/mock-internal/1234567890abcdef" + default_http_listener_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:listener/app/mock-internal/1234567890abcdef/abcdef1234567890" + load_balancer_arn_suffix = "app/mock-internal/1234567890abcdef" + target_group_arn_suffix = "targetgroup/mock-default/1234567890abcdef" + internal_invoke_url = "http://mock-internal-123456.eu-west-2.elb.amazonaws.com" + api_id = "mockapi123" + api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" + api_execution_arn = "arn:aws:execute-api:eu-west-2:111111111111:mockapi123" + api_stage_name = "$default" + vpc_link_id = "vpclink-mock123" + http_api_authorizer_id = "auth-mock123" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -20,7 +78,23 @@ inputs = merge( { ecs_security_group_id = dependency.security.outputs.ecs_sg }, - local.messaging.inputs, - local.cluster.inputs, - local.network.inputs, + dependency.messaging.outputs, + { + cluster_id = dependency.cluster.outputs.cluster_id + cluster_name = dependency.cluster.outputs.cluster_name + }, + { + network_default_target_group_arn = dependency.network.outputs.default_target_group_arn + network_load_balancer_arn = dependency.network.outputs.load_balancer_arn + network_default_http_listener_arn = dependency.network.outputs.default_http_listener_arn + network_load_balancer_arn_suffix = dependency.network.outputs.load_balancer_arn_suffix + network_target_group_arn_suffix = dependency.network.outputs.target_group_arn_suffix + network_internal_invoke_url = dependency.network.outputs.internal_invoke_url + network_api_id = dependency.network.outputs.api_id + network_api_invoke_url = dependency.network.outputs.api_invoke_url + network_api_execution_arn = dependency.network.outputs.api_execution_arn + network_api_stage_name = dependency.network.outputs.api_stage_name + network_vpc_link_id = dependency.network.outputs.vpc_link_id + network_http_api_authorizer_id = dependency.network.outputs.http_api_authorizer_id + }, ) diff --git a/infra/live/dev/aws/task_worker/terragrunt.hcl b/infra/live/dev/aws/task_worker/terragrunt.hcl index b28cbca9..c315b177 100644 --- a/infra/live/dev/aws/task_worker/terragrunt.hcl +++ b/infra/live/dev/aws/task_worker/terragrunt.hcl @@ -2,13 +2,52 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - messaging = read_terragrunt_config(find_in_parent_folders("dependencies/messaging.hcl")) - database = read_terragrunt_config(find_in_parent_folders("dependencies/database.hcl")) +dependency "messaging" { + config_path = "${get_original_terragrunt_dir()}/../messaging" + + mock_outputs = { + worker_topic_name = "mock-worker-events" + worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" + worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" + lambda_worker_queue_name = "mock-lambda-worker-queue" + lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" + lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" + lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" + lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" + lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" + ecs_worker_queue_name = "mock-ecs-worker-queue" + ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" + ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "database" { + config_path = "${get_original_terragrunt_dir()}/../database" + + mock_outputs = { + database_credentials_secret_arn = "arn:aws:secretsmanager:eu-west-2:111111111111:secret:mock-database-credentials" + database_readwrite_endpoint = "mock-database.cluster-abcdefghijkl.eu-west-2.rds.amazonaws.com" + database_name = "app" + database_port = 5432 + database_cluster_identifier = "mock-database-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { source = "../../../../modules//aws//task_worker" } -inputs = merge(local.messaging.inputs, local.database.inputs) +inputs = merge( + dependency.messaging.outputs, + { + database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn + database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint + database_name = dependency.database.outputs.database_name + database_port = dependency.database.outputs.database_port + database_cluster_identifier = dependency.database.outputs.database_cluster_identifier + }, +) diff --git a/infra/live/prod/aws/database/terragrunt.hcl b/infra/live/prod/aws/database/terragrunt.hcl index 5901ba70..752cffae 100644 --- a/infra/live/prod/aws/database/terragrunt.hcl +++ b/infra/live/prod/aws/database/terragrunt.hcl @@ -2,8 +2,19 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { diff --git a/infra/live/prod/aws/frontend/terragrunt.hcl b/infra/live/prod/aws/frontend/terragrunt.hcl index fa22a838..c7f13fce 100644 --- a/infra/live/prod/aws/frontend/terragrunt.hcl +++ b/infra/live/prod/aws/frontend/terragrunt.hcl @@ -2,12 +2,37 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - frontend = read_terragrunt_config(find_in_parent_folders("dependencies/frontend.hcl")) +dependency "network" { + config_path = "${get_original_terragrunt_dir()}/../network" + + mock_outputs = { + api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "cognito" { + config_path = "${get_original_terragrunt_dir()}/../cognito" + + mock_outputs = { + auth_user_pool_id = "eu-west-2_mock" + auth_user_pool_client_id = "mock-user-pool-client-id" + auth_hosted_ui_url = "https://mock-domain.auth.eu-west-2.amazoncognito.com" + auth_readonly_group_name = "readonly" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { source = "../../../../modules//aws//frontend" } -inputs = local.frontend.inputs +inputs = { + api_invoke_url = dependency.network.outputs.api_invoke_url + auth_user_pool_id = dependency.cognito.outputs.auth_user_pool_id + auth_user_pool_client_id = dependency.cognito.outputs.auth_user_pool_client_id + auth_hosted_ui_url = dependency.cognito.outputs.auth_hosted_ui_url + auth_readonly_group_name = dependency.cognito.outputs.auth_readonly_group_name +} diff --git a/infra/live/prod/aws/lambda_api/terragrunt.hcl b/infra/live/prod/aws/lambda_api/terragrunt.hcl index 8e29bf96..60cf23d2 100644 --- a/infra/live/prod/aws/lambda_api/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_api/terragrunt.hcl @@ -2,9 +2,46 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - network = read_terragrunt_config(find_in_parent_folders("dependencies/network.hcl")) - messaging = read_terragrunt_config(find_in_parent_folders("dependencies/messaging.hcl")) +dependency "network" { + config_path = "${get_original_terragrunt_dir()}/../network" + + mock_outputs = { + default_target_group_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:targetgroup/mock-default/1234567890abcdef" + load_balancer_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:loadbalancer/app/mock-internal/1234567890abcdef" + default_http_listener_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:listener/app/mock-internal/1234567890abcdef/abcdef1234567890" + load_balancer_arn_suffix = "app/mock-internal/1234567890abcdef" + target_group_arn_suffix = "targetgroup/mock-default/1234567890abcdef" + internal_invoke_url = "http://mock-internal-123456.eu-west-2.elb.amazonaws.com" + api_id = "mockapi123" + api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" + api_execution_arn = "arn:aws:execute-api:eu-west-2:111111111111:mockapi123" + api_stage_name = "$default" + vpc_link_id = "vpclink-mock123" + http_api_authorizer_id = "auth-mock123" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "messaging" { + config_path = "${get_original_terragrunt_dir()}/../messaging" + + mock_outputs = { + worker_topic_name = "mock-worker-events" + worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" + worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" + lambda_worker_queue_name = "mock-lambda-worker-queue" + lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" + lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" + lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" + lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" + lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" + ecs_worker_queue_name = "mock-ecs-worker-queue" + ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" + ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -12,8 +49,21 @@ terraform { } inputs = merge( - local.network.inputs, - local.messaging.inputs, + { + network_default_target_group_arn = dependency.network.outputs.default_target_group_arn + network_load_balancer_arn = dependency.network.outputs.load_balancer_arn + network_default_http_listener_arn = dependency.network.outputs.default_http_listener_arn + network_load_balancer_arn_suffix = dependency.network.outputs.load_balancer_arn_suffix + network_target_group_arn_suffix = dependency.network.outputs.target_group_arn_suffix + network_internal_invoke_url = dependency.network.outputs.internal_invoke_url + network_api_id = dependency.network.outputs.api_id + network_api_invoke_url = dependency.network.outputs.api_invoke_url + network_api_execution_arn = dependency.network.outputs.api_execution_arn + network_api_stage_name = dependency.network.outputs.api_stage_name + network_vpc_link_id = dependency.network.outputs.vpc_link_id + network_http_api_authorizer_id = dependency.network.outputs.http_api_authorizer_id + }, + dependency.messaging.outputs, { api_5xx_alarm_threshold = 5.0 api_5xx_alarm_evaluation_periods = 3 diff --git a/infra/live/prod/aws/lambda_worker/terragrunt.hcl b/infra/live/prod/aws/lambda_worker/terragrunt.hcl index 294342f8..c5981ca3 100644 --- a/infra/live/prod/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_worker/terragrunt.hcl @@ -2,8 +2,25 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - messaging = read_terragrunt_config(find_in_parent_folders("dependencies/messaging.hcl")) +dependency "messaging" { + config_path = "${get_original_terragrunt_dir()}/../messaging" + + mock_outputs = { + worker_topic_name = "mock-worker-events" + worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" + worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" + lambda_worker_queue_name = "mock-lambda-worker-queue" + lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" + lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" + lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" + lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" + lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" + ecs_worker_queue_name = "mock-ecs-worker-queue" + ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" + ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -11,7 +28,7 @@ terraform { } inputs = merge( - local.messaging.inputs, + dependency.messaging.outputs, { sqs_dlq_alarm_threshold = 5 # fail when there are 5 messages in the DLQ sqs_dlq_alarm_evaluation_periods = 3 diff --git a/infra/live/prod/aws/migrations/terragrunt.hcl b/infra/live/prod/aws/migrations/terragrunt.hcl index edc1b7f2..653e7061 100644 --- a/infra/live/prod/aws/migrations/terragrunt.hcl +++ b/infra/live/prod/aws/migrations/terragrunt.hcl @@ -2,12 +2,33 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -locals { - database = read_terragrunt_config(find_in_parent_folders("dependencies/database.hcl")) +dependency "database" { + config_path = "${get_original_terragrunt_dir()}/../database" + + mock_outputs = { + database_credentials_secret_arn = "arn:aws:secretsmanager:eu-west-2:111111111111:secret:mock-database-credentials" + database_readwrite_endpoint = "mock-database.cluster-abcdefghijkl.eu-west-2.rds.amazonaws.com" + database_name = "app" + database_port = 5432 + database_cluster_identifier = "mock-database-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -18,5 +39,11 @@ inputs = merge( { runtime_security_group_id = dependency.security.outputs.runtime_sg }, - local.database.inputs, + { + database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn + database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint + database_name = dependency.database.outputs.database_name + database_port = dependency.database.outputs.database_port + database_cluster_identifier = dependency.database.outputs.database_cluster_identifier + }, ) diff --git a/infra/live/prod/aws/network/terragrunt.hcl b/infra/live/prod/aws/network/terragrunt.hcl index d0db46e5..a3b77887 100644 --- a/infra/live/prod/aws/network/terragrunt.hcl +++ b/infra/live/prod/aws/network/terragrunt.hcl @@ -2,8 +2,19 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { diff --git a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl index bc618a77..ec3ff4d8 100644 --- a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl @@ -2,12 +2,28 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - database = read_terragrunt_config(find_in_parent_folders("dependencies/database.hcl")) +dependency "database" { + config_path = "${get_original_terragrunt_dir()}/../database" + + mock_outputs = { + database_credentials_secret_arn = "arn:aws:secretsmanager:eu-west-2:111111111111:secret:mock-database-credentials" + database_readwrite_endpoint = "mock-database.cluster-abcdefghijkl.eu-west-2.rds.amazonaws.com" + database_name = "app" + database_port = 5432 + database_cluster_identifier = "mock-database-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { source = "../../../../modules//aws//rds_reader_tagger" } -inputs = local.database.inputs +inputs = { + database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn + database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint + database_name = dependency.database.outputs.database_name + database_port = dependency.database.outputs.database_port + database_cluster_identifier = dependency.database.outputs.database_cluster_identifier +} diff --git a/infra/live/prod/aws/service_api/terragrunt.hcl b/infra/live/prod/aws/service_api/terragrunt.hcl index 501e7a44..4c136329 100644 --- a/infra/live/prod/aws/service_api/terragrunt.hcl +++ b/infra/live/prod/aws/service_api/terragrunt.hcl @@ -2,13 +2,51 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "cluster" { + config_path = "${get_original_terragrunt_dir()}/../cluster" + + mock_outputs = { + cluster_id = "mock-cluster-id" + cluster_name = "mock-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -locals { - cluster = read_terragrunt_config(find_in_parent_folders("dependencies/cluster.hcl")) - network = read_terragrunt_config(find_in_parent_folders("dependencies/network.hcl")) +dependency "network" { + config_path = "${get_original_terragrunt_dir()}/../network" + + mock_outputs = { + default_target_group_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:targetgroup/mock-default/1234567890abcdef" + load_balancer_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:loadbalancer/app/mock-internal/1234567890abcdef" + default_http_listener_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:listener/app/mock-internal/1234567890abcdef/abcdef1234567890" + load_balancer_arn_suffix = "app/mock-internal/1234567890abcdef" + target_group_arn_suffix = "targetgroup/mock-default/1234567890abcdef" + internal_invoke_url = "http://mock-internal-123456.eu-west-2.elb.amazonaws.com" + api_id = "mockapi123" + api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" + api_execution_arn = "arn:aws:execute-api:eu-west-2:111111111111:mockapi123" + api_stage_name = "$default" + vpc_link_id = "vpclink-mock123" + http_api_authorizer_id = "auth-mock123" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -19,6 +57,22 @@ inputs = merge( { ecs_security_group_id = dependency.security.outputs.ecs_sg }, - local.cluster.inputs, - local.network.inputs, + { + cluster_id = dependency.cluster.outputs.cluster_id + cluster_name = dependency.cluster.outputs.cluster_name + }, + { + network_default_target_group_arn = dependency.network.outputs.default_target_group_arn + network_load_balancer_arn = dependency.network.outputs.load_balancer_arn + network_default_http_listener_arn = dependency.network.outputs.default_http_listener_arn + network_load_balancer_arn_suffix = dependency.network.outputs.load_balancer_arn_suffix + network_target_group_arn_suffix = dependency.network.outputs.target_group_arn_suffix + network_internal_invoke_url = dependency.network.outputs.internal_invoke_url + network_api_id = dependency.network.outputs.api_id + network_api_invoke_url = dependency.network.outputs.api_invoke_url + network_api_execution_arn = dependency.network.outputs.api_execution_arn + network_api_stage_name = dependency.network.outputs.api_stage_name + network_vpc_link_id = dependency.network.outputs.vpc_link_id + network_http_api_authorizer_id = dependency.network.outputs.http_api_authorizer_id + }, ) diff --git a/infra/live/prod/aws/service_worker/terragrunt.hcl b/infra/live/prod/aws/service_worker/terragrunt.hcl index 7b23580e..6e0a23c5 100644 --- a/infra/live/prod/aws/service_worker/terragrunt.hcl +++ b/infra/live/prod/aws/service_worker/terragrunt.hcl @@ -2,14 +2,72 @@ include "root" { path = find_in_parent_folders("root.hcl") } -include "security" { - path = find_in_parent_folders("dependencies/security.hcl") +dependency "security" { + config_path = "${get_original_terragrunt_dir()}/../security" + + mock_outputs = { + load_balancer_sg = "sg-00000000000000001" + api_vpc_link_sg = "sg-00000000000000002" + vpc_endpoint_sg = "sg-00000000000000003" + ecs_sg = "sg-00000000000000004" + runtime_sg = "sg-00000000000000005" + postgres_sg = "sg-00000000000000006" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "messaging" { + config_path = "${get_original_terragrunt_dir()}/../messaging" + + mock_outputs = { + worker_topic_name = "mock-worker-events" + worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" + worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" + lambda_worker_queue_name = "mock-lambda-worker-queue" + lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" + lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" + lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" + lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" + lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" + ecs_worker_queue_name = "mock-ecs-worker-queue" + ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" + ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "cluster" { + config_path = "${get_original_terragrunt_dir()}/../cluster" + + mock_outputs = { + cluster_id = "mock-cluster-id" + cluster_name = "mock-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -locals { - messaging = read_terragrunt_config(find_in_parent_folders("dependencies/messaging.hcl")) - cluster = read_terragrunt_config(find_in_parent_folders("dependencies/cluster.hcl")) - network = read_terragrunt_config(find_in_parent_folders("dependencies/network.hcl")) +dependency "network" { + config_path = "${get_original_terragrunt_dir()}/../network" + + mock_outputs = { + default_target_group_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:targetgroup/mock-default/1234567890abcdef" + load_balancer_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:loadbalancer/app/mock-internal/1234567890abcdef" + default_http_listener_arn = "arn:aws:elasticloadbalancing:eu-west-2:111111111111:listener/app/mock-internal/1234567890abcdef/abcdef1234567890" + load_balancer_arn_suffix = "app/mock-internal/1234567890abcdef" + target_group_arn_suffix = "targetgroup/mock-default/1234567890abcdef" + internal_invoke_url = "http://mock-internal-123456.eu-west-2.elb.amazonaws.com" + api_id = "mockapi123" + api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" + api_execution_arn = "arn:aws:execute-api:eu-west-2:111111111111:mockapi123" + api_stage_name = "$default" + vpc_link_id = "vpclink-mock123" + http_api_authorizer_id = "auth-mock123" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { @@ -20,7 +78,23 @@ inputs = merge( { ecs_security_group_id = dependency.security.outputs.ecs_sg }, - local.messaging.inputs, - local.cluster.inputs, - local.network.inputs, + dependency.messaging.outputs, + { + cluster_id = dependency.cluster.outputs.cluster_id + cluster_name = dependency.cluster.outputs.cluster_name + }, + { + network_default_target_group_arn = dependency.network.outputs.default_target_group_arn + network_load_balancer_arn = dependency.network.outputs.load_balancer_arn + network_default_http_listener_arn = dependency.network.outputs.default_http_listener_arn + network_load_balancer_arn_suffix = dependency.network.outputs.load_balancer_arn_suffix + network_target_group_arn_suffix = dependency.network.outputs.target_group_arn_suffix + network_internal_invoke_url = dependency.network.outputs.internal_invoke_url + network_api_id = dependency.network.outputs.api_id + network_api_invoke_url = dependency.network.outputs.api_invoke_url + network_api_execution_arn = dependency.network.outputs.api_execution_arn + network_api_stage_name = dependency.network.outputs.api_stage_name + network_vpc_link_id = dependency.network.outputs.vpc_link_id + network_http_api_authorizer_id = dependency.network.outputs.http_api_authorizer_id + }, ) diff --git a/infra/live/prod/aws/task_worker/terragrunt.hcl b/infra/live/prod/aws/task_worker/terragrunt.hcl index b28cbca9..c315b177 100644 --- a/infra/live/prod/aws/task_worker/terragrunt.hcl +++ b/infra/live/prod/aws/task_worker/terragrunt.hcl @@ -2,13 +2,52 @@ include "root" { path = find_in_parent_folders("root.hcl") } -locals { - messaging = read_terragrunt_config(find_in_parent_folders("dependencies/messaging.hcl")) - database = read_terragrunt_config(find_in_parent_folders("dependencies/database.hcl")) +dependency "messaging" { + config_path = "${get_original_terragrunt_dir()}/../messaging" + + mock_outputs = { + worker_topic_name = "mock-worker-events" + worker_topic_arn = "arn:aws:sns:eu-west-2:111111111111:mock-worker-events" + worker_topic_publish_policy_arn = "arn:aws:iam::111111111111:policy/mock-worker-topic-publish" + lambda_worker_queue_name = "mock-lambda-worker-queue" + lambda_worker_queue_arn = "arn:aws:sqs:eu-west-2:111111111111:mock-lambda-worker-queue" + lambda_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-queue" + lambda_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-lambda-worker-queue-read" + lambda_worker_dead_letter_queue_name = "mock-lambda-worker-dlq" + lambda_worker_dead_letter_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-lambda-worker-dlq" + ecs_worker_queue_name = "mock-ecs-worker-queue" + ecs_worker_queue_url = "https://sqs.eu-west-2.amazonaws.com/111111111111/mock-ecs-worker-queue" + ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + +dependency "database" { + config_path = "${get_original_terragrunt_dir()}/../database" + + mock_outputs = { + database_credentials_secret_arn = "arn:aws:secretsmanager:eu-west-2:111111111111:secret:mock-database-credentials" + database_readwrite_endpoint = "mock-database.cluster-abcdefghijkl.eu-west-2.rds.amazonaws.com" + database_name = "app" + database_port = 5432 + database_cluster_identifier = "mock-database-cluster" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } terraform { source = "../../../../modules//aws//task_worker" } -inputs = merge(local.messaging.inputs, local.database.inputs) +inputs = merge( + dependency.messaging.outputs, + { + database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn + database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint + database_name = dependency.database.outputs.database_name + database_port = dependency.database.outputs.database_port + database_cluster_identifier = dependency.database.outputs.database_cluster_identifier + }, +) From 36c7ee3044f10ad710527fdfa7ed67d0ada4808e Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 10:59:29 +0100 Subject: [PATCH 05/73] chore: just tg-graph env --- justfile | 7 ------ justfile.ci | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 7 deletions(-) diff --git a/justfile b/justfile index 67d4cfdd..f29abaa9 100644 --- a/justfile +++ b/justfile @@ -152,13 +152,6 @@ tg-all op: terragrunt run-all {{op}} -# Print the Terragrunt dependency graph for one environment/provider root. -tg-graph env provider='aws': - #!/usr/bin/env bash - cd {{justfile_directory()}} - terragrunt graph-dependencies --terragrunt-working-dir infra/live/{{env}}/{{provider}} - - # Open an ECS Exec shell in the worker debug container. worker-debug-shell env: #!/usr/bin/env bash diff --git a/justfile.ci b/justfile.ci index c0ec334c..f657a532 100644 --- a/justfile.ci +++ b/justfile.ci @@ -12,6 +12,77 @@ EXTRA_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate EXTRA_CONTAI NON_SERVICE_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate NON_SERVICE_CONTAINER_DIRECTORIES` +# Print the Terragrunt dependency graph for one environment/provider root as JSON. +tg-graph env provider='aws': + #!/usr/bin/env bash + set -euo pipefail + cd "{{PROJECT_DIR}}" + graph_output="$( + terragrunt graph-dependencies --terragrunt-working-dir infra/live/{{env}}/{{provider}} + )" + + nodes_json="$( + printf '%s\n' "$graph_output" \ + | awk ' + /->/ { next } + /digraph[[:space:]]*\{/ { next } + /\}/ { next } + /"/ { + match($0, /"[^"]+"/) + if (RSTART > 0) { + print substr($0, RSTART + 1, RLENGTH - 2) + } + } + ' \ + | sort -u \ + | jq -Rsc 'split("\n") | map(select(length > 0))' + )" + + edges_json="$( + printf '%s\n' "$graph_output" \ + | awk ' + /->/ { + match($0, /"[^"]+"/) + from = substr($0, RSTART + 1, RLENGTH - 2) + remainder = substr($0, RSTART + RLENGTH) + match(remainder, /"[^"]+"/) + to = substr(remainder, RSTART + 1, RLENGTH - 2) + print from "\t" to + } + ' \ + | jq -Rsc ' + split("\n") + | map(select(length > 0)) + | map(split("\t")) + | map({from: .[0], to: .[1]}) + ' + )" + + jq -n \ + --arg environment "{{env}}" \ + --arg provider "{{provider}}" \ + --argjson nodes "$nodes_json" \ + --argjson edges "$edges_json" \ + ' + { + environment: $environment, + provider: $provider, + nodes: $nodes, + edges: $edges, + dependencies: ( + reduce $nodes[] as $node + ({}; + .[$node] = ( + $edges + | map(select(.from == $node) | .to) + | sort + ) + ) + ) + } + ' + + # Run `tflint` across Terraform module directories. tf-lint-check: #!/bin/bash From cf77cf5ddf6265da7fbe5baffa394af54d5fbffe Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 11:19:51 +0100 Subject: [PATCH 06/73] chore: just get dep array --- .github/docs/README.md | 2 + README.md | 15 +++ REPO_INSTRUCTIONS.md | 2 +- infra/scripts/render-terragrunt-graph.sh | 150 +++++++++++++++++++++++ justfile | 10 ++ justfile.ci | 71 ----------- 6 files changed, 178 insertions(+), 72 deletions(-) create mode 100755 infra/scripts/render-terragrunt-graph.sh diff --git a/.github/docs/README.md b/.github/docs/README.md index 9f82b5f9..c34d5d9a 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -89,6 +89,8 @@ flowchart LR Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` Pure ordered infra graph executor. It applies shared stacks first, then runtime stacks, then frontend infrastructure. Shared stacks now include the CloudWatch observability dashboard. It accepts `tg_action` so the same graph can run a normal apply, upload derived per-stack plan artifacts to the dedicated plan bucket under `terragrunt_plan/`, or apply from previously uploaded plan artifacts. The wrapper workflows now pass a single `plan_run_id`, exported to Terragrunt jobs as `PLAN_ARTIFACT_RUN_ID`, while each Terragrunt job configures AWS credentials at job start and then reuses that ambient session in the repo-local Terragrunt action. That means each infra run has one shared run-level metadata artifact (`infra-plan-metadata`) for the whole graph and one separate saved plan bundle per Terragrunt stack or module. Saved-plan transfer is opt-in: the shared workflow sets `TG_ENABLE_PLAN_ARTIFACTS=true` only for `plan` and `apply_plan`. In `plan` mode, the shared Terragrunt root `after_hook` renders and uploads each per-stack plan bundle. In `apply_plan` mode, the shared Terragrunt root `before_hook` downloads the saved plan bundle before `terragrunt apply` runs and fails if the saved metadata says mocked outputs were used. Its visible step labels now follow the high-level operation, so both direct apply and apply-from-plan render as `Apply` while plan still renders as `Plan`. Bootstrap-sensitive edges such as `security -> network` should be modeled with Terragrunt `dependency` blocks plus constrained `mock_outputs` in the live stack so `plan` and `validate` can run before upstream state exists, while `apply` still resolves real outputs. +- `infra/scripts/render-terragrunt-graph.sh` + Terragrunt-owned graph renderer used by the repo-local `just tg-graph` wrapper. It always converts `terragrunt graph-dependencies` output into JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files for that run from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. - `shared_deploy.yml` Rolls out Lambda code, optional migrations, optional reconciliation Lambdas, ECS task and service updates, and optional frontend deploys. Its multi-step AWS jobs now configure credentials once at job start and let the local `just` and Terragrunt actions reuse that ambient session. The reusable workflow renders its Lambda and ECS CodeDeploy AppSpec files from the shared templates under `config/deploy/`, and its mutating `just` steps should target `justfile.deploy` rather than the repo-root `justfile`. diff --git a/README.md b/README.md index 655da606..ebe6e574 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,21 @@ Given a Terragrunt file is found at `infra/live/dev/aws/lambda_api/terragrunt.hc just tg dev aws/lambda_api plan ``` +To print the Terragrunt dependency graph as JSON: + +```sh +just tg-graph dev +``` + +To join that graph with saved-plan metadata for one plan run, set `TG_GRAPH_METADATA_PLAN_RUN_ID` before running the same command: + +```sh +AWS_REGION=eu-west-2 \ +BUCKET_NAME=700060376888-eu-west-2-aws-serverless-github-deploy-tfplan \ +TG_GRAPH_METADATA_PLAN_RUN_ID=26105102715 \ +just tg-graph dev +``` + ### Publish A Worker Message To publish directly to the shared worker SNS topic from your shell: diff --git a/REPO_INSTRUCTIONS.md b/REPO_INSTRUCTIONS.md index 514bc439..0857ea37 100644 --- a/REPO_INSTRUCTIONS.md +++ b/REPO_INSTRUCTIONS.md @@ -43,6 +43,7 @@ These instructions apply to the entire repository. ## Script Ownership - reserve `infra/scripts/**` for Terraform or Terragrunt owned helper behavior that is part of the infra runtime contract +- Terragrunt graph rendering, saved-plan metadata lookups, and other helpers that shape the output of Terragrunt commands belong under `infra/scripts/**`, even if a `just` recipe or CI job invokes them - prefer implementing GitHub Actions or workflow-only helper logic directly in `justfile.ci` when practical - when a workflow-only helper needs more than a small recipe body, keep its ownership in the CI/workflow layer rather than under `infra/scripts/**` @@ -109,7 +110,6 @@ These instructions apply to the entire repository. - check apply/deploy/destroy, and avoid unnecessary `terraform_remote_state` coupling (especially for fast-changing outputs) - for bootstrap-sensitive or plan-sensitive cross-stack contracts, prefer Terragrunt `dependency` inputs in the live stack and `mock_outputs` for non-mutating commands rather than reading upstream state directly inside Terraform modules - if CI plan failures are caused by missing upstream state, fix the contract shape first instead of papering over the issue with more direct `terraform_remote_state` reads -- when the same Terragrunt dependency wiring or mocks are needed across environments, centralize that shared config under `infra/live/dependencies/` in a capability-scoped helper such as `network.hcl` and have environment stacks read it rather than duplicating the same blocks in `dev`, `prod`, or `ci` - keep this approach visible to users as well: when you introduce or expand this pattern, update the top-level `README.md` so the bootstrap-friendly mock strategy is documented outside agent-only instructions - if you intentionally add a Terraform `data "terraform_remote_state"` block, add a `# remote_state_reason: ...` comment immediately above it explaining why Terragrunt `dependency` plus `mock_outputs` is not practical for that case - if you intentionally add a Terraform `data "terraform_remote_state"` block, add a `# remote_state_reason: ...` comment immediately above it explaining why Terragrunt `dependency` plus `mock_outputs` is not practical for that case diff --git a/infra/scripts/render-terragrunt-graph.sh b/infra/scripts/render-terragrunt-graph.sh new file mode 100755 index 00000000..fb590a93 --- /dev/null +++ b/infra/scripts/render-terragrunt-graph.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +set -euo pipefail + +environment="${1:?environment is required}" +provider="${2:?provider is required}" + +infra_plan_dir="${INFRA_PLAN_DIR:-terragrunt_plan}" +plan_run_id="${TG_GRAPH_METADATA_PLAN_RUN_ID:-}" +aws_region="${AWS_REGION:-}" +bucket_name="${BUCKET_NAME:-}" + +graph_output="$(cat)" + +nodes_json="$( + printf '%s\n' "$graph_output" \ + | awk ' + /->/ { next } + /digraph[[:space:]]*\{/ { next } + /\}/ { next } + /"/ { + match($0, /"[^"]+"/) + if (RSTART > 0) { + print substr($0, RSTART + 1, RLENGTH - 2) + } + } + ' \ + | sort -u \ + | jq -Rsc 'split("\n") | map(select(length > 0))' +)" + +edges_json="$( + printf '%s\n' "$graph_output" \ + | awk ' + /->/ { + match($0, /"[^"]+"/) + from = substr($0, RSTART + 1, RLENGTH - 2) + remainder = substr($0, RSTART + RLENGTH) + match(remainder, /"[^"]+"/) + to = substr(remainder, RSTART + 1, RLENGTH - 2) + print from "\t" to + } + ' \ + | jq -Rsc ' + split("\n") + | map(select(length > 0)) + | map(split("\t")) + | map({from: .[0], to: .[1]}) + ' +)" + +graph_json="$( + jq -n \ + --arg environment "$environment" \ + --arg provider "$provider" \ + --argjson nodes "$nodes_json" \ + --argjson edges "$edges_json" \ + ' + { + environment: $environment, + provider: $provider, + nodes: $nodes, + edges: $edges, + dependencies: ( + reduce $nodes[] as $node + ({}; + .[$node] = ( + $edges + | map(select(.from == $node) | .to) + | sort + ) + ) + ) + } + ' +)" + +if [[ -z "$plan_run_id" ]]; then + printf '%s\n' "$graph_json" + exit 0 +fi + +if [[ -z "$aws_region" ]]; then + echo "AWS_REGION is required when TG_GRAPH_METADATA_PLAN_RUN_ID is set." >&2 + exit 1 +fi + +if [[ -z "$bucket_name" ]]; then + echo "BUCKET_NAME is required when TG_GRAPH_METADATA_PLAN_RUN_ID is set." >&2 + exit 1 +fi + +metadata_dir="$(mktemp -d)" +trap 'rm -rf "$metadata_dir"' EXIT + +run_prefix="s3://${bucket_name}/${infra_plan_dir}/${environment}/${plan_run_id}/" + +aws s3 sync \ + "$run_prefix" \ + "$metadata_dir" \ + --exclude "*" \ + --include "*/terragrunt.plan.meta.json" \ + >/dev/null + +metadata_files=() +while IFS= read -r -d '' file; do + metadata_files+=("$file") +done < <(find "$metadata_dir" -type f -name 'terragrunt.plan.meta.json' -print0 | sort -z) + +if [[ "${#metadata_files[@]}" -eq 0 ]]; then + jq -n \ + --arg environment "$environment" \ + --arg provider "$provider" \ + --arg plan_run_id "$plan_run_id" \ + '{environment: $environment, provider: $provider, plan_run_id: $plan_run_id, items: {}}' + exit 0 +fi + +jq -s \ + --arg environment "$environment" \ + --arg provider "$provider" \ + --arg plan_run_id "$plan_run_id" \ + --argjson graph "$graph_json" \ + ' + def basename_from_tg_directory: + split("/") | last; + + reduce ( + .[] + | select(.tg_directory != null) + ) as $meta ( + { + environment: $environment, + provider: $provider, + plan_run_id: $plan_run_id, + items: {} + }; + ($meta.tg_directory | basename_from_tg_directory) as $stack + | if ($graph.nodes | index($stack)) == null then + . + else + .items[$stack] = ( + $meta + + { + dependencies: ($graph.dependencies[$stack] // []) + } + ) + end + ) + ' \ + "${metadata_files[@]}" diff --git a/justfile b/justfile index f29abaa9..87fe0724 100644 --- a/justfile +++ b/justfile @@ -152,6 +152,16 @@ tg-all op: terragrunt run-all {{op}} +# Print the Terragrunt dependency graph for one environment/provider root as JSON. +# Set TG_GRAPH_METADATA_PLAN_RUN_ID to join saved-plan metadata into the output. +tg-graph env provider='aws': + #!/usr/bin/env bash + set -euo pipefail + cd {{justfile_directory()}} + terragrunt graph-dependencies --terragrunt-working-dir infra/live/{{env}}/{{provider}} \ + | "{{justfile_directory()}}/infra/scripts/render-terragrunt-graph.sh" "{{env}}" "{{provider}}" + + # Open an ECS Exec shell in the worker debug container. worker-debug-shell env: #!/usr/bin/env bash diff --git a/justfile.ci b/justfile.ci index f657a532..c0ec334c 100644 --- a/justfile.ci +++ b/justfile.ci @@ -12,77 +12,6 @@ EXTRA_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate EXTRA_CONTAI NON_SERVICE_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate NON_SERVICE_CONTAINER_DIRECTORIES` -# Print the Terragrunt dependency graph for one environment/provider root as JSON. -tg-graph env provider='aws': - #!/usr/bin/env bash - set -euo pipefail - cd "{{PROJECT_DIR}}" - graph_output="$( - terragrunt graph-dependencies --terragrunt-working-dir infra/live/{{env}}/{{provider}} - )" - - nodes_json="$( - printf '%s\n' "$graph_output" \ - | awk ' - /->/ { next } - /digraph[[:space:]]*\{/ { next } - /\}/ { next } - /"/ { - match($0, /"[^"]+"/) - if (RSTART > 0) { - print substr($0, RSTART + 1, RLENGTH - 2) - } - } - ' \ - | sort -u \ - | jq -Rsc 'split("\n") | map(select(length > 0))' - )" - - edges_json="$( - printf '%s\n' "$graph_output" \ - | awk ' - /->/ { - match($0, /"[^"]+"/) - from = substr($0, RSTART + 1, RLENGTH - 2) - remainder = substr($0, RSTART + RLENGTH) - match(remainder, /"[^"]+"/) - to = substr(remainder, RSTART + 1, RLENGTH - 2) - print from "\t" to - } - ' \ - | jq -Rsc ' - split("\n") - | map(select(length > 0)) - | map(split("\t")) - | map({from: .[0], to: .[1]}) - ' - )" - - jq -n \ - --arg environment "{{env}}" \ - --arg provider "{{provider}}" \ - --argjson nodes "$nodes_json" \ - --argjson edges "$edges_json" \ - ' - { - environment: $environment, - provider: $provider, - nodes: $nodes, - edges: $edges, - dependencies: ( - reduce $nodes[] as $node - ({}; - .[$node] = ( - $edges - | map(select(.from == $node) | .to) - | sort - ) - ) - ) - } - ' - - # Run `tflint` across Terraform module directories. tf-lint-check: #!/bin/bash From ad82fe2905b8c13e92491295e78e3f672931598f Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 14:08:45 +0100 Subject: [PATCH 07/73] feat: get list of dependencies --- .github/docs/README.md | 6 +- README.md | 21 +- REPO_INSTRUCTIONS.md | 1 + graph.json | 2319 ++++++++++++++++++++++ infra/scripts/render-terragrunt-graph.sh | 150 -- justfile | 139 +- justfile.ci | 14 + 7 files changed, 2490 insertions(+), 160 deletions(-) create mode 100644 graph.json delete mode 100755 infra/scripts/render-terragrunt-graph.sh diff --git a/.github/docs/README.md b/.github/docs/README.md index c34d5d9a..60eb1139 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -89,8 +89,10 @@ flowchart LR Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` Pure ordered infra graph executor. It applies shared stacks first, then runtime stacks, then frontend infrastructure. Shared stacks now include the CloudWatch observability dashboard. It accepts `tg_action` so the same graph can run a normal apply, upload derived per-stack plan artifacts to the dedicated plan bucket under `terragrunt_plan/`, or apply from previously uploaded plan artifacts. The wrapper workflows now pass a single `plan_run_id`, exported to Terragrunt jobs as `PLAN_ARTIFACT_RUN_ID`, while each Terragrunt job configures AWS credentials at job start and then reuses that ambient session in the repo-local Terragrunt action. That means each infra run has one shared run-level metadata artifact (`infra-plan-metadata`) for the whole graph and one separate saved plan bundle per Terragrunt stack or module. Saved-plan transfer is opt-in: the shared workflow sets `TG_ENABLE_PLAN_ARTIFACTS=true` only for `plan` and `apply_plan`. In `plan` mode, the shared Terragrunt root `after_hook` renders and uploads each per-stack plan bundle. In `apply_plan` mode, the shared Terragrunt root `before_hook` downloads the saved plan bundle before `terragrunt apply` runs and fails if the saved metadata says mocked outputs were used. Its visible step labels now follow the high-level operation, so both direct apply and apply-from-plan render as `Apply` while plan still renders as `Plan`. Bootstrap-sensitive edges such as `security -> network` should be modeled with Terragrunt `dependency` blocks plus constrained `mock_outputs` in the live stack so `plan` and `validate` can run before upstream state exists, while `apply` still resolves real outputs. -- `infra/scripts/render-terragrunt-graph.sh` - Terragrunt-owned graph renderer used by the repo-local `just tg-graph` wrapper. It always converts `terragrunt graph-dependencies` output into JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files for that run from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. +- `just tg-graph-process` + Standalone graph post-processor used by the repo-local `just` wrappers. It reads a saved Graphviz JSON file produced from `terragrunt graph-dependencies | dot -Tjson`, converts that into compact dependency JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. +- Local prerequisite for the graph helpers: + `brew install graphviz` - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. - `shared_deploy.yml` Rolls out Lambda code, optional migrations, optional reconciliation Lambdas, ECS task and service updates, and optional frontend deploys. Its multi-step AWS jobs now configure credentials once at job start and let the local `just` and Terragrunt actions reuse that ambient session. The reusable workflow renders its Lambda and ECS CodeDeploy AppSpec files from the shared templates under `config/deploy/`, and its mutating `just` steps should target `justfile.deploy` rather than the repo-root `justfile`. diff --git a/README.md b/README.md index ebe6e574..779d63aa 100644 --- a/README.md +++ b/README.md @@ -98,19 +98,30 @@ Given a Terragrunt file is found at `infra/live/dev/aws/lambda_api/terragrunt.hc just tg dev aws/lambda_api plan ``` -To print the Terragrunt dependency graph as JSON: +The Terragrunt graph helpers require Graphviz locally because they convert DOT output with `dot -Tjson`: ```sh -just tg-graph dev +brew install graphviz ``` -To join that graph with saved-plan metadata for one plan run, set `TG_GRAPH_METADATA_PLAN_RUN_ID` before running the same command: +To print the Terragrunt dependency graph as raw Graphviz JSON: + +```sh +just tg-graph dev > graph.json +``` + +To process that saved graph file into compact dependency JSON: + +```sh +just tg-graph-process graph.json dev +``` + +To join the processed graph with saved-plan metadata for one plan run, set `TG_GRAPH_METADATA_PLAN_RUN_ID` and the plan bucket before running the processing command: ```sh -AWS_REGION=eu-west-2 \ BUCKET_NAME=700060376888-eu-west-2-aws-serverless-github-deploy-tfplan \ TG_GRAPH_METADATA_PLAN_RUN_ID=26105102715 \ -just tg-graph dev +just tg-graph-process graph.json dev ``` ### Publish A Worker Message diff --git a/REPO_INSTRUCTIONS.md b/REPO_INSTRUCTIONS.md index 0857ea37..ecbd70df 100644 --- a/REPO_INSTRUCTIONS.md +++ b/REPO_INSTRUCTIONS.md @@ -61,6 +61,7 @@ These instructions apply to the entire repository. - when a request mentions external source code and asks how to build, make ready, or deploy it, interpret that as "understand the external app, then answer in terms of how this repository should implement or deploy it" unless the user explicitly redirects the work - read the relevant local contract docs before editing and follow them - prefer the smallest complete change that matches existing repo patterns +- remove stale code, temporary helpers, and abandoned experiment residue as part of the same change rather than leaving dead paths behind - verify related workflows, infra, docs, and downstream dependencies when the request affects shared behavior - state material assumptions when the intended shape is not fully explicit - when ambiguity is material or a wrong assumption could cause the repo shape or contract to drift, ask the user a clarifying question before editing diff --git a/graph.json b/graph.json new file mode 100644 index 00000000..82081fd2 --- /dev/null +++ b/graph.json @@ -0,0 +1,2319 @@ +{ + "name": "%1", + "directed": true, + "strict": false, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#fffffe00" + }, + { + "op": "C", + "grad": "none", + "color": "#ffffff" + }, + { + "op": "P", + "points": [ + [ + 0, + 0 + ], + [ + 0, + 180 + ], + [ + 1583.47, + 180 + ], + [ + 1583.47, + 0 + ] + ] + } + ], + "bb": "0,0,1583.5,180", + "xdotversion": "1.7", + "_subgraph_cnt": 0, + "objects": [ + { + "_gvid": 0, + "name": "cluster", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 869.84, + 90, + 35.49, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 869.84, + 84.95 + ], + "align": "c", + "width": 36, + "text": "cluster" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "869.84,90", + "width": "0.9857" + }, + { + "_gvid": 1, + "name": "code_bucket", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 1144.84, + 162, + 57.49, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 1144.84, + 156.95 + ], + "align": "c", + "width": 68.25, + "text": "code_bucket" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "1144.8,162", + "width": "1.597" + }, + { + "_gvid": 2, + "name": "cognito", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 900.84, + 18, + 38.56, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 900.84, + 12.95 + ], + "align": "c", + "width": 40.5, + "text": "cognito" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "900.84,18", + "width": "1.071" + }, + { + "_gvid": 3, + "name": "database", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 291.84, + 90, + 42.65, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 291.84, + 84.95 + ], + "align": "c", + "width": 46.5, + "text": "database" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "291.84,90", + "width": "1.1847" + }, + { + "_gvid": 4, + "name": "security", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 705.84, + 18, + 40.09, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 705.84, + 12.95 + ], + "align": "c", + "width": 42.75, + "text": "security" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "705.84,18", + "width": "1.1137" + }, + { + "_gvid": 5, + "name": "ecr", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 1246.84, + 162, + 27, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 1246.84, + 156.95 + ], + "align": "c", + "width": 16.5, + "text": "ecr" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "1246.8,162", + "width": "0.75" + }, + { + "_gvid": 6, + "name": "frontend", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 1026.84, + 162, + 42.14, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 1026.84, + 156.95 + ], + "align": "c", + "width": 45.75, + "text": "frontend" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "1026.8,162", + "width": "1.1705" + }, + { + "_gvid": 7, + "name": "network", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 774.84, + 90, + 41.12, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 774.84, + 84.95 + ], + "align": "c", + "width": 44.25, + "text": "network" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "774.84,90", + "width": "1.1421" + }, + { + "_gvid": 8, + "name": "lambda_api", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 634.84, + 162, + 54.42, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 634.84, + 156.95 + ], + "align": "c", + "width": 63.75, + "text": "lambda_api" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "634.84,162", + "width": "1.5117" + }, + { + "_gvid": 9, + "name": "messaging", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 559.84, + 90, + 50.33, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 559.84, + 84.95 + ], + "align": "c", + "width": 57.75, + "text": "messaging" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "559.84,90", + "width": "1.398" + }, + { + "_gvid": 10, + "name": "lambda_worker", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 492.84, + 162, + 69.26, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 492.84, + 156.95 + ], + "align": "c", + "width": 85.5, + "text": "lambda_worker" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "492.84,162", + "width": "1.924" + }, + { + "_gvid": 11, + "name": "migrations", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 50.84, + 162, + 50.84, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 50.84, + 156.95 + ], + "align": "c", + "width": 58.5, + "text": "migrations" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "50.84,162", + "width": "1.4122" + }, + { + "_gvid": 12, + "name": "observability", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 1350.84, + 162, + 59.03, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 1350.84, + 156.95 + ], + "align": "c", + "width": 70.5, + "text": "observability" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "1350.8,162", + "width": "1.6397" + }, + { + "_gvid": 13, + "name": "oidc", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 1454.84, + 162, + 27, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 1454.84, + 156.95 + ], + "align": "c", + "width": 23.25, + "text": "oidc" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "1454.8,162", + "width": "0.75" + }, + { + "_gvid": 14, + "name": "rds_reader_tagger", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 196.84, + 162, + 77.45, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 196.84, + 156.95 + ], + "align": "c", + "width": 97.5, + "text": "rds_reader_tagger" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "196.84,162", + "width": "2.1515" + }, + { + "_gvid": 15, + "name": "service_api", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 913.84, + 162, + 52.89, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 913.84, + 156.95 + ], + "align": "c", + "width": 61.5, + "text": "service_api" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "913.84,162", + "width": "1.4691" + }, + { + "_gvid": 16, + "name": "service_worker", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 774.84, + 162, + 67.73, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 774.84, + 156.95 + ], + "align": "c", + "width": 83.25, + "text": "service_worker" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "774.84,162", + "width": "1.8814" + }, + { + "_gvid": 17, + "name": "task_api", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 1541.84, + 162, + 41.63, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 1541.84, + 156.95 + ], + "align": "c", + "width": 45, + "text": "task_api" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "1541.8,162", + "width": "1.1563" + }, + { + "_gvid": 18, + "name": "task_worker", + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "e", + "rect": [ + 348.84, + 162, + 56.47, + 18 + ] + } + ], + "_ldraw_": [ + { + "op": "F", + "size": 14, + "face": "Times-Roman" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "T", + "pt": [ + 348.84, + 156.95 + ], + "align": "c", + "width": 66.75, + "text": "task_worker" + } + ], + "height": "0.5", + "label": "\\N", + "pos": "348.84,162", + "width": "1.5686" + } + ], + "edges": [ + { + "_gvid": 0, + "tail": 3, + "head": 4, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 330.79, + 82.41 + ], + [ + 406.82, + 69.56 + ], + [ + 574.8, + 41.16 + ], + [ + 657.68, + 27.14 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 658.02, + 30.64 + ], + [ + 667.3, + 25.52 + ], + [ + 656.85, + 23.73 + ] + ] + } + ], + "pos": "e,668.79,25.265 330.79,82.415 406.82,69.559 574.8,41.156 657.68,27.143" + }, + { + "_gvid": 2, + "tail": 6, + "head": 7, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 994.87, + 149.87 + ], + [ + 988.59, + 147.82 + ], + [ + 982.03, + 145.77 + ], + [ + 975.84, + 144 + ], + [ + 909.92, + 125.14 + ], + [ + 891.75, + 126.86 + ], + [ + 825.84, + 108 + ], + [ + 822.95, + 107.17 + ], + [ + 819.99, + 106.29 + ], + [ + 817.01, + 105.37 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 818.3, + 102.11 + ], + [ + 807.71, + 102.41 + ], + [ + 816.17, + 108.78 + ] + ] + } + ], + "pos": "e,806.27,101.95 994.87,149.87 988.59,147.82 982.03,145.77 975.84,144 909.92,125.14 891.75,126.86 825.84,108 822.95,107.17 819.99,106.29 817.01,105.37" + }, + { + "_gvid": 1, + "tail": 6, + "head": 2, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 1012.39, + 144.71 + ], + [ + 990.36, + 119.89 + ], + [ + 948.24, + 72.42 + ], + [ + 922.54, + 43.45 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 925.38, + 41.39 + ], + [ + 916.13, + 36.23 + ], + [ + 920.15, + 46.03 + ] + ] + } + ], + "pos": "e,915.12,35.099 1012.4,144.71 990.36,119.89 948.24,72.425 922.54,43.452" + }, + { + "_gvid": 5, + "tail": 8, + "head": 7, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 663.85, + 146.5 + ], + [ + 685.52, + 135.66 + ], + [ + 715.28, + 120.78 + ], + [ + 738.53, + 109.15 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 739.97, + 112.35 + ], + [ + 747.35, + 104.75 + ], + [ + 736.84, + 106.09 + ] + ] + } + ], + "pos": "e,748.7,104.07 663.85,146.5 685.52,135.66 715.28,120.78 738.53,109.15" + }, + { + "_gvid": 6, + "tail": 8, + "head": 9, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 617.44, + 144.76 + ], + [ + 607.88, + 135.84 + ], + [ + 595.85, + 124.61 + ], + [ + 585.25, + 114.72 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 587.95, + 112.44 + ], + [ + 578.25, + 108.18 + ], + [ + 583.17, + 117.56 + ] + ] + } + ], + "pos": "e,577.14,107.15 617.44,144.76 607.88,135.84 595.85,124.61 585.25,114.72" + }, + { + "_gvid": 7, + "tail": 10, + "head": 9, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 509.06, + 144.05 + ], + [ + 517.24, + 135.5 + ], + [ + 527.32, + 124.97 + ], + [ + 536.34, + 115.56 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 538.8, + 118.04 + ], + [ + 543.18, + 108.4 + ], + [ + 533.74, + 113.2 + ] + ] + } + ], + "pos": "e,544.23,107.31 509.06,144.05 517.24,135.5 527.32,124.97 536.34,115.56" + }, + { + "_gvid": 9, + "tail": 11, + "head": 4, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 75.79, + 145.99 + ], + [ + 111, + 125.57 + ], + [ + 177.91, + 89.58 + ], + [ + 239.84, + 72 + ], + [ + 386.05, + 30.48 + ], + [ + 566.81, + 21.33 + ], + [ + 653.99, + 19.42 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 653.96, + 22.92 + ], + [ + 663.89, + 19.23 + ], + [ + 653.83, + 15.92 + ] + ] + } + ], + "pos": "e,665.4,19.201 75.786,145.99 111,125.57 177.91,89.585 239.84,72 386.05,30.482 566.81,21.332 653.99,19.421" + }, + { + "_gvid": 8, + "tail": 11, + "head": 3, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 89.29, + 149.83 + ], + [ + 132.05, + 137.41 + ], + [ + 201.05, + 117.37 + ], + [ + 246.43, + 104.19 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 247.35, + 107.57 + ], + [ + 255.97, + 101.42 + ], + [ + 245.39, + 100.85 + ] + ] + } + ], + "pos": "e,257.43,101 89.293,149.83 132.05,137.41 201.05,117.37 246.43,104.19" + }, + { + "_gvid": 4, + "tail": 7, + "head": 4, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 759.18, + 73.12 + ], + [ + 750.36, + 64.17 + ], + [ + 739.17, + 52.81 + ], + [ + 729.31, + 42.81 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 732.01, + 40.56 + ], + [ + 722.5, + 35.9 + ], + [ + 727.02, + 45.48 + ] + ] + } + ], + "pos": "e,721.43,34.821 759.18,73.116 750.36,64.166 739.17,52.81 729.31,42.815" + }, + { + "_gvid": 3, + "tail": 7, + "head": 2, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 799.43, + 75.34 + ], + [ + 818.64, + 64.67 + ], + [ + 845.54, + 49.72 + ], + [ + 866.81, + 37.91 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 868.37, + 41.04 + ], + [ + 875.41, + 33.13 + ], + [ + 864.97, + 34.92 + ] + ] + } + ], + "pos": "e,876.74,32.391 799.43,75.337 818.64,64.669 845.54,49.724 866.81,37.908" + }, + { + "_gvid": 10, + "tail": 14, + "head": 3, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 219.35, + 144.41 + ], + [ + 232.19, + 134.95 + ], + [ + 248.44, + 122.98 + ], + [ + 262.29, + 112.77 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 264.24, + 115.69 + ], + [ + 270.21, + 106.94 + ], + [ + 260.09, + 110.05 + ] + ] + } + ], + "pos": "e,271.43,106.04 219.35,144.41 232.19,134.95 248.44,122.98 262.29,112.77" + }, + { + "_gvid": 12, + "tail": 15, + "head": 4, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 919.49, + 143.95 + ], + [ + 924.79, + 124.42 + ], + [ + 929.77, + 92.5 + ], + [ + 913.84, + 72 + ], + [ + 894.63, + 47.28 + ], + [ + 811.01, + 32.17 + ], + [ + 755.82, + 24.71 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 756.38, + 21.26 + ], + [ + 746.01, + 23.43 + ], + [ + 755.47, + 28.2 + ] + ] + } + ], + "pos": "e,744.51,23.238 919.49,143.95 924.79,124.42 929.77,92.497 913.84,72 894.63,47.277 811.01,32.174 755.82,24.714" + }, + { + "_gvid": 11, + "tail": 15, + "head": 0, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 903.19, + 144.05 + ], + [ + 898.09, + 135.94 + ], + [ + 891.87, + 126.04 + ], + [ + 886.19, + 117.01 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 889.18, + 115.19 + ], + [ + 880.9, + 108.59 + ], + [ + 883.25, + 118.92 + ] + ] + } + ], + "pos": "e,880.09,107.31 903.19,144.05 898.09,135.94 891.87,126.04 886.19,117.01" + }, + { + "_gvid": 13, + "tail": 15, + "head": 7, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 885.04, + 146.5 + ], + [ + 863.52, + 135.66 + ], + [ + 833.97, + 120.78 + ], + [ + 810.89, + 109.15 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 812.65, + 106.12 + ], + [ + 802.14, + 104.75 + ], + [ + 809.5, + 112.37 + ] + ] + } + ], + "pos": "e,800.79,104.07 885.04,146.5 863.52,135.66 833.97,120.78 810.89,109.15" + }, + { + "_gvid": 15, + "tail": 16, + "head": 4, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 754.73, + 144.57 + ], + [ + 744.28, + 134.97 + ], + [ + 732.19, + 122 + ], + [ + 724.84, + 108 + ], + [ + 715.01, + 89.27 + ], + [ + 710.26, + 65.79 + ], + [ + 707.97, + 47.65 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 711.47, + 47.41 + ], + [ + 706.93, + 37.84 + ], + [ + 704.51, + 48.15 + ] + ] + } + ], + "pos": "e,706.77,36.332 754.73,144.57 744.28,134.97 732.19,122 724.84,108 715.01,89.269 710.26,65.786 707.97,47.645" + }, + { + "_gvid": 17, + "tail": 16, + "head": 9, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 733.13, + 147.42 + ], + [ + 696.9, + 135.62 + ], + [ + 644.46, + 118.55 + ], + [ + 606.77, + 106.28 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 608.1, + 103.03 + ], + [ + 597.51, + 103.26 + ], + [ + 605.94, + 109.69 + ] + ] + } + ], + "pos": "e,596.07,102.8 733.13,147.42 696.9,135.62 644.46,118.55 606.77,106.28" + }, + { + "_gvid": 14, + "tail": 16, + "head": 0, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 796.87, + 144.76 + ], + [ + 810.09, + 135.02 + ], + [ + 827.05, + 122.53 + ], + [ + 841.29, + 112.04 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 842.99, + 115.13 + ], + [ + 848.97, + 106.38 + ], + [ + 838.84, + 109.49 + ] + ] + } + ], + "pos": "e,850.18,105.48 796.87,144.76 810.09,135.02 827.05,122.53 841.29,112.04" + }, + { + "_gvid": 16, + "tail": 16, + "head": 7, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 774.84, + 143.7 + ], + [ + 774.84, + 136.41 + ], + [ + 774.84, + 127.73 + ], + [ + 774.84, + 119.54 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 778.34, + 119.62 + ], + [ + 774.84, + 109.62 + ], + [ + 771.34, + 119.62 + ] + ] + } + ], + "pos": "e,774.84,108.1 774.84,143.7 774.84,136.41 774.84,127.73 774.84,119.54" + }, + { + "_gvid": 19, + "tail": 18, + "head": 9, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 387.05, + 148.32 + ], + [ + 422.4, + 136.6 + ], + [ + 475.03, + 119.13 + ], + [ + 512.92, + 106.57 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 513.85, + 109.95 + ], + [ + 522.24, + 103.47 + ], + [ + 511.65, + 103.3 + ] + ] + } + ], + "pos": "e,523.68,103 387.05,148.32 422.4,136.6 475.03,119.13 512.92,106.57" + }, + { + "_gvid": 18, + "tail": 18, + "head": 3, + "_draw_": [ + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "b", + "points": [ + [ + 335.04, + 144.05 + ], + [ + 328.22, + 135.68 + ], + [ + 319.85, + 125.4 + ], + [ + 312.31, + 116.13 + ] + ] + } + ], + "_hdraw_": [ + { + "op": "S", + "style": "solid" + }, + { + "op": "c", + "grad": "none", + "color": "#000000" + }, + { + "op": "C", + "grad": "none", + "color": "#000000" + }, + { + "op": "P", + "points": [ + [ + 315.1, + 114.03 + ], + [ + 306.07, + 108.48 + ], + [ + 309.67, + 118.45 + ] + ] + } + ], + "pos": "e,305.12,107.31 335.04,144.05 328.22,135.68 319.85,125.4 312.31,116.13" + } + ] +} diff --git a/infra/scripts/render-terragrunt-graph.sh b/infra/scripts/render-terragrunt-graph.sh deleted file mode 100755 index fb590a93..00000000 --- a/infra/scripts/render-terragrunt-graph.sh +++ /dev/null @@ -1,150 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -environment="${1:?environment is required}" -provider="${2:?provider is required}" - -infra_plan_dir="${INFRA_PLAN_DIR:-terragrunt_plan}" -plan_run_id="${TG_GRAPH_METADATA_PLAN_RUN_ID:-}" -aws_region="${AWS_REGION:-}" -bucket_name="${BUCKET_NAME:-}" - -graph_output="$(cat)" - -nodes_json="$( - printf '%s\n' "$graph_output" \ - | awk ' - /->/ { next } - /digraph[[:space:]]*\{/ { next } - /\}/ { next } - /"/ { - match($0, /"[^"]+"/) - if (RSTART > 0) { - print substr($0, RSTART + 1, RLENGTH - 2) - } - } - ' \ - | sort -u \ - | jq -Rsc 'split("\n") | map(select(length > 0))' -)" - -edges_json="$( - printf '%s\n' "$graph_output" \ - | awk ' - /->/ { - match($0, /"[^"]+"/) - from = substr($0, RSTART + 1, RLENGTH - 2) - remainder = substr($0, RSTART + RLENGTH) - match(remainder, /"[^"]+"/) - to = substr(remainder, RSTART + 1, RLENGTH - 2) - print from "\t" to - } - ' \ - | jq -Rsc ' - split("\n") - | map(select(length > 0)) - | map(split("\t")) - | map({from: .[0], to: .[1]}) - ' -)" - -graph_json="$( - jq -n \ - --arg environment "$environment" \ - --arg provider "$provider" \ - --argjson nodes "$nodes_json" \ - --argjson edges "$edges_json" \ - ' - { - environment: $environment, - provider: $provider, - nodes: $nodes, - edges: $edges, - dependencies: ( - reduce $nodes[] as $node - ({}; - .[$node] = ( - $edges - | map(select(.from == $node) | .to) - | sort - ) - ) - ) - } - ' -)" - -if [[ -z "$plan_run_id" ]]; then - printf '%s\n' "$graph_json" - exit 0 -fi - -if [[ -z "$aws_region" ]]; then - echo "AWS_REGION is required when TG_GRAPH_METADATA_PLAN_RUN_ID is set." >&2 - exit 1 -fi - -if [[ -z "$bucket_name" ]]; then - echo "BUCKET_NAME is required when TG_GRAPH_METADATA_PLAN_RUN_ID is set." >&2 - exit 1 -fi - -metadata_dir="$(mktemp -d)" -trap 'rm -rf "$metadata_dir"' EXIT - -run_prefix="s3://${bucket_name}/${infra_plan_dir}/${environment}/${plan_run_id}/" - -aws s3 sync \ - "$run_prefix" \ - "$metadata_dir" \ - --exclude "*" \ - --include "*/terragrunt.plan.meta.json" \ - >/dev/null - -metadata_files=() -while IFS= read -r -d '' file; do - metadata_files+=("$file") -done < <(find "$metadata_dir" -type f -name 'terragrunt.plan.meta.json' -print0 | sort -z) - -if [[ "${#metadata_files[@]}" -eq 0 ]]; then - jq -n \ - --arg environment "$environment" \ - --arg provider "$provider" \ - --arg plan_run_id "$plan_run_id" \ - '{environment: $environment, provider: $provider, plan_run_id: $plan_run_id, items: {}}' - exit 0 -fi - -jq -s \ - --arg environment "$environment" \ - --arg provider "$provider" \ - --arg plan_run_id "$plan_run_id" \ - --argjson graph "$graph_json" \ - ' - def basename_from_tg_directory: - split("/") | last; - - reduce ( - .[] - | select(.tg_directory != null) - ) as $meta ( - { - environment: $environment, - provider: $provider, - plan_run_id: $plan_run_id, - items: {} - }; - ($meta.tg_directory | basename_from_tg_directory) as $stack - | if ($graph.nodes | index($stack)) == null then - . - else - .items[$stack] = ( - $meta - + { - dependencies: ($graph.dependencies[$stack] // []) - } - ) - end - ) - ' \ - "${metadata_files[@]}" diff --git a/justfile b/justfile index 87fe0724..558d3b9a 100644 --- a/justfile +++ b/justfile @@ -152,14 +152,147 @@ tg-all op: terragrunt run-all {{op}} -# Print the Terragrunt dependency graph for one environment/provider root as JSON. -# Set TG_GRAPH_METADATA_PLAN_RUN_ID to join saved-plan metadata into the output. +# Print the Terragrunt dependency graph through Graphviz JSON. tg-graph env provider='aws': #!/usr/bin/env bash set -euo pipefail cd {{justfile_directory()}} + if ! command -v dot >/dev/null 2>&1; then + echo "❌ graphviz 'dot' is not installed or not on PATH." + exit 1 + fi + terragrunt graph-dependencies --terragrunt-working-dir infra/live/{{env}}/{{provider}} \ - | "{{justfile_directory()}}/infra/scripts/render-terragrunt-graph.sh" "{{env}}" "{{provider}}" + | dot -Tjson \ + | jq . + + +# Process a saved Graphviz JSON graph file into compact dependency JSON. +# Set TG_GRAPH_METADATA_PLAN_RUN_ID and BUCKET_NAME to join saved-plan metadata. +tg-graph-process graph_path env provider='aws': + #!/usr/bin/env bash + set -euo pipefail + cd {{justfile_directory()}} + + if [[ ! -f "{{graph_path}}" ]]; then + echo "❌ Graph file '{{graph_path}}' does not exist." + exit 1 + fi + + infra_plan_dir="${INFRA_PLAN_DIR:-terragrunt_plan}" + plan_run_id="${TG_GRAPH_METADATA_PLAN_RUN_ID:-}" + aws_region="${AWS_REGION:-}" + bucket_name="${BUCKET_NAME:-}" + + graph_json="$( + jq -c \ + --arg environment "{{env}}" \ + --arg provider "{{provider}}" \ + ' + . as $graph + | ($graph.objects // []) as $objects + | ($graph.edges // []) as $raw_edges + | ($objects | map(.name) | unique | sort) as $nodes + | ($raw_edges | map({from: $objects[.tail].name, to: $objects[.head].name})) as $edges + | { + environment: $environment, + provider: $provider, + nodes: $nodes, + edges: $edges, + dependencies: ( + reduce $nodes[] as $node + ({}; + .[$node] = ( + $edges + | map(select(.from == $node) | .to) + | unique + | sort + ) + ) + ) + } + ' \ + "{{graph_path}}" + )" + + if [[ -z "$plan_run_id" ]]; then + printf '%s\n' "$graph_json" + exit 0 + fi + + if [[ -z "$bucket_name" ]]; then + echo "❌ BUCKET_NAME is required when TG_GRAPH_METADATA_PLAN_RUN_ID is set." + exit 1 + fi + + metadata_dir="$(mktemp -d)" + trap 'rm -rf "$metadata_dir"' EXIT + run_prefix="s3://${bucket_name}/${infra_plan_dir}/{{env}}/${plan_run_id}/" + + if [[ -n "$aws_region" ]]; then + aws s3 sync \ + "$run_prefix" \ + "$metadata_dir" \ + --region "$aws_region" \ + --exclude "*" \ + --include "*/terragrunt.plan.meta.json" \ + >/dev/null + else + aws s3 sync \ + "$run_prefix" \ + "$metadata_dir" \ + --exclude "*" \ + --include "*/terragrunt.plan.meta.json" \ + >/dev/null + fi + + metadata_files=() + while IFS= read -r -d '' file; do + metadata_files+=("$file") + done < <(find "$metadata_dir" -type f -name 'terragrunt.plan.meta.json' -print0 | sort -z) + + if [[ "${#metadata_files[@]}" -eq 0 ]]; then + jq -n \ + --arg environment "{{env}}" \ + --arg provider "{{provider}}" \ + --arg plan_run_id "$plan_run_id" \ + '{environment: $environment, provider: $provider, plan_run_id: $plan_run_id, items: {}}' + exit 0 + fi + + jq -s \ + --arg environment "{{env}}" \ + --arg provider "{{provider}}" \ + --arg plan_run_id "$plan_run_id" \ + --argjson graph "$graph_json" \ + ' + def basename_from_tg_directory: + split("/") | last; + + reduce ( + .[] + | select(.tg_directory != null) + ) as $meta ( + { + environment: $environment, + provider: $provider, + plan_run_id: $plan_run_id, + items: {} + }; + ($meta.tg_directory | basename_from_tg_directory) as $stack + | if ($graph.nodes | index($stack)) == null then + . + else + .items[$stack] = ( + $meta + + { + dependencies: ($graph.dependencies[$stack] // []) + } + ) + end + ) + ' \ + "${metadata_files[@]}" # Open an ECS Exec shell in the worker debug container. diff --git a/justfile.ci b/justfile.ci index c0ec334c..ef1ddf64 100644 --- a/justfile.ci +++ b/justfile.ci @@ -12,6 +12,20 @@ EXTRA_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate EXTRA_CONTAI NON_SERVICE_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate NON_SERVICE_CONTAINER_DIRECTORIES` +# Generate compact Terragrunt graph JSON for workflow matrix use. +# Set TG_GRAPH_METADATA_PLAN_RUN_ID and BUCKET_NAME to join saved-plan metadata. +tg-graph-json environment provider='aws': + #!/usr/bin/env bash + set -euo pipefail + cd "{{PROJECT_DIR}}" + + tmp_graph="$(mktemp)" + trap 'rm -f "$tmp_graph"' EXIT + + just --justfile "{{PROJECT_DIR}}/justfile" tg-graph "{{environment}}" "{{provider}}" > "$tmp_graph" + just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-process "$tmp_graph" "{{environment}}" "{{provider}}" + + # Run `tflint` across Terraform module directories. tf-lint-check: #!/bin/bash From a2527d664b6339a8ff42e2a54a6ba3907b9c8fa1 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 15:31:08 +0100 Subject: [PATCH 08/73] chore: graph changed items --- .github/docs/README.md | 2 ++ README.md | 8 ++++++++ justfile | 23 +++++++++++++++++++++++ 3 files changed, 33 insertions(+) diff --git a/.github/docs/README.md b/.github/docs/README.md index 60eb1139..a69eedc3 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -91,6 +91,8 @@ flowchart LR Pure ordered infra graph executor. It applies shared stacks first, then runtime stacks, then frontend infrastructure. Shared stacks now include the CloudWatch observability dashboard. It accepts `tg_action` so the same graph can run a normal apply, upload derived per-stack plan artifacts to the dedicated plan bucket under `terragrunt_plan/`, or apply from previously uploaded plan artifacts. The wrapper workflows now pass a single `plan_run_id`, exported to Terragrunt jobs as `PLAN_ARTIFACT_RUN_ID`, while each Terragrunt job configures AWS credentials at job start and then reuses that ambient session in the repo-local Terragrunt action. That means each infra run has one shared run-level metadata artifact (`infra-plan-metadata`) for the whole graph and one separate saved plan bundle per Terragrunt stack or module. Saved-plan transfer is opt-in: the shared workflow sets `TG_ENABLE_PLAN_ARTIFACTS=true` only for `plan` and `apply_plan`. In `plan` mode, the shared Terragrunt root `after_hook` renders and uploads each per-stack plan bundle. In `apply_plan` mode, the shared Terragrunt root `before_hook` downloads the saved plan bundle before `terragrunt apply` runs and fails if the saved metadata says mocked outputs were used. Its visible step labels now follow the high-level operation, so both direct apply and apply-from-plan render as `Apply` while plan still renders as `Plan`. Bootstrap-sensitive edges such as `security -> network` should be modeled with Terragrunt `dependency` blocks plus constrained `mock_outputs` in the live stack so `plan` and `validate` can run before upstream state exists, while `apply` still resolves real outputs. - `just tg-graph-process` Standalone graph post-processor used by the repo-local `just` wrappers. It reads a saved Graphviz JSON file produced from `terragrunt graph-dependencies | dot -Tjson`, converts that into compact dependency JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. +- `just tg-graph-changed-items` + Small local helper layered on top of `just tg-graph-process`. In saved-plan metadata mode it converts the `.items` object into an array and keeps only entries where `has_changes: true`, adding the Terragrunt stack basename as `stack`. - Local prerequisite for the graph helpers: `brew install graphviz` - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. diff --git a/README.md b/README.md index 779d63aa..61643d40 100644 --- a/README.md +++ b/README.md @@ -116,6 +116,14 @@ To process that saved graph file into compact dependency JSON: just tg-graph-process graph.json dev ``` +To return only changed saved-plan items as an object array, set the saved-plan env vars and run: + +```sh +BUCKET_NAME=700060376888-eu-west-2-aws-serverless-github-deploy-tfplan \ +TG_GRAPH_METADATA_PLAN_RUN_ID=26105102715 \ +just tg-graph-changed-items graph.json dev +``` + To join the processed graph with saved-plan metadata for one plan run, set `TG_GRAPH_METADATA_PLAN_RUN_ID` and the plan bucket before running the processing command: ```sh diff --git a/justfile b/justfile index 558d3b9a..86f2ec12 100644 --- a/justfile +++ b/justfile @@ -295,6 +295,29 @@ tg-graph-process graph_path env provider='aws': "${metadata_files[@]}" +# Return only changed saved-plan graph items as an object array. +# Requires TG_GRAPH_METADATA_PLAN_RUN_ID and BUCKET_NAME so tg-graph-process +# emits saved-plan metadata under `.items`. +tg-graph-changed-items graph_path env provider='aws': + #!/usr/bin/env bash + set -euo pipefail + cd {{justfile_directory()}} + + just tg-graph-process "{{graph_path}}" "{{env}}" "{{provider}}" \ + | jq -c ' + if (.items? | type) != "object" then + error("tg-graph-changed-items requires tg-graph-process metadata mode.") + else + .items + | to_entries + | map( + select(.value.has_changes == true) + | (.value + {stack: .key}) + ) + end + ' + + # Open an ECS Exec shell in the worker debug container. worker-debug-shell env: #!/usr/bin/env bash From 3b51c868fa7ab76a6c73c1725816fa2fe3ef9d0e Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 15:47:01 +0100 Subject: [PATCH 09/73] debug: test --- .github/workflows/shared_infra.yml | 258 ++++------------------------- 1 file changed, 36 insertions(+), 222 deletions(-) diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index aead7d34..b02b4726 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -53,12 +53,13 @@ env: AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role AWS_REGION: ${{ vars.AWS_REGION }} DOMAIN_NAME: ${{ vars.DOMAIN_NAME }} + PLAN_BUCKET: ${{ vars.AWS_ACCOUNT_ID }}-${{ vars.AWS_REGION }}-${{ vars.PROJECT_NAME }}-tfplan TG_ENABLE_PLAN_ARTIFACTS: ${{ (inputs.tg_action == 'plan' || inputs.tg_action == 'apply_plan') && 'true' || 'false' }} PLAN_ARTIFACT_RUN_ID: ${{ inputs.plan_run_id }} TG_ACTION_LABEL: ${{ (inputs.tg_action == 'apply' || inputs.tg_action == 'apply_plan') && 'Apply' || inputs.tg_action == 'plan' && 'Plan' || inputs.tg_action == 'destroy' && 'Destroy' || inputs.tg_action == 'init' && 'Init' || 'Run' }} jobs: - oidc: + changed_items: runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 @@ -70,233 +71,46 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: ${{ env.TG_ACTION_LABEL }} oidc role infra + - name: Render Terragrunt graph + id: tg_graph uses: ./.github/actions/terragrunt - env: - TG_RESET_PLAN_ARTIFACT_BUCKET: "true" # this ensures that the plan artifact bucket is reset on every run, preventing stale artifacts from being used with: - tg_directory: infra/live/${{ inputs.environment }}/aws/oidc - tg_action: ${{ inputs.tg_action }} + tg_directory: infra/live/${{ inputs.environment }}/aws + tg_action: graph - messaging: - needs: oidc - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} messaging infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/messaging - tg_action: ${{ inputs.tg_action }} - - observability: - needs: oidc - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} observability infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/observability - tg_action: ${{ inputs.tg_action }} - - cognito: - needs: oidc - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} cognito infra - uses: ./.github/actions/terragrunt + - name: Write graph JSON for filter helper + shell: bash env: - TF_VAR_domain_name: ${{ env.DOMAIN_NAME }} - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/cognito - tg_action: ${{ inputs.tg_action }} - - frontend: - needs: - - network - - cognito - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} + TG_GRAPH_JSON: ${{ steps.tg_graph.outputs.tg_graph_json }} + run: | + printf '%s\n' "$TG_GRAPH_JSON" > "$RUNNER_TEMP/terragrunt-graph.json" - - name: ${{ env.TG_ACTION_LABEL }} frontend infra - uses: ./.github/actions/terragrunt + - name: Run changed-items helper + id: changed_items_json + uses: ./.github/actions/just env: - TF_VAR_domain_name: ${{ env.DOMAIN_NAME }} - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/frontend - tg_action: ${{ inputs.tg_action }} - - cluster: - needs: oidc - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} cluster infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/cluster - tg_action: ${{ inputs.tg_action }} - - security: - needs: oidc - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} security infra - id: deploy-security - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/security - tg_action: ${{ inputs.tg_action }} - - database: - needs: - - oidc - - security - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} database infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/database - tg_action: ${{ inputs.tg_action }} - - network: - needs: - - security - - cognito - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} network infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/network - tg_action: ${{ inputs.tg_action }} - - lambdas: - needs: - - oidc - - security - - network - - database - - messaging - runs-on: ubuntu-latest - strategy: - fail-fast: false # this is to prevent terraform lock issues - matrix: - value: ${{ fromJson(inputs.lambda_matrix) }} - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.value }} infra - uses: ./.github/actions/terragrunt + TG_GRAPH_METADATA_PLAN_RUN_ID: ${{ inputs.plan_run_id }} + BUCKET_NAME: ${{ env.PLAN_BUCKET }} with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.value }} - tg_action: ${{ inputs.tg_action }} + aws_region: ${{ env.AWS_REGION }} + justfile_path: justfile.ci + just_action: tg-graph-changed-items-json $RUNNER_TEMP/terragrunt-graph.json ${{ inputs.environment }} - services: - needs: - - oidc - - security - - cluster - - network - - database - - messaging - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - value: ${{ fromJson(inputs.service_matrix) }} - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.value }} bootstrap service infra - uses: ./.github/actions/terragrunt + - name: Print changed Terragrunt graph items + shell: bash env: - TF_VAR_bootstrap: "true" - TF_VAR_bootstrap_image_uri: ${{ inputs.bootstrap_image_uri }} - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.value }} - tg_action: ${{ inputs.tg_action }} + CHANGED_ITEMS_JSON: ${{ steps.changed_items_json.outputs.just_outputs }} + run: | + printf '%s\n' "$CHANGED_ITEMS_JSON" | tee changed-items.json + { + echo "## Changed Terragrunt Graph Items" + echo + echo '```json' + cat changed-items.json + echo + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + # Previous shared infra rollout jobs are intentionally disabled for this + # temporary graph-debug shape. Restore the ordered per-stack jobs after the + # changed-items output is wired into the next iteration. From 2d9d875f55d1f42a854e2264b87d25c4a9ac14a1 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 15:47:14 +0100 Subject: [PATCH 10/73] chore: graph changes --- .github/actions/terragrunt/README.md | 8 +++++-- .github/actions/terragrunt/action.yml | 30 ++++++++++++++++++++++++--- .github/docs/README.md | 2 +- justfile.ci | 15 ++++++++++++++ 4 files changed, 49 insertions(+), 6 deletions(-) diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index 22ed7683..6430765e 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -1,6 +1,6 @@ # Execute Terraform & Terragrunt -This GitHub Action sets up **Terraform** and **Terragrunt** and runs a specified `terragrunt` action: `apply`, `plan`, `apply_plan`, `destroy`, or `init`. When the action needs AWS, the workflow job should configure credentials first. +This GitHub Action sets up **Terraform** and **Terragrunt** and runs a specified `terragrunt` action: `apply`, `plan`, `apply_plan`, `destroy`, `init`, or `graph`. When the action needs AWS, the workflow job should configure credentials first. ## Features @@ -10,6 +10,7 @@ This GitHub Action sets up **Terraform** and **Terragrunt** and runs a specified - Optionally passes Terragrunt variables via JSON tfvars - Supports `plan` mode for producing local saved plan files - Supports `init` mode for outputs-only reads +- Supports `graph` mode for Graphviz JSON graph capture - Relies on shared Terragrunt root hooks for per-stack saved plan artifact upload and download - Exports Terragrunt outputs as compact JSON when state exists @@ -24,7 +25,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden | `aws_region` | AWS region to use | No | `eu-west-2` | | `override_tg_vars` | Terragrunt variables in JSON, written to `override_tg_vars.tfvars.json` | No | `{}` | | `tg_directory` | Directory containing the Terragrunt config | Yes | — | -| `tg_action` | Terragrunt action: `apply`, `plan`, `apply_plan`, `destroy`, or `init` | Yes | `apply` | +| `tg_action` | Terragrunt action: `apply`, `plan`, `apply_plan`, `destroy`, `init`, or `graph` | Yes | `apply` | `override_tg_vars` is written for `apply`, `plan`, and `destroy`, but not for `init`. @@ -33,6 +34,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden | Name | Description | |---|---| | `tg_outputs` | All Terraform outputs in compact JSON. If no state exists, returns `{}` | +| `tg_graph_json` | Terragrunt dependency graph rendered as compact Graphviz JSON. Set only for `tg_action: graph` | ## Behavior - `apply` @@ -45,6 +47,8 @@ The Terragrunt install step is kept in this repo-local action rather than hidden Runs `terragrunt destroy -auto-approve` - `init` Runs `terragrunt init -input=false -reconfigure` and then captures outputs +- `graph` + Installs Graphviz in the action, runs `terragrunt graph-dependencies | dot -Tjson`, and exposes the compact JSON as `tg_graph_json` ## Saved Plan Layout diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index 274b858e..baefe4e9 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -21,7 +21,7 @@ inputs: description: "Module directory to perform action upon" required: true tg_action: - description: "Terragrunt action to perform (`apply`, `plan`, `apply_plan`, `destroy`, or `init` for outputs-only)" + description: "Terragrunt action to perform (`apply`, `plan`, `apply_plan`, `destroy`, `init` for outputs-only, or `graph` for Graphviz JSON)" required: true default: apply @@ -29,6 +29,9 @@ outputs: tg_outputs: description: "All Terraform outputs in JSON format" value: ${{ steps.tg_outputs.outputs.terraform_json }} + tg_graph_json: + description: "Terragrunt dependency graph rendered as compact Graphviz JSON" + value: ${{ steps.tg_graph.outputs.graph_json }} runs: using: "composite" @@ -53,6 +56,13 @@ runs: shell: bash run: terragrunt --version + - name: Install Graphviz + if: inputs.tg_action == 'graph' + shell: bash + run: | + sudo apt-get update + sudo apt-get install -y graphviz + - name: Normalize and write override_tg_vars if: inputs.tg_action == 'apply' || inputs.tg_action == 'plan' || inputs.tg_action == 'destroy' shell: bash @@ -142,14 +152,28 @@ runs: echo "Running init only (no infra changes)..." terragrunt init -input=false -reconfigure ;; + graph) + echo "Graph mode runs in the dedicated graph step." + ;; *) - echo "Unknown tg_action: '${{ inputs.tg_action }}' (expected: apply|plan|apply_plan|destroy|init)" >&2 + echo "Unknown tg_action: '${{ inputs.tg_action }}' (expected: apply|plan|apply_plan|destroy|init|graph)" >&2 exit 2 ;; esac + - name: Capture Terragrunt graph JSON + if: inputs.tg_action == 'graph' + id: tg_graph + shell: bash + working-directory: ${{ inputs.tg_directory }} + run: | + echo "🕸️ Rendering Terragrunt dependency graph..." + graph_json="$(terragrunt graph-dependencies | dot -Tjson | jq -c .)" + echo "graph_json=$graph_json" >> "$GITHUB_OUTPUT" + echo "✅ Terragrunt graph captured." + - name: Capture Terraform Outputs - if: inputs.tg_action != 'destroy' + if: inputs.tg_action != 'destroy' && inputs.tg_action != 'graph' id: tg_outputs shell: bash working-directory: ${{ inputs.tg_directory }} diff --git a/.github/docs/README.md b/.github/docs/README.md index a69eedc3..5480eeb7 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -88,7 +88,7 @@ flowchart LR - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` - Pure ordered infra graph executor. It applies shared stacks first, then runtime stacks, then frontend infrastructure. Shared stacks now include the CloudWatch observability dashboard. It accepts `tg_action` so the same graph can run a normal apply, upload derived per-stack plan artifacts to the dedicated plan bucket under `terragrunt_plan/`, or apply from previously uploaded plan artifacts. The wrapper workflows now pass a single `plan_run_id`, exported to Terragrunt jobs as `PLAN_ARTIFACT_RUN_ID`, while each Terragrunt job configures AWS credentials at job start and then reuses that ambient session in the repo-local Terragrunt action. That means each infra run has one shared run-level metadata artifact (`infra-plan-metadata`) for the whole graph and one separate saved plan bundle per Terragrunt stack or module. Saved-plan transfer is opt-in: the shared workflow sets `TG_ENABLE_PLAN_ARTIFACTS=true` only for `plan` and `apply_plan`. In `plan` mode, the shared Terragrunt root `after_hook` renders and uploads each per-stack plan bundle. In `apply_plan` mode, the shared Terragrunt root `before_hook` downloads the saved plan bundle before `terragrunt apply` runs and fails if the saved metadata says mocked outputs were used. Its visible step labels now follow the high-level operation, so both direct apply and apply-from-plan render as `Apply` while plan still renders as `Plan`. Bootstrap-sensitive edges such as `security -> network` should be modeled with Terragrunt `dependency` blocks plus constrained `mock_outputs` in the live stack so `plan` and `validate` can run before upstream state exists, while `apply` still resolves real outputs. + Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode to render Graphviz JSON, applies `just tg-graph-changed-items` when `plan_run_id` is present, and prints the changed saved-plan items JSON into the job logs and step summary. When `plan_run_id` is empty it prints `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. - `just tg-graph-process` Standalone graph post-processor used by the repo-local `just` wrappers. It reads a saved Graphviz JSON file produced from `terragrunt graph-dependencies | dot -Tjson`, converts that into compact dependency JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. - `just tg-graph-changed-items` diff --git a/justfile.ci b/justfile.ci index ef1ddf64..7ccd1afc 100644 --- a/justfile.ci +++ b/justfile.ci @@ -26,6 +26,21 @@ tg-graph-json environment provider='aws': just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-process "$tmp_graph" "{{environment}}" "{{provider}}" +# Generate only changed saved-plan graph items as a JSON array. +# If TG_GRAPH_METADATA_PLAN_RUN_ID is empty, emit []. +tg-graph-changed-items-json graph_path environment provider='aws': + #!/usr/bin/env bash + set -euo pipefail + cd "{{PROJECT_DIR}}" + + if [[ -z "${TG_GRAPH_METADATA_PLAN_RUN_ID:-}" ]]; then + printf '[]\n' + exit 0 + fi + + just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-changed-items "{{graph_path}}" "{{environment}}" "{{provider}}" + + # Run `tflint` across Terraform module directories. tf-lint-check: #!/bin/bash From d3cd85ae43bbe91f334cd8836cf49f83ff712bc1 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 15:52:11 +0100 Subject: [PATCH 11/73] fix: pass var --- .github/docs/README.md | 2 ++ .github/workflows/shared_infra.yml | 10 ++-------- justfile.ci | 14 ++++++++++++-- 3 files changed, 16 insertions(+), 10 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 5480eeb7..0551dce5 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -89,6 +89,8 @@ flowchart LR Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode to render Graphviz JSON, applies `just tg-graph-changed-items` when `plan_run_id` is present, and prints the changed saved-plan items JSON into the job logs and step summary. When `plan_run_id` is empty it prints `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. +- `just --justfile justfile.ci tg-graph-changed-items-json` + CI helper for the temporary debug workflow. It expects `TG_GRAPH_JSON` from the shared Terragrunt action output, returns `[]` when `TG_GRAPH_METADATA_PLAN_RUN_ID` is empty, and otherwise filters the saved-plan graph down to changed items only. - `just tg-graph-process` Standalone graph post-processor used by the repo-local `just` wrappers. It reads a saved Graphviz JSON file produced from `terragrunt graph-dependencies | dot -Tjson`, converts that into compact dependency JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. - `just tg-graph-changed-items` diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index b02b4726..12a31bbf 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -78,23 +78,17 @@ jobs: tg_directory: infra/live/${{ inputs.environment }}/aws tg_action: graph - - name: Write graph JSON for filter helper - shell: bash - env: - TG_GRAPH_JSON: ${{ steps.tg_graph.outputs.tg_graph_json }} - run: | - printf '%s\n' "$TG_GRAPH_JSON" > "$RUNNER_TEMP/terragrunt-graph.json" - - name: Run changed-items helper id: changed_items_json uses: ./.github/actions/just env: + TG_GRAPH_JSON: ${{ steps.tg_graph.outputs.tg_graph_json }} TG_GRAPH_METADATA_PLAN_RUN_ID: ${{ inputs.plan_run_id }} BUCKET_NAME: ${{ env.PLAN_BUCKET }} with: aws_region: ${{ env.AWS_REGION }} justfile_path: justfile.ci - just_action: tg-graph-changed-items-json $RUNNER_TEMP/terragrunt-graph.json ${{ inputs.environment }} + just_action: tg-graph-changed-items-json ${{ inputs.environment }} - name: Print changed Terragrunt graph items shell: bash diff --git a/justfile.ci b/justfile.ci index 7ccd1afc..1d7a8599 100644 --- a/justfile.ci +++ b/justfile.ci @@ -28,7 +28,8 @@ tg-graph-json environment provider='aws': # Generate only changed saved-plan graph items as a JSON array. # If TG_GRAPH_METADATA_PLAN_RUN_ID is empty, emit []. -tg-graph-changed-items-json graph_path environment provider='aws': +# Requires TG_GRAPH_JSON to be set from the shared Terragrunt action output. +tg-graph-changed-items-json environment provider='aws': #!/usr/bin/env bash set -euo pipefail cd "{{PROJECT_DIR}}" @@ -38,7 +39,16 @@ tg-graph-changed-items-json graph_path environment provider='aws': exit 0 fi - just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-changed-items "{{graph_path}}" "{{environment}}" "{{provider}}" + if [[ -z "${TG_GRAPH_JSON:-}" ]]; then + echo "❌ TG_GRAPH_JSON is required for tg-graph-changed-items-json." + exit 1 + fi + + tmp_graph="$(mktemp)" + trap 'rm -f "$tmp_graph"' EXIT + printf '%s\n' "$TG_GRAPH_JSON" > "$tmp_graph" + + just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-changed-items "$tmp_graph" "{{environment}}" "{{provider}}" # Run `tflint` across Terraform module directories. From 84ed89f1e006446481dbe383ed6631ad69414e63 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 16:07:32 +0100 Subject: [PATCH 12/73] debug: output test --- .github/actions/terragrunt/README.md | 6 +++--- .github/actions/terragrunt/action.yml | 6 +++--- .github/docs/README.md | 2 +- justfile.ci | 5 ++--- 4 files changed, 9 insertions(+), 10 deletions(-) diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index 6430765e..174a2c9d 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -10,7 +10,7 @@ This GitHub Action sets up **Terraform** and **Terragrunt** and runs a specified - Optionally passes Terragrunt variables via JSON tfvars - Supports `plan` mode for producing local saved plan files - Supports `init` mode for outputs-only reads -- Supports `graph` mode for Graphviz JSON graph capture +- Supports `graph` mode for `terragrunt run-all graph-dependencies` rendered as Graphviz JSON - Relies on shared Terragrunt root hooks for per-stack saved plan artifact upload and download - Exports Terragrunt outputs as compact JSON when state exists @@ -34,7 +34,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden | Name | Description | |---|---| | `tg_outputs` | All Terraform outputs in compact JSON. If no state exists, returns `{}` | -| `tg_graph_json` | Terragrunt dependency graph rendered as compact Graphviz JSON. Set only for `tg_action: graph` | +| `tg_graph_json` | Terragrunt `run-all graph-dependencies` rendered as compact Graphviz JSON. Set only for `tg_action: graph` | ## Behavior - `apply` @@ -48,7 +48,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - `init` Runs `terragrunt init -input=false -reconfigure` and then captures outputs - `graph` - Installs Graphviz in the action, runs `terragrunt graph-dependencies | dot -Tjson`, and exposes the compact JSON as `tg_graph_json` + Installs Graphviz in the action, runs `terragrunt run-all graph-dependencies | dot -Tjson`, and exposes the compact JSON as `tg_graph_json` ## Saved Plan Layout diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index baefe4e9..5c8e0b32 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -30,7 +30,7 @@ outputs: description: "All Terraform outputs in JSON format" value: ${{ steps.tg_outputs.outputs.terraform_json }} tg_graph_json: - description: "Terragrunt dependency graph rendered as compact Graphviz JSON" + description: "Terragrunt run-all dependency graph rendered as compact Graphviz JSON" value: ${{ steps.tg_graph.outputs.graph_json }} runs: @@ -167,8 +167,8 @@ runs: shell: bash working-directory: ${{ inputs.tg_directory }} run: | - echo "🕸️ Rendering Terragrunt dependency graph..." - graph_json="$(terragrunt graph-dependencies | dot -Tjson | jq -c .)" + echo "🕸️ Rendering Terragrunt run-all dependency graph..." + graph_json="$(terragrunt run-all graph-dependencies | dot -Tjson | jq -c .)" echo "graph_json=$graph_json" >> "$GITHUB_OUTPUT" echo "✅ Terragrunt graph captured." diff --git a/.github/docs/README.md b/.github/docs/README.md index 0551dce5..8de4782d 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -88,7 +88,7 @@ flowchart LR - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` - Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode to render Graphviz JSON, applies `just tg-graph-changed-items` when `plan_run_id` is present, and prints the changed saved-plan items JSON into the job logs and step summary. When `plan_run_id` is empty it prints `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. + Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode, which runs `terragrunt run-all graph-dependencies` from the target live environment and renders Graphviz JSON. It then applies `just tg-graph-changed-items` when `plan_run_id` is present and prints the changed saved-plan items JSON into the job logs and step summary. When `plan_run_id` is empty it prints `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. - `just --justfile justfile.ci tg-graph-changed-items-json` CI helper for the temporary debug workflow. It expects `TG_GRAPH_JSON` from the shared Terragrunt action output, returns `[]` when `TG_GRAPH_METADATA_PLAN_RUN_ID` is empty, and otherwise filters the saved-plan graph down to changed items only. - `just tg-graph-process` diff --git a/justfile.ci b/justfile.ci index 1d7a8599..0ba8e7de 100644 --- a/justfile.ci +++ b/justfile.ci @@ -39,13 +39,12 @@ tg-graph-changed-items-json environment provider='aws': exit 0 fi + tmp_graph="$(mktemp)" + trap 'rm -f "$tmp_graph"' EXIT if [[ -z "${TG_GRAPH_JSON:-}" ]]; then echo "❌ TG_GRAPH_JSON is required for tg-graph-changed-items-json." exit 1 fi - - tmp_graph="$(mktemp)" - trap 'rm -f "$tmp_graph"' EXIT printf '%s\n' "$TG_GRAPH_JSON" > "$tmp_graph" just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-changed-items "$tmp_graph" "{{environment}}" "{{provider}}" From 32a445480ab83bba4ac31fb4fa68e7c4a8809bb9 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 16:14:07 +0100 Subject: [PATCH 13/73] fix: tg flags for graph --- .github/actions/terragrunt/README.md | 2 +- .github/actions/terragrunt/action.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index 174a2c9d..380ae0d9 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -48,7 +48,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - `init` Runs `terragrunt init -input=false -reconfigure` and then captures outputs - `graph` - Installs Graphviz in the action, runs `terragrunt run-all graph-dependencies | dot -Tjson`, and exposes the compact JSON as `tg_graph_json` + Installs Graphviz in the action, runs `terragrunt run-all graph-dependencies --terragrunt-non-interactive --terragrunt-include-external-dependencies | dot -Tjson`, and exposes the compact JSON as `tg_graph_json` ## Saved Plan Layout diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index 5c8e0b32..dd025a23 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -168,7 +168,7 @@ runs: working-directory: ${{ inputs.tg_directory }} run: | echo "🕸️ Rendering Terragrunt run-all dependency graph..." - graph_json="$(terragrunt run-all graph-dependencies | dot -Tjson | jq -c .)" + graph_json="$(terragrunt run-all graph-dependencies --terragrunt-non-interactive --terragrunt-include-external-dependencies | dot -Tjson | jq -c .)" echo "graph_json=$graph_json" >> "$GITHUB_OUTPUT" echo "✅ Terragrunt graph captured." From 768311fa88e110743acba396a28765f752921240 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 16:29:14 +0100 Subject: [PATCH 14/73] feat: process graph output --- .github/actions/terragrunt/README.md | 6 +- .github/actions/terragrunt/action.yml | 22 ++--- .github/docs/README.md | 8 +- .github/workflows/shared_infra.yml | 2 +- README.md | 11 ++- justfile | 115 ++++++++++++++++++-------- justfile.ci | 8 +- 7 files changed, 107 insertions(+), 65 deletions(-) diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index 380ae0d9..3acef8ea 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -10,7 +10,7 @@ This GitHub Action sets up **Terraform** and **Terragrunt** and runs a specified - Optionally passes Terragrunt variables via JSON tfvars - Supports `plan` mode for producing local saved plan files - Supports `init` mode for outputs-only reads -- Supports `graph` mode for `terragrunt run-all graph-dependencies` rendered as Graphviz JSON +- Supports `graph` mode for raw `terragrunt run-all graph-dependencies` output capture - Relies on shared Terragrunt root hooks for per-stack saved plan artifact upload and download - Exports Terragrunt outputs as compact JSON when state exists @@ -34,7 +34,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden | Name | Description | |---|---| | `tg_outputs` | All Terraform outputs in compact JSON. If no state exists, returns `{}` | -| `tg_graph_json` | Terragrunt `run-all graph-dependencies` rendered as compact Graphviz JSON. Set only for `tg_action: graph` | +| `tg_graph_output` | Raw Terragrunt `run-all graph-dependencies` output. Set only for `tg_action: graph` | ## Behavior - `apply` @@ -48,7 +48,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - `init` Runs `terragrunt init -input=false -reconfigure` and then captures outputs - `graph` - Installs Graphviz in the action, runs `terragrunt run-all graph-dependencies --terragrunt-non-interactive --terragrunt-include-external-dependencies | dot -Tjson`, and exposes the compact JSON as `tg_graph_json` + Runs `terragrunt run-all graph-dependencies --terragrunt-non-interactive --terragrunt-include-external-dependencies` and exposes the raw output as `tg_graph_output` ## Saved Plan Layout diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index dd025a23..1ad26633 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -21,7 +21,7 @@ inputs: description: "Module directory to perform action upon" required: true tg_action: - description: "Terragrunt action to perform (`apply`, `plan`, `apply_plan`, `destroy`, `init` for outputs-only, or `graph` for Graphviz JSON)" + description: "Terragrunt action to perform (`apply`, `plan`, `apply_plan`, `destroy`, `init` for outputs-only, or `graph` for raw run-all graph output)" required: true default: apply @@ -29,9 +29,9 @@ outputs: tg_outputs: description: "All Terraform outputs in JSON format" value: ${{ steps.tg_outputs.outputs.terraform_json }} - tg_graph_json: - description: "Terragrunt run-all dependency graph rendered as compact Graphviz JSON" - value: ${{ steps.tg_graph.outputs.graph_json }} + tg_graph_output: + description: "Raw Terragrunt run-all dependency graph output" + value: ${{ steps.tg_graph.outputs.graph_output }} runs: using: "composite" @@ -56,13 +56,6 @@ runs: shell: bash run: terragrunt --version - - name: Install Graphviz - if: inputs.tg_action == 'graph' - shell: bash - run: | - sudo apt-get update - sudo apt-get install -y graphviz - - name: Normalize and write override_tg_vars if: inputs.tg_action == 'apply' || inputs.tg_action == 'plan' || inputs.tg_action == 'destroy' shell: bash @@ -168,8 +161,11 @@ runs: working-directory: ${{ inputs.tg_directory }} run: | echo "🕸️ Rendering Terragrunt run-all dependency graph..." - graph_json="$(terragrunt run-all graph-dependencies --terragrunt-non-interactive --terragrunt-include-external-dependencies | dot -Tjson | jq -c .)" - echo "graph_json=$graph_json" >> "$GITHUB_OUTPUT" + { + echo "graph_output<> "$GITHUB_OUTPUT" echo "✅ Terragrunt graph captured." - name: Capture Terraform Outputs diff --git a/.github/docs/README.md b/.github/docs/README.md index 8de4782d..b76107b1 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -88,15 +88,13 @@ flowchart LR - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` - Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode, which runs `terragrunt run-all graph-dependencies` from the target live environment and renders Graphviz JSON. It then applies `just tg-graph-changed-items` when `plan_run_id` is present and prints the changed saved-plan items JSON into the job logs and step summary. When `plan_run_id` is empty it prints `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. + Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode, which runs `terragrunt run-all graph-dependencies` from the target live environment and returns the raw Terragrunt graph output. It then applies `just tg-graph-changed-items` when `plan_run_id` is present and prints the changed saved-plan items JSON into the job logs and step summary. When `plan_run_id` is empty it prints `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. - `just --justfile justfile.ci tg-graph-changed-items-json` - CI helper for the temporary debug workflow. It expects `TG_GRAPH_JSON` from the shared Terragrunt action output, returns `[]` when `TG_GRAPH_METADATA_PLAN_RUN_ID` is empty, and otherwise filters the saved-plan graph down to changed items only. + CI helper for the temporary debug workflow. It expects `TG_GRAPH_OUTPUT` from the shared Terragrunt action output, returns `[]` when `TG_GRAPH_METADATA_PLAN_RUN_ID` is empty, and otherwise filters the saved-plan graph down to changed items only. - `just tg-graph-process` - Standalone graph post-processor used by the repo-local `just` wrappers. It reads a saved Graphviz JSON file produced from `terragrunt graph-dependencies | dot -Tjson`, converts that into compact dependency JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. + Standalone graph post-processor used by the repo-local `just` wrappers. It reads a saved raw Terragrunt graph file produced from `terragrunt run-all graph-dependencies`, converts that into compact dependency JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. - `just tg-graph-changed-items` Small local helper layered on top of `just tg-graph-process`. In saved-plan metadata mode it converts the `.items` object into an array and keeps only entries where `has_changes: true`, adding the Terragrunt stack basename as `stack`. -- Local prerequisite for the graph helpers: - `brew install graphviz` - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. - `shared_deploy.yml` Rolls out Lambda code, optional migrations, optional reconciliation Lambdas, ECS task and service updates, and optional frontend deploys. Its multi-step AWS jobs now configure credentials once at job start and let the local `just` and Terragrunt actions reuse that ambient session. The reusable workflow renders its Lambda and ECS CodeDeploy AppSpec files from the shared templates under `config/deploy/`, and its mutating `just` steps should target `justfile.deploy` rather than the repo-root `justfile`. diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 12a31bbf..7a81cdcc 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -82,7 +82,7 @@ jobs: id: changed_items_json uses: ./.github/actions/just env: - TG_GRAPH_JSON: ${{ steps.tg_graph.outputs.tg_graph_json }} + TG_GRAPH_OUTPUT: ${{ steps.tg_graph.outputs.tg_graph_output }} TG_GRAPH_METADATA_PLAN_RUN_ID: ${{ inputs.plan_run_id }} BUCKET_NAME: ${{ env.PLAN_BUCKET }} with: diff --git a/README.md b/README.md index 61643d40..7d4bacea 100644 --- a/README.md +++ b/README.md @@ -98,16 +98,19 @@ Given a Terragrunt file is found at `infra/live/dev/aws/lambda_api/terragrunt.hc just tg dev aws/lambda_api plan ``` -The Terragrunt graph helpers require Graphviz locally because they convert DOT output with `dot -Tjson`: +To print the Terragrunt `run-all` dependency graph as raw Terragrunt output: ```sh -brew install graphviz +just tg-graph dev > graph.json ``` -To print the Terragrunt dependency graph as raw Graphviz JSON: +That runs the same non-interactive Terragrunt graph command used in CI: ```sh -just tg-graph dev > graph.json +cd infra/live/dev/aws +terragrunt run-all graph-dependencies \ + --terragrunt-non-interactive \ + --terragrunt-include-external-dependencies ``` To process that saved graph file into compact dependency JSON: diff --git a/justfile b/justfile index 86f2ec12..7ce7a952 100644 --- a/justfile +++ b/justfile @@ -152,22 +152,18 @@ tg-all op: terragrunt run-all {{op}} -# Print the Terragrunt dependency graph through Graphviz JSON. +# Print the raw Terragrunt run-all dependency graph. tg-graph env provider='aws': #!/usr/bin/env bash set -euo pipefail - cd {{justfile_directory()}} - if ! command -v dot >/dev/null 2>&1; then - echo "❌ graphviz 'dot' is not installed or not on PATH." - exit 1 - fi + cd {{justfile_directory()}}/infra/live/{{env}}/{{provider}} - terragrunt graph-dependencies --terragrunt-working-dir infra/live/{{env}}/{{provider}} \ - | dot -Tjson \ - | jq . + terragrunt run-all graph-dependencies \ + --terragrunt-non-interactive \ + --terragrunt-include-external-dependencies -# Process a saved Graphviz JSON graph file into compact dependency JSON. +# Process a saved raw Terragrunt graph file into compact dependency JSON. # Set TG_GRAPH_METADATA_PLAN_RUN_ID and BUCKET_NAME to join saved-plan metadata. tg-graph-process graph_path env provider='aws': #!/usr/bin/env bash @@ -183,36 +179,85 @@ tg-graph-process graph_path env provider='aws': plan_run_id="${TG_GRAPH_METADATA_PLAN_RUN_ID:-}" aws_region="${AWS_REGION:-}" bucket_name="${BUCKET_NAME:-}" + tmp_nodes="$(mktemp)" + tmp_edges="$(mktemp)" + trap 'rm -f "$tmp_nodes" "$tmp_edges"' EXIT + + awk -F'"' ' + /->/ { + if (NF >= 4) { + print $2 "\t" $4 + } + next + } + /^[[:space:]]*"/ && /;[[:space:]]*$/ { + if (NF >= 2) { + print $2 + } + } + ' "{{graph_path}}" \ + | while IFS= read -r line; do + if [[ "$line" == *$'\t'* ]]; then + printf '%s\n' "$line" >> "$tmp_edges" + elif [[ -n "$line" ]]; then + printf '%s\n' "$line" >> "$tmp_nodes" + fi + done + + nodes_json="$( + { + cat "$tmp_nodes" + awk -F'\t' 'NF >= 2 { print $1; print $2 }' "$tmp_edges" + } \ + | jq -R -s ' + split("\n") + | map(select(length > 0)) + | map(split("/") | last) + | unique + | sort + ' + )" + + edges_json="$( + jq -R -s ' + split("\n") + | map(select(length > 0)) + | map(split("\t")) + | map(select(length == 2)) + | map({ + from: (.[0] | split("/") | last), + to: (.[1] | split("/") | last) + }) + | unique + | sort_by(.from, .to) + ' "$tmp_edges" + )" graph_json="$( - jq -c \ + jq -cn \ --arg environment "{{env}}" \ --arg provider "{{provider}}" \ + --argjson nodes "$nodes_json" \ + --argjson edges "$edges_json" \ + ' + { + environment: $environment, + provider: $provider, + nodes: $nodes, + edges: $edges, + dependencies: ( + reduce $nodes[] as $node + ({}; + .[$node] = ( + $edges + | map(select(.from == $node) | .to) + | unique + | sort + ) + ) + ) + } ' - . as $graph - | ($graph.objects // []) as $objects - | ($graph.edges // []) as $raw_edges - | ($objects | map(.name) | unique | sort) as $nodes - | ($raw_edges | map({from: $objects[.tail].name, to: $objects[.head].name})) as $edges - | { - environment: $environment, - provider: $provider, - nodes: $nodes, - edges: $edges, - dependencies: ( - reduce $nodes[] as $node - ({}; - .[$node] = ( - $edges - | map(select(.from == $node) | .to) - | unique - | sort - ) - ) - ) - } - ' \ - "{{graph_path}}" )" if [[ -z "$plan_run_id" ]]; then diff --git a/justfile.ci b/justfile.ci index 0ba8e7de..98527064 100644 --- a/justfile.ci +++ b/justfile.ci @@ -28,7 +28,7 @@ tg-graph-json environment provider='aws': # Generate only changed saved-plan graph items as a JSON array. # If TG_GRAPH_METADATA_PLAN_RUN_ID is empty, emit []. -# Requires TG_GRAPH_JSON to be set from the shared Terragrunt action output. +# Requires TG_GRAPH_OUTPUT to be set from the shared Terragrunt action output. tg-graph-changed-items-json environment provider='aws': #!/usr/bin/env bash set -euo pipefail @@ -41,11 +41,11 @@ tg-graph-changed-items-json environment provider='aws': tmp_graph="$(mktemp)" trap 'rm -f "$tmp_graph"' EXIT - if [[ -z "${TG_GRAPH_JSON:-}" ]]; then - echo "❌ TG_GRAPH_JSON is required for tg-graph-changed-items-json." + if [[ -z "${TG_GRAPH_OUTPUT:-}" ]]; then + echo "❌ TG_GRAPH_OUTPUT is required for tg-graph-changed-items-json." exit 1 fi - printf '%s\n' "$TG_GRAPH_JSON" > "$tmp_graph" + printf '%s\n' "$TG_GRAPH_OUTPUT" > "$tmp_graph" just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-changed-items "$tmp_graph" "{{environment}}" "{{provider}}" From fb33c670ba94315fed9522d14fca7eb30a474986 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 17:23:44 +0100 Subject: [PATCH 15/73] chore: just tg-all-module-dependencies --- README.md | 10 +++++++-- justfile | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7d4bacea..47985406 100644 --- a/README.md +++ b/README.md @@ -98,10 +98,16 @@ Given a Terragrunt file is found at `infra/live/dev/aws/lambda_api/terragrunt.hc just tg dev aws/lambda_api plan ``` -To print the Terragrunt `run-all` dependency graph as raw Terragrunt output: +To return the direct dependencies for every module as a JSON object: ```sh -just tg-graph dev > graph.json +just tg-all-module-dependencies dev +``` + +If you only need the raw Terragrunt graph output: + +```sh +just tg-graph dev > graph.txt ``` That runs the same non-interactive Terragrunt graph command used in CI: diff --git a/justfile b/justfile index 7ce7a952..660cc3bd 100644 --- a/justfile +++ b/justfile @@ -162,6 +162,72 @@ tg-graph env provider='aws': --terragrunt-non-interactive \ --terragrunt-include-external-dependencies +# Return the direct Terragrunt dependencies for all modules as a JSON object. +tg-all-module-dependencies env provider='aws': + #!/usr/bin/env bash + set -euo pipefail + cd {{justfile_directory()}}/infra/live/{{env}}/{{provider}} + + tmp_graph="$(mktemp)" + tmp_nodes="$(mktemp)" + tmp_edges="$(mktemp)" + trap 'rm -f "$tmp_graph" "$tmp_nodes" "$tmp_edges"' EXIT + + terragrunt run-all graph-dependencies \ + --terragrunt-non-interactive \ + --terragrunt-include-external-dependencies \ + > "$tmp_graph" + + awk -F'"' ' + /->/ && NF >= 4 { + from = $2 + to = $4 + sub(".*/", "", from) + sub(".*/", "", to) + print from "\t" to + next + } + /^[[:space:]]*"/ && /;[[:space:]]*$/ && NF >= 2 { + node = $2 + sub(".*/", "", node) + print node + } + ' "$tmp_graph" \ + | while IFS= read -r line; do + if [[ "$line" == *$'\t'* ]]; then + printf '%s\n' "$line" >> "$tmp_edges" + elif [[ -n "$line" ]]; then + printf '%s\n' "$line" >> "$tmp_nodes" + fi + done + + { + cat "$tmp_nodes" + awk -F'\t' 'NF >= 2 { print $1; print $2 }' "$tmp_edges" + } \ + | sort -u \ + | while IFS= read -r node; do + [[ -n "$node" ]] || continue + deps="$( + awk -F'\t' -v target="$node" '$1 == target { print $2 }' "$tmp_edges" \ + | sort -u \ + | jq -R . \ + | jq -s -c . + )" + printf '%s\t%s\n' "$node" "$deps" + done \ + | jq -R -s ' + split("\n") + | map(select(length > 0)) + | map(split("\t")) + | map(select(length == 2)) + | reduce .[] as $pair + ({}; + .[$pair[0]] = ($pair[1] | fromjson) + ) + ' \ + | jq -c . + # Process a saved raw Terragrunt graph file into compact dependency JSON. # Set TG_GRAPH_METADATA_PLAN_RUN_ID and BUCKET_NAME to join saved-plan metadata. From 8ccc12d562f76350049c7343c07d13b4425b951c Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 19:20:37 +0100 Subject: [PATCH 16/73] chore: output all deps json --- .github/actions/terragrunt/README.md | 2 +- .github/actions/terragrunt/action.yml | 2 +- .github/docs/README.md | 2 +- .github/workflows/shared_infra.yml | 8 +- README.md | 6 + graph.json | 2319 ------------------------- justfile | 265 +-- justfile.ci | 101 +- 8 files changed, 94 insertions(+), 2611 deletions(-) delete mode 100644 graph.json diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index 3acef8ea..bcbef605 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -48,7 +48,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - `init` Runs `terragrunt init -input=false -reconfigure` and then captures outputs - `graph` - Runs `terragrunt run-all graph-dependencies --terragrunt-non-interactive --terragrunt-include-external-dependencies` and exposes the raw output as `tg_graph_output` + Runs `terragrunt run-all graph-dependencies --terragrunt-non-interactive --terragrunt-include-external-dependencies --terragrunt-log-level error` and exposes the raw output as `tg_graph_output` ## Saved Plan Layout diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index 1ad26633..15ac4084 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -163,7 +163,7 @@ runs: echo "🕸️ Rendering Terragrunt run-all dependency graph..." { echo "graph_output<> "$GITHUB_OUTPUT" echo "✅ Terragrunt graph captured." diff --git a/.github/docs/README.md b/.github/docs/README.md index b76107b1..d880774b 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -88,7 +88,7 @@ flowchart LR - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` - Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode, which runs `terragrunt run-all graph-dependencies` from the target live environment and returns the raw Terragrunt graph output. It then applies `just tg-graph-changed-items` when `plan_run_id` is present and prints the changed saved-plan items JSON into the job logs and step summary. When `plan_run_id` is empty it prints `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. + Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode, which runs `terragrunt run-all graph-dependencies` from the target live environment and returns the raw Terragrunt graph output. It then applies `just tg-graph-changed-items` when `plan_run_id` is present, prints the changed saved-plan items JSON into the job logs and step summary, and exposes that same JSON as the reusable-workflow output `changed_items_json`. When `plan_run_id` is empty it prints and returns `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. - `just --justfile justfile.ci tg-graph-changed-items-json` CI helper for the temporary debug workflow. It expects `TG_GRAPH_OUTPUT` from the shared Terragrunt action output, returns `[]` when `TG_GRAPH_METADATA_PLAN_RUN_ID` is empty, and otherwise filters the saved-plan graph down to changed items only. - `just tg-graph-process` diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 7a81cdcc..01aed391 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -38,6 +38,10 @@ on: required: false type: string default: "" + outputs: + changed_items_json: + description: "Changed Terragrunt graph items JSON from the temporary graph-debug path" + value: ${{ jobs.changed_items.outputs.changed_items_json }} concurrency: # only run one instance of workflow at any one time @@ -61,6 +65,8 @@ env: jobs: changed_items: runs-on: ubuntu-latest + outputs: + changed_items_json: ${{ steps.changed_items_json.outputs.just_outputs }} steps: - uses: actions/checkout@v6 with: @@ -88,7 +94,7 @@ jobs: with: aws_region: ${{ env.AWS_REGION }} justfile_path: justfile.ci - just_action: tg-graph-changed-items-json ${{ inputs.environment }} + just_action: tg-graph-output-to-json ${{ inputs.environment }} - name: Print changed Terragrunt graph items shell: bash diff --git a/README.md b/README.md index 47985406..80be8395 100644 --- a/README.md +++ b/README.md @@ -104,6 +104,12 @@ To return the direct dependencies for every module as a JSON object: just tg-all-module-dependencies dev ``` +To test the JSON processor locally through the same split used by CI, run: + +```sh +just tg-graph-json dev +``` + If you only need the raw Terragrunt graph output: ```sh diff --git a/graph.json b/graph.json deleted file mode 100644 index 82081fd2..00000000 --- a/graph.json +++ /dev/null @@ -1,2319 +0,0 @@ -{ - "name": "%1", - "directed": true, - "strict": false, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#fffffe00" - }, - { - "op": "C", - "grad": "none", - "color": "#ffffff" - }, - { - "op": "P", - "points": [ - [ - 0, - 0 - ], - [ - 0, - 180 - ], - [ - 1583.47, - 180 - ], - [ - 1583.47, - 0 - ] - ] - } - ], - "bb": "0,0,1583.5,180", - "xdotversion": "1.7", - "_subgraph_cnt": 0, - "objects": [ - { - "_gvid": 0, - "name": "cluster", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 869.84, - 90, - 35.49, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 869.84, - 84.95 - ], - "align": "c", - "width": 36, - "text": "cluster" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "869.84,90", - "width": "0.9857" - }, - { - "_gvid": 1, - "name": "code_bucket", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 1144.84, - 162, - 57.49, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 1144.84, - 156.95 - ], - "align": "c", - "width": 68.25, - "text": "code_bucket" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "1144.8,162", - "width": "1.597" - }, - { - "_gvid": 2, - "name": "cognito", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 900.84, - 18, - 38.56, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 900.84, - 12.95 - ], - "align": "c", - "width": 40.5, - "text": "cognito" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "900.84,18", - "width": "1.071" - }, - { - "_gvid": 3, - "name": "database", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 291.84, - 90, - 42.65, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 291.84, - 84.95 - ], - "align": "c", - "width": 46.5, - "text": "database" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "291.84,90", - "width": "1.1847" - }, - { - "_gvid": 4, - "name": "security", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 705.84, - 18, - 40.09, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 705.84, - 12.95 - ], - "align": "c", - "width": 42.75, - "text": "security" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "705.84,18", - "width": "1.1137" - }, - { - "_gvid": 5, - "name": "ecr", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 1246.84, - 162, - 27, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 1246.84, - 156.95 - ], - "align": "c", - "width": 16.5, - "text": "ecr" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "1246.8,162", - "width": "0.75" - }, - { - "_gvid": 6, - "name": "frontend", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 1026.84, - 162, - 42.14, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 1026.84, - 156.95 - ], - "align": "c", - "width": 45.75, - "text": "frontend" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "1026.8,162", - "width": "1.1705" - }, - { - "_gvid": 7, - "name": "network", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 774.84, - 90, - 41.12, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 774.84, - 84.95 - ], - "align": "c", - "width": 44.25, - "text": "network" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "774.84,90", - "width": "1.1421" - }, - { - "_gvid": 8, - "name": "lambda_api", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 634.84, - 162, - 54.42, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 634.84, - 156.95 - ], - "align": "c", - "width": 63.75, - "text": "lambda_api" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "634.84,162", - "width": "1.5117" - }, - { - "_gvid": 9, - "name": "messaging", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 559.84, - 90, - 50.33, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 559.84, - 84.95 - ], - "align": "c", - "width": 57.75, - "text": "messaging" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "559.84,90", - "width": "1.398" - }, - { - "_gvid": 10, - "name": "lambda_worker", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 492.84, - 162, - 69.26, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 492.84, - 156.95 - ], - "align": "c", - "width": 85.5, - "text": "lambda_worker" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "492.84,162", - "width": "1.924" - }, - { - "_gvid": 11, - "name": "migrations", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 50.84, - 162, - 50.84, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 50.84, - 156.95 - ], - "align": "c", - "width": 58.5, - "text": "migrations" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "50.84,162", - "width": "1.4122" - }, - { - "_gvid": 12, - "name": "observability", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 1350.84, - 162, - 59.03, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 1350.84, - 156.95 - ], - "align": "c", - "width": 70.5, - "text": "observability" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "1350.8,162", - "width": "1.6397" - }, - { - "_gvid": 13, - "name": "oidc", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 1454.84, - 162, - 27, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 1454.84, - 156.95 - ], - "align": "c", - "width": 23.25, - "text": "oidc" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "1454.8,162", - "width": "0.75" - }, - { - "_gvid": 14, - "name": "rds_reader_tagger", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 196.84, - 162, - 77.45, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 196.84, - 156.95 - ], - "align": "c", - "width": 97.5, - "text": "rds_reader_tagger" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "196.84,162", - "width": "2.1515" - }, - { - "_gvid": 15, - "name": "service_api", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 913.84, - 162, - 52.89, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 913.84, - 156.95 - ], - "align": "c", - "width": 61.5, - "text": "service_api" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "913.84,162", - "width": "1.4691" - }, - { - "_gvid": 16, - "name": "service_worker", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 774.84, - 162, - 67.73, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 774.84, - 156.95 - ], - "align": "c", - "width": 83.25, - "text": "service_worker" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "774.84,162", - "width": "1.8814" - }, - { - "_gvid": 17, - "name": "task_api", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 1541.84, - 162, - 41.63, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 1541.84, - 156.95 - ], - "align": "c", - "width": 45, - "text": "task_api" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "1541.8,162", - "width": "1.1563" - }, - { - "_gvid": 18, - "name": "task_worker", - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "e", - "rect": [ - 348.84, - 162, - 56.47, - 18 - ] - } - ], - "_ldraw_": [ - { - "op": "F", - "size": 14, - "face": "Times-Roman" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "T", - "pt": [ - 348.84, - 156.95 - ], - "align": "c", - "width": 66.75, - "text": "task_worker" - } - ], - "height": "0.5", - "label": "\\N", - "pos": "348.84,162", - "width": "1.5686" - } - ], - "edges": [ - { - "_gvid": 0, - "tail": 3, - "head": 4, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 330.79, - 82.41 - ], - [ - 406.82, - 69.56 - ], - [ - 574.8, - 41.16 - ], - [ - 657.68, - 27.14 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 658.02, - 30.64 - ], - [ - 667.3, - 25.52 - ], - [ - 656.85, - 23.73 - ] - ] - } - ], - "pos": "e,668.79,25.265 330.79,82.415 406.82,69.559 574.8,41.156 657.68,27.143" - }, - { - "_gvid": 2, - "tail": 6, - "head": 7, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 994.87, - 149.87 - ], - [ - 988.59, - 147.82 - ], - [ - 982.03, - 145.77 - ], - [ - 975.84, - 144 - ], - [ - 909.92, - 125.14 - ], - [ - 891.75, - 126.86 - ], - [ - 825.84, - 108 - ], - [ - 822.95, - 107.17 - ], - [ - 819.99, - 106.29 - ], - [ - 817.01, - 105.37 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 818.3, - 102.11 - ], - [ - 807.71, - 102.41 - ], - [ - 816.17, - 108.78 - ] - ] - } - ], - "pos": "e,806.27,101.95 994.87,149.87 988.59,147.82 982.03,145.77 975.84,144 909.92,125.14 891.75,126.86 825.84,108 822.95,107.17 819.99,106.29 817.01,105.37" - }, - { - "_gvid": 1, - "tail": 6, - "head": 2, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 1012.39, - 144.71 - ], - [ - 990.36, - 119.89 - ], - [ - 948.24, - 72.42 - ], - [ - 922.54, - 43.45 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 925.38, - 41.39 - ], - [ - 916.13, - 36.23 - ], - [ - 920.15, - 46.03 - ] - ] - } - ], - "pos": "e,915.12,35.099 1012.4,144.71 990.36,119.89 948.24,72.425 922.54,43.452" - }, - { - "_gvid": 5, - "tail": 8, - "head": 7, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 663.85, - 146.5 - ], - [ - 685.52, - 135.66 - ], - [ - 715.28, - 120.78 - ], - [ - 738.53, - 109.15 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 739.97, - 112.35 - ], - [ - 747.35, - 104.75 - ], - [ - 736.84, - 106.09 - ] - ] - } - ], - "pos": "e,748.7,104.07 663.85,146.5 685.52,135.66 715.28,120.78 738.53,109.15" - }, - { - "_gvid": 6, - "tail": 8, - "head": 9, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 617.44, - 144.76 - ], - [ - 607.88, - 135.84 - ], - [ - 595.85, - 124.61 - ], - [ - 585.25, - 114.72 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 587.95, - 112.44 - ], - [ - 578.25, - 108.18 - ], - [ - 583.17, - 117.56 - ] - ] - } - ], - "pos": "e,577.14,107.15 617.44,144.76 607.88,135.84 595.85,124.61 585.25,114.72" - }, - { - "_gvid": 7, - "tail": 10, - "head": 9, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 509.06, - 144.05 - ], - [ - 517.24, - 135.5 - ], - [ - 527.32, - 124.97 - ], - [ - 536.34, - 115.56 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 538.8, - 118.04 - ], - [ - 543.18, - 108.4 - ], - [ - 533.74, - 113.2 - ] - ] - } - ], - "pos": "e,544.23,107.31 509.06,144.05 517.24,135.5 527.32,124.97 536.34,115.56" - }, - { - "_gvid": 9, - "tail": 11, - "head": 4, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 75.79, - 145.99 - ], - [ - 111, - 125.57 - ], - [ - 177.91, - 89.58 - ], - [ - 239.84, - 72 - ], - [ - 386.05, - 30.48 - ], - [ - 566.81, - 21.33 - ], - [ - 653.99, - 19.42 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 653.96, - 22.92 - ], - [ - 663.89, - 19.23 - ], - [ - 653.83, - 15.92 - ] - ] - } - ], - "pos": "e,665.4,19.201 75.786,145.99 111,125.57 177.91,89.585 239.84,72 386.05,30.482 566.81,21.332 653.99,19.421" - }, - { - "_gvid": 8, - "tail": 11, - "head": 3, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 89.29, - 149.83 - ], - [ - 132.05, - 137.41 - ], - [ - 201.05, - 117.37 - ], - [ - 246.43, - 104.19 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 247.35, - 107.57 - ], - [ - 255.97, - 101.42 - ], - [ - 245.39, - 100.85 - ] - ] - } - ], - "pos": "e,257.43,101 89.293,149.83 132.05,137.41 201.05,117.37 246.43,104.19" - }, - { - "_gvid": 4, - "tail": 7, - "head": 4, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 759.18, - 73.12 - ], - [ - 750.36, - 64.17 - ], - [ - 739.17, - 52.81 - ], - [ - 729.31, - 42.81 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 732.01, - 40.56 - ], - [ - 722.5, - 35.9 - ], - [ - 727.02, - 45.48 - ] - ] - } - ], - "pos": "e,721.43,34.821 759.18,73.116 750.36,64.166 739.17,52.81 729.31,42.815" - }, - { - "_gvid": 3, - "tail": 7, - "head": 2, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 799.43, - 75.34 - ], - [ - 818.64, - 64.67 - ], - [ - 845.54, - 49.72 - ], - [ - 866.81, - 37.91 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 868.37, - 41.04 - ], - [ - 875.41, - 33.13 - ], - [ - 864.97, - 34.92 - ] - ] - } - ], - "pos": "e,876.74,32.391 799.43,75.337 818.64,64.669 845.54,49.724 866.81,37.908" - }, - { - "_gvid": 10, - "tail": 14, - "head": 3, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 219.35, - 144.41 - ], - [ - 232.19, - 134.95 - ], - [ - 248.44, - 122.98 - ], - [ - 262.29, - 112.77 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 264.24, - 115.69 - ], - [ - 270.21, - 106.94 - ], - [ - 260.09, - 110.05 - ] - ] - } - ], - "pos": "e,271.43,106.04 219.35,144.41 232.19,134.95 248.44,122.98 262.29,112.77" - }, - { - "_gvid": 12, - "tail": 15, - "head": 4, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 919.49, - 143.95 - ], - [ - 924.79, - 124.42 - ], - [ - 929.77, - 92.5 - ], - [ - 913.84, - 72 - ], - [ - 894.63, - 47.28 - ], - [ - 811.01, - 32.17 - ], - [ - 755.82, - 24.71 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 756.38, - 21.26 - ], - [ - 746.01, - 23.43 - ], - [ - 755.47, - 28.2 - ] - ] - } - ], - "pos": "e,744.51,23.238 919.49,143.95 924.79,124.42 929.77,92.497 913.84,72 894.63,47.277 811.01,32.174 755.82,24.714" - }, - { - "_gvid": 11, - "tail": 15, - "head": 0, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 903.19, - 144.05 - ], - [ - 898.09, - 135.94 - ], - [ - 891.87, - 126.04 - ], - [ - 886.19, - 117.01 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 889.18, - 115.19 - ], - [ - 880.9, - 108.59 - ], - [ - 883.25, - 118.92 - ] - ] - } - ], - "pos": "e,880.09,107.31 903.19,144.05 898.09,135.94 891.87,126.04 886.19,117.01" - }, - { - "_gvid": 13, - "tail": 15, - "head": 7, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 885.04, - 146.5 - ], - [ - 863.52, - 135.66 - ], - [ - 833.97, - 120.78 - ], - [ - 810.89, - 109.15 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 812.65, - 106.12 - ], - [ - 802.14, - 104.75 - ], - [ - 809.5, - 112.37 - ] - ] - } - ], - "pos": "e,800.79,104.07 885.04,146.5 863.52,135.66 833.97,120.78 810.89,109.15" - }, - { - "_gvid": 15, - "tail": 16, - "head": 4, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 754.73, - 144.57 - ], - [ - 744.28, - 134.97 - ], - [ - 732.19, - 122 - ], - [ - 724.84, - 108 - ], - [ - 715.01, - 89.27 - ], - [ - 710.26, - 65.79 - ], - [ - 707.97, - 47.65 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 711.47, - 47.41 - ], - [ - 706.93, - 37.84 - ], - [ - 704.51, - 48.15 - ] - ] - } - ], - "pos": "e,706.77,36.332 754.73,144.57 744.28,134.97 732.19,122 724.84,108 715.01,89.269 710.26,65.786 707.97,47.645" - }, - { - "_gvid": 17, - "tail": 16, - "head": 9, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 733.13, - 147.42 - ], - [ - 696.9, - 135.62 - ], - [ - 644.46, - 118.55 - ], - [ - 606.77, - 106.28 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 608.1, - 103.03 - ], - [ - 597.51, - 103.26 - ], - [ - 605.94, - 109.69 - ] - ] - } - ], - "pos": "e,596.07,102.8 733.13,147.42 696.9,135.62 644.46,118.55 606.77,106.28" - }, - { - "_gvid": 14, - "tail": 16, - "head": 0, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 796.87, - 144.76 - ], - [ - 810.09, - 135.02 - ], - [ - 827.05, - 122.53 - ], - [ - 841.29, - 112.04 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 842.99, - 115.13 - ], - [ - 848.97, - 106.38 - ], - [ - 838.84, - 109.49 - ] - ] - } - ], - "pos": "e,850.18,105.48 796.87,144.76 810.09,135.02 827.05,122.53 841.29,112.04" - }, - { - "_gvid": 16, - "tail": 16, - "head": 7, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 774.84, - 143.7 - ], - [ - 774.84, - 136.41 - ], - [ - 774.84, - 127.73 - ], - [ - 774.84, - 119.54 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 778.34, - 119.62 - ], - [ - 774.84, - 109.62 - ], - [ - 771.34, - 119.62 - ] - ] - } - ], - "pos": "e,774.84,108.1 774.84,143.7 774.84,136.41 774.84,127.73 774.84,119.54" - }, - { - "_gvid": 19, - "tail": 18, - "head": 9, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 387.05, - 148.32 - ], - [ - 422.4, - 136.6 - ], - [ - 475.03, - 119.13 - ], - [ - 512.92, - 106.57 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 513.85, - 109.95 - ], - [ - 522.24, - 103.47 - ], - [ - 511.65, - 103.3 - ] - ] - } - ], - "pos": "e,523.68,103 387.05,148.32 422.4,136.6 475.03,119.13 512.92,106.57" - }, - { - "_gvid": 18, - "tail": 18, - "head": 3, - "_draw_": [ - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "b", - "points": [ - [ - 335.04, - 144.05 - ], - [ - 328.22, - 135.68 - ], - [ - 319.85, - 125.4 - ], - [ - 312.31, - 116.13 - ] - ] - } - ], - "_hdraw_": [ - { - "op": "S", - "style": "solid" - }, - { - "op": "c", - "grad": "none", - "color": "#000000" - }, - { - "op": "C", - "grad": "none", - "color": "#000000" - }, - { - "op": "P", - "points": [ - [ - 315.1, - 114.03 - ], - [ - 306.07, - 108.48 - ], - [ - 309.67, - 118.45 - ] - ] - } - ], - "pos": "e,305.12,107.31 335.04,144.05 328.22,135.68 319.85,125.4 312.31,116.13" - } - ] -} diff --git a/justfile b/justfile index 660cc3bd..8cf3a057 100644 --- a/justfile +++ b/justfile @@ -158,275 +158,20 @@ tg-graph env provider='aws': set -euo pipefail cd {{justfile_directory()}}/infra/live/{{env}}/{{provider}} - terragrunt run-all graph-dependencies \ - --terragrunt-non-interactive \ - --terragrunt-include-external-dependencies - -# Return the direct Terragrunt dependencies for all modules as a JSON object. -tg-all-module-dependencies env provider='aws': - #!/usr/bin/env bash - set -euo pipefail - cd {{justfile_directory()}}/infra/live/{{env}}/{{provider}} - - tmp_graph="$(mktemp)" - tmp_nodes="$(mktemp)" - tmp_edges="$(mktemp)" - trap 'rm -f "$tmp_graph" "$tmp_nodes" "$tmp_edges"' EXIT - terragrunt run-all graph-dependencies \ --terragrunt-non-interactive \ --terragrunt-include-external-dependencies \ - > "$tmp_graph" - - awk -F'"' ' - /->/ && NF >= 4 { - from = $2 - to = $4 - sub(".*/", "", from) - sub(".*/", "", to) - print from "\t" to - next - } - /^[[:space:]]*"/ && /;[[:space:]]*$/ && NF >= 2 { - node = $2 - sub(".*/", "", node) - print node - } - ' "$tmp_graph" \ - | while IFS= read -r line; do - if [[ "$line" == *$'\t'* ]]; then - printf '%s\n' "$line" >> "$tmp_edges" - elif [[ -n "$line" ]]; then - printf '%s\n' "$line" >> "$tmp_nodes" - fi - done - - { - cat "$tmp_nodes" - awk -F'\t' 'NF >= 2 { print $1; print $2 }' "$tmp_edges" - } \ - | sort -u \ - | while IFS= read -r node; do - [[ -n "$node" ]] || continue - deps="$( - awk -F'\t' -v target="$node" '$1 == target { print $2 }' "$tmp_edges" \ - | sort -u \ - | jq -R . \ - | jq -s -c . - )" - printf '%s\t%s\n' "$node" "$deps" - done \ - | jq -R -s ' - split("\n") - | map(select(length > 0)) - | map(split("\t")) - | map(select(length == 2)) - | reduce .[] as $pair - ({}; - .[$pair[0]] = ($pair[1] | fromjson) - ) - ' \ - | jq -c . - - -# Process a saved raw Terragrunt graph file into compact dependency JSON. -# Set TG_GRAPH_METADATA_PLAN_RUN_ID and BUCKET_NAME to join saved-plan metadata. -tg-graph-process graph_path env provider='aws': - #!/usr/bin/env bash - set -euo pipefail - cd {{justfile_directory()}} - - if [[ ! -f "{{graph_path}}" ]]; then - echo "❌ Graph file '{{graph_path}}' does not exist." - exit 1 - fi - - infra_plan_dir="${INFRA_PLAN_DIR:-terragrunt_plan}" - plan_run_id="${TG_GRAPH_METADATA_PLAN_RUN_ID:-}" - aws_region="${AWS_REGION:-}" - bucket_name="${BUCKET_NAME:-}" - tmp_nodes="$(mktemp)" - tmp_edges="$(mktemp)" - trap 'rm -f "$tmp_nodes" "$tmp_edges"' EXIT - - awk -F'"' ' - /->/ { - if (NF >= 4) { - print $2 "\t" $4 - } - next - } - /^[[:space:]]*"/ && /;[[:space:]]*$/ { - if (NF >= 2) { - print $2 - } - } - ' "{{graph_path}}" \ - | while IFS= read -r line; do - if [[ "$line" == *$'\t'* ]]; then - printf '%s\n' "$line" >> "$tmp_edges" - elif [[ -n "$line" ]]; then - printf '%s\n' "$line" >> "$tmp_nodes" - fi - done - - nodes_json="$( - { - cat "$tmp_nodes" - awk -F'\t' 'NF >= 2 { print $1; print $2 }' "$tmp_edges" - } \ - | jq -R -s ' - split("\n") - | map(select(length > 0)) - | map(split("/") | last) - | unique - | sort - ' - )" - - edges_json="$( - jq -R -s ' - split("\n") - | map(select(length > 0)) - | map(split("\t")) - | map(select(length == 2)) - | map({ - from: (.[0] | split("/") | last), - to: (.[1] | split("/") | last) - }) - | unique - | sort_by(.from, .to) - ' "$tmp_edges" - )" - - graph_json="$( - jq -cn \ - --arg environment "{{env}}" \ - --arg provider "{{provider}}" \ - --argjson nodes "$nodes_json" \ - --argjson edges "$edges_json" \ - ' - { - environment: $environment, - provider: $provider, - nodes: $nodes, - edges: $edges, - dependencies: ( - reduce $nodes[] as $node - ({}; - .[$node] = ( - $edges - | map(select(.from == $node) | .to) - | unique - | sort - ) - ) - ) - } - ' - )" - - if [[ -z "$plan_run_id" ]]; then - printf '%s\n' "$graph_json" - exit 0 - fi + --terragrunt-log-level error - if [[ -z "$bucket_name" ]]; then - echo "❌ BUCKET_NAME is required when TG_GRAPH_METADATA_PLAN_RUN_ID is set." - exit 1 - fi - - metadata_dir="$(mktemp -d)" - trap 'rm -rf "$metadata_dir"' EXIT - run_prefix="s3://${bucket_name}/${infra_plan_dir}/{{env}}/${plan_run_id}/" - - if [[ -n "$aws_region" ]]; then - aws s3 sync \ - "$run_prefix" \ - "$metadata_dir" \ - --region "$aws_region" \ - --exclude "*" \ - --include "*/terragrunt.plan.meta.json" \ - >/dev/null - else - aws s3 sync \ - "$run_prefix" \ - "$metadata_dir" \ - --exclude "*" \ - --include "*/terragrunt.plan.meta.json" \ - >/dev/null - fi - - metadata_files=() - while IFS= read -r -d '' file; do - metadata_files+=("$file") - done < <(find "$metadata_dir" -type f -name 'terragrunt.plan.meta.json' -print0 | sort -z) - - if [[ "${#metadata_files[@]}" -eq 0 ]]; then - jq -n \ - --arg environment "{{env}}" \ - --arg provider "{{provider}}" \ - --arg plan_run_id "$plan_run_id" \ - '{environment: $environment, provider: $provider, plan_run_id: $plan_run_id, items: {}}' - exit 0 - fi - jq -s \ - --arg environment "{{env}}" \ - --arg provider "{{provider}}" \ - --arg plan_run_id "$plan_run_id" \ - --argjson graph "$graph_json" \ - ' - def basename_from_tg_directory: - split("/") | last; - - reduce ( - .[] - | select(.tg_directory != null) - ) as $meta ( - { - environment: $environment, - provider: $provider, - plan_run_id: $plan_run_id, - items: {} - }; - ($meta.tg_directory | basename_from_tg_directory) as $stack - | if ($graph.nodes | index($stack)) == null then - . - else - .items[$stack] = ( - $meta - + { - dependencies: ($graph.dependencies[$stack] // []) - } - ) - end - ) - ' \ - "${metadata_files[@]}" - - -# Return only changed saved-plan graph items as an object array. -# Requires TG_GRAPH_METADATA_PLAN_RUN_ID and BUCKET_NAME so tg-graph-process -# emits saved-plan metadata under `.items`. -tg-graph-changed-items graph_path env provider='aws': +# Run tg-graph once locally and feed the raw output into the CI parser. +tg-graph-json env provider='aws': #!/usr/bin/env bash set -euo pipefail cd {{justfile_directory()}} - just tg-graph-process "{{graph_path}}" "{{env}}" "{{provider}}" \ - | jq -c ' - if (.items? | type) != "object" then - error("tg-graph-changed-items requires tg-graph-process metadata mode.") - else - .items - | to_entries - | map( - select(.value.has_changes == true) - | (.value + {stack: .key}) - ) - end - ' + TG_GRAPH_OUTPUT="$(just tg-graph "{{env}}" "{{provider}}")" \ + just --justfile "{{justfile_directory()}}/justfile.ci" tg-graph-output-to-json "{{env}}" "{{provider}}" # Open an ECS Exec shell in the worker debug container. diff --git a/justfile.ci b/justfile.ci index 98527064..ad60e04f 100644 --- a/justfile.ci +++ b/justfile.ci @@ -12,42 +12,87 @@ EXTRA_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate EXTRA_CONTAI NON_SERVICE_CONTAINER_DIRECTORIES := `just --justfile justfile --evaluate NON_SERVICE_CONTAINER_DIRECTORIES` -# Generate compact Terragrunt graph JSON for workflow matrix use. -# Set TG_GRAPH_METADATA_PLAN_RUN_ID and BUCKET_NAME to join saved-plan metadata. -tg-graph-json environment provider='aws': +# Convert raw Terragrunt graph output from TG_GRAPH_OUTPUT into compact JSON. +tg-graph-output-to-json environment provider='aws': #!/usr/bin/env bash set -euo pipefail cd "{{PROJECT_DIR}}" - tmp_graph="$(mktemp)" - trap 'rm -f "$tmp_graph"' EXIT - - just --justfile "{{PROJECT_DIR}}/justfile" tg-graph "{{environment}}" "{{provider}}" > "$tmp_graph" - just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-process "$tmp_graph" "{{environment}}" "{{provider}}" - - -# Generate only changed saved-plan graph items as a JSON array. -# If TG_GRAPH_METADATA_PLAN_RUN_ID is empty, emit []. -# Requires TG_GRAPH_OUTPUT to be set from the shared Terragrunt action output. -tg-graph-changed-items-json environment provider='aws': - #!/usr/bin/env bash - set -euo pipefail - cd "{{PROJECT_DIR}}" - - if [[ -z "${TG_GRAPH_METADATA_PLAN_RUN_ID:-}" ]]; then - printf '[]\n' - exit 0 - fi - - tmp_graph="$(mktemp)" - trap 'rm -f "$tmp_graph"' EXIT if [[ -z "${TG_GRAPH_OUTPUT:-}" ]]; then - echo "❌ TG_GRAPH_OUTPUT is required for tg-graph-changed-items-json." + echo "❌ TG_GRAPH_OUTPUT is required for tg-graph-output-to-json." exit 1 fi - printf '%s\n' "$TG_GRAPH_OUTPUT" > "$tmp_graph" - just --justfile "{{PROJECT_DIR}}/justfile" tg-graph-changed-items "$tmp_graph" "{{environment}}" "{{provider}}" + tmp_nodes="$(mktemp)" + tmp_edges="$(mktemp)" + trap 'rm -f "$tmp_nodes" "$tmp_edges"' EXIT + + awk -F'"' ' + /->/ && NF >= 4 { + from = $2 + to = $4 + sub(".*/", "", from) + sub(".*/", "", to) + print from "\t" to + next + } + /^[[:space:]]*"/ && /;[[:space:]]*$/ && NF >= 2 { + node = $2 + sub(".*/", "", node) + print node + } + ' <(printf '%s\n' "$TG_GRAPH_OUTPUT") \ + | while IFS= read -r line; do + if [[ "$line" == *$'\t'* ]]; then + printf '%s\n' "$line" >> "$tmp_edges" + elif [[ -n "$line" ]]; then + printf '%s\n' "$line" >> "$tmp_nodes" + fi + done + + { + cat "$tmp_nodes" + awk -F'\t' 'NF >= 2 { print $1; print $2 }' "$tmp_edges" + } \ + | sort -u \ + | while IFS= read -r node; do + [[ -n "$node" ]] || continue + deps="$( + awk -F'\t' -v target="$node" '$1 == target { print $2 }' "$tmp_edges" \ + | sort -u \ + | jq -R . \ + | jq -s -c . + )" + printf '%s\t%s\n' "$node" "$deps" + done \ + | jq -R -s --arg environment "{{environment}}" --arg provider "{{provider}}" ' + split("\n") + | map(select(length > 0)) + | map(split("\t")) + | map(select(length == 2)) + | { + environment: $environment, + provider: $provider, + dependencies: ( + reduce .[] as $pair + ({}; + .[$pair[0]] = ($pair[1] | fromjson) + ) + ) + } + ' \ + | jq -c ' + .dependencies as $deps + | . + { + nodes: ($deps | keys | sort), + edges: ( + $deps + | to_entries + | map(.key as $from | .value[]? | {from: $from, to: .}) + | sort_by(.from, .to) + ) + } + ' # Run `tflint` across Terraform module directories. From 92cfa27b311b9378186a32d7170bed4a2cc34831 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 20:06:24 +0100 Subject: [PATCH 17/73] chore: just tg-graph-waves --- README.md | 4 ++-- justfile | 14 ++++++++++---- justfile.ci | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 80be8395..1ba2456a 100644 --- a/README.md +++ b/README.md @@ -104,10 +104,10 @@ To return the direct dependencies for every module as a JSON object: just tg-all-module-dependencies dev ``` -To test the JSON processor locally through the same split used by CI, run: +To test the wave-matrix processor locally through the same split used by CI, run: ```sh -just tg-graph-json dev +just tg-graph-waves dev ``` If you only need the raw Terragrunt graph output: diff --git a/justfile b/justfile index 8cf3a057..c9af103b 100644 --- a/justfile +++ b/justfile @@ -164,14 +164,20 @@ tg-graph env provider='aws': --terragrunt-log-level error -# Run tg-graph once locally and feed the raw output into the CI parser. -tg-graph-json env provider='aws': +# Run tg-graph once locally and feed the raw output through the CI graph and +# wave processors. +tg-graph-waves env provider='aws': #!/usr/bin/env bash set -euo pipefail cd {{justfile_directory()}} - TG_GRAPH_OUTPUT="$(just tg-graph "{{env}}" "{{provider}}")" \ - just --justfile "{{justfile_directory()}}/justfile.ci" tg-graph-output-to-json "{{env}}" "{{provider}}" + tg_graph_json="$( + TG_GRAPH_OUTPUT="$(just tg-graph "{{env}}" "{{provider}}")" \ + just --justfile "{{justfile_directory()}}/justfile.ci" tg-graph-output-to-json "{{env}}" "{{provider}}" + )" + + TG_GRAPH_JSON="$tg_graph_json" \ + just --justfile "{{justfile_directory()}}/justfile.ci" tg-graph-json-to-waves # Open an ECS Exec shell in the worker debug container. diff --git a/justfile.ci b/justfile.ci index ad60e04f..a2dd19c1 100644 --- a/justfile.ci +++ b/justfile.ci @@ -95,6 +95,47 @@ tg-graph-output-to-json environment provider='aws': ' +# Convert compact Terragrunt graph JSON from TG_GRAPH_JSON into a dependency-safe +# wave matrix JSON array for GitHub Actions. +tg-graph-json-to-waves: + #!/usr/bin/env bash + set -euo pipefail + cd "{{PROJECT_DIR}}" + + if [[ -z "${TG_GRAPH_JSON:-}" ]]; then + echo "❌ TG_GRAPH_JSON is required for tg-graph-json-to-waves." + exit 1 + fi + + jq -cn --argjson graph "$TG_GRAPH_JSON" ' + def build_waves($remaining): + if ($remaining | length) == 0 then + [] + else + ($remaining | to_entries | map(select((.value | length) == 0) | .key) | sort) as $ready + | if ($ready | length) == 0 then + error("Cycle detected in Terragrunt dependency graph.") + else + [{modules: $ready}] + build_waves( + reduce ($remaining | to_entries[]) as $entry + ({}; + if ($ready | index($entry.key)) != null then + . + else + .[$entry.key] = ($entry.value - $ready) + end + ) + ) + end + end; + + ($graph.dependencies // {}) as $deps + | build_waves($deps) + | to_entries + | map({wave: .key, modules: .value.modules}) + ' | jq -c . + + # Run `tflint` across Terraform module directories. tf-lint-check: #!/bin/bash From 4889c54342fcbc99fa170bb730785e7b6db60072 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Wed, 20 May 2026 20:36:00 +0100 Subject: [PATCH 18/73] chore: note --- .github/workflows/shared_infra.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 01aed391..5d9b2842 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -111,6 +111,10 @@ jobs: echo '```' } >> "$GITHUB_STEP_SUMMARY" + + + # do wave-1 2 3 4 jibs + # Previous shared infra rollout jobs are intentionally disabled for this # temporary graph-debug shape. Restore the ordered per-stack jobs after the # changed-items output is wired into the next iteration. From 470d45c3a8509e8fd427a51c47c56b9f5ddeee7b Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 13:12:47 +0100 Subject: [PATCH 19/73] debug: test for modules --- .github/docs/README.md | 10 +--- .github/workflows/shared_infra.yml | 88 ++++++++++++++++++++++++------ 2 files changed, 73 insertions(+), 25 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index d880774b..dad5a0ed 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -88,13 +88,9 @@ flowchart LR - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` - Temporary graph-debug executor. It currently runs a single job that uses the shared repo-local Terragrunt action in `tg_action: graph` mode, which runs `terragrunt run-all graph-dependencies` from the target live environment and returns the raw Terragrunt graph output. It then applies `just tg-graph-changed-items` when `plan_run_id` is present, prints the changed saved-plan items JSON into the job logs and step summary, and exposes that same JSON as the reusable-workflow output `changed_items_json`. When `plan_run_id` is empty it prints and returns `[]`. The previous ordered per-stack rollout jobs are intentionally disabled while this graph-filter shape is being iterated. -- `just --justfile justfile.ci tg-graph-changed-items-json` - CI helper for the temporary debug workflow. It expects `TG_GRAPH_OUTPUT` from the shared Terragrunt action output, returns `[]` when `TG_GRAPH_METADATA_PLAN_RUN_ID` is empty, and otherwise filters the saved-plan graph down to changed items only. -- `just tg-graph-process` - Standalone graph post-processor used by the repo-local `just` wrappers. It reads a saved raw Terragrunt graph file produced from `terragrunt run-all graph-dependencies`, converts that into compact dependency JSON, and when `TG_GRAPH_METADATA_PLAN_RUN_ID` is set it also downloads per-stack `terragrunt.plan.meta.json` files from `BUCKET_NAME` and emits only graph items that have saved-plan metadata. -- `just tg-graph-changed-items` - Small local helper layered on top of `just tg-graph-process`. In saved-plan metadata mode it converts the `.items` object into an array and keeps only entries where `has_changes: true`, adding the Terragrunt stack basename as `stack`. + Temporary wave-placeholder executor. It renders the Terragrunt `run-all graph-dependencies` output, converts that graph into compact JSON, derives dependency-safe wave groups, prints the resulting wave matrix JSON into the logs and step summary, and exposes it as the reusable-workflow output `waves_json`. It then runs placeholder `wave_0`, `wave_1`, and `wave_2` jobs in dependency order, and each placeholder job only runs when its module array is non-empty. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. +- `just --justfile justfile.ci tg-graph-json-to-waves` + CI helper that expects compact graph JSON in `TG_GRAPH_JSON` and returns a sequential JSON array of wave objects like `[{ "wave": 0, "modules": [...] }, ...]`, with each wave containing only modules whose direct dependencies were satisfied by earlier waves. - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. - `shared_deploy.yml` Rolls out Lambda code, optional migrations, optional reconciliation Lambdas, ECS task and service updates, and optional frontend deploys. Its multi-step AWS jobs now configure credentials once at job start and let the local `just` and Terragrunt actions reuse that ambient session. The reusable workflow renders its Lambda and ECS CodeDeploy AppSpec files from the shared templates under `config/deploy/`, and its mutating `just` steps should target `justfile.deploy` rather than the repo-root `justfile`. diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 5d9b2842..40d9e970 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -39,9 +39,12 @@ on: type: string default: "" outputs: + waves_json: + description: "Dependency-safe Terragrunt wave matrix JSON" + value: ${{ jobs.generate_waves.outputs.waves_json }} changed_items_json: - description: "Changed Terragrunt graph items JSON from the temporary graph-debug path" - value: ${{ jobs.changed_items.outputs.changed_items_json }} + description: "Deprecated compatibility output; currently mirrors waves_json" + value: ${{ jobs.generate_waves.outputs.waves_json }} concurrency: # only run one instance of workflow at any one time @@ -63,10 +66,13 @@ env: TG_ACTION_LABEL: ${{ (inputs.tg_action == 'apply' || inputs.tg_action == 'apply_plan') && 'Apply' || inputs.tg_action == 'plan' && 'Plan' || inputs.tg_action == 'destroy' && 'Destroy' || inputs.tg_action == 'init' && 'Init' || 'Run' }} jobs: - changed_items: + generate_waves: runs-on: ubuntu-latest outputs: - changed_items_json: ${{ steps.changed_items_json.outputs.just_outputs }} + waves_json: ${{ steps.waves_json.outputs.just_outputs }} + wave_0_modules: ${{ steps.wave_outputs.outputs.wave_0_modules }} + wave_1_modules: ${{ steps.wave_outputs.outputs.wave_1_modules }} + wave_2_modules: ${{ steps.wave_outputs.outputs.wave_2_modules }} steps: - uses: actions/checkout@v6 with: @@ -84,37 +90,83 @@ jobs: tg_directory: infra/live/${{ inputs.environment }}/aws tg_action: graph - - name: Run changed-items helper - id: changed_items_json + - name: Convert graph output to compact JSON + id: tg_graph_json uses: ./.github/actions/just env: TG_GRAPH_OUTPUT: ${{ steps.tg_graph.outputs.tg_graph_output }} - TG_GRAPH_METADATA_PLAN_RUN_ID: ${{ inputs.plan_run_id }} - BUCKET_NAME: ${{ env.PLAN_BUCKET }} with: - aws_region: ${{ env.AWS_REGION }} justfile_path: justfile.ci just_action: tg-graph-output-to-json ${{ inputs.environment }} - - name: Print changed Terragrunt graph items + - name: Build wave matrix JSON + id: waves_json + uses: ./.github/actions/just + env: + TG_GRAPH_JSON: ${{ steps.tg_graph_json.outputs.just_outputs }} + with: + justfile_path: justfile.ci + just_action: tg-graph-json-to-waves + + - name: Expose placeholder wave module arrays + id: wave_outputs + shell: bash + env: + WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} + run: | + echo "wave_0_modules=$(jq -c '.[0].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" + echo "wave_1_modules=$(jq -c '.[1].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" + echo "wave_2_modules=$(jq -c '.[2].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" + + - name: Print Terragrunt wave matrix shell: bash env: - CHANGED_ITEMS_JSON: ${{ steps.changed_items_json.outputs.just_outputs }} + WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} run: | - printf '%s\n' "$CHANGED_ITEMS_JSON" | tee changed-items.json + printf '%s\n' "$WAVES_JSON" | tee waves.json { - echo "## Changed Terragrunt Graph Items" + echo "## Terragrunt Wave Matrix" echo echo '```json' - cat changed-items.json + cat waves.json echo echo '```' } >> "$GITHUB_STEP_SUMMARY" + wave_0: + needs: generate_waves + if: ${{ needs.generate_waves.outputs.wave_0_modules != '[]' }} + runs-on: ubuntu-latest + steps: + - name: Placeholder wave 0 + env: + MODULES: ${{ needs.generate_waves.outputs.wave_0_modules }} + run: echo "wave_0 modules=$MODULES" + wave_1: + needs: + - generate_waves + - wave_0 + if: ${{ always() && needs.generate_waves.outputs.wave_1_modules != '[]' && (needs.wave_0.result == 'success' || needs.wave_0.result == 'skipped') }} + runs-on: ubuntu-latest + steps: + - name: Placeholder wave 1 + env: + MODULES: ${{ needs.generate_waves.outputs.wave_1_modules }} + run: echo "wave_1 modules=$MODULES" - # do wave-1 2 3 4 jibs + wave_2: + needs: + - generate_waves + - wave_1 + if: ${{ always() && needs.generate_waves.outputs.wave_2_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} + runs-on: ubuntu-latest + steps: + - name: Placeholder wave 2 + env: + MODULES: ${{ needs.generate_waves.outputs.wave_2_modules }} + run: echo "wave_2 modules=$MODULES" - # Previous shared infra rollout jobs are intentionally disabled for this - # temporary graph-debug shape. Restore the ordered per-stack jobs after the - # changed-items output is wired into the next iteration. + # Placeholder wave jobs intentionally stop at logging module arrays. Replace + # them with real per-wave deployment fanout once the runtime override contract + # is restored for every stack type. From dbba76b8bd38bd6c6160e05ebbccf9b7e944167a Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 13:19:31 +0100 Subject: [PATCH 20/73] chore: matrix test --- .github/docs/README.md | 2 +- .github/workflows/shared_infra.yml | 30 ++++++++++++++++++------------ 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index dad5a0ed..41763ca9 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -88,7 +88,7 @@ flowchart LR - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` - Temporary wave-placeholder executor. It renders the Terragrunt `run-all graph-dependencies` output, converts that graph into compact JSON, derives dependency-safe wave groups, prints the resulting wave matrix JSON into the logs and step summary, and exposes it as the reusable-workflow output `waves_json`. It then runs placeholder `wave_0`, `wave_1`, and `wave_2` jobs in dependency order, and each placeholder job only runs when its module array is non-empty. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. + Temporary wave-placeholder executor. It renders the Terragrunt `run-all graph-dependencies` output, converts that graph into compact JSON, derives dependency-safe wave groups, prints the resulting wave matrix JSON into the logs and step summary, and exposes it as the reusable-workflow output `waves_json`. It then runs placeholder `wave_0`, `wave_1`, and `wave_2` jobs in dependency order; each placeholder job only runs when its module array is non-empty and fans out that array as a matrix that currently just prints the module name. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. - `just --justfile justfile.ci tg-graph-json-to-waves` CI helper that expects compact graph JSON in `TG_GRAPH_JSON` and returns a sequential JSON array of wave objects like `[{ "wave": 0, "modules": [...] }, ...]`, with each wave containing only modules whose direct dependencies were satisfied by earlier waves. - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 40d9e970..80188a18 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -137,11 +137,13 @@ jobs: needs: generate_waves if: ${{ needs.generate_waves.outputs.wave_0_modules != '[]' }} runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_0_modules) }} steps: - - name: Placeholder wave 0 - env: - MODULES: ${{ needs.generate_waves.outputs.wave_0_modules }} - run: echo "wave_0 modules=$MODULES" + - name: ${{ matrix.module }} + run: echo "${{ matrix.module }}" wave_1: needs: @@ -149,11 +151,13 @@ jobs: - wave_0 if: ${{ always() && needs.generate_waves.outputs.wave_1_modules != '[]' && (needs.wave_0.result == 'success' || needs.wave_0.result == 'skipped') }} runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_1_modules) }} steps: - - name: Placeholder wave 1 - env: - MODULES: ${{ needs.generate_waves.outputs.wave_1_modules }} - run: echo "wave_1 modules=$MODULES" + - name: ${{ matrix.module }} + run: echo "${{ matrix.module }}" wave_2: needs: @@ -161,11 +165,13 @@ jobs: - wave_1 if: ${{ always() && needs.generate_waves.outputs.wave_2_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_2_modules) }} steps: - - name: Placeholder wave 2 - env: - MODULES: ${{ needs.generate_waves.outputs.wave_2_modules }} - run: echo "wave_2 modules=$MODULES" + - name: ${{ matrix.module }} + run: echo "${{ matrix.module }}" # Placeholder wave jobs intentionally stop at logging module arrays. Replace # them with real per-wave deployment fanout once the runtime override contract From e2ff2aeb603d4ad1cb9995cd4be0b07ab47ec274 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 13:35:42 +0100 Subject: [PATCH 21/73] chore: add dependency on oidc --- infra/live/ci/aws/code_bucket/terragrunt.hcl | 4 ++++ infra/live/ci/aws/ecr/terragrunt.hcl | 4 ++++ infra/live/dev/aws/code_bucket/terragrunt.hcl | 4 ++++ infra/live/dev/aws/ecr/terragrunt.hcl | 4 ++++ infra/live/dev/aws/security/terragrunt.hcl | 4 ++++ infra/live/prod/aws/security/terragrunt.hcl | 4 ++++ 6 files changed, 24 insertions(+) diff --git a/infra/live/ci/aws/code_bucket/terragrunt.hcl b/infra/live/ci/aws/code_bucket/terragrunt.hcl index ed01f2ed..1d6a045b 100644 --- a/infra/live/ci/aws/code_bucket/terragrunt.hcl +++ b/infra/live/ci/aws/code_bucket/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//_shared//code_bucket" } diff --git a/infra/live/ci/aws/ecr/terragrunt.hcl b/infra/live/ci/aws/ecr/terragrunt.hcl index 1d671831..881bf7b4 100644 --- a/infra/live/ci/aws/ecr/terragrunt.hcl +++ b/infra/live/ci/aws/ecr/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//_shared//ecr" } diff --git a/infra/live/dev/aws/code_bucket/terragrunt.hcl b/infra/live/dev/aws/code_bucket/terragrunt.hcl index ed01f2ed..1d6a045b 100644 --- a/infra/live/dev/aws/code_bucket/terragrunt.hcl +++ b/infra/live/dev/aws/code_bucket/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//_shared//code_bucket" } diff --git a/infra/live/dev/aws/ecr/terragrunt.hcl b/infra/live/dev/aws/ecr/terragrunt.hcl index 1d671831..881bf7b4 100644 --- a/infra/live/dev/aws/ecr/terragrunt.hcl +++ b/infra/live/dev/aws/ecr/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//_shared//ecr" } diff --git a/infra/live/dev/aws/security/terragrunt.hcl b/infra/live/dev/aws/security/terragrunt.hcl index 7b418119..cecf3764 100644 --- a/infra/live/dev/aws/security/terragrunt.hcl +++ b/infra/live/dev/aws/security/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//security" } diff --git a/infra/live/prod/aws/security/terragrunt.hcl b/infra/live/prod/aws/security/terragrunt.hcl index 7b418119..cecf3764 100644 --- a/infra/live/prod/aws/security/terragrunt.hcl +++ b/infra/live/prod/aws/security/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//security" } From 938f56adf4eaf8fab2d263b33c2a068ad5c41aba Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 13:39:01 +0100 Subject: [PATCH 22/73] chore: mock code bucket --- infra/live/dev/aws/lambda_api/terragrunt.hcl | 13 +++++++++++++ infra/live/dev/aws/lambda_worker/terragrunt.hcl | 13 +++++++++++++ infra/live/dev/aws/migrations/terragrunt.hcl | 13 +++++++++++++ infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl | 11 +++++++++++ infra/live/prod/aws/lambda_api/terragrunt.hcl | 13 +++++++++++++ infra/live/prod/aws/lambda_worker/terragrunt.hcl | 13 +++++++++++++ infra/live/prod/aws/migrations/terragrunt.hcl | 13 +++++++++++++ .../live/prod/aws/rds_reader_tagger/terragrunt.hcl | 11 +++++++++++ 8 files changed, 100 insertions(+) diff --git a/infra/live/dev/aws/lambda_api/terragrunt.hcl b/infra/live/dev/aws/lambda_api/terragrunt.hcl index 5ff0ef12..e11cf036 100644 --- a/infra/live/dev/aws/lambda_api/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_api/terragrunt.hcl @@ -44,11 +44,24 @@ dependency "messaging" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } +dependency "code_bucket" { + config_path = "${get_original_terragrunt_dir()}/../code_bucket" + + mock_outputs = { + bucket = "mock-code-bucket" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + terraform { source = "../../../../modules//aws//lambda_api" } inputs = merge( + { + code_bucket = dependency.code_bucket.outputs.bucket + }, { network_default_target_group_arn = dependency.network.outputs.default_target_group_arn network_load_balancer_arn = dependency.network.outputs.load_balancer_arn diff --git a/infra/live/dev/aws/lambda_worker/terragrunt.hcl b/infra/live/dev/aws/lambda_worker/terragrunt.hcl index 5d60f8d2..5028ff8b 100644 --- a/infra/live/dev/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_worker/terragrunt.hcl @@ -23,11 +23,24 @@ dependency "messaging" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } +dependency "code_bucket" { + config_path = "${get_original_terragrunt_dir()}/../code_bucket" + + mock_outputs = { + bucket = "mock-code-bucket" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + terraform { source = "../../../../modules//aws//lambda_worker" } inputs = merge( + { + code_bucket = dependency.code_bucket.outputs.bucket + }, dependency.messaging.outputs, { sqs_dlq_alarm_threshold = 1 # fail when any messages are in the DLQ (quick fail for testing) diff --git a/infra/live/dev/aws/migrations/terragrunt.hcl b/infra/live/dev/aws/migrations/terragrunt.hcl index 653e7061..d003d86f 100644 --- a/infra/live/dev/aws/migrations/terragrunt.hcl +++ b/infra/live/dev/aws/migrations/terragrunt.hcl @@ -31,11 +31,24 @@ dependency "database" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } +dependency "code_bucket" { + config_path = "${get_original_terragrunt_dir()}/../code_bucket" + + mock_outputs = { + bucket = "mock-code-bucket" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + terraform { source = "../../../../modules//aws//migrations" } inputs = merge( + { + code_bucket = dependency.code_bucket.outputs.bucket + }, { runtime_security_group_id = dependency.security.outputs.runtime_sg }, diff --git a/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl index ec3ff4d8..04aad825 100644 --- a/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl @@ -16,11 +16,22 @@ dependency "database" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } +dependency "code_bucket" { + config_path = "${get_original_terragrunt_dir()}/../code_bucket" + + mock_outputs = { + bucket = "mock-code-bucket" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + terraform { source = "../../../../modules//aws//rds_reader_tagger" } inputs = { + code_bucket = dependency.code_bucket.outputs.bucket database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint database_name = dependency.database.outputs.database_name diff --git a/infra/live/prod/aws/lambda_api/terragrunt.hcl b/infra/live/prod/aws/lambda_api/terragrunt.hcl index 60cf23d2..6f2e8d65 100644 --- a/infra/live/prod/aws/lambda_api/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_api/terragrunt.hcl @@ -44,11 +44,24 @@ dependency "messaging" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } +dependency "code_bucket" { + config_path = "${get_original_terragrunt_dir()}/../code_bucket" + + mock_outputs = { + bucket = "mock-code-bucket" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + terraform { source = "../../../../modules//aws//lambda_api" } inputs = merge( + { + code_bucket = dependency.code_bucket.outputs.bucket + }, { network_default_target_group_arn = dependency.network.outputs.default_target_group_arn network_load_balancer_arn = dependency.network.outputs.load_balancer_arn diff --git a/infra/live/prod/aws/lambda_worker/terragrunt.hcl b/infra/live/prod/aws/lambda_worker/terragrunt.hcl index c5981ca3..8e2dfb29 100644 --- a/infra/live/prod/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_worker/terragrunt.hcl @@ -23,11 +23,24 @@ dependency "messaging" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } +dependency "code_bucket" { + config_path = "${get_original_terragrunt_dir()}/../code_bucket" + + mock_outputs = { + bucket = "mock-code-bucket" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + terraform { source = "../../../../modules//aws//lambda_worker" } inputs = merge( + { + code_bucket = dependency.code_bucket.outputs.bucket + }, dependency.messaging.outputs, { sqs_dlq_alarm_threshold = 5 # fail when there are 5 messages in the DLQ diff --git a/infra/live/prod/aws/migrations/terragrunt.hcl b/infra/live/prod/aws/migrations/terragrunt.hcl index 653e7061..d003d86f 100644 --- a/infra/live/prod/aws/migrations/terragrunt.hcl +++ b/infra/live/prod/aws/migrations/terragrunt.hcl @@ -31,11 +31,24 @@ dependency "database" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } +dependency "code_bucket" { + config_path = "${get_original_terragrunt_dir()}/../code_bucket" + + mock_outputs = { + bucket = "mock-code-bucket" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + terraform { source = "../../../../modules//aws//migrations" } inputs = merge( + { + code_bucket = dependency.code_bucket.outputs.bucket + }, { runtime_security_group_id = dependency.security.outputs.runtime_sg }, diff --git a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl index ec3ff4d8..04aad825 100644 --- a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl @@ -16,11 +16,22 @@ dependency "database" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } +dependency "code_bucket" { + config_path = "${get_original_terragrunt_dir()}/../code_bucket" + + mock_outputs = { + bucket = "mock-code-bucket" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + terraform { source = "../../../../modules//aws//rds_reader_tagger" } inputs = { + code_bucket = dependency.code_bucket.outputs.bucket database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint database_name = dependency.database.outputs.database_name From 5b6267e49bd611dd2f126a04ff66f5c4de447286 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 13:45:18 +0100 Subject: [PATCH 23/73] chore: ecr deps --- REPO_INSTRUCTIONS.md | 1 + infra/live/dev/aws/service_api/terragrunt.hcl | 12 ++++++++++++ infra/live/dev/aws/service_worker/terragrunt.hcl | 12 ++++++++++++ infra/live/prod/aws/lambda_api/terragrunt.hcl | 13 ------------- infra/live/prod/aws/lambda_worker/terragrunt.hcl | 13 ------------- infra/live/prod/aws/migrations/terragrunt.hcl | 13 ------------- .../live/prod/aws/rds_reader_tagger/terragrunt.hcl | 11 ----------- 7 files changed, 25 insertions(+), 50 deletions(-) diff --git a/REPO_INSTRUCTIONS.md b/REPO_INSTRUCTIONS.md index ecbd70df..56184580 100644 --- a/REPO_INSTRUCTIONS.md +++ b/REPO_INSTRUCTIONS.md @@ -105,6 +105,7 @@ These instructions apply to the entire repository. - verify runtime type (Lambda/ECS), deploy mode, and (for ECS) connection type and load-balancer shape - verify required infra resources exist (CodeDeploy app/deployment group, listeners/target groups, alarms, VPC link if applicable) +- before adding a Terragrunt `dependency` or `dependencies` path, verify the target live stack actually exists in that environment/repo slice - when changing reusable workflow contracts, compare every caller `with:` block to the callee `workflow_call.inputs` - when adding or renaming Terraform module `output` values that are intended for Terragrunt `dependency..outputs` passthrough, verify every downstream consumer wrapper declares a `variable` with the exact same name - if that same-name output-to-variable contract does not hold yet, do not leave it implicit: either add the matching variables, or call out the mismatch explicitly before closing the task diff --git a/infra/live/dev/aws/service_api/terragrunt.hcl b/infra/live/dev/aws/service_api/terragrunt.hcl index 4c136329..8495dba7 100644 --- a/infra/live/dev/aws/service_api/terragrunt.hcl +++ b/infra/live/dev/aws/service_api/terragrunt.hcl @@ -2,6 +2,18 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependency "ecr" { + config_path = "${get_original_terragrunt_dir()}/../ecr" + + mock_outputs = { + repository_url = "111111111111.dkr.ecr.eu-west-2.amazonaws.com/mock-ecr" + repository_name = "mock-ecr" + repository_arn = "arn:aws:ecr:eu-west-2:111111111111:repository/mock-ecr" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + dependency "security" { config_path = "${get_original_terragrunt_dir()}/../security" diff --git a/infra/live/dev/aws/service_worker/terragrunt.hcl b/infra/live/dev/aws/service_worker/terragrunt.hcl index 6e0a23c5..9c1d3c2d 100644 --- a/infra/live/dev/aws/service_worker/terragrunt.hcl +++ b/infra/live/dev/aws/service_worker/terragrunt.hcl @@ -2,6 +2,18 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependency "ecr" { + config_path = "${get_original_terragrunt_dir()}/../ecr" + + mock_outputs = { + repository_url = "111111111111.dkr.ecr.eu-west-2.amazonaws.com/mock-ecr" + repository_name = "mock-ecr" + repository_arn = "arn:aws:ecr:eu-west-2:111111111111:repository/mock-ecr" + } + + mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] +} + dependency "security" { config_path = "${get_original_terragrunt_dir()}/../security" diff --git a/infra/live/prod/aws/lambda_api/terragrunt.hcl b/infra/live/prod/aws/lambda_api/terragrunt.hcl index 6f2e8d65..60cf23d2 100644 --- a/infra/live/prod/aws/lambda_api/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_api/terragrunt.hcl @@ -44,24 +44,11 @@ dependency "messaging" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -dependency "code_bucket" { - config_path = "${get_original_terragrunt_dir()}/../code_bucket" - - mock_outputs = { - bucket = "mock-code-bucket" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - terraform { source = "../../../../modules//aws//lambda_api" } inputs = merge( - { - code_bucket = dependency.code_bucket.outputs.bucket - }, { network_default_target_group_arn = dependency.network.outputs.default_target_group_arn network_load_balancer_arn = dependency.network.outputs.load_balancer_arn diff --git a/infra/live/prod/aws/lambda_worker/terragrunt.hcl b/infra/live/prod/aws/lambda_worker/terragrunt.hcl index 8e2dfb29..c5981ca3 100644 --- a/infra/live/prod/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_worker/terragrunt.hcl @@ -23,24 +23,11 @@ dependency "messaging" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -dependency "code_bucket" { - config_path = "${get_original_terragrunt_dir()}/../code_bucket" - - mock_outputs = { - bucket = "mock-code-bucket" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - terraform { source = "../../../../modules//aws//lambda_worker" } inputs = merge( - { - code_bucket = dependency.code_bucket.outputs.bucket - }, dependency.messaging.outputs, { sqs_dlq_alarm_threshold = 5 # fail when there are 5 messages in the DLQ diff --git a/infra/live/prod/aws/migrations/terragrunt.hcl b/infra/live/prod/aws/migrations/terragrunt.hcl index d003d86f..653e7061 100644 --- a/infra/live/prod/aws/migrations/terragrunt.hcl +++ b/infra/live/prod/aws/migrations/terragrunt.hcl @@ -31,24 +31,11 @@ dependency "database" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -dependency "code_bucket" { - config_path = "${get_original_terragrunt_dir()}/../code_bucket" - - mock_outputs = { - bucket = "mock-code-bucket" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - terraform { source = "../../../../modules//aws//migrations" } inputs = merge( - { - code_bucket = dependency.code_bucket.outputs.bucket - }, { runtime_security_group_id = dependency.security.outputs.runtime_sg }, diff --git a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl index 04aad825..ec3ff4d8 100644 --- a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl @@ -16,22 +16,11 @@ dependency "database" { mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } -dependency "code_bucket" { - config_path = "${get_original_terragrunt_dir()}/../code_bucket" - - mock_outputs = { - bucket = "mock-code-bucket" - } - - mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] -} - terraform { source = "../../../../modules//aws//rds_reader_tagger" } inputs = { - code_bucket = dependency.code_bucket.outputs.bucket database_credentials_secret_arn = dependency.database.outputs.database_credentials_secret_arn database_readwrite_endpoint = dependency.database.outputs.database_readwrite_endpoint database_name = dependency.database.outputs.database_name From 591ead1f87028d41174ed4deb4726cd96c69283b Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 13:49:09 +0100 Subject: [PATCH 24/73] chore: no need for dev release setups --- .github/docs/README.md | 2 +- .github/workflows/dev_infra_apply.yml | 22 ++++------------------ 2 files changed, 5 insertions(+), 19 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 41763ca9..6907d6d9 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -109,7 +109,7 @@ flowchart LR ### Wrapper Workflows - `dev_infra_apply.yml` - Entry point for dev infra apply. + Entry point for dev infra apply. It currently calls the shared infra workflow directly with empty placeholder values for `code_bucket`, `lambda_matrix`, `bootstrap_image_uri`, and `service_matrix`, because the temporary wave-placeholder executor does not yet consume the old discovery/artifact inputs. - `dev_infra_plan.yml` Entry point for dev infra plan. It discovers runtime directories, prepares dev artifact references, and then runs the shared infra wrapper in direct-input `plan` mode. - `dev_infra_plan_and_apply.yml` diff --git a/.github/workflows/dev_infra_apply.yml b/.github/workflows/dev_infra_apply.yml index 1c088d5d..ee4b2974 100644 --- a/.github/workflows/dev_infra_apply.yml +++ b/.github/workflows/dev_infra_apply.yml @@ -9,27 +9,13 @@ permissions: actions: read jobs: - setup: - name: Discover - uses: ./.github/workflows/shared_directories_get.yml - - code: - name: Artifacts - uses: ./.github/workflows/shared_infra_releases.yml - with: - environment: dev - infra_version: ${{ github.sha }} - infra: name: Apply - needs: - - setup - - code uses: ./.github/workflows/shared_infra_apply.yml with: environment: dev infra_version: ${{ github.sha }} - code_bucket: ${{ needs.code.outputs.code_bucket }} - lambda_matrix: ${{ needs.setup.outputs.lambda_dirs }} - bootstrap_image_uri: ${{ needs.code.outputs.bootstrap_image_uri }} - service_matrix: ${{ needs.setup.outputs.ecs_service_dirs }} + code_bucket: "" + lambda_matrix: "[]" + bootstrap_image_uri: "" + service_matrix: "[]" From 9587c948148fbf2304a3941f2131568c3d188cd5 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 13:51:03 +0100 Subject: [PATCH 25/73] chore: same for plan --- .github/docs/README.md | 2 +- .github/workflows/dev_infra_plan.yml | 22 ++++------------------ 2 files changed, 5 insertions(+), 19 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 6907d6d9..4db9ad74 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -111,7 +111,7 @@ flowchart LR - `dev_infra_apply.yml` Entry point for dev infra apply. It currently calls the shared infra workflow directly with empty placeholder values for `code_bucket`, `lambda_matrix`, `bootstrap_image_uri`, and `service_matrix`, because the temporary wave-placeholder executor does not yet consume the old discovery/artifact inputs. - `dev_infra_plan.yml` - Entry point for dev infra plan. It discovers runtime directories, prepares dev artifact references, and then runs the shared infra wrapper in direct-input `plan` mode. + Entry point for dev infra plan. It currently calls the shared infra plan wrapper directly with empty placeholder values for `code_bucket`, `lambda_matrix`, `bootstrap_image_uri`, and `service_matrix`, because the temporary wave-placeholder executor does not yet consume the old discovery/artifact inputs. - `dev_infra_plan_and_apply.yml` Entry point for dev infra plan-then-apply. It captures the current workflow `run_id` as plan context, runs the shared infra wrapper in direct-input `plan` mode so that the wrapper emits both plan artifacts and `infra-plan-metadata`, and then reruns the same ordered infra graph in metadata-backed `apply_plan` mode. - `prod_infra_plan.yml` diff --git a/.github/workflows/dev_infra_plan.yml b/.github/workflows/dev_infra_plan.yml index d50eb069..34a661df 100644 --- a/.github/workflows/dev_infra_plan.yml +++ b/.github/workflows/dev_infra_plan.yml @@ -9,27 +9,13 @@ permissions: actions: read jobs: - setup: - name: Discover - uses: ./.github/workflows/shared_directories_get.yml - - code: - name: Artifacts - uses: ./.github/workflows/shared_infra_releases.yml - with: - environment: dev - infra_version: ${{ github.sha }} - infra: name: Plan - needs: - - setup - - code uses: ./.github/workflows/shared_infra_plan.yml with: environment: dev infra_version: ${{ github.sha }} - code_bucket: ${{ needs.code.outputs.code_bucket }} - lambda_matrix: ${{ needs.setup.outputs.lambda_dirs }} - bootstrap_image_uri: ${{ needs.code.outputs.bootstrap_image_uri }} - service_matrix: ${{ needs.setup.outputs.ecs_service_dirs }} + code_bucket: "" + lambda_matrix: "[]" + bootstrap_image_uri: "" + service_matrix: "[]" From 3791b79a674a38a63a18a5cf12c78fea12dbfa6c Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 14:01:26 +0100 Subject: [PATCH 26/73] chore: rm unused inputs --- .github/docs/README.md | 6 ++++-- .github/workflows/dev_infra_apply.yml | 3 --- .github/workflows/dev_infra_plan.yml | 3 --- .github/workflows/prod_infra_apply.yml | 3 --- .github/workflows/prod_infra_plan.yml | 3 --- .github/workflows/shared_infra.yml | 12 ----------- .github/workflows/shared_infra_apply.yml | 15 -------------- .../shared_infra_apply_from_plan.yml | 9 --------- .github/workflows/shared_infra_plan.yml | 20 +------------------ REPO_INSTRUCTIONS.md | 1 + 10 files changed, 6 insertions(+), 69 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 4db9ad74..f5e37eab 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -92,6 +92,8 @@ flowchart LR - `just --justfile justfile.ci tg-graph-json-to-waves` CI helper that expects compact graph JSON in `TG_GRAPH_JSON` and returns a sequential JSON array of wave objects like `[{ "wave": 0, "modules": [...] }, ...]`, with each wave containing only modules whose direct dependencies were satisfied by earlier waves. - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. +- The shared infra wrappers no longer accept `lambda_matrix` or `service_matrix`. Infra selection now comes from the Terragrunt dependency graph and derived waves, not from precomputed runtime directory matrices. +- The shared infra wrappers no longer accept `code_bucket` either. The current graph-wave placeholder path only needs `environment`, `infra_version`, optional `bootstrap_image_uri`, and the Terragrunt action context. - `shared_deploy.yml` Rolls out Lambda code, optional migrations, optional reconciliation Lambdas, ECS task and service updates, and optional frontend deploys. Its multi-step AWS jobs now configure credentials once at job start and let the local `just` and Terragrunt actions reuse that ambient session. The reusable workflow renders its Lambda and ECS CodeDeploy AppSpec files from the shared templates under `config/deploy/`, and its mutating `just` steps should target `justfile.deploy` rather than the repo-root `justfile`. @@ -109,9 +111,9 @@ flowchart LR ### Wrapper Workflows - `dev_infra_apply.yml` - Entry point for dev infra apply. It currently calls the shared infra workflow directly with empty placeholder values for `code_bucket`, `lambda_matrix`, `bootstrap_image_uri`, and `service_matrix`, because the temporary wave-placeholder executor does not yet consume the old discovery/artifact inputs. + Entry point for dev infra apply. It currently calls the shared infra workflow directly with an empty placeholder `bootstrap_image_uri`, because the temporary wave-placeholder executor does not yet consume that old artifact input. - `dev_infra_plan.yml` - Entry point for dev infra plan. It currently calls the shared infra plan wrapper directly with empty placeholder values for `code_bucket`, `lambda_matrix`, `bootstrap_image_uri`, and `service_matrix`, because the temporary wave-placeholder executor does not yet consume the old discovery/artifact inputs. + Entry point for dev infra plan. It currently calls the shared infra plan wrapper directly with an empty placeholder `bootstrap_image_uri`, because the temporary wave-placeholder executor does not yet consume that old artifact input. - `dev_infra_plan_and_apply.yml` Entry point for dev infra plan-then-apply. It captures the current workflow `run_id` as plan context, runs the shared infra wrapper in direct-input `plan` mode so that the wrapper emits both plan artifacts and `infra-plan-metadata`, and then reruns the same ordered infra graph in metadata-backed `apply_plan` mode. - `prod_infra_plan.yml` diff --git a/.github/workflows/dev_infra_apply.yml b/.github/workflows/dev_infra_apply.yml index ee4b2974..1ac7f726 100644 --- a/.github/workflows/dev_infra_apply.yml +++ b/.github/workflows/dev_infra_apply.yml @@ -15,7 +15,4 @@ jobs: with: environment: dev infra_version: ${{ github.sha }} - code_bucket: "" - lambda_matrix: "[]" bootstrap_image_uri: "" - service_matrix: "[]" diff --git a/.github/workflows/dev_infra_plan.yml b/.github/workflows/dev_infra_plan.yml index 34a661df..43e87210 100644 --- a/.github/workflows/dev_infra_plan.yml +++ b/.github/workflows/dev_infra_plan.yml @@ -15,7 +15,4 @@ jobs: with: environment: dev infra_version: ${{ github.sha }} - code_bucket: "" - lambda_matrix: "[]" bootstrap_image_uri: "" - service_matrix: "[]" diff --git a/.github/workflows/prod_infra_apply.yml b/.github/workflows/prod_infra_apply.yml index 3fcbd64a..1f57574d 100644 --- a/.github/workflows/prod_infra_apply.yml +++ b/.github/workflows/prod_infra_apply.yml @@ -26,7 +26,4 @@ jobs: with: environment: prod infra_version: 0.19.13 - code_bucket: ${{ needs.get_build.outputs.code_bucket }} - lambda_matrix: ${{ needs.get_build.outputs.lambda_version_files }} bootstrap_image_uri: ${{ needs.get_build.outputs.bootstrap_image_uri }} - service_matrix: ${{ needs.get_build.outputs.ecs_service_matrix }} diff --git a/.github/workflows/prod_infra_plan.yml b/.github/workflows/prod_infra_plan.yml index 35f94ea8..6170eea9 100644 --- a/.github/workflows/prod_infra_plan.yml +++ b/.github/workflows/prod_infra_plan.yml @@ -47,7 +47,4 @@ jobs: with: environment: prod infra_version: ${{ inputs.infra_version }} - code_bucket: ${{ needs.get_build.outputs.code_bucket }} - lambda_matrix: ${{ needs.get_build.outputs.lambda_version_files }} bootstrap_image_uri: ${{ needs.get_build.outputs.bootstrap_image_uri }} - service_matrix: ${{ needs.get_build.outputs.ecs_service_matrix }} diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 80188a18..cbc246c3 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -11,23 +11,11 @@ on: description: "Version of infrastructure (terraform) to be deployed" required: true type: string - code_bucket: - description: "Bucket containing build artifacts" - required: true - type: string bootstrap_image_uri: description: "Bootstrap ECS image URI" required: false type: string default: "" - lambda_matrix: - required: false - type: string - default: "[]" - service_matrix: - required: false - type: string - default: "[]" tg_action: description: "Terragrunt action to run across the infra graph" required: false diff --git a/.github/workflows/shared_infra_apply.yml b/.github/workflows/shared_infra_apply.yml index 98ec6e1a..5c03610e 100644 --- a/.github/workflows/shared_infra_apply.yml +++ b/.github/workflows/shared_infra_apply.yml @@ -11,23 +11,11 @@ on: description: "Version of infrastructure (terraform) to be applied" required: true type: string - code_bucket: - description: "Bucket containing build artifacts" - required: true - type: string bootstrap_image_uri: description: "Bootstrap ECS image URI" required: false type: string default: "" - lambda_matrix: - required: false - type: string - default: "[]" - service_matrix: - required: false - type: string - default: "[]" permissions: id-token: write @@ -40,8 +28,5 @@ jobs: with: environment: ${{ inputs.environment }} infra_version: ${{ inputs.infra_version }} - code_bucket: ${{ inputs.code_bucket }} - lambda_matrix: ${{ inputs.lambda_matrix }} bootstrap_image_uri: ${{ inputs.bootstrap_image_uri }} - service_matrix: ${{ inputs.service_matrix }} tg_action: apply diff --git a/.github/workflows/shared_infra_apply_from_plan.yml b/.github/workflows/shared_infra_apply_from_plan.yml index 8b1a8bdb..d96de1a7 100644 --- a/.github/workflows/shared_infra_apply_from_plan.yml +++ b/.github/workflows/shared_infra_apply_from_plan.yml @@ -26,10 +26,7 @@ jobs: runs-on: ubuntu-latest outputs: infra_version: ${{ steps.read_metadata.outputs.infra_version }} - code_bucket: ${{ steps.read_metadata.outputs.code_bucket }} - lambda_matrix: ${{ steps.read_metadata.outputs.lambda_matrix }} bootstrap_image_uri: ${{ steps.read_metadata.outputs.bootstrap_image_uri }} - service_matrix: ${{ steps.read_metadata.outputs.service_matrix }} steps: - uses: actions/checkout@v6 @@ -54,10 +51,7 @@ jobs: shell: bash run: | echo "infra_version=$(jq -r '.infra_version' plan-metadata.json)" >> "$GITHUB_OUTPUT" - echo "code_bucket=$(jq -r '.code_bucket' plan-metadata.json)" >> "$GITHUB_OUTPUT" - echo "lambda_matrix=$(jq -c '.lambda_matrix' plan-metadata.json)" >> "$GITHUB_OUTPUT" echo "bootstrap_image_uri=$(jq -r '.bootstrap_image_uri' plan-metadata.json)" >> "$GITHUB_OUTPUT" - echo "service_matrix=$(jq -c '.service_matrix' plan-metadata.json)" >> "$GITHUB_OUTPUT" infra: needs: @@ -66,9 +60,6 @@ jobs: with: environment: ${{ inputs.environment }} infra_version: ${{ needs.metadata.outputs.infra_version }} - code_bucket: ${{ needs.metadata.outputs.code_bucket }} - lambda_matrix: ${{ needs.metadata.outputs.lambda_matrix }} bootstrap_image_uri: ${{ needs.metadata.outputs.bootstrap_image_uri }} - service_matrix: ${{ needs.metadata.outputs.service_matrix }} tg_action: apply_plan plan_run_id: ${{ inputs.plan_artifact_run_id }} diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index 1a31a7d0..ddfbe75d 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -11,23 +11,11 @@ on: description: "Version of infrastructure (terraform) to be planned" required: true type: string - code_bucket: - description: "Bucket containing build artifacts" - required: true - type: string bootstrap_image_uri: description: "Bootstrap ECS image URI" required: false type: string default: "" - lambda_matrix: - required: false - type: string - default: "[]" - service_matrix: - required: false - type: string - default: "[]" outputs: plan_artifact_run_id: description: "Workflow run ID for the saved plan artifacts and metadata" @@ -55,10 +43,7 @@ jobs: cat > plan-metadata.json <.outputs` passthrough, verify every downstream consumer wrapper declares a `variable` with the exact same name - if that same-name output-to-variable contract does not hold yet, do not leave it implicit: either add the matching variables, or call out the mismatch explicitly before closing the task - check apply/deploy/destroy, and avoid unnecessary `terraform_remote_state` coupling (especially for fast-changing outputs) From 7bc935cf56b5b95d3c4457a3af470cd7d1750946 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 15:05:00 +0100 Subject: [PATCH 27/73] chore: shared get modules --- .github/docs/README.md | 2 + .github/workflows/shared_get_modules.yml | 103 +++++++++++++++++++++++ 2 files changed, 105 insertions(+) create mode 100644 .github/workflows/shared_get_modules.yml diff --git a/.github/docs/README.md b/.github/docs/README.md index f5e37eab..62329c2c 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -136,6 +136,8 @@ flowchart LR - `shared_directories_get.yml` Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. +- `shared_get_modules.yml` + Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` as reusable-workflow outputs. ## Feasibility Checks diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml new file mode 100644 index 00000000..b4d57a98 --- /dev/null +++ b/.github/workflows/shared_get_modules.yml @@ -0,0 +1,103 @@ +name: Shared Get Modules + +on: + workflow_call: + inputs: + environment: + description: environment reference i.e. 'prod' or 'dev' + required: true + type: string + infra_version: + description: "Version of infrastructure (terraform) to inspect" + required: true + type: string + outputs: + waves_json: + description: "Dependency-safe Terragrunt wave matrix JSON" + value: ${{ jobs.generate_waves.outputs.waves_json }} + wave_0_modules: + description: "Wave 0 module array" + value: ${{ jobs.generate_waves.outputs.wave_0_modules }} + wave_1_modules: + description: "Wave 1 module array" + value: ${{ jobs.generate_waves.outputs.wave_1_modules }} + wave_2_modules: + description: "Wave 2 module array" + value: ${{ jobs.generate_waves.outputs.wave_2_modules }} + +permissions: + id-token: write + contents: read + actions: read + +env: + AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role + AWS_REGION: ${{ vars.AWS_REGION }} + +jobs: + generate_waves: + runs-on: ubuntu-latest + outputs: + waves_json: ${{ steps.waves_json.outputs.just_outputs }} + wave_0_modules: ${{ steps.wave_outputs.outputs.wave_0_modules }} + wave_1_modules: ${{ steps.wave_outputs.outputs.wave_1_modules }} + wave_2_modules: ${{ steps.wave_outputs.outputs.wave_2_modules }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: Render Terragrunt graph + id: tg_graph + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws + tg_action: graph + + - name: Convert graph output to compact JSON + id: tg_graph_json + uses: ./.github/actions/just + env: + TG_GRAPH_OUTPUT: ${{ steps.tg_graph.outputs.tg_graph_output }} + with: + justfile_path: justfile.ci + just_action: tg-graph-output-to-json ${{ inputs.environment }} + + - name: Build wave matrix JSON + id: waves_json + uses: ./.github/actions/just + env: + TG_GRAPH_JSON: ${{ steps.tg_graph_json.outputs.just_outputs }} + with: + justfile_path: justfile.ci + just_action: tg-graph-json-to-waves + + - name: Expose wave module arrays + id: wave_outputs + shell: bash + env: + WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} + run: | + echo "wave_0_modules=$(jq -c '.[0].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" + echo "wave_1_modules=$(jq -c '.[1].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" + echo "wave_2_modules=$(jq -c '.[2].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" + + - name: Print Terragrunt wave matrix + shell: bash + env: + WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} + run: | + printf '%s\n' "$WAVES_JSON" | tee waves.json + { + echo "## Terragrunt Wave Matrix" + echo + echo '```json' + cat waves.json + echo + echo '```' + } >> "$GITHUB_STEP_SUMMARY" From 8d3fe8d95a5193e13cf1efc7170f1fb4a6e5f354 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 15:12:52 +0100 Subject: [PATCH 28/73] debug: use shared get modules --- .github/docs/README.md | 2 +- .github/workflows/shared_infra.yml | 69 ++---------------------------- 2 files changed, 5 insertions(+), 66 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 62329c2c..742f02ce 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -82,7 +82,7 @@ flowchart LR ### Infra And Code Rollout - `shared_infra_plan.yml` - Plan wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly, uploads one run-level `plan-metadata.json` file as a GitHub Actions artifact named `infra-plan-metadata`, and then calls `shared_infra.yml` with `tg_action: plan` plus `plan_run_id: ${{ github.run_id }}`. After the plan completes, it prints the current workflow `github.run_id` into both the logs and the GitHub Actions step summary as `plan_artifact_run_id`, and exposes that value as a reusable-workflow output. + Plan wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly, uploads one run-level `plan-metadata.json` file as a GitHub Actions artifact named `infra-plan-metadata`, starts `shared_get_modules.yml` in parallel to derive the current wave outputs, and then calls `shared_infra.yml` with `tg_action: plan` plus `plan_run_id: ${{ github.run_id }}`. After the plan completes, it prints the current workflow `github.run_id` into both the logs and the GitHub Actions step summary as `plan_artifact_run_id`, and exposes that value as a reusable-workflow output. - `shared_infra_apply.yml` Direct-input apply wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly and calls `shared_infra.yml` with `tg_action: apply`. - `shared_infra_apply_from_plan.yml` diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index cbc246c3..496e4da4 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -55,71 +55,10 @@ env: jobs: generate_waves: - runs-on: ubuntu-latest - outputs: - waves_json: ${{ steps.waves_json.outputs.just_outputs }} - wave_0_modules: ${{ steps.wave_outputs.outputs.wave_0_modules }} - wave_1_modules: ${{ steps.wave_outputs.outputs.wave_1_modules }} - wave_2_modules: ${{ steps.wave_outputs.outputs.wave_2_modules }} - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Render Terragrunt graph - id: tg_graph - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws - tg_action: graph - - - name: Convert graph output to compact JSON - id: tg_graph_json - uses: ./.github/actions/just - env: - TG_GRAPH_OUTPUT: ${{ steps.tg_graph.outputs.tg_graph_output }} - with: - justfile_path: justfile.ci - just_action: tg-graph-output-to-json ${{ inputs.environment }} - - - name: Build wave matrix JSON - id: waves_json - uses: ./.github/actions/just - env: - TG_GRAPH_JSON: ${{ steps.tg_graph_json.outputs.just_outputs }} - with: - justfile_path: justfile.ci - just_action: tg-graph-json-to-waves - - - name: Expose placeholder wave module arrays - id: wave_outputs - shell: bash - env: - WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} - run: | - echo "wave_0_modules=$(jq -c '.[0].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" - echo "wave_1_modules=$(jq -c '.[1].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" - echo "wave_2_modules=$(jq -c '.[2].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" - - - name: Print Terragrunt wave matrix - shell: bash - env: - WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} - run: | - printf '%s\n' "$WAVES_JSON" | tee waves.json - { - echo "## Terragrunt Wave Matrix" - echo - echo '```json' - cat waves.json - echo - echo '```' - } >> "$GITHUB_STEP_SUMMARY" + uses: ./.github/workflows/shared_get_modules.yml + with: + environment: ${{ inputs.environment }} + infra_version: ${{ inputs.infra_version }} wave_0: needs: generate_waves From cce0a15fef01b523015897a1c4fd82048b31c702 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 15:22:43 +0100 Subject: [PATCH 29/73] chore: run a proper plan --- .github/docs/README.md | 4 +-- .github/workflows/shared_infra.yml | 54 +++++++++++++++++++++++++----- REPO_INSTRUCTIONS.md | 1 + 3 files changed, 48 insertions(+), 11 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 742f02ce..33cbd6a3 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -88,7 +88,7 @@ flowchart LR - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. - `shared_infra.yml` - Temporary wave-placeholder executor. It renders the Terragrunt `run-all graph-dependencies` output, converts that graph into compact JSON, derives dependency-safe wave groups, prints the resulting wave matrix JSON into the logs and step summary, and exposes it as the reusable-workflow output `waves_json`. It then runs placeholder `wave_0`, `wave_1`, and `wave_2` jobs in dependency order; each placeholder job only runs when its module array is non-empty and fans out that array as a matrix that currently just prints the module name. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. + Temporary wave-based executor. It delegates graph and wave discovery to `shared_get_modules.yml`, exposes the resulting `waves_json` as the reusable-workflow output, and then runs `wave_0`, `wave_1`, and `wave_2` jobs in dependency order. Each wave only runs when its module array is non-empty, fans that array out as a matrix, checks out the requested ref, configures AWS credentials once per matrix job, and invokes the shared repo-local Terragrunt action against `infra/live//aws/`. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. - `just --justfile justfile.ci tg-graph-json-to-waves` CI helper that expects compact graph JSON in `TG_GRAPH_JSON` and returns a sequential JSON array of wave objects like `[{ "wave": 0, "modules": [...] }, ...]`, with each wave containing only modules whose direct dependencies were satisfied by earlier waves. - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. @@ -137,7 +137,7 @@ flowchart LR Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. - `shared_get_modules.yml` - Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` as reusable-workflow outputs. + Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` as reusable-workflow outputs. The current shared workflow contract therefore assumes the graph fits within three waves; if Terragrunt HCL dependency changes introduce a fourth wave, the reusable outputs and `shared_infra.yml` wave jobs must be extended in the same change. ## Feasibility Checks diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 496e4da4..9ea535a8 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -69,8 +69,20 @@ jobs: matrix: module: ${{ fromJson(needs.generate_waves.outputs.wave_0_modules) }} steps: - - name: ${{ matrix.module }} - run: echo "${{ matrix.module }}" + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: ${{ inputs.tg_action }} wave_1: needs: @@ -83,8 +95,20 @@ jobs: matrix: module: ${{ fromJson(needs.generate_waves.outputs.wave_1_modules) }} steps: - - name: ${{ matrix.module }} - run: echo "${{ matrix.module }}" + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: ${{ inputs.tg_action }} wave_2: needs: @@ -97,9 +121,21 @@ jobs: matrix: module: ${{ fromJson(needs.generate_waves.outputs.wave_2_modules) }} steps: - - name: ${{ matrix.module }} - run: echo "${{ matrix.module }}" + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: ${{ inputs.tg_action }} - # Placeholder wave jobs intentionally stop at logging module arrays. Replace - # them with real per-wave deployment fanout once the runtime override contract - # is restored for every stack type. + # Wave jobs currently apply the graph-derived module order directly through the + # shared Terragrunt action. Keep any runtime-specific override shaping explicit + # if a stack later needs more than the raw module path and tg_action. diff --git a/REPO_INSTRUCTIONS.md b/REPO_INSTRUCTIONS.md index 9a91c5b1..4979c723 100644 --- a/REPO_INSTRUCTIONS.md +++ b/REPO_INSTRUCTIONS.md @@ -108,6 +108,7 @@ These instructions apply to the entire repository. - before adding a Terragrunt `dependency` or `dependencies` path, verify the target live stack actually exists in that environment/repo slice - when changing reusable workflow contracts, compare every caller `with:` block to the callee `workflow_call.inputs` - when a workflow input, output, or metadata field is no longer consumed, remove it from the shared contract and callers in the same change rather than leaving dead plumbing behind +- when changing Terragrunt `*.hcl` dependency edges, re-check the derived infra wave count; the current shared module-discovery/workflow contract only exposes `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` - when adding or renaming Terraform module `output` values that are intended for Terragrunt `dependency..outputs` passthrough, verify every downstream consumer wrapper declares a `variable` with the exact same name - if that same-name output-to-variable contract does not hold yet, do not leave it implicit: either add the matching variables, or call out the mismatch explicitly before closing the task - check apply/deploy/destroy, and avoid unnecessary `terraform_remote_state` coupling (especially for fast-changing outputs) From 7bb65b437b210a42c57aed3021085876047fb24f Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 15:29:49 +0100 Subject: [PATCH 30/73] chore: fix vars --- .github/workflows/shared_infra.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 9ea535a8..aba7922b 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -47,7 +47,9 @@ permissions: env: AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role AWS_REGION: ${{ vars.AWS_REGION }} - DOMAIN_NAME: ${{ vars.DOMAIN_NAME }} + TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }} + TF_VAR_bootstrap: "true" + TF_VAR_bootstrap_image_uri: ${{ inputs.bootstrap_image_uri }} PLAN_BUCKET: ${{ vars.AWS_ACCOUNT_ID }}-${{ vars.AWS_REGION }}-${{ vars.PROJECT_NAME }}-tfplan TG_ENABLE_PLAN_ARTIFACTS: ${{ (inputs.tg_action == 'plan' || inputs.tg_action == 'apply_plan') && 'true' || 'false' }} PLAN_ARTIFACT_RUN_ID: ${{ inputs.plan_run_id }} From 0d60273da233bfc29f44632ffce52f06684bd543 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 16:48:19 +0100 Subject: [PATCH 31/73] fix: filter out tasks --- .github/docs/README.md | 2 +- .github/workflows/shared_get_modules.yml | 19 ++++++++++++++++--- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 33cbd6a3..4fe4db78 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -137,7 +137,7 @@ flowchart LR Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. - `shared_get_modules.yml` - Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` as reusable-workflow outputs. The current shared workflow contract therefore assumes the graph fits within three waves; if Terragrunt HCL dependency changes introduce a fourth wave, the reusable outputs and `shared_infra.yml` wave jobs must be extended in the same change. + Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, excludes `task_api` and `task_worker` from the emitted rollout waves for the current bootstrap-oriented infra path, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` as reusable-workflow outputs. The current shared workflow contract therefore assumes the graph fits within three waves; if Terragrunt HCL dependency changes introduce a fourth wave, the reusable outputs and `shared_infra.yml` wave jobs must be extended in the same change. ## Feasibility Checks diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index b4d57a98..1fd8ca54 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -38,7 +38,7 @@ jobs: generate_waves: runs-on: ubuntu-latest outputs: - waves_json: ${{ steps.waves_json.outputs.just_outputs }} + waves_json: ${{ steps.filtered_waves.outputs.waves_json }} wave_0_modules: ${{ steps.wave_outputs.outputs.wave_0_modules }} wave_1_modules: ${{ steps.wave_outputs.outputs.wave_1_modules }} wave_2_modules: ${{ steps.wave_outputs.outputs.wave_2_modules }} @@ -77,11 +77,24 @@ jobs: justfile_path: justfile.ci just_action: tg-graph-json-to-waves + - name: Exclude task modules from wave matrix + id: filtered_waves + shell: bash + env: + RAW_WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} + run: | + echo "waves_json=$(jq -c ' + map(.modules |= map(select(. != "task_api" and . != "task_worker"))) + | map(select(.modules | length > 0)) + | to_entries + | map({wave: .key, modules: .value.modules}) + ' <<<"$RAW_WAVES_JSON")" >> "$GITHUB_OUTPUT" + - name: Expose wave module arrays id: wave_outputs shell: bash env: - WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} + WAVES_JSON: ${{ steps.filtered_waves.outputs.waves_json }} run: | echo "wave_0_modules=$(jq -c '.[0].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" echo "wave_1_modules=$(jq -c '.[1].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" @@ -90,7 +103,7 @@ jobs: - name: Print Terragrunt wave matrix shell: bash env: - WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} + WAVES_JSON: ${{ steps.filtered_waves.outputs.waves_json }} run: | printf '%s\n' "$WAVES_JSON" | tee waves.json { From 8cff8ccbc0484cc089df94a4aa7fd43bc5495426 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 16:53:01 +0100 Subject: [PATCH 32/73] chore: rn jobs --- .github/workflows/shared_infra.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index aba7922b..28ebbef1 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -64,6 +64,7 @@ jobs: wave_0: needs: generate_waves + name: 0 / ${{ matrix.module }} if: ${{ needs.generate_waves.outputs.wave_0_modules != '[]' }} runs-on: ubuntu-latest strategy: @@ -90,6 +91,7 @@ jobs: needs: - generate_waves - wave_0 + name: 1 / ${{ matrix.module }} if: ${{ always() && needs.generate_waves.outputs.wave_1_modules != '[]' && (needs.wave_0.result == 'success' || needs.wave_0.result == 'skipped') }} runs-on: ubuntu-latest strategy: @@ -116,6 +118,7 @@ jobs: needs: - generate_waves - wave_1 + name: 2 / ${{ matrix.module }} if: ${{ always() && needs.generate_waves.outputs.wave_2_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} runs-on: ubuntu-latest strategy: From ed3a4665e5f02be5731d18818c01b6ce03b8260f Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 20:02:22 +0100 Subject: [PATCH 33/73] chore: mv to shared plan wf --- .github/docs/README.md | 2 +- .github/workflows/shared_get_modules.yml | 16 +++- .github/workflows/shared_infra.yml | 1 + .github/workflows/shared_infra_plan.yml | 93 +++++++++++++++++++++--- 4 files changed, 99 insertions(+), 13 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 4fe4db78..1f0c1093 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -137,7 +137,7 @@ flowchart LR Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. - `shared_get_modules.yml` - Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, excludes `task_api` and `task_worker` from the emitted rollout waves for the current bootstrap-oriented infra path, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` as reusable-workflow outputs. The current shared workflow contract therefore assumes the graph fits within three waves; if Terragrunt HCL dependency changes introduce a fourth wave, the reusable outputs and `shared_infra.yml` wave jobs must be extended in the same change. + Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` as reusable-workflow outputs. Callers can pass `ignore_task_modules: true` to exclude any `task_*` modules from the emitted rollout waves for the current bootstrap-oriented infra path. The current shared workflow contract therefore assumes the graph fits within three waves; if Terragrunt HCL dependency changes introduce a fourth wave, the reusable outputs and `shared_infra.yml` wave jobs must be extended in the same change. ## Feasibility Checks diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index 1fd8ca54..b29c7179 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -11,6 +11,11 @@ on: description: "Version of infrastructure (terraform) to inspect" required: true type: string + ignore_task_modules: + description: "Whether to exclude task_* modules from emitted waves" + required: false + type: boolean + default: false outputs: waves_json: description: "Dependency-safe Terragrunt wave matrix JSON" @@ -77,18 +82,23 @@ jobs: justfile_path: justfile.ci just_action: tg-graph-json-to-waves - - name: Exclude task modules from wave matrix + - name: Optionally exclude task modules from wave matrix id: filtered_waves shell: bash env: RAW_WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} + IGNORE_TASK_MODULES: ${{ inputs.ignore_task_modules }} run: | echo "waves_json=$(jq -c ' - map(.modules |= map(select(. != "task_api" and . != "task_worker"))) + if $ignore_tasks then + map(.modules |= map(select(startswith("task_") | not))) + else + . + end | map(select(.modules | length > 0)) | to_entries | map({wave: .key, modules: .value.modules}) - ' <<<"$RAW_WAVES_JSON")" >> "$GITHUB_OUTPUT" + ' --argjson ignore_tasks "$IGNORE_TASK_MODULES" <<<"$RAW_WAVES_JSON")" >> "$GITHUB_OUTPUT" - name: Expose wave module arrays id: wave_outputs diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 28ebbef1..9ae9dbbb 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -61,6 +61,7 @@ jobs: with: environment: ${{ inputs.environment }} infra_version: ${{ inputs.infra_version }} + ignore_task_modules: true wave_0: needs: generate_waves diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index ddfbe75d..21b95ceb 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -30,6 +30,13 @@ env: AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role jobs: + generate_waves: + uses: ./.github/workflows/shared_get_modules.yml + with: + environment: ${{ inputs.environment }} + infra_version: ${{ inputs.infra_version }} + ignore_task_modules: true + metadata: runs-on: ubuntu-latest steps: @@ -54,16 +61,84 @@ jobs: path: plan-metadata.json retention-days: 14 - infra: + wave_0: + needs: generate_waves + name: 0 / ${{ matrix.module }} + if: ${{ needs.generate_waves.outputs.wave_0_modules != '[]' }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_0_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: ${{ inputs.tg_action }} + + wave_1: needs: - - metadata - uses: ./.github/workflows/shared_infra.yml - with: - environment: ${{ inputs.environment }} - infra_version: ${{ inputs.infra_version }} - bootstrap_image_uri: ${{ inputs.bootstrap_image_uri }} - tg_action: plan - plan_run_id: ${{ github.run_id }} + - generate_waves + - wave_0 + name: 1 / ${{ matrix.module }} + if: ${{ always() && needs.generate_waves.outputs.wave_1_modules != '[]' && (needs.wave_0.result == 'success' || needs.wave_0.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_1_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: ${{ inputs.tg_action }} + + wave_2: + needs: + - generate_waves + - wave_1 + name: 2 / ${{ matrix.module }} + if: ${{ always() && needs.generate_waves.outputs.wave_2_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_2_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: ${{ inputs.tg_action }} plan_context: name: Plan Context From d2719d1d466a7a968ef35e2279e23439d2a5be51 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 20:24:52 +0100 Subject: [PATCH 34/73] chore: upload plans --- .github/workflows/shared_infra_plan.yml | 31 +++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index 21b95ceb..f936d030 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -86,6 +86,17 @@ jobs: tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: ${{ inputs.tg_action }} + - name: Upload plan files + if: ${{ success() }} + uses: actions/upload-artifact@v7 + with: + name: terragrunt-plan-${{ inputs.environment }}-${{ matrix.module }} + path: | + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.meta.json + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.tfplan + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.txt + if-no-files-found: warn + wave_1: needs: - generate_waves @@ -112,6 +123,16 @@ jobs: with: tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: ${{ inputs.tg_action }} + - name: Upload plan files + if: ${{ success() }} + uses: actions/upload-artifact@v7 + with: + name: terragrunt-plan-${{ inputs.environment }}-${{ matrix.module }} + path: | + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.meta.json + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.tfplan + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.txt + if-no-files-found: warn wave_2: needs: @@ -139,6 +160,16 @@ jobs: with: tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: ${{ inputs.tg_action }} + - name: Upload plan files + if: ${{ success() }} + uses: actions/upload-artifact@v7 + with: + name: terragrunt-plan-${{ inputs.environment }}-${{ matrix.module }} + path: | + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.meta.json + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.tfplan + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.txt + if-no-files-found: warn plan_context: name: Plan Context From 573405f33994fc5ef4a471ea162321dab6240bae Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 20:38:07 +0100 Subject: [PATCH 35/73] fix: plan vars + apply matrix --- .github/workflows/shared_infra_apply.yml | 91 ++++++++++++++++++++++-- .github/workflows/shared_infra_plan.yml | 2 + 2 files changed, 89 insertions(+), 4 deletions(-) diff --git a/.github/workflows/shared_infra_apply.yml b/.github/workflows/shared_infra_apply.yml index 5c03610e..367dc1bd 100644 --- a/.github/workflows/shared_infra_apply.yml +++ b/.github/workflows/shared_infra_apply.yml @@ -22,11 +22,94 @@ permissions: contents: read actions: read +env: + AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role + AWS_REGION: ${{ vars.AWS_REGION }} + TG_ACTION_LABEL: "Apply" + jobs: - infra: - uses: ./.github/workflows/shared_infra.yml + generate_waves: + uses: ./.github/workflows/shared_get_modules.yml with: environment: ${{ inputs.environment }} infra_version: ${{ inputs.infra_version }} - bootstrap_image_uri: ${{ inputs.bootstrap_image_uri }} - tg_action: apply + ignore_task_modules: true + + wave_0: + needs: generate_waves + name: 0 / ${{ matrix.module }} + if: ${{ needs.generate_waves.outputs.wave_0_modules != '[]' }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_0_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: apply + + wave_1: + needs: + - generate_waves + - wave_0 + name: 1 / ${{ matrix.module }} + if: ${{ always() && needs.generate_waves.outputs.wave_1_modules != '[]' && (needs.wave_0.result == 'success' || needs.wave_0.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_1_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: apply + + wave_2: + needs: + - generate_waves + - wave_1 + name: 2 / ${{ matrix.module }} + if: ${{ always() && needs.generate_waves.outputs.wave_2_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_2_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: apply diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index f936d030..759b628e 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -28,6 +28,8 @@ permissions: env: AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role + AWS_REGION: ${{ vars.AWS_REGION }} + TG_ACTION_LABEL: "Plan" jobs: generate_waves: From 9652fc972770e016d233d6eb0b513e3dee87ffb5 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Thu, 21 May 2026 20:46:02 +0100 Subject: [PATCH 36/73] fix: set plan action --- .github/workflows/shared_infra_plan.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index 759b628e..65c1bb73 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -86,7 +86,7 @@ jobs: uses: ./.github/actions/terragrunt with: tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - tg_action: ${{ inputs.tg_action }} + tg_action: plan - name: Upload plan files if: ${{ success() }} @@ -124,7 +124,7 @@ jobs: uses: ./.github/actions/terragrunt with: tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - tg_action: ${{ inputs.tg_action }} + tg_action: plan - name: Upload plan files if: ${{ success() }} uses: actions/upload-artifact@v7 @@ -161,7 +161,7 @@ jobs: uses: ./.github/actions/terragrunt with: tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - tg_action: ${{ inputs.tg_action }} + tg_action: plan - name: Upload plan files if: ${{ success() }} uses: actions/upload-artifact@v7 From f1cbb0813b83d84c73c4f8c24a526580554b3225 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 09:52:08 +0100 Subject: [PATCH 37/73] chore: domain_name var in ci --- .github/workflows/shared_infra_apply.yml | 2 ++ .github/workflows/shared_infra_plan.yml | 2 ++ 2 files changed, 4 insertions(+) diff --git a/.github/workflows/shared_infra_apply.yml b/.github/workflows/shared_infra_apply.yml index 367dc1bd..fa2ab25f 100644 --- a/.github/workflows/shared_infra_apply.yml +++ b/.github/workflows/shared_infra_apply.yml @@ -27,6 +27,8 @@ env: AWS_REGION: ${{ vars.AWS_REGION }} TG_ACTION_LABEL: "Apply" + TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }} + jobs: generate_waves: uses: ./.github/workflows/shared_get_modules.yml diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index 65c1bb73..fad26a78 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -31,6 +31,8 @@ env: AWS_REGION: ${{ vars.AWS_REGION }} TG_ACTION_LABEL: "Plan" + TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }} + jobs: generate_waves: uses: ./.github/workflows/shared_get_modules.yml From 9ffe21924455000e51b5638a8e4792b04f46ded3 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 10:10:00 +0100 Subject: [PATCH 38/73] chore: pass in domain via global vars --- .github/workflows/destroy.yml | 4 ---- .github/workflows/shared_infra.yml | 1 - .github/workflows/shared_infra_apply.yml | 2 -- .github/workflows/shared_infra_plan.yml | 2 -- README.md | 8 +------- infra/README.md | 2 +- infra/live/global_vars.hcl | 2 ++ 7 files changed, 4 insertions(+), 17 deletions(-) diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index 0ef17300..5b6acd24 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -28,7 +28,6 @@ env: TF_VAR_lambda_version: this AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role AWS_REGION: ${{ vars.AWS_REGION }} - DOMAIN_NAME: ${{ vars.DOMAIN_NAME }} jobs: setup: @@ -91,7 +90,6 @@ jobs: uses: ./.github/actions/terragrunt env: TF_VAR_api_invoke_url: "https://placeholder.execute-api.us-east-1.amazonaws.com" - TF_VAR_domain_name: ${{ env.DOMAIN_NAME }} TF_VAR_auth_user_pool_id: "destroy-placeholder" TF_VAR_auth_user_pool_client_id: "destroy-placeholder" TF_VAR_auth_hosted_ui_url: "https://destroy-placeholder" @@ -116,8 +114,6 @@ jobs: - name: Destroy cognito infra uses: ./.github/actions/terragrunt - env: - TF_VAR_domain_name: ${{ env.DOMAIN_NAME }} with: tg_directory: infra/live/${{ inputs.environment }}/aws/cognito tg_action: destroy diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml index 9ae9dbbb..12209a9f 100644 --- a/.github/workflows/shared_infra.yml +++ b/.github/workflows/shared_infra.yml @@ -47,7 +47,6 @@ permissions: env: AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role AWS_REGION: ${{ vars.AWS_REGION }} - TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }} TF_VAR_bootstrap: "true" TF_VAR_bootstrap_image_uri: ${{ inputs.bootstrap_image_uri }} PLAN_BUCKET: ${{ vars.AWS_ACCOUNT_ID }}-${{ vars.AWS_REGION }}-${{ vars.PROJECT_NAME }}-tfplan diff --git a/.github/workflows/shared_infra_apply.yml b/.github/workflows/shared_infra_apply.yml index fa2ab25f..367dc1bd 100644 --- a/.github/workflows/shared_infra_apply.yml +++ b/.github/workflows/shared_infra_apply.yml @@ -27,8 +27,6 @@ env: AWS_REGION: ${{ vars.AWS_REGION }} TG_ACTION_LABEL: "Apply" - TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }} - jobs: generate_waves: uses: ./.github/workflows/shared_get_modules.yml diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index fad26a78..65c1bb73 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -31,8 +31,6 @@ env: AWS_REGION: ${{ vars.AWS_REGION }} TG_ACTION_LABEL: "Plan" - TF_VAR_domain_name: ${{ vars.DOMAIN_NAME }} - jobs: generate_waves: uses: ./.github/workflows/shared_get_modules.yml diff --git a/README.md b/README.md index 1ba2456a..01f014a6 100644 --- a/README.md +++ b/README.md @@ -284,13 +284,7 @@ The Cognito stack creates the user pool, app client, Hosted UI domain, and `read just cognito-create-readonly-user dev readonly@example.com 'ChangeMe123!' ``` -Set the GitHub environment variable `DOMAIN_NAME` to the hosted zone base domain, for example: - -```text -chrispsheehan.com -``` - -When that value is present, the frontend and Cognito stacks derive the deployed domain and auth callback/logout URLs automatically. Local Vite login still coexists through `http://localhost:5173`. +The frontend and Cognito stacks read `domain_name` from the shared Terragrunt global inputs in [infra/live/global_vars.hcl](infra/live/global_vars.hcl), which currently sets it to `chrispsheehan.com`. That keeps the deployed domain and auth callback/logout URLs consistent without extra CI wiring. Local Vite login still coexists through `http://localhost:5173`. ## Infra Deployment Use Cases diff --git a/infra/README.md b/infra/README.md index 1bb410ac..bf1ef627 100644 --- a/infra/README.md +++ b/infra/README.md @@ -71,7 +71,7 @@ stores state at: - `cognito` Owns the Cognito user pool, frontend app client, Hosted UI domain, and read-only user group. - `frontend` - Owns the derived CloudFront custom domain, ACM certificate in `us-east-1`, and Route53 alias records using the required `DOMAIN_NAME` workflow env input. + Owns the derived CloudFront custom domain, ACM certificate in `us-east-1`, and Route53 alias records using the Terragrunt `domain_name` input from `infra/live/global_vars.hcl`. - `observability` Owns the shared CloudWatch dashboard used to inspect Lambda and ECS logs in one console place. - `database` diff --git a/infra/live/global_vars.hcl b/infra/live/global_vars.hcl index 4c04287b..302d794e 100644 --- a/infra/live/global_vars.hcl +++ b/infra/live/global_vars.hcl @@ -1,6 +1,7 @@ locals { vpc_name = "vpc" aws_region = "eu-west-2" + domain_name = "chrispsheehan.com" allowed_role_actions = [ "s3:*", "iam:*", @@ -35,6 +36,7 @@ locals { inputs = { vpc_name = local.vpc_name aws_region = local.aws_region + domain_name = local.domain_name allowed_role_actions = local.allowed_role_actions code_artifact_expiration_days = local.code_artifact_expiration_days infra_plan_artifact_expiration_days = local.infra_plan_artifact_expiration_days From 9d2b55c170c3d705cadd00981fb44c68609e0feb Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 10:10:17 +0100 Subject: [PATCH 39/73] fix: fmt --- infra/live/global_vars.hcl | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/infra/live/global_vars.hcl b/infra/live/global_vars.hcl index 302d794e..bfef183d 100644 --- a/infra/live/global_vars.hcl +++ b/infra/live/global_vars.hcl @@ -1,6 +1,6 @@ locals { - vpc_name = "vpc" - aws_region = "eu-west-2" + vpc_name = "vpc" + aws_region = "eu-west-2" domain_name = "chrispsheehan.com" allowed_role_actions = [ "s3:*", From a98ff2a2d701f409ea95c2cc8eaf07f92695d577 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 10:13:22 +0100 Subject: [PATCH 40/73] fix: TG_ENABLE_PLAN_ARTIFACTS: "true" --- .github/workflows/shared_infra_plan.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index 65c1bb73..2cc3788a 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -30,6 +30,7 @@ env: AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role AWS_REGION: ${{ vars.AWS_REGION }} TG_ACTION_LABEL: "Plan" + TG_ENABLE_PLAN_ARTIFACTS: "true" jobs: generate_waves: From 2ce37bfa12152f6f5022913c9ff2ba4a9b160270 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 10:20:12 +0100 Subject: [PATCH 41/73] fix: set PLAN_ARTIFACT_RUN_ID --- .github/workflows/shared_infra_plan.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index 2cc3788a..f66025fb 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -31,6 +31,7 @@ env: AWS_REGION: ${{ vars.AWS_REGION }} TG_ACTION_LABEL: "Plan" TG_ENABLE_PLAN_ARTIFACTS: "true" + PLAN_ARTIFACT_RUN_ID: ${{ github.run_id }} jobs: generate_waves: From cf48322c8217b9cf3ab8935ec9875b2a38b01b7d Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 11:29:37 +0100 Subject: [PATCH 42/73] feat: try artifact upload/download --- .github/actions/terragrunt/README.md | 18 +-- .github/actions/terragrunt/action.yml | 32 ++-- .github/docs/README.md | 11 +- .github/workflows/shared_infra.yml | 146 ------------------ .../shared_infra_apply_from_plan.yml | 11 -- .github/workflows/shared_infra_plan.yml | 2 - README.md | 2 +- infra/README.md | 17 +- infra/root.hcl | 43 ------ infra/scripts/ensure-plan-artifact-bucket.sh | 61 -------- infra/scripts/handle-plan-artifact.sh | 88 ----------- 11 files changed, 39 insertions(+), 392 deletions(-) delete mode 100644 .github/workflows/shared_infra.yml delete mode 100644 infra/scripts/ensure-plan-artifact-bucket.sh delete mode 100644 infra/scripts/handle-plan-artifact.sh diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index bcbef605..e6eeb4b1 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -11,7 +11,7 @@ This GitHub Action sets up **Terraform** and **Terragrunt** and runs a specified - Supports `plan` mode for producing local saved plan files - Supports `init` mode for outputs-only reads - Supports `graph` mode for raw `terragrunt run-all graph-dependencies` output capture -- Relies on shared Terragrunt root hooks for per-stack saved plan artifact upload and download +- Writes saved plan files into the live stack directory so workflows can upload and download them with GitHub artifacts - Exports Terragrunt outputs as compact JSON when state exists The Terragrunt install step is kept in this repo-local action rather than hidden behind a third-party Terragrunt wrapper action so the repo can control the exact setup-action revision and react quickly to GitHub Actions runtime deprecations or nested dependency warnings. @@ -40,9 +40,9 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - `apply` Runs `terragrunt apply -auto-approve` - `plan` - Runs `terragrunt plan -detailed-exitcode -out=terragrunt.tfplan`. The shared Terragrunt root `after_hook` renders `terragrunt.plan.txt`, writes `terragrunt.plan.meta.json`, always uploads the metadata, and only uploads `terragrunt.tfplan` plus `terragrunt.plan.txt` when the metadata says the stack has changes. + Runs `terragrunt plan -detailed-exitcode -out=/terragrunt.tfplan`. The action writes `terragrunt.plan.meta.json` into the live stack directory for every plan run and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes. - `apply_plan` - Runs `terragrunt apply terragrunt.tfplan`. The shared Terragrunt root `before_hook` downloads the saved plan bundle into the Terragrunt working directory when `TG_ENABLE_PLAN_ARTIFACTS=true` and `PLAN_ARTIFACT_RUN_ID` is set, and fails early if the saved metadata reports mocked dependency outputs. + Runs `terragrunt apply /terragrunt.tfplan`. The calling workflow is expected to download that stack's saved plan artifact into the live stack directory before invoking `apply_plan`. - `destroy` Runs `terragrunt destroy -auto-approve` - `init` @@ -55,10 +55,10 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - One run-level metadata file is stored separately by the shared infra wrapper as a GitHub Actions artifact: - artifact name: `infra-plan-metadata` - file: `plan-metadata.json` -- Each Terragrunt stack or module stores its own plan bundle at: - - `s3:///terragrunt_plan///terragrunt-plan-/terragrunt.plan.meta.json` - - `s3:///terragrunt_plan///terragrunt-plan-/terragrunt.tfplan` only when changes exist - - `s3:///terragrunt_plan///terragrunt-plan-/terragrunt.plan.txt` only when changes exist +- Each Terragrunt stack or module stores its own plan bundle as a GitHub Actions artifact named `terragrunt-plan--`: + - `terragrunt.plan.meta.json` + - `terragrunt.tfplan` only when changes exist + - `terragrunt.plan.txt` only when changes exist ## AWS Credentials @@ -143,7 +143,7 @@ jobs: tg_action: plan ``` -### Apply From Uploaded Plan In S3 +### Apply From Downloaded GitHub Artifact ```yaml jobs: @@ -169,4 +169,4 @@ jobs: tg_action: apply_plan ``` -This action expects the workflow to set both `TG_ENABLE_PLAN_ARTIFACTS=true` and `PLAN_ARTIFACT_RUN_ID` when using cross-run saved plans so the shared Terragrunt root hooks can resolve the per-stack plan bundle location from the derived plan bucket and environment. +This action expects the workflow to download the matching per-stack plan artifact into the live stack directory before using `tg_action: apply_plan`. diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index 15ac4084..bef4549d 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -73,8 +73,12 @@ runs: TG_PLAN_LOG_ABS_PATH: ${{ github.workspace }}/${{ inputs.tg_directory }}/terragrunt.plan.log working-directory: ${{ inputs.tg_directory }} run: | - PLAN_PATH="terragrunt.tfplan" - PLAN_LOG_PATH="$(pwd)/terragrunt.plan.log" + PLAN_DIR="${{ github.workspace }}/${{ inputs.tg_directory }}" + PLAN_PATH="${PLAN_DIR}/terragrunt.tfplan" + PLAN_TEXT_PATH="${PLAN_DIR}/terragrunt.plan.txt" + PLAN_META_PATH="${PLAN_DIR}/terragrunt.plan.meta.json" + PLAN_JSON_PATH="${PLAN_DIR}/terragrunt.plan.json" + PLAN_LOG_PATH="${PLAN_DIR}/terragrunt.plan.log" case "${{ inputs.tg_action }}" in apply) @@ -96,12 +100,25 @@ runs: echo "::warning title=Mock outputs used during plan::Terragrunt used dependency mock outputs while creating a saved plan. This plan artifact should not be used with apply_plan until a fresh plan is created from real upstream outputs." fi + jq -n \ + --arg tg_directory "${{ inputs.tg_directory }}" \ + --argjson contains_mocked_outputs "$plan_contains_mocked_outputs" \ + '{tg_directory: $tg_directory, contains_mocked_outputs: $contains_mocked_outputs}' \ + > "$PLAN_META_PATH" + + terraform show -no-color "$PLAN_PATH" > "$PLAN_TEXT_PATH" + echo "plan_exit_code=$plan_exit_code" >> "$GITHUB_OUTPUT" echo "plan_contains_mocked_outputs=$plan_contains_mocked_outputs" >> "$GITHUB_OUTPUT" - echo "Terragrunt binary plan path in cache: $PLAN_PATH" + echo "Terragrunt binary plan path: $PLAN_PATH" ;; apply_plan) + if [ ! -f "$PLAN_PATH" ]; then + echo "::error title=Missing saved plan artifact::Expected '${PLAN_PATH}' to exist before apply_plan." + exit 1 + fi + set +e APPLY_LOG_PATH="$(pwd)/terragrunt.apply.log" terragrunt apply -auto-approve "$PLAN_PATH" 2>&1 | tee "$APPLY_LOG_PATH" @@ -126,15 +143,6 @@ runs: emit_error "Saved plan is stale" "Saved plan is stale" fi - emit_error "Saved plan artifact missing" "Saved plan artifact not found for" - emit_error "Missing PLAN_ARTIFACT_RUN_ID" "PLAN_ARTIFACT_RUN_ID is required when TG_ENABLE_PLAN_ARTIFACTS=true" - - if grep -Fq "Plan bucket" "$APPLY_LOG_PATH" && grep -Fq "does not exist" "$APPLY_LOG_PATH"; then - emit_error "Plan bucket missing" "Plan bucket" - fi - - emit_error "Plan bucket creation declined" "Plan bucket creation declined." - emit_error "Plan bucket confirmation unavailable" "no interactive terminal is available for confirmation" exit "$apply_exit_code" fi ;; diff --git a/.github/docs/README.md b/.github/docs/README.md index 1f0c1093..609c17fa 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -148,17 +148,16 @@ Run these checks on every CI, workflow, or deploy-contract change. - compare every caller `with:` block against the callee `workflow_call.inputs` - compare expected outputs against actual `jobs..outputs.*` - verify optional inputs are intentionally omitted, not accidentally missing -- the repo-local `./.github/actions/terragrunt` action supports `tg_action: plan` for producing the binary plan locally; the shared Terragrunt root `after_hook` then renders `terragrunt.plan.txt` and writes `terragrunt.plan.meta.json` -- shared Terragrunt root hooks now always upload per-stack `terragrunt.plan.meta.json` on saved `plan`, but only upload `terragrunt.tfplan` and `terragrunt.plan.txt` when the metadata reports real changes; `apply_plan` still downloads the normal plan bundle when one exists +- the repo-local `./.github/actions/terragrunt` action supports `tg_action: plan` for producing the binary plan in the live stack directory, writes `terragrunt.plan.meta.json` there for every saved plan, and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes +- `apply_plan` now expects the calling workflow job to download the matching per-stack GitHub artifact into the live stack directory before invoking Terragrunt - both repo-local composite actions, `./.github/actions/just` and `./.github/actions/terragrunt`, now assume AWS credentials are already configured in the current job when they need AWS access. The repo pattern is to run `aws-actions/configure-aws-credentials` at the top of each AWS-using job and then call the local actions without extra auth inputs - `./.github/actions/just` installs the requested `just` version through `extractions/setup-crate@v2` in the same minimal composite-action shape as `extractions/setup-just`, rather than depending on `extractions/setup-just` itself - `./.github/actions/terragrunt` installs the requested Terragrunt version through `jdx/mise-action@v4`, while Terraform stays pinned separately through `hashicorp/setup-terraform` - saved infra-plan storage is intentionally split into two levels: - one run-level metadata artifact named `infra-plan-metadata` containing `plan-metadata.json` - - one per-stack plan bundle under `s3:///terragrunt_plan///terragrunt-plan-/` -- the dedicated plan bucket is repo-wide, derived as `---tfplan`, and plan uniqueness comes from `terragrunt_plan///...` + - one per-stack GitHub artifact named `terragrunt-plan--` - `./.github/actions/terragrunt` derives its plan artifact name from `tg_directory`, so callers do not need to pass artifact naming inputs -- if `apply_plan` is used across separate workflow runs, pass the earlier workflow `run_id` through `plan_artifact_run_id`; the shared wrappers recover both metadata and per-stack plan files from the dedicated plan bucket under `terragrunt_plan///...` +- if `apply_plan` is used across separate workflow runs, pass the earlier workflow `run_id` through `plan_artifact_run_id`; the shared wrappers recover both metadata and per-stack plan files from GitHub artifacts in that earlier run - if a cross-run apply should not ask the operator to re-enter versions or recompute artifact resolution, store both the input versions and the resolved reusable-workflow outputs in a metadata artifact during plan and recover them in the apply wrapper from the earlier `run_id` - keep `shared_infra.yml` as the pure graph executor and prefer handling metadata creation/recovery in the dedicated plan/apply wrappers - when using `./.github/actions/just`, check whether the caller needs the repo-root `justfile` or an explicit `justfile_path` @@ -235,7 +234,7 @@ These are the workflows most users trigger directly. - `prod_infra_apply.yml` Resolves released artifacts from `ci` and applies prod infrastructure. - `prod_infra_apply_from_plan.yml` - Reapplies the ordered prod infra graph from plan artifacts created by a prior `prod_infra_plan` run, using the shared `plan_artifact_run_id` contract end-to-end. `shared_infra_apply_from_plan.yml` reads the matching metadata artifact before delegating to `shared_infra.yml`. + Reapplies the ordered prod infra graph from plan artifacts created by a prior `prod_infra_plan` run, using the shared `plan_artifact_run_id` contract end-to-end. `shared_infra_apply_from_plan.yml` reads the matching metadata artifact first, then each apply job downloads its matching per-stack GitHub artifact before invoking `apply_plan`. - `dev_code_deploy.yml` Discovers directories, builds fresh dev artifacts, resolves deploy inputs, and deploys code to dev. - `prod_code_deploy.yml` diff --git a/.github/workflows/shared_infra.yml b/.github/workflows/shared_infra.yml deleted file mode 100644 index 12209a9f..00000000 --- a/.github/workflows/shared_infra.yml +++ /dev/null @@ -1,146 +0,0 @@ -name: Shared Infra - -on: - workflow_call: - inputs: - environment: - description: environment reference i.e. 'prod' or 'dev' - required: true - type: string - infra_version: - description: "Version of infrastructure (terraform) to be deployed" - required: true - type: string - bootstrap_image_uri: - description: "Bootstrap ECS image URI" - required: false - type: string - default: "" - tg_action: - description: "Terragrunt action to run across the infra graph" - required: false - type: string - default: "apply" - plan_run_id: - description: "Optional unique run id used to derive saved plan artifact paths" - required: false - type: string - default: "" - outputs: - waves_json: - description: "Dependency-safe Terragrunt wave matrix JSON" - value: ${{ jobs.generate_waves.outputs.waves_json }} - changed_items_json: - description: "Deprecated compatibility output; currently mirrors waves_json" - value: ${{ jobs.generate_waves.outputs.waves_json }} - - -concurrency: # only run one instance of workflow at any one time - group: infra-${{ inputs.environment }} - cancel-in-progress: false - -permissions: - id-token: write - contents: read - actions: read - -env: - AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role - AWS_REGION: ${{ vars.AWS_REGION }} - TF_VAR_bootstrap: "true" - TF_VAR_bootstrap_image_uri: ${{ inputs.bootstrap_image_uri }} - PLAN_BUCKET: ${{ vars.AWS_ACCOUNT_ID }}-${{ vars.AWS_REGION }}-${{ vars.PROJECT_NAME }}-tfplan - TG_ENABLE_PLAN_ARTIFACTS: ${{ (inputs.tg_action == 'plan' || inputs.tg_action == 'apply_plan') && 'true' || 'false' }} - PLAN_ARTIFACT_RUN_ID: ${{ inputs.plan_run_id }} - TG_ACTION_LABEL: ${{ (inputs.tg_action == 'apply' || inputs.tg_action == 'apply_plan') && 'Apply' || inputs.tg_action == 'plan' && 'Plan' || inputs.tg_action == 'destroy' && 'Destroy' || inputs.tg_action == 'init' && 'Init' || 'Run' }} - -jobs: - generate_waves: - uses: ./.github/workflows/shared_get_modules.yml - with: - environment: ${{ inputs.environment }} - infra_version: ${{ inputs.infra_version }} - ignore_task_modules: true - - wave_0: - needs: generate_waves - name: 0 / ${{ matrix.module }} - if: ${{ needs.generate_waves.outputs.wave_0_modules != '[]' }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - module: ${{ fromJson(needs.generate_waves.outputs.wave_0_modules) }} - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - tg_action: ${{ inputs.tg_action }} - - wave_1: - needs: - - generate_waves - - wave_0 - name: 1 / ${{ matrix.module }} - if: ${{ always() && needs.generate_waves.outputs.wave_1_modules != '[]' && (needs.wave_0.result == 'success' || needs.wave_0.result == 'skipped') }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - module: ${{ fromJson(needs.generate_waves.outputs.wave_1_modules) }} - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - tg_action: ${{ inputs.tg_action }} - - wave_2: - needs: - - generate_waves - - wave_1 - name: 2 / ${{ matrix.module }} - if: ${{ always() && needs.generate_waves.outputs.wave_2_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - module: ${{ fromJson(needs.generate_waves.outputs.wave_2_modules) }} - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ inputs.infra_version }} - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - tg_action: ${{ inputs.tg_action }} - - # Wave jobs currently apply the graph-derived module order directly through the - # shared Terragrunt action. Keep any runtime-specific override shaping explicit - # if a stack later needs more than the raw module path and tg_action. diff --git a/.github/workflows/shared_infra_apply_from_plan.yml b/.github/workflows/shared_infra_apply_from_plan.yml index d96de1a7..c5b049a4 100644 --- a/.github/workflows/shared_infra_apply_from_plan.yml +++ b/.github/workflows/shared_infra_apply_from_plan.yml @@ -52,14 +52,3 @@ jobs: run: | echo "infra_version=$(jq -r '.infra_version' plan-metadata.json)" >> "$GITHUB_OUTPUT" echo "bootstrap_image_uri=$(jq -r '.bootstrap_image_uri' plan-metadata.json)" >> "$GITHUB_OUTPUT" - - infra: - needs: - - metadata - uses: ./.github/workflows/shared_infra.yml - with: - environment: ${{ inputs.environment }} - infra_version: ${{ needs.metadata.outputs.infra_version }} - bootstrap_image_uri: ${{ needs.metadata.outputs.bootstrap_image_uri }} - tg_action: apply_plan - plan_run_id: ${{ inputs.plan_artifact_run_id }} diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index f66025fb..65c1bb73 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -30,8 +30,6 @@ env: AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role AWS_REGION: ${{ vars.AWS_REGION }} TG_ACTION_LABEL: "Plan" - TG_ENABLE_PLAN_ARTIFACTS: "true" - PLAN_ARTIFACT_RUN_ID: ${{ github.run_id }} jobs: generate_waves: diff --git a/README.md b/README.md index 01f014a6..ab7085e5 100644 --- a/README.md +++ b/README.md @@ -311,7 +311,7 @@ Infrastructure apply and feature-code rollout are intentionally decoupled in thi - `*_infra` workflows apply infrastructure only - `*_code` workflows deploy feature code only - code deploy workflows publish the real Lambda versions and ECS task revisions into that pre-created deploy surface -- saved infra plans are stored in the shared S3 code bucket under `terragrunt_plan///...`, using the same artifact split as build outputs: `dev` writes to the `dev` code bucket and non-`dev` environments reuse the `ci` code bucket +- saved infra plans are stored as GitHub Actions artifacts keyed by workflow run id, with one run-level metadata artifact plus one per-stack plan artifact - Code artifact retention and infra-plan retention are configured separately in the shared code bucket module - rerunning infrastructure apply does not roll out new feature code - the shared Lambda and ECS module READMEs are the canonical source for bootstrap, rollout, and rollback details for each runtime shape diff --git a/infra/README.md b/infra/README.md index bf1ef627..e90e07ed 100644 --- a/infra/README.md +++ b/infra/README.md @@ -34,10 +34,7 @@ Shared artifact names also follow naming conventions from `infra/root.hcl`: - dedicated saved-plan bucket: `---tfplan` - code bucket: `-code` - ECS ECR repository: `-ecr` -- saved Terragrunt plan artifacts: `s3:///terragrunt_plan///...` -- plan-bucket retention: `infra_plan_artifact_expiration_days` applies an S3 lifecycle rule to `terragrunt_plan/` in the dedicated saved-plan bucket -- during `terragrunt init` and saved-plan `plan`, the root hook ensures the dedicated saved-plan bucket exists; interactive runs prompt before creation and non-interactive runs fail if no prompt is possible -- to reapply the configured `infra_plan_artifact_expiration_days` lifecycle rule locally for an existing bucket, rerun with `TG_RESET_PLAN_ARTIFACT_BUCKET=true` +- saved Terragrunt plan artifacts are stored as GitHub Actions artifacts keyed by workflow run id So a stack at: @@ -156,7 +153,7 @@ That `containers/lib` directory is helper code only and is not treated as a depl - build workflows produce Lambda zips and container images - `*_infra` wrappers need the inputs required to apply infra safely, such as directory-derived stack matrices and any artifact-derived bootstrap references - in `prod`, the `*_infra` wrappers read shared artifact resources from `ci` but only apply service and task stacks in `prod` -- saved `plan` / `apply_plan` artifacts live in the dedicated plan bucket under `terragrunt_plan///...` +- saved `plan` / `apply_plan` artifacts live in GitHub Actions artifacts keyed by workflow run id - each saved-plan stack always uploads `terragrunt.plan.meta.json`; the binary `terragrunt.tfplan` and rendered `terragrunt.plan.txt` are uploaded only when the plan contains real changes - deploy workflows: - publish Lambda versions and use Lambda CodeDeploy @@ -199,23 +196,17 @@ just --justfile justfile.deploy lambda-get-version just --justfile justfile.deploy frontend-build ``` -For a local saved-plan run that can upload plan artifacts through the normal repo wrapper, enable artifact mode, provide a unique run id, and pass the Terragrunt operation as one quoted argument: +For a local saved-plan run, pass the Terragrunt operation as one quoted argument: ```sh -TG_ENABLE_PLAN_ARTIFACTS=true \ -PLAN_ARTIFACT_RUN_ID="local-example-run" \ just tg dev aws/oidc 'plan -out=terragrunt.tfplan' ``` -The `tg` recipe treats the final argument as the Terragrunt operation string, so quoting lets you pass flags such as `-out=...` through the wrapper. The current saved-plan hook expects the binary plan filename to be `terragrunt.tfplan`; if you choose a different `-out` filename, the upload hook will not find it. - -Per-stack saved-plan bundles in S3 use the live stack identity rather than your full local filesystem path, for example `terragrunt-plan-dev-aws-oidc`. +The `tg` recipe treats the final argument as the Terragrunt operation string, so quoting lets you pass flags such as `-out=...` through the wrapper. The workflow saved-plan path expects the binary plan filename to be `terragrunt.tfplan`. To apply that same saved plan later, reuse the same run id: ```sh -TG_ENABLE_PLAN_ARTIFACTS=true \ -PLAN_ARTIFACT_RUN_ID="local-example-run" \ just tg dev aws/oidc 'apply terragrunt.tfplan' ``` diff --git a/infra/root.hcl b/infra/root.hcl index d8a18c33..1f86be72 100644 --- a/infra/root.hcl +++ b/infra/root.hcl @@ -12,7 +12,6 @@ locals { global_vars = read_terragrunt_config(find_in_parent_folders("global_vars.hcl")) environment_vars = read_terragrunt_config(find_in_parent_folders("environment_vars.hcl")) - infra_root_dir = abspath(dirname(find_in_parent_folders("root.hcl"))) project_name = element(split("/", local.github_repo), 1) @@ -21,13 +20,7 @@ locals { deploy_role_name = "${local.project_name}-${local.environment}-github-oidc-role" deploy_role_arn = "arn:aws:iam::${local.aws_account_id}:role/${local.deploy_role_name}" state_bucket = "${local.base_reference}-tfstate" - plan_bucket = "${local.base_reference}-tfplan" state_key = "${local.environment}/${local.provider}/${local.module}/terraform.tfstate" - plan_artifact_stack_key = "${local.environment}/${local.provider}/${local.module}" - plan_artifact_retention_days = try( - local.environment_vars.inputs.infra_plan_artifact_expiration_days, - 1, - ) # separate shared artifact resources when dev, otherwise ci artifact_base = local.environment == "dev" ? "${local.base_reference}-${local.environment}" : "${local.base_reference}-ci" code_bucket = "${local.artifact_base}-code" @@ -41,41 +34,6 @@ terraform { "bash", "-c", "echo STATE:${local.state_bucket}/${local.state_key} LOCKFILE:${local.state_key}.tflock" ] } - - before_hook "ensure_plan_artifact_bucket" { - commands = ["init", "plan"] - execute = [ - "bash", - "${local.infra_root_dir}/scripts/ensure-plan-artifact-bucket.sh", - local.plan_bucket, - local.aws_region, - tostring(local.plan_artifact_retention_days), - ] - } - - before_hook "download_saved_plan" { - commands = ["apply"] - execute = [ - "bash", - "${local.infra_root_dir}/scripts/handle-plan-artifact.sh", - "download", - local.plan_artifact_stack_key, - local.plan_bucket, - local.environment, - ] - } - - after_hook "upload_saved_plan" { - commands = ["plan"] - execute = [ - "bash", - "${local.infra_root_dir}/scripts/handle-plan-artifact.sh", - "upload", - local.plan_artifact_stack_key, - local.plan_bucket, - local.environment, - ] - } } remote_state { @@ -142,7 +100,6 @@ inputs = merge( deploy_role_name = local.deploy_role_name deploy_role_arn = local.deploy_role_arn state_bucket = local.state_bucket - plan_bucket = local.plan_bucket code_bucket = local.code_bucket ecr_repository_name = local.ecr_repository_name } diff --git a/infra/scripts/ensure-plan-artifact-bucket.sh b/infra/scripts/ensure-plan-artifact-bucket.sh deleted file mode 100644 index b1596b3e..00000000 --- a/infra/scripts/ensure-plan-artifact-bucket.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -bucket_name="${1:?bucket name is required}" -aws_region="${2:?aws region is required}" -retention_days="${3:-0}" -plan_prefix="${INFRA_PLAN_DIR:-terragrunt_plan/}" -reset_flag="${TG_RESET_PLAN_ARTIFACT_BUCKET:-false}" - -if [[ "$plan_prefix" != */ ]]; then - plan_prefix="${plan_prefix}/" -fi - -ensure_lifecycle() { - if [[ "$retention_days" =~ ^[0-9]+$ ]] && [ "$retention_days" -gt 0 ]; then - aws s3api put-bucket-lifecycle-configuration \ - --bucket "$bucket_name" \ - --lifecycle-configuration "{ - \"Rules\": [ - { - \"ID\": \"expire-plan-artifacts\", - \"Status\": \"Enabled\", - \"Filter\": {\"Prefix\": \"$plan_prefix\"}, - \"Expiration\": {\"Days\": $retention_days} - } - ] - }" >/dev/null - echo "Ensured plan artifact retention of ${retention_days} days on s3://${bucket_name}/${plan_prefix}" - fi -} - -if aws s3api head-bucket --bucket "$bucket_name" >/dev/null 2>&1; then - if [ "$reset_flag" = "true" ]; then - ensure_lifecycle - fi - exit 0 -fi - -if [ -r /dev/tty ] && [ -w /dev/tty ]; then - printf "Plan bucket '%s' does not exist. Create it in %s? [y/N] " "$bucket_name" "$aws_region" > /dev/tty - read -r response < /dev/tty - case "$response" in - [yY]|[yY][eE][sS]) ;; - *) - echo "Plan bucket creation declined." >&2 - exit 1 - ;; - esac -else - echo "Plan bucket '$bucket_name' does not exist and no interactive terminal is available for confirmation." >&2 - echo "Create it manually or rerun from a terminal where Terragrunt hooks can prompt." >&2 - exit 1 -fi - -if [ "$aws_region" = "us-east-1" ]; then - aws s3api create-bucket --bucket "$bucket_name" >/dev/null -else - aws s3api create-bucket --bucket "$bucket_name" --create-bucket-configuration "LocationConstraint=$aws_region" >/dev/null -fi - -ensure_lifecycle diff --git a/infra/scripts/handle-plan-artifact.sh b/infra/scripts/handle-plan-artifact.sh deleted file mode 100644 index 7d5e35e9..00000000 --- a/infra/scripts/handle-plan-artifact.sh +++ /dev/null @@ -1,88 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -mode="${1:?mode is required}" -logical_tg_dir="${2:?terragrunt directory is required}" -plan_bucket="${3:?plan bucket is required}" -environment="${4:?environment is required}" -infra_plan_dir="${INFRA_PLAN_DIR:-terragrunt_plan}" - -plan_path="${PWD}/terragrunt.tfplan" -plan_text_path="${PWD}/terragrunt.plan.txt" -plan_meta_path="${PWD}/terragrunt.plan.meta.json" -plan_json_path="${PWD}/terragrunt.plan.json" -plan_log_path="${PWD}/${TG_PLAN_LOG_FILENAME:-terragrunt.plan.log}" -fallback_plan_log_path="${TG_PLAN_LOG_ABS_PATH:-}" - -if [[ "${TG_ENABLE_PLAN_ARTIFACTS:-false}" != "true" ]]; then - echo "TG_ENABLE_PLAN_ARTIFACTS=false, skipping plan artifact ${mode}." >&2 - exit 0 -fi - -if [[ -z "${PLAN_ARTIFACT_RUN_ID:-}" ]]; then - echo "PLAN_ARTIFACT_RUN_ID is required when TG_ENABLE_PLAN_ARTIFACTS=true." >&2 - exit 1 -fi - -sanitized_dir="$(echo "$logical_tg_dir" | tr '/.' '--')" -artifact_s3_prefix="s3://${plan_bucket}/${infra_plan_dir}/${environment}/${PLAN_ARTIFACT_RUN_ID}/terragrunt-plan-${sanitized_dir}" - -case "$mode" in - download) - echo "Downloading plan artifacts from ${artifact_s3_prefix}" >&2 - if ! aws s3 ls "${artifact_s3_prefix}/terragrunt.tfplan" >/dev/null 2>&1; then - echo "Saved plan artifact not found for ${logical_tg_dir} and PLAN_ARTIFACT_RUN_ID=${PLAN_ARTIFACT_RUN_ID}." >&2 - echo "Expected plan bundle at ${artifact_s3_prefix}" >&2 - exit 1 - fi - - aws s3 cp "${artifact_s3_prefix}/terragrunt.tfplan" "$plan_path" - aws s3 cp "${artifact_s3_prefix}/terragrunt.plan.txt" "$plan_text_path" - aws s3 cp "${artifact_s3_prefix}/terragrunt.plan.meta.json" "$plan_meta_path" - echo "Downloaded plan artifacts for ${logical_tg_dir}" >&2 - - if [[ "$(jq -r '.contains_mocked_outputs // false' "$plan_meta_path")" == "true" ]]; then - echo "Saved plan for '${logical_tg_dir}' contains mocked outputs. Regenerate it after upstream real outputs exist." >&2 - exit 1 - fi - ;; - upload) - if [[ ! -f "$plan_path" ]]; then - exit 0 - fi - - terraform show -no-color "$plan_path" > "$plan_text_path" - terraform show -json "$plan_path" > "$plan_json_path" - - contains_mocked_outputs=false - if [[ -f "$plan_log_path" ]] && grep -Fq "mock outputs provided and returning those in dependency output" "$plan_log_path"; then - contains_mocked_outputs=true - elif [[ -n "$fallback_plan_log_path" ]] && [[ -f "$fallback_plan_log_path" ]] && grep -Fq "mock outputs provided and returning those in dependency output" "$fallback_plan_log_path"; then - contains_mocked_outputs=true - fi - - jq -n \ - --arg tg_directory "$logical_tg_dir" \ - --argjson has_changes "$(jq -r '([(.resource_changes // [])[]?.change.actions[]?] | any(. != "no-op")) or ((.output_changes // {}) | length > 0)' "$plan_json_path")" \ - --argjson contains_mocked_outputs "$contains_mocked_outputs" \ - '{tg_directory: $tg_directory, has_changes: $has_changes, contains_mocked_outputs: $contains_mocked_outputs}' \ - > "$plan_meta_path" - - echo "Uploading plan metadata for ${logical_tg_dir} to ${artifact_s3_prefix}" >&2 - aws s3 cp "$plan_meta_path" "${artifact_s3_prefix}/terragrunt.plan.meta.json" - - if [[ "$(jq -r '.has_changes' "$plan_meta_path")" == "true" ]]; then - aws s3 cp "$plan_path" "${artifact_s3_prefix}/terragrunt.tfplan" - aws s3 cp "$plan_text_path" "${artifact_s3_prefix}/terragrunt.plan.txt" - echo "Uploaded plan artifacts for ${logical_tg_dir}" >&2 - else - echo "Plan for ${logical_tg_dir} has no changes. Uploaded metadata only." >&2 - fi - - rm -f "$plan_json_path" - ;; - *) - echo "Unknown mode '$mode'." >&2 - exit 2 - ;; -esac From 96604f3a57f7de04ef42f1807f0d89454e3cc2c5 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 11:33:16 +0100 Subject: [PATCH 43/73] fix: fix use tg not tf --- .github/actions/terragrunt/action.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index bef4549d..5b375ade 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -106,7 +106,7 @@ runs: '{tg_directory: $tg_directory, contains_mocked_outputs: $contains_mocked_outputs}' \ > "$PLAN_META_PATH" - terraform show -no-color "$PLAN_PATH" > "$PLAN_TEXT_PATH" + terragrunt show -no-color "$PLAN_PATH" > "$PLAN_TEXT_PATH" echo "plan_exit_code=$plan_exit_code" >> "$GITHUB_OUTPUT" echo "plan_contains_mocked_outputs=$plan_contains_mocked_outputs" >> "$GITHUB_OUTPUT" From dbaab91ca1ae34bc624cb49a2ab3cc7d27c0426c Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 11:42:43 +0100 Subject: [PATCH 44/73] debug: test pull artifacts --- .github/workflows/shared_infra_apply_from_plan.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/shared_infra_apply_from_plan.yml b/.github/workflows/shared_infra_apply_from_plan.yml index c5b049a4..0a7bfca6 100644 --- a/.github/workflows/shared_infra_apply_from_plan.yml +++ b/.github/workflows/shared_infra_apply_from_plan.yml @@ -30,6 +30,13 @@ jobs: steps: - uses: actions/checkout@v6 + - name: Download all artifacts from plan run + uses: actions/download-artifact@v7 + with: + github-token: ${{ github.token }} + run-id: ${{ inputs.plan_artifact_run_id }} + path: downloaded-artifacts + - name: Download plan metadata artifact uses: actions/download-artifact@v7 with: From dd4751bbb99660de7b9bb45154e7e90ec6126500 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 11:50:28 +0100 Subject: [PATCH 45/73] debug: print out metadata --- .../workflows/shared_infra_apply_from_plan.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/.github/workflows/shared_infra_apply_from_plan.yml b/.github/workflows/shared_infra_apply_from_plan.yml index 0a7bfca6..f99278d6 100644 --- a/.github/workflows/shared_infra_apply_from_plan.yml +++ b/.github/workflows/shared_infra_apply_from_plan.yml @@ -37,6 +37,23 @@ jobs: run-id: ${{ inputs.plan_artifact_run_id }} path: downloaded-artifacts + - name: Print downloaded plan metadata JSON + shell: bash + run: | + shopt -s nullglob + mapfile -t metadata_files < <(find downloaded-artifacts -type f -name 'terragrunt.plan.meta.json' | sort) + + if [ "${#metadata_files[@]}" -eq 0 ]; then + echo "No terragrunt.plan.meta.json files found under downloaded-artifacts/" + exit 0 + fi + + for metadata_file in "${metadata_files[@]}"; do + echo "=== ${metadata_file} ===" + cat "${metadata_file}" + echo + done + - name: Download plan metadata artifact uses: actions/download-artifact@v7 with: From 5dae8422596ffc315e2fcb15fa4af82de86b797e Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 12:02:47 +0100 Subject: [PATCH 46/73] chore: mermaid for full flow --- .github/workflows/shared_get_modules.yml | 36 ++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index b29c7179..f863992e 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -116,9 +116,45 @@ jobs: WAVES_JSON: ${{ steps.filtered_waves.outputs.waves_json }} run: | printf '%s\n' "$WAVES_JSON" | tee waves.json + jq -r ' + def wave_id($wave): "wave_" + ($wave | tostring); + def node_id($module): "module_" + ($module | gsub("[^A-Za-z0-9_]"; "_")); + def wave_label($wave): "Wave " + ($wave | tostring); + def module_label($module): $module; + + [ + "flowchart LR", + ( + .[] + | " " + wave_id(.wave) + "[\"" + wave_label(.wave) + "\"]" + ), + ( + .[] + | .wave as $wave + | .modules[] + | " " + node_id(.) + "[\"" + module_label(.) + "\"]" + ), + ( + .[] + | .wave as $wave + | .modules[] + | " " + wave_id($wave) + " --> " + node_id(.) + ), + ( + [ .[] | .wave ] | sort | .[] as $wave + | select($wave > 0) + | " " + wave_id($wave - 1) + " --> " + wave_id($wave) + ) + ] + | .[] + ' waves.json > waves.mmd { echo "## Terragrunt Wave Matrix" echo + echo '```mermaid' + cat waves.mmd + echo '```' + echo echo '```json' cat waves.json echo From 8444716b09964168726fb1e9fa9ac187e9f02fac Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 13:13:12 +0100 Subject: [PATCH 47/73] chore: upload waves in plan --- .github/actions/terragrunt/README.md | 2 +- .github/docs/README.md | 2 +- .github/workflows/shared_infra_plan.yml | 4 +++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index e6eeb4b1..9c3b66b9 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -54,7 +54,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - One run-level metadata file is stored separately by the shared infra wrapper as a GitHub Actions artifact: - artifact name: `infra-plan-metadata` - - file: `plan-metadata.json` + - file: `plan-metadata.json` containing the frozen workflow inputs and derived `waves` - Each Terragrunt stack or module stores its own plan bundle as a GitHub Actions artifact named `terragrunt-plan--`: - `terragrunt.plan.meta.json` - `terragrunt.tfplan` only when changes exist diff --git a/.github/docs/README.md b/.github/docs/README.md index 609c17fa..4d10c18e 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -82,7 +82,7 @@ flowchart LR ### Infra And Code Rollout - `shared_infra_plan.yml` - Plan wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly, uploads one run-level `plan-metadata.json` file as a GitHub Actions artifact named `infra-plan-metadata`, starts `shared_get_modules.yml` in parallel to derive the current wave outputs, and then calls `shared_infra.yml` with `tg_action: plan` plus `plan_run_id: ${{ github.run_id }}`. After the plan completes, it prints the current workflow `github.run_id` into both the logs and the GitHub Actions step summary as `plan_artifact_run_id`, and exposes that value as a reusable-workflow output. + Plan wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly, starts `shared_get_modules.yml` to derive the current wave outputs, writes those waves plus the direct workflow inputs into one run-level `plan-metadata.json` file, uploads that file as a GitHub Actions artifact named `infra-plan-metadata`, and then calls `shared_infra.yml` with `tg_action: plan` plus `plan_run_id: ${{ github.run_id }}`. After the plan completes, it prints the current workflow `github.run_id` into both the logs and the GitHub Actions step summary as `plan_artifact_run_id`, and exposes that value as a reusable-workflow output. - `shared_infra_apply.yml` Direct-input apply wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly and calls `shared_infra.yml` with `tg_action: apply`. - `shared_infra_apply_from_plan.yml` diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index 65c1bb73..4b8ee81d 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -40,6 +40,7 @@ jobs: ignore_task_modules: true metadata: + needs: generate_waves runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 @@ -52,7 +53,8 @@ jobs: cat > plan-metadata.json < Date: Fri, 22 May 2026 13:20:43 +0100 Subject: [PATCH 48/73] debug: echo out plan files --- .github/docs/README.md | 2 +- .../shared_infra_apply_from_plan.yml | 210 ++++++++++++++++-- 2 files changed, 187 insertions(+), 25 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 4d10c18e..7ea5bf19 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -86,7 +86,7 @@ flowchart LR - `shared_infra_apply.yml` Direct-input apply wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly and calls `shared_infra.yml` with `tg_action: apply`. - `shared_infra_apply_from_plan.yml` - Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs back out, and then calls `shared_infra.yml` with `tg_action: apply_plan` plus `plan_run_id: `. Per-stack plan bundle download still happens inside the shared Terragrunt root `before_hook`. + Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs and saved wave arrays back out, and then reruns the same `wave_0`, `wave_1`, and `wave_2` module order. For now each per-module job only downloads its matching `terragrunt-plan--` GitHub artifact and prints the recovered plan metadata/text files plus the binary plan file presence for inspection. - `shared_infra.yml` Temporary wave-based executor. It delegates graph and wave discovery to `shared_get_modules.yml`, exposes the resulting `waves_json` as the reusable-workflow output, and then runs `wave_0`, `wave_1`, and `wave_2` jobs in dependency order. Each wave only runs when its module array is non-empty, fans that array out as a matrix, checks out the requested ref, configures AWS credentials once per matrix job, and invokes the shared repo-local Terragrunt action against `infra/live//aws/`. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. - `just --justfile justfile.ci tg-graph-json-to-waves` diff --git a/.github/workflows/shared_infra_apply_from_plan.yml b/.github/workflows/shared_infra_apply_from_plan.yml index f99278d6..864b057d 100644 --- a/.github/workflows/shared_infra_apply_from_plan.yml +++ b/.github/workflows/shared_infra_apply_from_plan.yml @@ -27,33 +27,13 @@ jobs: outputs: infra_version: ${{ steps.read_metadata.outputs.infra_version }} bootstrap_image_uri: ${{ steps.read_metadata.outputs.bootstrap_image_uri }} + waves_json: ${{ steps.read_metadata.outputs.waves_json }} + wave_0_modules: ${{ steps.read_metadata.outputs.wave_0_modules }} + wave_1_modules: ${{ steps.read_metadata.outputs.wave_1_modules }} + wave_2_modules: ${{ steps.read_metadata.outputs.wave_2_modules }} steps: - uses: actions/checkout@v6 - - name: Download all artifacts from plan run - uses: actions/download-artifact@v7 - with: - github-token: ${{ github.token }} - run-id: ${{ inputs.plan_artifact_run_id }} - path: downloaded-artifacts - - - name: Print downloaded plan metadata JSON - shell: bash - run: | - shopt -s nullglob - mapfile -t metadata_files < <(find downloaded-artifacts -type f -name 'terragrunt.plan.meta.json' | sort) - - if [ "${#metadata_files[@]}" -eq 0 ]; then - echo "No terragrunt.plan.meta.json files found under downloaded-artifacts/" - exit 0 - fi - - for metadata_file in "${metadata_files[@]}"; do - echo "=== ${metadata_file} ===" - cat "${metadata_file}" - echo - done - - name: Download plan metadata artifact uses: actions/download-artifact@v7 with: @@ -76,3 +56,185 @@ jobs: run: | echo "infra_version=$(jq -r '.infra_version' plan-metadata.json)" >> "$GITHUB_OUTPUT" echo "bootstrap_image_uri=$(jq -r '.bootstrap_image_uri' plan-metadata.json)" >> "$GITHUB_OUTPUT" + echo "waves_json=$(jq -c '.waves // []' plan-metadata.json)" >> "$GITHUB_OUTPUT" + echo "wave_0_modules=$(jq -c '.waves[0].modules // []' plan-metadata.json)" >> "$GITHUB_OUTPUT" + echo "wave_1_modules=$(jq -c '.waves[1].modules // []' plan-metadata.json)" >> "$GITHUB_OUTPUT" + echo "wave_2_modules=$(jq -c '.waves[2].modules // []' plan-metadata.json)" >> "$GITHUB_OUTPUT" + + wave_0: + needs: metadata + name: 0 / ${{ matrix.module }} + if: ${{ needs.metadata.outputs.wave_0_modules != '[]' }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.metadata.outputs.wave_0_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ needs.metadata.outputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: Download saved plan artifact + uses: actions/download-artifact@v7 + with: + name: terragrunt-plan-${{ inputs.environment }}-${{ matrix.module }} + github-token: ${{ github.token }} + run-id: ${{ inputs.plan_artifact_run_id }} + path: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + + - name: Print saved plan files + shell: bash + run: | + PLAN_DIR="infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}" + META_PATH="$PLAN_DIR/terragrunt.plan.meta.json" + TEXT_PATH="$PLAN_DIR/terragrunt.plan.txt" + BINARY_PATH="$PLAN_DIR/terragrunt.tfplan" + + echo "=== ${META_PATH} ===" + if [ -f "$META_PATH" ]; then + cat "$META_PATH" + else + echo "missing" + fi + echo + + echo "=== ${TEXT_PATH} ===" + if [ -f "$TEXT_PATH" ]; then + cat "$TEXT_PATH" + else + echo "missing" + fi + echo + + echo "=== ${BINARY_PATH} ===" + if [ -f "$BINARY_PATH" ]; then + ls -lh "$BINARY_PATH" + else + echo "missing" + fi + + wave_1: + needs: + - metadata + - wave_0 + name: 1 / ${{ matrix.module }} + if: ${{ always() && needs.metadata.outputs.wave_1_modules != '[]' && (needs.wave_0.result == 'success' || needs.wave_0.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.metadata.outputs.wave_1_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ needs.metadata.outputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: Download saved plan artifact + uses: actions/download-artifact@v7 + with: + name: terragrunt-plan-${{ inputs.environment }}-${{ matrix.module }} + github-token: ${{ github.token }} + run-id: ${{ inputs.plan_artifact_run_id }} + path: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + + - name: Print saved plan files + shell: bash + run: | + PLAN_DIR="infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}" + META_PATH="$PLAN_DIR/terragrunt.plan.meta.json" + TEXT_PATH="$PLAN_DIR/terragrunt.plan.txt" + BINARY_PATH="$PLAN_DIR/terragrunt.tfplan" + + echo "=== ${META_PATH} ===" + if [ -f "$META_PATH" ]; then + cat "$META_PATH" + else + echo "missing" + fi + echo + + echo "=== ${TEXT_PATH} ===" + if [ -f "$TEXT_PATH" ]; then + cat "$TEXT_PATH" + else + echo "missing" + fi + echo + + echo "=== ${BINARY_PATH} ===" + if [ -f "$BINARY_PATH" ]; then + ls -lh "$BINARY_PATH" + else + echo "missing" + fi + + wave_2: + needs: + - metadata + - wave_1 + name: 2 / ${{ matrix.module }} + if: ${{ always() && needs.metadata.outputs.wave_2_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.metadata.outputs.wave_2_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ needs.metadata.outputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: Download saved plan artifact + uses: actions/download-artifact@v7 + with: + name: terragrunt-plan-${{ inputs.environment }}-${{ matrix.module }} + github-token: ${{ github.token }} + run-id: ${{ inputs.plan_artifact_run_id }} + path: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + + - name: Print saved plan files + shell: bash + run: | + PLAN_DIR="infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}" + META_PATH="$PLAN_DIR/terragrunt.plan.meta.json" + TEXT_PATH="$PLAN_DIR/terragrunt.plan.txt" + BINARY_PATH="$PLAN_DIR/terragrunt.tfplan" + + echo "=== ${META_PATH} ===" + if [ -f "$META_PATH" ]; then + cat "$META_PATH" + else + echo "missing" + fi + echo + + echo "=== ${TEXT_PATH} ===" + if [ -f "$TEXT_PATH" ]; then + cat "$TEXT_PATH" + else + echo "missing" + fi + echo + + echo "=== ${BINARY_PATH} ===" + if [ -f "$BINARY_PATH" ]; then + ls -lh "$BINARY_PATH" + else + echo "missing" + fi From 89653b738667d4f9cc3505d39db684cf4518cdab Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 13:35:54 +0100 Subject: [PATCH 49/73] chore: cat plan-metadata.json --- .github/workflows/shared_infra_apply_from_plan.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/shared_infra_apply_from_plan.yml b/.github/workflows/shared_infra_apply_from_plan.yml index 864b057d..70ff562e 100644 --- a/.github/workflows/shared_infra_apply_from_plan.yml +++ b/.github/workflows/shared_infra_apply_from_plan.yml @@ -50,6 +50,12 @@ jobs: exit 1 fi + - name: Print recovered plan metadata JSON + shell: bash + run: | + echo "=== plan-metadata.json ===" + cat plan-metadata.json + - name: Read plan metadata id: read_metadata shell: bash From 638b8d73c0144f7bd2c67422cb99d19476396cf2 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 13:45:56 +0100 Subject: [PATCH 50/73] chore: rn to no plan --- .github/docs/README.md | 4 ++-- .github/workflows/dev_infra_apply.yml | 2 +- .github/workflows/prod_infra_apply.yml | 2 +- ...{shared_infra_apply.yml => shared_infra_apply_no_plan.yml} | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) rename .github/workflows/{shared_infra_apply.yml => shared_infra_apply_no_plan.yml} (99%) diff --git a/.github/docs/README.md b/.github/docs/README.md index 7ea5bf19..8150c821 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -19,7 +19,7 @@ Use it when you need to understand: - Release and validation: `release.yml`, `pull_request.yml` - Shared artifact prep and build: `shared_infra_releases.yml`, `shared_build.yml`, `shared_build_get.yml` -- Shared infra and code rollout: `shared_infra_plan.yml`, `shared_infra_apply.yml`, `shared_infra_apply_from_plan.yml`, `shared_infra.yml`, `shared_deploy.yml`, `shared_directories_get.yml` +- Shared infra and code rollout: `shared_infra_plan.yml`, `shared_infra_apply_no_plan.yml`, `shared_infra_apply_from_plan.yml`, `shared_infra.yml`, `shared_deploy.yml`, `shared_directories_get.yml` - Environment entry points: `dev_infra_apply.yml`, `dev_infra_plan.yml`, `dev_infra_plan_and_apply.yml`, `dev_infra_apply_from_plan.yml`, `dev_code_deploy.yml`, `prod_infra_apply.yml`, `prod_infra_plan.yml`, `prod_infra_apply_from_plan.yml` - Cleanup: `destroy.yml` @@ -83,7 +83,7 @@ flowchart LR - `shared_infra_plan.yml` Plan wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly, starts `shared_get_modules.yml` to derive the current wave outputs, writes those waves plus the direct workflow inputs into one run-level `plan-metadata.json` file, uploads that file as a GitHub Actions artifact named `infra-plan-metadata`, and then calls `shared_infra.yml` with `tg_action: plan` plus `plan_run_id: ${{ github.run_id }}`. After the plan completes, it prints the current workflow `github.run_id` into both the logs and the GitHub Actions step summary as `plan_artifact_run_id`, and exposes that value as a reusable-workflow output. -- `shared_infra_apply.yml` +- `shared_infra_apply_no_plan.yml` Direct-input apply wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly and calls `shared_infra.yml` with `tg_action: apply`. - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs and saved wave arrays back out, and then reruns the same `wave_0`, `wave_1`, and `wave_2` module order. For now each per-module job only downloads its matching `terragrunt-plan--` GitHub artifact and prints the recovered plan metadata/text files plus the binary plan file presence for inspection. diff --git a/.github/workflows/dev_infra_apply.yml b/.github/workflows/dev_infra_apply.yml index 1ac7f726..4af79338 100644 --- a/.github/workflows/dev_infra_apply.yml +++ b/.github/workflows/dev_infra_apply.yml @@ -11,7 +11,7 @@ permissions: jobs: infra: name: Apply - uses: ./.github/workflows/shared_infra_apply.yml + uses: ./.github/workflows/shared_infra_apply_no_plan.yml with: environment: dev infra_version: ${{ github.sha }} diff --git a/.github/workflows/prod_infra_apply.yml b/.github/workflows/prod_infra_apply.yml index 1f57574d..24d7a7c3 100644 --- a/.github/workflows/prod_infra_apply.yml +++ b/.github/workflows/prod_infra_apply.yml @@ -22,7 +22,7 @@ jobs: name: Apply needs: - get_build - uses: ./.github/workflows/shared_infra_apply.yml + uses: ./.github/workflows/shared_infra_apply_no_plan.yml with: environment: prod infra_version: 0.19.13 diff --git a/.github/workflows/shared_infra_apply.yml b/.github/workflows/shared_infra_apply_no_plan.yml similarity index 99% rename from .github/workflows/shared_infra_apply.yml rename to .github/workflows/shared_infra_apply_no_plan.yml index 367dc1bd..274486d4 100644 --- a/.github/workflows/shared_infra_apply.yml +++ b/.github/workflows/shared_infra_apply_no_plan.yml @@ -1,4 +1,4 @@ -name: Shared Infra Apply +name: Shared Infra Apply No Plan on: workflow_call: From 884865c2d32b260ba8db758020dc198c5793fe52 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 13:48:25 +0100 Subject: [PATCH 51/73] chore: rn to no_plan other ymls --- .github/docs/README.md | 10 +++++----- ...dev_infra_apply.yml => dev_infra_apply_no_plan.yml} | 0 ...od_infra_apply.yml => prod_infra_apply_no_plan.yml} | 0 3 files changed, 5 insertions(+), 5 deletions(-) rename .github/workflows/{dev_infra_apply.yml => dev_infra_apply_no_plan.yml} (100%) rename .github/workflows/{prod_infra_apply.yml => prod_infra_apply_no_plan.yml} (100%) diff --git a/.github/docs/README.md b/.github/docs/README.md index 8150c821..3e439edc 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -20,7 +20,7 @@ Use it when you need to understand: - Release and validation: `release.yml`, `pull_request.yml` - Shared artifact prep and build: `shared_infra_releases.yml`, `shared_build.yml`, `shared_build_get.yml` - Shared infra and code rollout: `shared_infra_plan.yml`, `shared_infra_apply_no_plan.yml`, `shared_infra_apply_from_plan.yml`, `shared_infra.yml`, `shared_deploy.yml`, `shared_directories_get.yml` -- Environment entry points: `dev_infra_apply.yml`, `dev_infra_plan.yml`, `dev_infra_plan_and_apply.yml`, `dev_infra_apply_from_plan.yml`, `dev_code_deploy.yml`, `prod_infra_apply.yml`, `prod_infra_plan.yml`, `prod_infra_apply_from_plan.yml` +- Environment entry points: `dev_infra_apply_no_plan.yml`, `dev_infra_plan.yml`, `dev_infra_plan_and_apply.yml`, `dev_infra_apply_from_plan.yml`, `dev_code_deploy.yml`, `prod_infra_apply_no_plan.yml`, `prod_infra_plan.yml`, `prod_infra_apply_from_plan.yml` - Cleanup: `destroy.yml` ## Workflow Contracts @@ -110,7 +110,7 @@ flowchart LR ### Wrapper Workflows -- `dev_infra_apply.yml` +- `dev_infra_apply_no_plan.yml` Entry point for dev infra apply. It currently calls the shared infra workflow directly with an empty placeholder `bootstrap_image_uri`, because the temporary wave-placeholder executor does not yet consume that old artifact input. - `dev_infra_plan.yml` Entry point for dev infra plan. It currently calls the shared infra plan wrapper directly with an empty placeholder `bootstrap_image_uri`, because the temporary wave-placeholder executor does not yet consume that old artifact input. @@ -118,7 +118,7 @@ flowchart LR Entry point for dev infra plan-then-apply. It captures the current workflow `run_id` as plan context, runs the shared infra wrapper in direct-input `plan` mode so that the wrapper emits both plan artifacts and `infra-plan-metadata`, and then reruns the same ordered infra graph in metadata-backed `apply_plan` mode. - `prod_infra_plan.yml` Entry point for prod infra plan. It resolves released artifacts from `ci` and then runs the shared infra wrapper in direct-input `plan` mode so that it emits both the reusable metadata artifact and the derived per-stack plan artifacts for that resolved input set. -- `prod_infra_apply.yml` +- `prod_infra_apply_no_plan.yml` Entry point for prod infra apply using shared artifacts from `ci`. - `dev_infra_apply_from_plan.yml` Entry point for dev infra apply-from-plan. It takes a prior `plan_artifact_run_id` from an earlier `dev_infra_plan.yml` or `dev_infra_plan_and_apply.yml` run and reruns the ordered dev infra graph through `shared_infra_apply_from_plan.yml`. @@ -221,7 +221,7 @@ Run these checks on every CI, workflow, or deploy-contract change. These are the workflows most users trigger directly. -- `dev_infra_apply.yml` +- `dev_infra_apply_no_plan.yml` Discovers directories, prepares dev artifacts, and applies dev infrastructure. - `dev_infra_plan.yml` Discovers directories, prepares dev artifacts, and plans the ordered dev infra graph through `shared_infra_plan.yml`. @@ -231,7 +231,7 @@ These are the workflows most users trigger directly. Reapplies the ordered dev infra graph from plan artifacts created by an earlier dev plan run, using the shared `plan_artifact_run_id` contract end-to-end. - `prod_infra_plan.yml` Resolves released artifacts from `ci`, then plans the ordered prod infra graph so that shared infra emits both the metadata artifact and the derived per-stack plan artifacts. -- `prod_infra_apply.yml` +- `prod_infra_apply_no_plan.yml` Resolves released artifacts from `ci` and applies prod infrastructure. - `prod_infra_apply_from_plan.yml` Reapplies the ordered prod infra graph from plan artifacts created by a prior `prod_infra_plan` run, using the shared `plan_artifact_run_id` contract end-to-end. `shared_infra_apply_from_plan.yml` reads the matching metadata artifact first, then each apply job downloads its matching per-stack GitHub artifact before invoking `apply_plan`. diff --git a/.github/workflows/dev_infra_apply.yml b/.github/workflows/dev_infra_apply_no_plan.yml similarity index 100% rename from .github/workflows/dev_infra_apply.yml rename to .github/workflows/dev_infra_apply_no_plan.yml diff --git a/.github/workflows/prod_infra_apply.yml b/.github/workflows/prod_infra_apply_no_plan.yml similarity index 100% rename from .github/workflows/prod_infra_apply.yml rename to .github/workflows/prod_infra_apply_no_plan.yml From 61ab74c56c644310c744f9fbe8c15cb5dbfc851f Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 14:04:28 +0100 Subject: [PATCH 52/73] fix: amend waves to correct order --- .github/docs/README.md | 4 +- .github/workflows/shared_get_modules.yml | 5 ++ .../shared_infra_apply_from_plan.yml | 62 +++++++++++++++++++ .../workflows/shared_infra_apply_no_plan.yml | 27 ++++++++ .github/workflows/shared_infra_plan.yml | 38 ++++++++++++ infra/live/dev/aws/cluster/terragrunt.hcl | 4 ++ infra/live/dev/aws/cognito/terragrunt.hcl | 4 ++ infra/live/dev/aws/lambda_api/terragrunt.hcl | 4 ++ .../live/dev/aws/lambda_worker/terragrunt.hcl | 4 ++ infra/live/dev/aws/messaging/terragrunt.hcl | 4 ++ .../live/dev/aws/observability/terragrunt.hcl | 4 ++ infra/live/dev/aws/service_api/terragrunt.hcl | 4 ++ .../dev/aws/service_worker/terragrunt.hcl | 4 ++ infra/live/dev/aws/task_api/terragrunt.hcl | 4 ++ infra/live/dev/aws/task_worker/terragrunt.hcl | 4 ++ infra/live/prod/aws/cluster/terragrunt.hcl | 4 ++ infra/live/prod/aws/cognito/terragrunt.hcl | 4 ++ infra/live/prod/aws/lambda_api/terragrunt.hcl | 4 ++ .../prod/aws/lambda_worker/terragrunt.hcl | 4 ++ infra/live/prod/aws/messaging/terragrunt.hcl | 4 ++ .../prod/aws/observability/terragrunt.hcl | 4 ++ .../live/prod/aws/service_api/terragrunt.hcl | 4 ++ .../prod/aws/service_worker/terragrunt.hcl | 4 ++ infra/live/prod/aws/task_api/terragrunt.hcl | 4 ++ .../live/prod/aws/task_worker/terragrunt.hcl | 4 ++ 25 files changed, 214 insertions(+), 2 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 3e439edc..9919c556 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -88,7 +88,7 @@ flowchart LR - `shared_infra_apply_from_plan.yml` Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs and saved wave arrays back out, and then reruns the same `wave_0`, `wave_1`, and `wave_2` module order. For now each per-module job only downloads its matching `terragrunt-plan--` GitHub artifact and prints the recovered plan metadata/text files plus the binary plan file presence for inspection. - `shared_infra.yml` - Temporary wave-based executor. It delegates graph and wave discovery to `shared_get_modules.yml`, exposes the resulting `waves_json` as the reusable-workflow output, and then runs `wave_0`, `wave_1`, and `wave_2` jobs in dependency order. Each wave only runs when its module array is non-empty, fans that array out as a matrix, checks out the requested ref, configures AWS credentials once per matrix job, and invokes the shared repo-local Terragrunt action against `infra/live//aws/`. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. + Temporary wave-based executor. It delegates graph and wave discovery to `shared_get_modules.yml`, exposes the resulting `waves_json` as the reusable-workflow output, and then runs `wave_0`, `wave_1`, `wave_2`, and `wave_3` jobs in dependency order. Each wave only runs when its module array is non-empty, fans that array out as a matrix, checks out the requested ref, configures AWS credentials once per matrix job, and invokes the shared repo-local Terragrunt action against `infra/live//aws/`. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. - `just --justfile justfile.ci tg-graph-json-to-waves` CI helper that expects compact graph JSON in `TG_GRAPH_JSON` and returns a sequential JSON array of wave objects like `[{ "wave": 0, "modules": [...] }, ...]`, with each wave containing only modules whose direct dependencies were satisfied by earlier waves. - The shared infra wrappers must forward the permissions required by the nested reusable call chain. In practice that means `id-token: write` everywhere the Terragrunt action may assume AWS OIDC and `contents: read` for checkout. The shared plan/apply wrappers now rely on AWS access to the shared code bucket rather than GitHub artifact permissions for cross-run recovery. @@ -137,7 +137,7 @@ flowchart LR Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. - `shared_get_modules.yml` - Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, and `wave_2_modules` as reusable-workflow outputs. Callers can pass `ignore_task_modules: true` to exclude any `task_*` modules from the emitted rollout waves for the current bootstrap-oriented infra path. The current shared workflow contract therefore assumes the graph fits within three waves; if Terragrunt HCL dependency changes introduce a fourth wave, the reusable outputs and `shared_infra.yml` wave jobs must be extended in the same change. + Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, `wave_2_modules`, and `wave_3_modules` as reusable-workflow outputs. Callers can pass `ignore_task_modules: true` to exclude any `task_*` modules from the emitted rollout waves for the current bootstrap-oriented infra path. ## Feasibility Checks diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index f863992e..026a6f45 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -29,6 +29,9 @@ on: wave_2_modules: description: "Wave 2 module array" value: ${{ jobs.generate_waves.outputs.wave_2_modules }} + wave_3_modules: + description: "Wave 3 module array" + value: ${{ jobs.generate_waves.outputs.wave_3_modules }} permissions: id-token: write @@ -47,6 +50,7 @@ jobs: wave_0_modules: ${{ steps.wave_outputs.outputs.wave_0_modules }} wave_1_modules: ${{ steps.wave_outputs.outputs.wave_1_modules }} wave_2_modules: ${{ steps.wave_outputs.outputs.wave_2_modules }} + wave_3_modules: ${{ steps.wave_outputs.outputs.wave_3_modules }} steps: - uses: actions/checkout@v6 with: @@ -109,6 +113,7 @@ jobs: echo "wave_0_modules=$(jq -c '.[0].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" echo "wave_1_modules=$(jq -c '.[1].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" echo "wave_2_modules=$(jq -c '.[2].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" + echo "wave_3_modules=$(jq -c '.[3].modules // []' <<<"$WAVES_JSON")" >> "$GITHUB_OUTPUT" - name: Print Terragrunt wave matrix shell: bash diff --git a/.github/workflows/shared_infra_apply_from_plan.yml b/.github/workflows/shared_infra_apply_from_plan.yml index 70ff562e..799d8dec 100644 --- a/.github/workflows/shared_infra_apply_from_plan.yml +++ b/.github/workflows/shared_infra_apply_from_plan.yml @@ -31,6 +31,7 @@ jobs: wave_0_modules: ${{ steps.read_metadata.outputs.wave_0_modules }} wave_1_modules: ${{ steps.read_metadata.outputs.wave_1_modules }} wave_2_modules: ${{ steps.read_metadata.outputs.wave_2_modules }} + wave_3_modules: ${{ steps.read_metadata.outputs.wave_3_modules }} steps: - uses: actions/checkout@v6 @@ -66,6 +67,7 @@ jobs: echo "wave_0_modules=$(jq -c '.waves[0].modules // []' plan-metadata.json)" >> "$GITHUB_OUTPUT" echo "wave_1_modules=$(jq -c '.waves[1].modules // []' plan-metadata.json)" >> "$GITHUB_OUTPUT" echo "wave_2_modules=$(jq -c '.waves[2].modules // []' plan-metadata.json)" >> "$GITHUB_OUTPUT" + echo "wave_3_modules=$(jq -c '.waves[3].modules // []' plan-metadata.json)" >> "$GITHUB_OUTPUT" wave_0: needs: metadata @@ -244,3 +246,63 @@ jobs: else echo "missing" fi + + wave_3: + needs: + - metadata + - wave_2 + name: 3 / ${{ matrix.module }} + if: ${{ always() && needs.metadata.outputs.wave_3_modules != '[]' && (needs.wave_2.result == 'success' || needs.wave_2.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.metadata.outputs.wave_3_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ needs.metadata.outputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: Download saved plan artifact + uses: actions/download-artifact@v7 + with: + name: terragrunt-plan-${{ inputs.environment }}-${{ matrix.module }} + github-token: ${{ github.token }} + run-id: ${{ inputs.plan_artifact_run_id }} + path: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + + - name: Print saved plan files + shell: bash + run: | + PLAN_DIR="infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}" + META_PATH="$PLAN_DIR/terragrunt.plan.meta.json" + TEXT_PATH="$PLAN_DIR/terragrunt.plan.txt" + BINARY_PATH="$PLAN_DIR/terragrunt.tfplan" + + echo "=== ${META_PATH} ===" + if [ -f "$META_PATH" ]; then + cat "$META_PATH" + else + echo "missing" + fi + echo + + echo "=== ${TEXT_PATH} ===" + if [ -f "$TEXT_PATH" ]; then + cat "$TEXT_PATH" + else + echo "missing" + fi + echo + + echo "=== ${BINARY_PATH} ===" + if [ -f "$BINARY_PATH" ]; then + ls -lh "$BINARY_PATH" + else + echo "missing" + fi diff --git a/.github/workflows/shared_infra_apply_no_plan.yml b/.github/workflows/shared_infra_apply_no_plan.yml index 274486d4..002bf3be 100644 --- a/.github/workflows/shared_infra_apply_no_plan.yml +++ b/.github/workflows/shared_infra_apply_no_plan.yml @@ -113,3 +113,30 @@ jobs: with: tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: apply + + wave_3: + needs: + - generate_waves + - wave_2 + name: 3 / ${{ matrix.module }} + if: ${{ always() && needs.generate_waves.outputs.wave_3_modules != '[]' && (needs.wave_2.result == 'success' || needs.wave_2.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_3_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: apply diff --git a/.github/workflows/shared_infra_plan.yml b/.github/workflows/shared_infra_plan.yml index 4b8ee81d..8836cbdf 100644 --- a/.github/workflows/shared_infra_plan.yml +++ b/.github/workflows/shared_infra_plan.yml @@ -175,6 +175,44 @@ jobs: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.txt if-no-files-found: warn + wave_3: + needs: + - generate_waves + - wave_2 + name: 3 / ${{ matrix.module }} + if: ${{ always() && needs.generate_waves.outputs.wave_3_modules != '[]' && (needs.wave_2.result == 'success' || needs.wave_2.result == 'skipped') }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.generate_waves.outputs.wave_3_modules) }} + steps: + - uses: actions/checkout@v6 + with: + ref: ${{ inputs.infra_version }} + + - uses: aws-actions/configure-aws-credentials@v6 + with: + role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} + aws-region: ${{ env.AWS_REGION }} + + - name: ${{ env.TG_ACTION_LABEL }} ${{ matrix.module }} infra + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: plan + + - name: Upload plan files + if: ${{ success() }} + uses: actions/upload-artifact@v7 + with: + name: terragrunt-plan-${{ inputs.environment }}-${{ matrix.module }} + path: | + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.meta.json + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.tfplan + infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}/terragrunt.plan.txt + if-no-files-found: warn + plan_context: name: Plan Context runs-on: ubuntu-latest diff --git a/infra/live/dev/aws/cluster/terragrunt.hcl b/infra/live/dev/aws/cluster/terragrunt.hcl index 8d29bd5d..b52fb59a 100644 --- a/infra/live/dev/aws/cluster/terragrunt.hcl +++ b/infra/live/dev/aws/cluster/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//_shared//cluster" } diff --git a/infra/live/dev/aws/cognito/terragrunt.hcl b/infra/live/dev/aws/cognito/terragrunt.hcl index 6e2da3c4..5f937331 100644 --- a/infra/live/dev/aws/cognito/terragrunt.hcl +++ b/infra/live/dev/aws/cognito/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//cognito" } diff --git a/infra/live/dev/aws/lambda_api/terragrunt.hcl b/infra/live/dev/aws/lambda_api/terragrunt.hcl index e11cf036..1fed78c8 100644 --- a/infra/live/dev/aws/lambda_api/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_api/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../security", "../database"] +} + dependency "network" { config_path = "${get_original_terragrunt_dir()}/../network" diff --git a/infra/live/dev/aws/lambda_worker/terragrunt.hcl b/infra/live/dev/aws/lambda_worker/terragrunt.hcl index 5028ff8b..09340257 100644 --- a/infra/live/dev/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_worker/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../security", "../network", "../database"] +} + dependency "messaging" { config_path = "${get_original_terragrunt_dir()}/../messaging" diff --git a/infra/live/dev/aws/messaging/terragrunt.hcl b/infra/live/dev/aws/messaging/terragrunt.hcl index b85cf717..8cf46acc 100644 --- a/infra/live/dev/aws/messaging/terragrunt.hcl +++ b/infra/live/dev/aws/messaging/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//messaging" } diff --git a/infra/live/dev/aws/observability/terragrunt.hcl b/infra/live/dev/aws/observability/terragrunt.hcl index 3a37c9ba..d3cae1c1 100644 --- a/infra/live/dev/aws/observability/terragrunt.hcl +++ b/infra/live/dev/aws/observability/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//observability" } diff --git a/infra/live/dev/aws/service_api/terragrunt.hcl b/infra/live/dev/aws/service_api/terragrunt.hcl index 8495dba7..e216b716 100644 --- a/infra/live/dev/aws/service_api/terragrunt.hcl +++ b/infra/live/dev/aws/service_api/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../database", "../messaging"] +} + dependency "ecr" { config_path = "${get_original_terragrunt_dir()}/../ecr" diff --git a/infra/live/dev/aws/service_worker/terragrunt.hcl b/infra/live/dev/aws/service_worker/terragrunt.hcl index 9c1d3c2d..8a75266c 100644 --- a/infra/live/dev/aws/service_worker/terragrunt.hcl +++ b/infra/live/dev/aws/service_worker/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../database"] +} + dependency "ecr" { config_path = "${get_original_terragrunt_dir()}/../ecr" diff --git a/infra/live/dev/aws/task_api/terragrunt.hcl b/infra/live/dev/aws/task_api/terragrunt.hcl index af324e86..0d192fda 100644 --- a/infra/live/dev/aws/task_api/terragrunt.hcl +++ b/infra/live/dev/aws/task_api/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../security", "../cluster", "../network"] +} + terraform { source = "../../../../modules//aws//task_api" } diff --git a/infra/live/dev/aws/task_worker/terragrunt.hcl b/infra/live/dev/aws/task_worker/terragrunt.hcl index c315b177..fb0d4fc2 100644 --- a/infra/live/dev/aws/task_worker/terragrunt.hcl +++ b/infra/live/dev/aws/task_worker/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../security", "../network"] +} + dependency "messaging" { config_path = "${get_original_terragrunt_dir()}/../messaging" diff --git a/infra/live/prod/aws/cluster/terragrunt.hcl b/infra/live/prod/aws/cluster/terragrunt.hcl index 8d29bd5d..b52fb59a 100644 --- a/infra/live/prod/aws/cluster/terragrunt.hcl +++ b/infra/live/prod/aws/cluster/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//_shared//cluster" } diff --git a/infra/live/prod/aws/cognito/terragrunt.hcl b/infra/live/prod/aws/cognito/terragrunt.hcl index 6e2da3c4..5f937331 100644 --- a/infra/live/prod/aws/cognito/terragrunt.hcl +++ b/infra/live/prod/aws/cognito/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//cognito" } diff --git a/infra/live/prod/aws/lambda_api/terragrunt.hcl b/infra/live/prod/aws/lambda_api/terragrunt.hcl index 60cf23d2..22f87c7c 100644 --- a/infra/live/prod/aws/lambda_api/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_api/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../security", "../database"] +} + dependency "network" { config_path = "${get_original_terragrunt_dir()}/../network" diff --git a/infra/live/prod/aws/lambda_worker/terragrunt.hcl b/infra/live/prod/aws/lambda_worker/terragrunt.hcl index c5981ca3..2b609221 100644 --- a/infra/live/prod/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_worker/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../security", "../network", "../database"] +} + dependency "messaging" { config_path = "${get_original_terragrunt_dir()}/../messaging" diff --git a/infra/live/prod/aws/messaging/terragrunt.hcl b/infra/live/prod/aws/messaging/terragrunt.hcl index b85cf717..8cf46acc 100644 --- a/infra/live/prod/aws/messaging/terragrunt.hcl +++ b/infra/live/prod/aws/messaging/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//messaging" } diff --git a/infra/live/prod/aws/observability/terragrunt.hcl b/infra/live/prod/aws/observability/terragrunt.hcl index 3a37c9ba..d3cae1c1 100644 --- a/infra/live/prod/aws/observability/terragrunt.hcl +++ b/infra/live/prod/aws/observability/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../oidc"] +} + terraform { source = "../../../../modules//aws//observability" } diff --git a/infra/live/prod/aws/service_api/terragrunt.hcl b/infra/live/prod/aws/service_api/terragrunt.hcl index 4c136329..d53eaf3e 100644 --- a/infra/live/prod/aws/service_api/terragrunt.hcl +++ b/infra/live/prod/aws/service_api/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../database", "../messaging"] +} + dependency "security" { config_path = "${get_original_terragrunt_dir()}/../security" diff --git a/infra/live/prod/aws/service_worker/terragrunt.hcl b/infra/live/prod/aws/service_worker/terragrunt.hcl index 6e0a23c5..26dd12fd 100644 --- a/infra/live/prod/aws/service_worker/terragrunt.hcl +++ b/infra/live/prod/aws/service_worker/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../database"] +} + dependency "security" { config_path = "${get_original_terragrunt_dir()}/../security" diff --git a/infra/live/prod/aws/task_api/terragrunt.hcl b/infra/live/prod/aws/task_api/terragrunt.hcl index af324e86..0d192fda 100644 --- a/infra/live/prod/aws/task_api/terragrunt.hcl +++ b/infra/live/prod/aws/task_api/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../security", "../cluster", "../network"] +} + terraform { source = "../../../../modules//aws//task_api" } diff --git a/infra/live/prod/aws/task_worker/terragrunt.hcl b/infra/live/prod/aws/task_worker/terragrunt.hcl index c315b177..fb0d4fc2 100644 --- a/infra/live/prod/aws/task_worker/terragrunt.hcl +++ b/infra/live/prod/aws/task_worker/terragrunt.hcl @@ -2,6 +2,10 @@ include "root" { path = find_in_parent_folders("root.hcl") } +dependencies { + paths = ["../security", "../network"] +} + dependency "messaging" { config_path = "${get_original_terragrunt_dir()}/../messaging" From 2ab7f650f1919e9c7eb07e4fe5da9c835ff521f0 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 14:10:20 +0100 Subject: [PATCH 53/73] chore: plan mermaid update --- .github/workflows/shared_get_modules.yml | 28 +++++++++--------------- 1 file changed, 10 insertions(+), 18 deletions(-) diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index 026a6f45..a0dd84bf 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -122,33 +122,25 @@ jobs: run: | printf '%s\n' "$WAVES_JSON" | tee waves.json jq -r ' - def wave_id($wave): "wave_" + ($wave | tostring); def node_id($module): "module_" + ($module | gsub("[^A-Za-z0-9_]"; "_")); - def wave_label($wave): "Wave " + ($wave | tostring); def module_label($module): $module; - [ + . as $waves + | [ "flowchart LR", ( - .[] - | " " + wave_id(.wave) + "[\"" + wave_label(.wave) + "\"]" - ), - ( - .[] - | .wave as $wave + $waves[] | .modules[] | " " + node_id(.) + "[\"" + module_label(.) + "\"]" ), ( - .[] - | .wave as $wave - | .modules[] - | " " + wave_id($wave) + " --> " + node_id(.) - ), - ( - [ .[] | .wave ] | sort | .[] as $wave - | select($wave > 0) - | " " + wave_id($wave - 1) + " --> " + wave_id($wave) + $waves + | to_entries[] + | select(.key > 0) + | .value.modules[] as $current + | .key as $wave_index + | $waves[$wave_index - 1].modules[]? + | " " + node_id(.) + " --> " + node_id($current) ) ] | .[] From 43ad9e1cc15f09df0baf9c4da209e866391e6d1f Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 14:10:41 +0100 Subject: [PATCH 54/73] fix: fmt --- infra/root.hcl | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/infra/root.hcl b/infra/root.hcl index 1f86be72..40099e1e 100644 --- a/infra/root.hcl +++ b/infra/root.hcl @@ -15,12 +15,12 @@ locals { project_name = element(split("/", local.github_repo), 1) - aws_region = local.global_vars.inputs.aws_region - base_reference = "${local.aws_account_id}-${local.aws_region}-${local.project_name}" - deploy_role_name = "${local.project_name}-${local.environment}-github-oidc-role" - deploy_role_arn = "arn:aws:iam::${local.aws_account_id}:role/${local.deploy_role_name}" - state_bucket = "${local.base_reference}-tfstate" - state_key = "${local.environment}/${local.provider}/${local.module}/terraform.tfstate" + aws_region = local.global_vars.inputs.aws_region + base_reference = "${local.aws_account_id}-${local.aws_region}-${local.project_name}" + deploy_role_name = "${local.project_name}-${local.environment}-github-oidc-role" + deploy_role_arn = "arn:aws:iam::${local.aws_account_id}:role/${local.deploy_role_name}" + state_bucket = "${local.base_reference}-tfstate" + state_key = "${local.environment}/${local.provider}/${local.module}/terraform.tfstate" # separate shared artifact resources when dev, otherwise ci artifact_base = local.environment == "dev" ? "${local.base_reference}-${local.environment}" : "${local.base_reference}-ci" code_bucket = "${local.artifact_base}-code" From 210ece7552b986073dd6d8dc8aa1c7d38526f369 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 14:16:17 +0100 Subject: [PATCH 55/73] chore: better mermaid --- .github/workflows/shared_get_modules.yml | 26 +++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index a0dd84bf..0764301b 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -123,24 +123,40 @@ jobs: printf '%s\n' "$WAVES_JSON" | tee waves.json jq -r ' def node_id($module): "module_" + ($module | gsub("[^A-Za-z0-9_]"; "_")); - def module_label($module): $module; + def wave_id($wave): "wave_" + ($wave | tostring); + def module_line($module): " " + node_id($module) + "[\"" + $module + "\"]"; + def edge_lines($prior; $current): + if ($prior.modules | length) == 0 or ($current.modules | length) == 0 then + [] + else + ( + ($prior.modules[0]) as $anchor + | ($current.modules | map(" " + node_id($anchor) + " --> " + node_id(.))) + ) + end; . as $waves | [ "flowchart LR", + ( + $waves[] + | " subgraph " + wave_id(.wave) + "[\"Wave " + (.wave | tostring) + "\"]" + ), ( $waves[] | .modules[] - | " " + node_id(.) + "[\"" + module_label(.) + "\"]" + | module_line(.) + ), + ( + $waves[] + | " end" ), ( $waves | to_entries[] | select(.key > 0) - | .value.modules[] as $current | .key as $wave_index - | $waves[$wave_index - 1].modules[]? - | " " + node_id(.) + " --> " + node_id($current) + | edge_lines($waves[$wave_index - 1]; .value)[] ) ] | .[] From 34f6036afe75ac526e5f143b7c7f808ef55e8ab4 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 14:19:22 +0100 Subject: [PATCH 56/73] chore: rm mermaid --- .github/workflows/shared_get_modules.yml | 46 +----------------------- 1 file changed, 1 insertion(+), 45 deletions(-) diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index 0764301b..950dc554 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -121,55 +121,11 @@ jobs: WAVES_JSON: ${{ steps.filtered_waves.outputs.waves_json }} run: | printf '%s\n' "$WAVES_JSON" | tee waves.json - jq -r ' - def node_id($module): "module_" + ($module | gsub("[^A-Za-z0-9_]"; "_")); - def wave_id($wave): "wave_" + ($wave | tostring); - def module_line($module): " " + node_id($module) + "[\"" + $module + "\"]"; - def edge_lines($prior; $current): - if ($prior.modules | length) == 0 or ($current.modules | length) == 0 then - [] - else - ( - ($prior.modules[0]) as $anchor - | ($current.modules | map(" " + node_id($anchor) + " --> " + node_id(.))) - ) - end; - - . as $waves - | [ - "flowchart LR", - ( - $waves[] - | " subgraph " + wave_id(.wave) + "[\"Wave " + (.wave | tostring) + "\"]" - ), - ( - $waves[] - | .modules[] - | module_line(.) - ), - ( - $waves[] - | " end" - ), - ( - $waves - | to_entries[] - | select(.key > 0) - | .key as $wave_index - | edge_lines($waves[$wave_index - 1]; .value)[] - ) - ] - | .[] - ' waves.json > waves.mmd { echo "## Terragrunt Wave Matrix" echo - echo '```mermaid' - cat waves.mmd - echo '```' - echo echo '```json' - cat waves.json + jq . waves.json echo echo '```' } >> "$GITHUB_STEP_SUMMARY" From cd8e52bc16a9d293dce35b353af569e9f27389bd Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 15:29:24 +0100 Subject: [PATCH 57/73] debug: try apply from plan --- .github/docs/README.md | 2 +- .../shared_infra_apply_from_plan.yml | 140 +++--------------- 2 files changed, 21 insertions(+), 121 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 9919c556..bfced1b5 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -86,7 +86,7 @@ flowchart LR - `shared_infra_apply_no_plan.yml` Direct-input apply wrapper around `shared_infra.yml`. It takes resolved workflow inputs directly and calls `shared_infra.yml` with `tg_action: apply`. - `shared_infra_apply_from_plan.yml` - Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs and saved wave arrays back out, and then reruns the same `wave_0`, `wave_1`, and `wave_2` module order. For now each per-module job only downloads its matching `terragrunt-plan--` GitHub artifact and prints the recovered plan metadata/text files plus the binary plan file presence for inspection. + Apply-from-plan wrapper around `shared_infra.yml`. It takes `plan_artifact_run_id`, downloads the `infra-plan-metadata` GitHub artifact from that earlier workflow run, reads the frozen graph inputs and saved wave arrays back out, and then reruns the same `wave_0`, `wave_1`, `wave_2`, and `wave_3` module order. Each per-module job downloads its matching `terragrunt-plan--` GitHub artifact into the live stack directory and then invokes the repo-local Terragrunt action with `tg_action: apply_plan`. - `shared_infra.yml` Temporary wave-based executor. It delegates graph and wave discovery to `shared_get_modules.yml`, exposes the resulting `waves_json` as the reusable-workflow output, and then runs `wave_0`, `wave_1`, `wave_2`, and `wave_3` jobs in dependency order. Each wave only runs when its module array is non-empty, fans that array out as a matrix, checks out the requested ref, configures AWS credentials once per matrix job, and invokes the shared repo-local Terragrunt action against `infra/live//aws/`. The deprecated `changed_items_json` workflow output is still present for compatibility and currently mirrors `waves_json`. - `just --justfile justfile.ci tg-graph-json-to-waves` diff --git a/.github/workflows/shared_infra_apply_from_plan.yml b/.github/workflows/shared_infra_apply_from_plan.yml index 799d8dec..bf43ffe0 100644 --- a/.github/workflows/shared_infra_apply_from_plan.yml +++ b/.github/workflows/shared_infra_apply_from_plan.yml @@ -96,36 +96,11 @@ jobs: run-id: ${{ inputs.plan_artifact_run_id }} path: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - - name: Print saved plan files - shell: bash - run: | - PLAN_DIR="infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}" - META_PATH="$PLAN_DIR/terragrunt.plan.meta.json" - TEXT_PATH="$PLAN_DIR/terragrunt.plan.txt" - BINARY_PATH="$PLAN_DIR/terragrunt.tfplan" - - echo "=== ${META_PATH} ===" - if [ -f "$META_PATH" ]; then - cat "$META_PATH" - else - echo "missing" - fi - echo - - echo "=== ${TEXT_PATH} ===" - if [ -f "$TEXT_PATH" ]; then - cat "$TEXT_PATH" - else - echo "missing" - fi - echo - - echo "=== ${BINARY_PATH} ===" - if [ -f "$BINARY_PATH" ]; then - ls -lh "$BINARY_PATH" - else - echo "missing" - fi + - name: Apply ${{ matrix.module }} infra from saved plan + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: apply_plan wave_1: needs: @@ -156,36 +131,11 @@ jobs: run-id: ${{ inputs.plan_artifact_run_id }} path: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - - name: Print saved plan files - shell: bash - run: | - PLAN_DIR="infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}" - META_PATH="$PLAN_DIR/terragrunt.plan.meta.json" - TEXT_PATH="$PLAN_DIR/terragrunt.plan.txt" - BINARY_PATH="$PLAN_DIR/terragrunt.tfplan" - - echo "=== ${META_PATH} ===" - if [ -f "$META_PATH" ]; then - cat "$META_PATH" - else - echo "missing" - fi - echo - - echo "=== ${TEXT_PATH} ===" - if [ -f "$TEXT_PATH" ]; then - cat "$TEXT_PATH" - else - echo "missing" - fi - echo - - echo "=== ${BINARY_PATH} ===" - if [ -f "$BINARY_PATH" ]; then - ls -lh "$BINARY_PATH" - else - echo "missing" - fi + - name: Apply ${{ matrix.module }} infra from saved plan + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: apply_plan wave_2: needs: @@ -216,36 +166,11 @@ jobs: run-id: ${{ inputs.plan_artifact_run_id }} path: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - - name: Print saved plan files - shell: bash - run: | - PLAN_DIR="infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}" - META_PATH="$PLAN_DIR/terragrunt.plan.meta.json" - TEXT_PATH="$PLAN_DIR/terragrunt.plan.txt" - BINARY_PATH="$PLAN_DIR/terragrunt.tfplan" - - echo "=== ${META_PATH} ===" - if [ -f "$META_PATH" ]; then - cat "$META_PATH" - else - echo "missing" - fi - echo - - echo "=== ${TEXT_PATH} ===" - if [ -f "$TEXT_PATH" ]; then - cat "$TEXT_PATH" - else - echo "missing" - fi - echo - - echo "=== ${BINARY_PATH} ===" - if [ -f "$BINARY_PATH" ]; then - ls -lh "$BINARY_PATH" - else - echo "missing" - fi + - name: Apply ${{ matrix.module }} infra from saved plan + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: apply_plan wave_3: needs: @@ -276,33 +201,8 @@ jobs: run-id: ${{ inputs.plan_artifact_run_id }} path: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - - name: Print saved plan files - shell: bash - run: | - PLAN_DIR="infra/live/${{ inputs.environment }}/aws/${{ matrix.module }}" - META_PATH="$PLAN_DIR/terragrunt.plan.meta.json" - TEXT_PATH="$PLAN_DIR/terragrunt.plan.txt" - BINARY_PATH="$PLAN_DIR/terragrunt.tfplan" - - echo "=== ${META_PATH} ===" - if [ -f "$META_PATH" ]; then - cat "$META_PATH" - else - echo "missing" - fi - echo - - echo "=== ${TEXT_PATH} ===" - if [ -f "$TEXT_PATH" ]; then - cat "$TEXT_PATH" - else - echo "missing" - fi - echo - - echo "=== ${BINARY_PATH} ===" - if [ -f "$BINARY_PATH" ]; then - ls -lh "$BINARY_PATH" - else - echo "missing" - fi + - name: Apply ${{ matrix.module }} infra from saved plan + uses: ./.github/actions/terragrunt + with: + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} + tg_action: apply_plan From 59026925f5454598539b3899faa2cba5e8ae4bb4 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 15:42:33 +0100 Subject: [PATCH 58/73] chore: waves in destroy --- .github/docs/README.md | 2 +- .github/workflows/destroy.yml | 318 ++++++++++------------------------ 2 files changed, 92 insertions(+), 228 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index bfced1b5..349ef1af 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -132,7 +132,7 @@ flowchart LR ### Cleanup And Discovery - `destroy.yml` - Tears down app layers before shared dependencies, including the shared observability dashboard and any environment-owned shared artifact stacks such as the `dev` code bucket. The workflow-dispatch input `allow_prod_cleanup` now gates every cleanup or destroy job that is normally skipped for `prod`, including the `Code Bucket`, `ECR`, and final tagged-resource cleanup jobs. After the main graph completes, the workflow first counts tagged leftovers through `justfile.destroy`, prints a warning only when any remain, and then runs the cleanup recipe. That cleanup currently deletes leaked Cognito user pools, deregisters and then deletes leaked ECS task-definition revisions, deletes leftover ECS clusters, and force-deletes leftover Secrets Manager secrets, then validates the remaining tagged ARNs against the underlying service APIs rather than treating the tagging index as the source of truth. Already-removed Cognito pools, ECS task-definition revisions, ECS clusters, or Secrets Manager secrets are treated as successful no-ops so stale tagging API results do not fail cleanup. If unsupported or still-live tagged resources remain after that sweep, the workflow now records a warning and step summary instead of failing the whole destroy run. `prod` runs that same path only when `allow_prod_cleanup` is enabled, and the workflow prints a conspicuous warning first. + Tears down infrastructure through the same Terragrunt graph contract as plan/apply, but in reverse wave order so downstream runtimes are removed before shared dependencies. It derives the current module waves through `shared_get_modules.yml`, optionally removes `code_bucket` and `ecr` from the destroy set for `prod` unless `allow_prod_cleanup` is enabled, and then runs `wave_3`, `wave_2`, `wave_1`, and `wave_0`. The only remaining module-specific destroy placeholder vars are the required ECS task image inputs for `task_*`; the frontend, service, and database stacks now rely on Terragrunt dependency mocks instead of workflow-injected destroy placeholders. After the main graph completes, the workflow first counts tagged leftovers through `justfile.destroy`, prints a warning only when any remain, and then runs the cleanup recipe. That cleanup currently deletes leaked Cognito user pools, deregisters and then deletes leaked ECS task-definition revisions, deletes leftover ECS clusters, and force-deletes leftover Secrets Manager secrets, then validates the remaining tagged ARNs against the underlying service APIs rather than treating the tagging index as the source of truth. Already-removed Cognito pools, ECS task-definition revisions, ECS clusters, or Secrets Manager secrets are treated as successful no-ops so stale tagging API results do not fail cleanup. If unsupported or still-live tagged resources remain after that sweep, the workflow now records a warning and step summary instead of failing the whole destroy run. `prod` runs that same path only when `allow_prod_cleanup` is enabled, and the workflow prints a conspicuous warning first. - `shared_directories_get.yml` Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index 5b6acd24..3234d8bc 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -30,54 +30,70 @@ env: AWS_REGION: ${{ vars.AWS_REGION }} jobs: - setup: - name: Discover - uses: ./.github/workflows/shared_directories_get.yml - - observability: - name: Observability + generate_waves: + name: Discover Waves + uses: ./.github/workflows/shared_get_modules.yml + with: + environment: ${{ inputs.environment }} + infra_version: ${{ github.sha }} + + prepare_waves: + name: Prepare Destroy Waves + needs: generate_waves runs-on: ubuntu-latest + outputs: + waves_json: ${{ steps.filter.outputs.waves_json }} + wave_0_modules: ${{ steps.filter.outputs.wave_0_modules }} + wave_1_modules: ${{ steps.filter.outputs.wave_1_modules }} + wave_2_modules: ${{ steps.filter.outputs.wave_2_modules }} + wave_3_modules: ${{ steps.filter.outputs.wave_3_modules }} steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy observability infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/observability - tg_action: destroy + - name: Filter destroy waves + id: filter + shell: bash + env: + RAW_WAVES_JSON: ${{ needs.generate_waves.outputs.waves_json }} + ENVIRONMENT: ${{ inputs.environment }} + ALLOW_PROD_CLEANUP: ${{ inputs.allow_prod_cleanup }} + run: | + FILTERED_WAVES_JSON="$(jq -c ' + if $environment == "prod" and ($allow_prod_cleanup | not) then + map(.modules |= map(select(. != "code_bucket" and . != "ecr"))) + else + . + end + | map(select(.modules | length > 0)) + ' --arg environment "$ENVIRONMENT" --argjson allow_prod_cleanup "$ALLOW_PROD_CLEANUP" <<<"$RAW_WAVES_JSON")" + + echo "waves_json=$FILTERED_WAVES_JSON" >> "$GITHUB_OUTPUT" + echo "wave_0_modules=$(jq -c '.[0].modules // []' <<<"$FILTERED_WAVES_JSON")" >> "$GITHUB_OUTPUT" + echo "wave_1_modules=$(jq -c '.[1].modules // []' <<<"$FILTERED_WAVES_JSON")" >> "$GITHUB_OUTPUT" + echo "wave_2_modules=$(jq -c '.[2].modules // []' <<<"$FILTERED_WAVES_JSON")" >> "$GITHUB_OUTPUT" + echo "wave_3_modules=$(jq -c '.[3].modules // []' <<<"$FILTERED_WAVES_JSON")" >> "$GITHUB_OUTPUT" + + - name: Print destroy wave matrix + shell: bash + env: + WAVES_JSON: ${{ steps.filter.outputs.waves_json }} + run: | + { + echo "## Destroy Wave Matrix" + echo + echo '```json' + printf '%s\n' "$WAVES_JSON" | jq . + echo + echo '```' + } >> "$GITHUB_STEP_SUMMARY" - lambdas: - name: Lambdas + wave_3: + name: 3 / ${{ matrix.module }} + needs: prepare_waves + if: ${{ needs.prepare_waves.outputs.wave_3_modules != '[]' }} runs-on: ubuntu-latest - needs: - - setup - - services strategy: fail-fast: false matrix: - value: ${{ fromJson(needs.setup.outputs.lambda_dirs) }} - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Deploy ${{ matrix.value }} infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.value }} - tg_action: destroy - - frontend: - name: Frontend - runs-on: ubuntu-latest + module: ${{ fromJson(needs.prepare_waves.outputs.wave_3_modules) }} steps: - uses: actions/checkout@v6 @@ -86,46 +102,27 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Destroy frontend infra + - name: Destroy ${{ matrix.module }} infra uses: ./.github/actions/terragrunt env: - TF_VAR_api_invoke_url: "https://placeholder.execute-api.us-east-1.amazonaws.com" - TF_VAR_auth_user_pool_id: "destroy-placeholder" - TF_VAR_auth_user_pool_client_id: "destroy-placeholder" - TF_VAR_auth_hosted_ui_url: "https://destroy-placeholder" - TF_VAR_auth_readonly_group_name: "readonly" + TF_VAR_image_uri: ${{ startsWith(matrix.module, 'task_') && 'destroy-placeholder' || '' }} + TF_VAR_debug_image_uri: ${{ startsWith(matrix.module, 'task_') && 'destroy-placeholder' || '' }} + TF_VAR_aws_otel_collector_image_uri: ${{ startsWith(matrix.module, 'task_') && 'destroy-placeholder' || '' }} with: - tg_directory: infra/live/${{ inputs.environment }}/aws/frontend + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: destroy - cognito: - name: Cognito - runs-on: ubuntu-latest + wave_2: + name: 2 / ${{ matrix.module }} needs: - - frontend - - network - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy cognito infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/cognito - tg_action: destroy - - services: - name: Services + - prepare_waves + - wave_3 + if: ${{ always() && needs.prepare_waves.outputs.wave_2_modules != '[]' && (needs.wave_3.result == 'success' || needs.wave_3.result == 'skipped') }} runs-on: ubuntu-latest - needs: setup strategy: fail-fast: false matrix: - value: ${{ fromJson(needs.setup.outputs.ecs_service_dirs) }} + module: ${{ fromJson(needs.prepare_waves.outputs.wave_2_modules) }} steps: - uses: actions/checkout@v6 @@ -134,95 +131,23 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Destroy ${{ matrix.value }} infra + - name: Destroy ${{ matrix.module }} infra uses: ./.github/actions/terragrunt - env: - TF_VAR_bootstrap: "true" - TF_VAR_bootstrap_image_uri: "destroy-placeholder" with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.value }} + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: destroy - tasks: - name: Tasks - runs-on: ubuntu-latest + wave_1: + name: 1 / ${{ matrix.module }} needs: - - setup - - services + - prepare_waves + - wave_2 + if: ${{ always() && needs.prepare_waves.outputs.wave_1_modules != '[]' && (needs.wave_2.result == 'success' || needs.wave_2.result == 'skipped') }} + runs-on: ubuntu-latest strategy: fail-fast: false matrix: - value: ${{ fromJson(needs.setup.outputs.task_dirs) }} - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy ${{ matrix.value }} infra - uses: ./.github/actions/terragrunt - env: - TF_VAR_image_uri: "destroy-placeholder" - TF_VAR_debug_image_uri: "destroy-placeholder" - TF_VAR_aws_otel_collector_image_uri: "destroy-placeholder" - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.value }} - tg_action: destroy - - messaging: - name: Messaging - runs-on: ubuntu-latest - needs: - - lambdas - - services - - tasks - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy messaging infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/messaging - tg_action: destroy - - database: - name: Database - runs-on: ubuntu-latest - needs: - - lambdas - - services - - tasks - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy database infra - uses: ./.github/actions/terragrunt - env: - TF_VAR_database_security_group_id: "destroy-placeholder" - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/database - tg_action: destroy - - network: - name: Network - needs: - - frontend - - services - - tasks - - database - runs-on: ubuntu-latest + module: ${{ fromJson(needs.prepare_waves.outputs.wave_1_modules) }} steps: - uses: actions/checkout@v6 @@ -231,79 +156,23 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Destroy network infra + - name: Destroy ${{ matrix.module }} infra uses: ./.github/actions/terragrunt with: - tg_directory: infra/live/${{ inputs.environment }}/aws/network + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: destroy - security: - name: Security + wave_0: + name: 0 / ${{ matrix.module }} needs: - - network - - lambdas - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy security infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/security - tg_action: destroy - - build-bucket: - name: Code Bucket - if: inputs.environment != 'prod' || inputs.allow_prod_cleanup - needs: - - lambdas - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy code - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/code_bucket - tg_action: destroy - - ecr: - name: ECR - if: inputs.environment != 'prod' || inputs.allow_prod_cleanup - needs: - - network - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy code - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/ecr - tg_action: destroy - - cluster: - name: Cluster - needs: - - services - - tasks - - database + - prepare_waves + - wave_1 + if: ${{ always() && needs.prepare_waves.outputs.wave_0_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + module: ${{ fromJson(needs.prepare_waves.outputs.wave_0_modules) }} steps: - uses: actions/checkout@v6 @@ -312,22 +181,17 @@ jobs: role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Destroy cluster infra + - name: Destroy ${{ matrix.module }} infra uses: ./.github/actions/terragrunt with: - tg_directory: infra/live/${{ inputs.environment }}/aws/cluster + tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: destroy cleanup: name: Cleanup if: inputs.environment != 'prod' || inputs.allow_prod_cleanup needs: - - observability - - cognito - - security - - build-bucket - - ecr - - cluster + - wave_0 runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 From d34992b80f847fbd366fce8da3b2d8af8dee27a9 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 15:46:05 +0100 Subject: [PATCH 59/73] fix: destroy perms --- .github/workflows/destroy.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index 3234d8bc..43212ea2 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -23,6 +23,7 @@ concurrency: # only run destroy when no other deploy/destroy is running for the permissions: id-token: write contents: write + actions: read env: TF_VAR_lambda_version: this From 4729ff4c85164e61d6bb3007918d657b5cb33168 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 15:49:28 +0100 Subject: [PATCH 60/73] chore: ignore_shared_artifact_modules bool --- .github/docs/README.md | 4 +- .github/workflows/destroy.yml | 73 +++++------------------- .github/workflows/shared_get_modules.yml | 13 ++++- 3 files changed, 27 insertions(+), 63 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 349ef1af..a70b1aed 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -132,12 +132,12 @@ flowchart LR ### Cleanup And Discovery - `destroy.yml` - Tears down infrastructure through the same Terragrunt graph contract as plan/apply, but in reverse wave order so downstream runtimes are removed before shared dependencies. It derives the current module waves through `shared_get_modules.yml`, optionally removes `code_bucket` and `ecr` from the destroy set for `prod` unless `allow_prod_cleanup` is enabled, and then runs `wave_3`, `wave_2`, `wave_1`, and `wave_0`. The only remaining module-specific destroy placeholder vars are the required ECS task image inputs for `task_*`; the frontend, service, and database stacks now rely on Terragrunt dependency mocks instead of workflow-injected destroy placeholders. After the main graph completes, the workflow first counts tagged leftovers through `justfile.destroy`, prints a warning only when any remain, and then runs the cleanup recipe. That cleanup currently deletes leaked Cognito user pools, deregisters and then deletes leaked ECS task-definition revisions, deletes leftover ECS clusters, and force-deletes leftover Secrets Manager secrets, then validates the remaining tagged ARNs against the underlying service APIs rather than treating the tagging index as the source of truth. Already-removed Cognito pools, ECS task-definition revisions, ECS clusters, or Secrets Manager secrets are treated as successful no-ops so stale tagging API results do not fail cleanup. If unsupported or still-live tagged resources remain after that sweep, the workflow now records a warning and step summary instead of failing the whole destroy run. `prod` runs that same path only when `allow_prod_cleanup` is enabled, and the workflow prints a conspicuous warning first. + Tears down infrastructure through the same Terragrunt graph contract as plan/apply, but in reverse wave order so downstream runtimes are removed before shared dependencies. It derives the current module waves through `shared_get_modules.yml`, using that reusable workflow's filtering inputs to omit `code_bucket` and `ecr` from the destroy set for `prod` unless `allow_prod_cleanup` is enabled, and then runs `wave_3`, `wave_2`, `wave_1`, and `wave_0`. The only remaining module-specific destroy placeholder vars are the required ECS task image inputs for `task_*`; the frontend, service, and database stacks now rely on Terragrunt dependency mocks instead of workflow-injected destroy placeholders. After the main graph completes, the workflow first counts tagged leftovers through `justfile.destroy`, prints a warning only when any remain, and then runs the cleanup recipe. That cleanup currently deletes leaked Cognito user pools, deregisters and then deletes leaked ECS task-definition revisions, deletes leftover ECS clusters, and force-deletes leftover Secrets Manager secrets, then validates the remaining tagged ARNs against the underlying service APIs rather than treating the tagging index as the source of truth. Already-removed Cognito pools, ECS task-definition revisions, ECS clusters, or Secrets Manager secrets are treated as successful no-ops so stale tagging API results do not fail cleanup. If unsupported or still-live tagged resources remain after that sweep, the workflow now records a warning and step summary instead of failing the whole destroy run. `prod` runs that same path only when `allow_prod_cleanup` is enabled, and the workflow prints a conspicuous warning first. - `shared_directories_get.yml` Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. - `shared_get_modules.yml` - Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, `wave_2_modules`, and `wave_3_modules` as reusable-workflow outputs. Callers can pass `ignore_task_modules: true` to exclude any `task_*` modules from the emitted rollout waves for the current bootstrap-oriented infra path. + Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, `wave_2_modules`, and `wave_3_modules` as reusable-workflow outputs. Callers can pass `ignore_task_modules: true` to exclude any `task_*` modules from the emitted rollout waves for the current bootstrap-oriented infra path, and `ignore_shared_artifact_modules: true` to omit shared artifact stacks such as `code_bucket` and `ecr`. ## Feasibility Checks diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index 43212ea2..9f3c1b53 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -37,64 +37,17 @@ jobs: with: environment: ${{ inputs.environment }} infra_version: ${{ github.sha }} - - prepare_waves: - name: Prepare Destroy Waves - needs: generate_waves - runs-on: ubuntu-latest - outputs: - waves_json: ${{ steps.filter.outputs.waves_json }} - wave_0_modules: ${{ steps.filter.outputs.wave_0_modules }} - wave_1_modules: ${{ steps.filter.outputs.wave_1_modules }} - wave_2_modules: ${{ steps.filter.outputs.wave_2_modules }} - wave_3_modules: ${{ steps.filter.outputs.wave_3_modules }} - steps: - - name: Filter destroy waves - id: filter - shell: bash - env: - RAW_WAVES_JSON: ${{ needs.generate_waves.outputs.waves_json }} - ENVIRONMENT: ${{ inputs.environment }} - ALLOW_PROD_CLEANUP: ${{ inputs.allow_prod_cleanup }} - run: | - FILTERED_WAVES_JSON="$(jq -c ' - if $environment == "prod" and ($allow_prod_cleanup | not) then - map(.modules |= map(select(. != "code_bucket" and . != "ecr"))) - else - . - end - | map(select(.modules | length > 0)) - ' --arg environment "$ENVIRONMENT" --argjson allow_prod_cleanup "$ALLOW_PROD_CLEANUP" <<<"$RAW_WAVES_JSON")" - - echo "waves_json=$FILTERED_WAVES_JSON" >> "$GITHUB_OUTPUT" - echo "wave_0_modules=$(jq -c '.[0].modules // []' <<<"$FILTERED_WAVES_JSON")" >> "$GITHUB_OUTPUT" - echo "wave_1_modules=$(jq -c '.[1].modules // []' <<<"$FILTERED_WAVES_JSON")" >> "$GITHUB_OUTPUT" - echo "wave_2_modules=$(jq -c '.[2].modules // []' <<<"$FILTERED_WAVES_JSON")" >> "$GITHUB_OUTPUT" - echo "wave_3_modules=$(jq -c '.[3].modules // []' <<<"$FILTERED_WAVES_JSON")" >> "$GITHUB_OUTPUT" - - - name: Print destroy wave matrix - shell: bash - env: - WAVES_JSON: ${{ steps.filter.outputs.waves_json }} - run: | - { - echo "## Destroy Wave Matrix" - echo - echo '```json' - printf '%s\n' "$WAVES_JSON" | jq . - echo - echo '```' - } >> "$GITHUB_STEP_SUMMARY" + ignore_shared_artifact_modules: ${{ inputs.environment == 'prod' && !inputs.allow_prod_cleanup }} wave_3: name: 3 / ${{ matrix.module }} - needs: prepare_waves - if: ${{ needs.prepare_waves.outputs.wave_3_modules != '[]' }} + needs: generate_waves + if: ${{ needs.generate_waves.outputs.wave_3_modules != '[]' }} runs-on: ubuntu-latest strategy: fail-fast: false matrix: - module: ${{ fromJson(needs.prepare_waves.outputs.wave_3_modules) }} + module: ${{ fromJson(needs.generate_waves.outputs.wave_3_modules) }} steps: - uses: actions/checkout@v6 @@ -116,14 +69,14 @@ jobs: wave_2: name: 2 / ${{ matrix.module }} needs: - - prepare_waves + - generate_waves - wave_3 - if: ${{ always() && needs.prepare_waves.outputs.wave_2_modules != '[]' && (needs.wave_3.result == 'success' || needs.wave_3.result == 'skipped') }} + if: ${{ always() && needs.generate_waves.outputs.wave_2_modules != '[]' && (needs.wave_3.result == 'success' || needs.wave_3.result == 'skipped') }} runs-on: ubuntu-latest strategy: fail-fast: false matrix: - module: ${{ fromJson(needs.prepare_waves.outputs.wave_2_modules) }} + module: ${{ fromJson(needs.generate_waves.outputs.wave_2_modules) }} steps: - uses: actions/checkout@v6 @@ -141,14 +94,14 @@ jobs: wave_1: name: 1 / ${{ matrix.module }} needs: - - prepare_waves + - generate_waves - wave_2 - if: ${{ always() && needs.prepare_waves.outputs.wave_1_modules != '[]' && (needs.wave_2.result == 'success' || needs.wave_2.result == 'skipped') }} + if: ${{ always() && needs.generate_waves.outputs.wave_1_modules != '[]' && (needs.wave_2.result == 'success' || needs.wave_2.result == 'skipped') }} runs-on: ubuntu-latest strategy: fail-fast: false matrix: - module: ${{ fromJson(needs.prepare_waves.outputs.wave_1_modules) }} + module: ${{ fromJson(needs.generate_waves.outputs.wave_1_modules) }} steps: - uses: actions/checkout@v6 @@ -166,14 +119,14 @@ jobs: wave_0: name: 0 / ${{ matrix.module }} needs: - - prepare_waves + - generate_waves - wave_1 - if: ${{ always() && needs.prepare_waves.outputs.wave_0_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} + if: ${{ always() && needs.generate_waves.outputs.wave_0_modules != '[]' && (needs.wave_1.result == 'success' || needs.wave_1.result == 'skipped') }} runs-on: ubuntu-latest strategy: fail-fast: false matrix: - module: ${{ fromJson(needs.prepare_waves.outputs.wave_0_modules) }} + module: ${{ fromJson(needs.generate_waves.outputs.wave_0_modules) }} steps: - uses: actions/checkout@v6 diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index 950dc554..3e89e430 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -16,6 +16,11 @@ on: required: false type: boolean default: false + ignore_shared_artifact_modules: + description: "Whether to exclude shared artifact modules such as code_bucket and ecr from emitted waves" + required: false + type: boolean + default: false outputs: waves_json: description: "Dependency-safe Terragrunt wave matrix JSON" @@ -92,6 +97,7 @@ jobs: env: RAW_WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} IGNORE_TASK_MODULES: ${{ inputs.ignore_task_modules }} + IGNORE_SHARED_ARTIFACT_MODULES: ${{ inputs.ignore_shared_artifact_modules }} run: | echo "waves_json=$(jq -c ' if $ignore_tasks then @@ -99,10 +105,15 @@ jobs: else . end + | if $ignore_shared_artifact_modules then + map(.modules |= map(select(. != "code_bucket" and . != "ecr"))) + else + . + end | map(select(.modules | length > 0)) | to_entries | map({wave: .key, modules: .value.modules}) - ' --argjson ignore_tasks "$IGNORE_TASK_MODULES" <<<"$RAW_WAVES_JSON")" >> "$GITHUB_OUTPUT" + ' --argjson ignore_tasks "$IGNORE_TASK_MODULES" --argjson ignore_shared_artifact_modules "$IGNORE_SHARED_ARTIFACT_MODULES" <<<"$RAW_WAVES_JSON")" >> "$GITHUB_OUTPUT" - name: Expose wave module arrays id: wave_outputs From ec560cd057c6daeafac3bf7a601b739b8de4c22b Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 15:55:09 +0100 Subject: [PATCH 61/73] fix: mock_outputs_merge_strategy_with_state = "shallow" --- infra/live/dev/aws/service_api/terragrunt.hcl | 4 ++++ infra/live/dev/aws/service_worker/terragrunt.hcl | 5 +++++ infra/live/dev/aws/task_worker/terragrunt.hcl | 2 ++ infra/live/prod/aws/service_api/terragrunt.hcl | 3 +++ infra/live/prod/aws/service_worker/terragrunt.hcl | 4 ++++ infra/live/prod/aws/task_worker/terragrunt.hcl | 2 ++ 6 files changed, 20 insertions(+) diff --git a/infra/live/dev/aws/service_api/terragrunt.hcl b/infra/live/dev/aws/service_api/terragrunt.hcl index e216b716..94cae671 100644 --- a/infra/live/dev/aws/service_api/terragrunt.hcl +++ b/infra/live/dev/aws/service_api/terragrunt.hcl @@ -15,6 +15,7 @@ dependency "ecr" { repository_arn = "arn:aws:ecr:eu-west-2:111111111111:repository/mock-ecr" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -30,6 +31,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -41,6 +43,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -62,6 +65,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/service_worker/terragrunt.hcl b/infra/live/dev/aws/service_worker/terragrunt.hcl index 8a75266c..ac2ebac3 100644 --- a/infra/live/dev/aws/service_worker/terragrunt.hcl +++ b/infra/live/dev/aws/service_worker/terragrunt.hcl @@ -15,6 +15,7 @@ dependency "ecr" { repository_arn = "arn:aws:ecr:eu-west-2:111111111111:repository/mock-ecr" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -30,6 +31,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -51,6 +53,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -62,6 +65,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -83,6 +87,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/task_worker/terragrunt.hcl b/infra/live/dev/aws/task_worker/terragrunt.hcl index fb0d4fc2..c9ef3ee8 100644 --- a/infra/live/dev/aws/task_worker/terragrunt.hcl +++ b/infra/live/dev/aws/task_worker/terragrunt.hcl @@ -24,6 +24,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -38,6 +39,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/service_api/terragrunt.hcl b/infra/live/prod/aws/service_api/terragrunt.hcl index d53eaf3e..4c5dcdb6 100644 --- a/infra/live/prod/aws/service_api/terragrunt.hcl +++ b/infra/live/prod/aws/service_api/terragrunt.hcl @@ -18,6 +18,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -29,6 +30,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -50,6 +52,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/service_worker/terragrunt.hcl b/infra/live/prod/aws/service_worker/terragrunt.hcl index 26dd12fd..a62abc29 100644 --- a/infra/live/prod/aws/service_worker/terragrunt.hcl +++ b/infra/live/prod/aws/service_worker/terragrunt.hcl @@ -18,6 +18,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -39,6 +40,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -50,6 +52,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -71,6 +74,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/task_worker/terragrunt.hcl b/infra/live/prod/aws/task_worker/terragrunt.hcl index fb0d4fc2..c9ef3ee8 100644 --- a/infra/live/prod/aws/task_worker/terragrunt.hcl +++ b/infra/live/prod/aws/task_worker/terragrunt.hcl @@ -24,6 +24,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -38,6 +39,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } From d3487fa386a9c6f26e145c677cd7bb74369681c9 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 15:57:46 +0100 Subject: [PATCH 62/73] chore: update notes --- .github/docs/README.md | 1 + README.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index a70b1aed..6e12932f 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -150,6 +150,7 @@ Run these checks on every CI, workflow, or deploy-contract change. - verify optional inputs are intentionally omitted, not accidentally missing - the repo-local `./.github/actions/terragrunt` action supports `tg_action: plan` for producing the binary plan in the live stack directory, writes `terragrunt.plan.meta.json` there for every saved plan, and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes - `apply_plan` now expects the calling workflow job to download the matching per-stack GitHub artifact into the live stack directory before invoking Terragrunt +- when a live Terragrunt `dependency` block uses `mock_outputs` for planability or destroy safety, default it to `mock_outputs_merge_strategy_with_state = "shallow"` so partial real upstream state does not suppress missing mock keys - both repo-local composite actions, `./.github/actions/just` and `./.github/actions/terragrunt`, now assume AWS credentials are already configured in the current job when they need AWS access. The repo pattern is to run `aws-actions/configure-aws-credentials` at the top of each AWS-using job and then call the local actions without extra auth inputs - `./.github/actions/just` installs the requested `just` version through `extractions/setup-crate@v2` in the same minimal composite-action shape as `extractions/setup-just`, rather than depending on `extractions/setup-just` itself - `./.github/actions/terragrunt` installs the requested Terragrunt version through `jdx/mise-action@v4`, while Terraform stays pinned separately through `hashicorp/setup-terraform` diff --git a/README.md b/README.md index ab7085e5..ce523217 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ Lambda + ECS with CodeDeploy rollouts, plus provisioned concurrency controls for ## Bootstrap-Friendly Plans -For cross-stack contracts that often block CI plans before upstream stacks exist, this repo prefers Terragrunt `dependency` wiring in the live stack plus `mock_outputs` for non-mutating commands such as `plan` and `validate`. Keep those `dependency` blocks in the consuming stack instead of hiding them behind `read_terragrunt_config(...)` helper indirection, because Terragrunt graph commands only emit direct stack edges. The Terraform modules should consume explicit inputs rather than reaching back into sibling stack state directly when the contract needs bootstrap-friendly plan behavior. +For cross-stack contracts that often block CI plans before upstream stacks exist, this repo prefers Terragrunt `dependency` wiring in the live stack plus `mock_outputs` for non-mutating commands such as `plan` and `validate`. Keep those `dependency` blocks in the consuming stack instead of hiding them behind `read_terragrunt_config(...)` helper indirection, because Terragrunt graph commands only emit direct stack edges. The Terraform modules should consume explicit inputs rather than reaching back into sibling stack state directly when the contract needs bootstrap-friendly plan behavior. When a dependency may have partial real state during bootstrap, drift, or destroy, default the live Terragrunt `dependency` block to `mock_outputs_merge_strategy_with_state = "shallow"` so missing output keys can still fall back to mocks. Use [CONTRIBUTING.md](CONTRIBUTING.md) for expectations when changing the repo itself. From 7fd22bc4ce6ed2561b680b553718553390f8147a Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 16:00:23 +0100 Subject: [PATCH 63/73] chore: ignore_oidc_module true --- .github/docs/README.md | 4 ++-- .github/workflows/destroy.yml | 1 + .github/workflows/shared_get_modules.yml | 13 ++++++++++++- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 6e12932f..25ac1029 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -132,12 +132,12 @@ flowchart LR ### Cleanup And Discovery - `destroy.yml` - Tears down infrastructure through the same Terragrunt graph contract as plan/apply, but in reverse wave order so downstream runtimes are removed before shared dependencies. It derives the current module waves through `shared_get_modules.yml`, using that reusable workflow's filtering inputs to omit `code_bucket` and `ecr` from the destroy set for `prod` unless `allow_prod_cleanup` is enabled, and then runs `wave_3`, `wave_2`, `wave_1`, and `wave_0`. The only remaining module-specific destroy placeholder vars are the required ECS task image inputs for `task_*`; the frontend, service, and database stacks now rely on Terragrunt dependency mocks instead of workflow-injected destroy placeholders. After the main graph completes, the workflow first counts tagged leftovers through `justfile.destroy`, prints a warning only when any remain, and then runs the cleanup recipe. That cleanup currently deletes leaked Cognito user pools, deregisters and then deletes leaked ECS task-definition revisions, deletes leftover ECS clusters, and force-deletes leftover Secrets Manager secrets, then validates the remaining tagged ARNs against the underlying service APIs rather than treating the tagging index as the source of truth. Already-removed Cognito pools, ECS task-definition revisions, ECS clusters, or Secrets Manager secrets are treated as successful no-ops so stale tagging API results do not fail cleanup. If unsupported or still-live tagged resources remain after that sweep, the workflow now records a warning and step summary instead of failing the whole destroy run. `prod` runs that same path only when `allow_prod_cleanup` is enabled, and the workflow prints a conspicuous warning first. + Tears down infrastructure through the same Terragrunt graph contract as plan/apply, but in reverse wave order so downstream runtimes are removed before shared dependencies. It derives the current module waves through `shared_get_modules.yml`, using that reusable workflow's filtering inputs to omit `oidc` entirely and to omit `code_bucket` and `ecr` from the destroy set for `prod` unless `allow_prod_cleanup` is enabled, and then runs `wave_3`, `wave_2`, `wave_1`, and `wave_0`. The only remaining module-specific destroy placeholder vars are the required ECS task image inputs for `task_*`; the frontend, service, and database stacks now rely on Terragrunt dependency mocks instead of workflow-injected destroy placeholders. After the main graph completes, the workflow first counts tagged leftovers through `justfile.destroy`, prints a warning only when any remain, and then runs the cleanup recipe. That cleanup currently deletes leaked Cognito user pools, deregisters and then deletes leaked ECS task-definition revisions, deletes leftover ECS clusters, and force-deletes leftover Secrets Manager secrets, then validates the remaining tagged ARNs against the underlying service APIs rather than treating the tagging index as the source of truth. Already-removed Cognito pools, ECS task-definition revisions, ECS clusters, or Secrets Manager secrets are treated as successful no-ops so stale tagging API results do not fail cleanup. If unsupported or still-live tagged resources remain after that sweep, the workflow now records a warning and step summary instead of failing the whole destroy run. `prod` runs that same path only when `allow_prod_cleanup` is enabled, and the workflow prints a conspicuous warning first. - `shared_directories_get.yml` Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. - `shared_get_modules.yml` - Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, `wave_2_modules`, and `wave_3_modules` as reusable-workflow outputs. Callers can pass `ignore_task_modules: true` to exclude any `task_*` modules from the emitted rollout waves for the current bootstrap-oriented infra path, and `ignore_shared_artifact_modules: true` to omit shared artifact stacks such as `code_bucket` and `ecr`. + Reusable module-discovery workflow for infra waves. It renders the Terragrunt graph for the target environment, converts that graph into compact JSON, derives dependency-safe waves, and exposes `waves_json`, `wave_0_modules`, `wave_1_modules`, `wave_2_modules`, and `wave_3_modules` as reusable-workflow outputs. Callers can pass `ignore_task_modules: true` to exclude any `task_*` modules from the emitted rollout waves for the current bootstrap-oriented infra path, `ignore_shared_artifact_modules: true` to omit shared artifact stacks such as `code_bucket` and `ecr`, and `ignore_oidc_module: true` to exclude `oidc` entirely. ## Feasibility Checks diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index 9f3c1b53..fe155fae 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -38,6 +38,7 @@ jobs: environment: ${{ inputs.environment }} infra_version: ${{ github.sha }} ignore_shared_artifact_modules: ${{ inputs.environment == 'prod' && !inputs.allow_prod_cleanup }} + ignore_oidc_module: true wave_3: name: 3 / ${{ matrix.module }} diff --git a/.github/workflows/shared_get_modules.yml b/.github/workflows/shared_get_modules.yml index 3e89e430..b36553f3 100644 --- a/.github/workflows/shared_get_modules.yml +++ b/.github/workflows/shared_get_modules.yml @@ -21,6 +21,11 @@ on: required: false type: boolean default: false + ignore_oidc_module: + description: "Whether to exclude the oidc module from emitted waves" + required: false + type: boolean + default: false outputs: waves_json: description: "Dependency-safe Terragrunt wave matrix JSON" @@ -98,6 +103,7 @@ jobs: RAW_WAVES_JSON: ${{ steps.waves_json.outputs.just_outputs }} IGNORE_TASK_MODULES: ${{ inputs.ignore_task_modules }} IGNORE_SHARED_ARTIFACT_MODULES: ${{ inputs.ignore_shared_artifact_modules }} + IGNORE_OIDC_MODULE: ${{ inputs.ignore_oidc_module }} run: | echo "waves_json=$(jq -c ' if $ignore_tasks then @@ -110,10 +116,15 @@ jobs: else . end + | if $ignore_oidc_module then + map(.modules |= map(select(. != "oidc"))) + else + . + end | map(select(.modules | length > 0)) | to_entries | map({wave: .key, modules: .value.modules}) - ' --argjson ignore_tasks "$IGNORE_TASK_MODULES" --argjson ignore_shared_artifact_modules "$IGNORE_SHARED_ARTIFACT_MODULES" <<<"$RAW_WAVES_JSON")" >> "$GITHUB_OUTPUT" + ' --argjson ignore_tasks "$IGNORE_TASK_MODULES" --argjson ignore_shared_artifact_modules "$IGNORE_SHARED_ARTIFACT_MODULES" --argjson ignore_oidc_module "$IGNORE_OIDC_MODULE" <<<"$RAW_WAVES_JSON")" >> "$GITHUB_OUTPUT" - name: Expose wave module arrays id: wave_outputs From a0c872094937d1cbf15fa99e03899b5a5c257b49 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 16:02:02 +0100 Subject: [PATCH 64/73] fix: more mock_outputs_merge_strategy_with_state = "shallow" --- infra/live/dev/aws/lambda_api/terragrunt.hcl | 3 +++ infra/live/dev/aws/lambda_worker/terragrunt.hcl | 2 ++ infra/live/prod/aws/lambda_api/terragrunt.hcl | 2 ++ infra/live/prod/aws/lambda_worker/terragrunt.hcl | 1 + 4 files changed, 8 insertions(+) diff --git a/infra/live/dev/aws/lambda_api/terragrunt.hcl b/infra/live/dev/aws/lambda_api/terragrunt.hcl index 1fed78c8..f8ccbd76 100644 --- a/infra/live/dev/aws/lambda_api/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_api/terragrunt.hcl @@ -24,6 +24,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -45,6 +46,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -55,6 +57,7 @@ dependency "code_bucket" { bucket = "mock-code-bucket" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/lambda_worker/terragrunt.hcl b/infra/live/dev/aws/lambda_worker/terragrunt.hcl index 09340257..b6a7c672 100644 --- a/infra/live/dev/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_worker/terragrunt.hcl @@ -24,6 +24,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -34,6 +35,7 @@ dependency "code_bucket" { bucket = "mock-code-bucket" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/lambda_api/terragrunt.hcl b/infra/live/prod/aws/lambda_api/terragrunt.hcl index 22f87c7c..ff766ad7 100644 --- a/infra/live/prod/aws/lambda_api/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_api/terragrunt.hcl @@ -24,6 +24,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -45,6 +46,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/lambda_worker/terragrunt.hcl b/infra/live/prod/aws/lambda_worker/terragrunt.hcl index 2b609221..dbf634dd 100644 --- a/infra/live/prod/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_worker/terragrunt.hcl @@ -24,6 +24,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } From ce8cd9078444b42cef150cffd5d4f6f75851b58d Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 16:03:29 +0100 Subject: [PATCH 65/73] chore: more mock_outputs_merge_strategy_with_state = "shallow" --- infra/live/dev/aws/database/terragrunt.hcl | 1 + infra/live/dev/aws/frontend/terragrunt.hcl | 2 ++ infra/live/dev/aws/migrations/terragrunt.hcl | 3 +++ infra/live/dev/aws/network/terragrunt.hcl | 2 ++ infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl | 2 ++ infra/live/prod/aws/database/terragrunt.hcl | 1 + infra/live/prod/aws/frontend/terragrunt.hcl | 2 ++ infra/live/prod/aws/migrations/terragrunt.hcl | 2 ++ infra/live/prod/aws/network/terragrunt.hcl | 2 ++ infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl | 1 + 10 files changed, 18 insertions(+) diff --git a/infra/live/dev/aws/database/terragrunt.hcl b/infra/live/dev/aws/database/terragrunt.hcl index d11b514c..bb659ce4 100644 --- a/infra/live/dev/aws/database/terragrunt.hcl +++ b/infra/live/dev/aws/database/terragrunt.hcl @@ -14,6 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/frontend/terragrunt.hcl b/infra/live/dev/aws/frontend/terragrunt.hcl index c7f13fce..5468eac9 100644 --- a/infra/live/dev/aws/frontend/terragrunt.hcl +++ b/infra/live/dev/aws/frontend/terragrunt.hcl @@ -9,6 +9,7 @@ dependency "network" { api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -22,6 +23,7 @@ dependency "cognito" { auth_readonly_group_name = "readonly" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/migrations/terragrunt.hcl b/infra/live/dev/aws/migrations/terragrunt.hcl index d003d86f..414bc39f 100644 --- a/infra/live/dev/aws/migrations/terragrunt.hcl +++ b/infra/live/dev/aws/migrations/terragrunt.hcl @@ -14,6 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -28,6 +29,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -38,6 +40,7 @@ dependency "code_bucket" { bucket = "mock-code-bucket" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/network/terragrunt.hcl b/infra/live/dev/aws/network/terragrunt.hcl index a3b77887..5422b42c 100644 --- a/infra/live/dev/aws/network/terragrunt.hcl +++ b/infra/live/dev/aws/network/terragrunt.hcl @@ -14,6 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -29,6 +30,7 @@ dependency "cognito" { auth_issuer_url = "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_mock" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl index 04aad825..62ff2df0 100644 --- a/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl @@ -13,6 +13,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -23,6 +24,7 @@ dependency "code_bucket" { bucket = "mock-code-bucket" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/database/terragrunt.hcl b/infra/live/prod/aws/database/terragrunt.hcl index 752cffae..40ff90a9 100644 --- a/infra/live/prod/aws/database/terragrunt.hcl +++ b/infra/live/prod/aws/database/terragrunt.hcl @@ -14,6 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/frontend/terragrunt.hcl b/infra/live/prod/aws/frontend/terragrunt.hcl index c7f13fce..5468eac9 100644 --- a/infra/live/prod/aws/frontend/terragrunt.hcl +++ b/infra/live/prod/aws/frontend/terragrunt.hcl @@ -9,6 +9,7 @@ dependency "network" { api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -22,6 +23,7 @@ dependency "cognito" { auth_readonly_group_name = "readonly" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/migrations/terragrunt.hcl b/infra/live/prod/aws/migrations/terragrunt.hcl index 653e7061..f0ecb4ae 100644 --- a/infra/live/prod/aws/migrations/terragrunt.hcl +++ b/infra/live/prod/aws/migrations/terragrunt.hcl @@ -14,6 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -28,6 +29,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/network/terragrunt.hcl b/infra/live/prod/aws/network/terragrunt.hcl index a3b77887..5422b42c 100644 --- a/infra/live/prod/aws/network/terragrunt.hcl +++ b/infra/live/prod/aws/network/terragrunt.hcl @@ -14,6 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -29,6 +30,7 @@ dependency "cognito" { auth_issuer_url = "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_mock" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl index ec3ff4d8..5a5f452f 100644 --- a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl @@ -13,6 +13,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } From c86db4f341c556d5ffb8db2f8e7d214f30e91728 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 16:07:36 +0100 Subject: [PATCH 66/73] chore: image destroy place holder --- .github/workflows/destroy.yml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index fe155fae..25b81d11 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -27,6 +27,9 @@ permissions: env: TF_VAR_lambda_version: this + TF_VAR_image_uri: destroy-placeholder + TF_VAR_aws_otel_collector_image_uri: destroy-placeholder + TF_VAR_debug_image_uri: destroy-placeholder AWS_OIDC_ROLE_ARN: arn:aws:iam::${{ vars.AWS_ACCOUNT_ID }}:role/${{ vars.PROJECT_NAME }}-${{ inputs.environment }}-github-oidc-role AWS_REGION: ${{ vars.AWS_REGION }} @@ -59,10 +62,6 @@ jobs: - name: Destroy ${{ matrix.module }} infra uses: ./.github/actions/terragrunt - env: - TF_VAR_image_uri: ${{ startsWith(matrix.module, 'task_') && 'destroy-placeholder' || '' }} - TF_VAR_debug_image_uri: ${{ startsWith(matrix.module, 'task_') && 'destroy-placeholder' || '' }} - TF_VAR_aws_otel_collector_image_uri: ${{ startsWith(matrix.module, 'task_') && 'destroy-placeholder' || '' }} with: tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} tg_action: destroy From 455598e5cd48d83783e70c53c568b11aad541525 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 16:09:43 +0100 Subject: [PATCH 67/73] chore: rm wave 3 from destroy --- .github/docs/README.md | 2 +- .github/workflows/destroy.yml | 29 ++--------------------------- 2 files changed, 3 insertions(+), 28 deletions(-) diff --git a/.github/docs/README.md b/.github/docs/README.md index 25ac1029..216a0c8f 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -132,7 +132,7 @@ flowchart LR ### Cleanup And Discovery - `destroy.yml` - Tears down infrastructure through the same Terragrunt graph contract as plan/apply, but in reverse wave order so downstream runtimes are removed before shared dependencies. It derives the current module waves through `shared_get_modules.yml`, using that reusable workflow's filtering inputs to omit `oidc` entirely and to omit `code_bucket` and `ecr` from the destroy set for `prod` unless `allow_prod_cleanup` is enabled, and then runs `wave_3`, `wave_2`, `wave_1`, and `wave_0`. The only remaining module-specific destroy placeholder vars are the required ECS task image inputs for `task_*`; the frontend, service, and database stacks now rely on Terragrunt dependency mocks instead of workflow-injected destroy placeholders. After the main graph completes, the workflow first counts tagged leftovers through `justfile.destroy`, prints a warning only when any remain, and then runs the cleanup recipe. That cleanup currently deletes leaked Cognito user pools, deregisters and then deletes leaked ECS task-definition revisions, deletes leftover ECS clusters, and force-deletes leftover Secrets Manager secrets, then validates the remaining tagged ARNs against the underlying service APIs rather than treating the tagging index as the source of truth. Already-removed Cognito pools, ECS task-definition revisions, ECS clusters, or Secrets Manager secrets are treated as successful no-ops so stale tagging API results do not fail cleanup. If unsupported or still-live tagged resources remain after that sweep, the workflow now records a warning and step summary instead of failing the whole destroy run. `prod` runs that same path only when `allow_prod_cleanup` is enabled, and the workflow prints a conspicuous warning first. + Tears down infrastructure through the same Terragrunt graph contract as plan/apply, but in reverse shared-infra wave order. It derives the current module waves through `shared_get_modules.yml`, using that reusable workflow's filtering inputs to omit `oidc` entirely and to omit `code_bucket` and `ecr` from the destroy set for `prod` unless `allow_prod_cleanup` is enabled, and then runs `wave_2`, `wave_1`, and `wave_0`. The dedicated `wave_3` destroy stage is intentionally omitted. The only remaining module-specific destroy placeholder vars are the required ECS task image inputs for `task_*`; the frontend, service, and database stacks now rely on Terragrunt dependency mocks instead of workflow-injected destroy placeholders. After the main graph completes, the workflow first counts tagged leftovers through `justfile.destroy`, prints a warning only when any remain, and then runs the cleanup recipe. That cleanup currently deletes leaked Cognito user pools, deregisters and then deletes leaked ECS task-definition revisions, deletes leftover ECS clusters, and force-deletes leftover Secrets Manager secrets, then validates the remaining tagged ARNs against the underlying service APIs rather than treating the tagging index as the source of truth. Already-removed Cognito pools, ECS task-definition revisions, ECS clusters, or Secrets Manager secrets are treated as successful no-ops so stale tagging API results do not fail cleanup. If unsupported or still-live tagged resources remain after that sweep, the workflow now records a warning and step summary instead of failing the whole destroy run. `prod` runs that same path only when `allow_prod_cleanup` is enabled, and the workflow prints a conspicuous warning first. - `shared_directories_get.yml` Derives the directory-based matrices used by wrapper workflows and PR action-test discovery. The distinction between `service_dirs` and `container_dirs` is intentional: `service_dirs` contains deployable ECS service image directories only, while `container_dirs` also includes shared ECS sidecar image targets such as `debug` and `otel_collector`. ECS artifact builds that feed `shared_build.yml` should use `container_dirs` for `ecs_matrix`, because ECS task deploys need the shared sidecar images as well as the service images. Workflows that only need app service names or task/service stack derivation should use `service_dirs`. diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index 25b81d11..00730e7a 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -43,35 +43,10 @@ jobs: ignore_shared_artifact_modules: ${{ inputs.environment == 'prod' && !inputs.allow_prod_cleanup }} ignore_oidc_module: true - wave_3: - name: 3 / ${{ matrix.module }} - needs: generate_waves - if: ${{ needs.generate_waves.outputs.wave_3_modules != '[]' }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - module: ${{ fromJson(needs.generate_waves.outputs.wave_3_modules) }} - steps: - - uses: actions/checkout@v6 - - - uses: aws-actions/configure-aws-credentials@v6 - with: - role-to-assume: ${{ env.AWS_OIDC_ROLE_ARN }} - aws-region: ${{ env.AWS_REGION }} - - - name: Destroy ${{ matrix.module }} infra - uses: ./.github/actions/terragrunt - with: - tg_directory: infra/live/${{ inputs.environment }}/aws/${{ matrix.module }} - tg_action: destroy - wave_2: name: 2 / ${{ matrix.module }} - needs: - - generate_waves - - wave_3 - if: ${{ always() && needs.generate_waves.outputs.wave_2_modules != '[]' && (needs.wave_3.result == 'success' || needs.wave_3.result == 'skipped') }} + needs: generate_waves + if: ${{ needs.generate_waves.outputs.wave_2_modules != '[]' }} runs-on: ubuntu-latest strategy: fail-fast: false From 3a0544ae58e5d587b52ecc5ef3257cf7f8080451 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 16:44:12 +0100 Subject: [PATCH 68/73] fix: fmt --- infra/live/dev/aws/database/terragrunt.hcl | 2 +- infra/live/dev/aws/frontend/terragrunt.hcl | 4 ++-- infra/live/dev/aws/lambda_api/terragrunt.hcl | 6 +++--- infra/live/dev/aws/lambda_worker/terragrunt.hcl | 4 ++-- infra/live/dev/aws/migrations/terragrunt.hcl | 6 +++--- infra/live/dev/aws/network/terragrunt.hcl | 4 ++-- infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl | 4 ++-- infra/live/dev/aws/service_api/terragrunt.hcl | 8 ++++---- infra/live/dev/aws/service_worker/terragrunt.hcl | 10 +++++----- infra/live/dev/aws/task_worker/terragrunt.hcl | 4 ++-- infra/live/prod/aws/database/terragrunt.hcl | 2 +- infra/live/prod/aws/frontend/terragrunt.hcl | 4 ++-- infra/live/prod/aws/lambda_api/terragrunt.hcl | 4 ++-- infra/live/prod/aws/lambda_worker/terragrunt.hcl | 2 +- infra/live/prod/aws/migrations/terragrunt.hcl | 4 ++-- infra/live/prod/aws/network/terragrunt.hcl | 4 ++-- infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl | 2 +- infra/live/prod/aws/service_api/terragrunt.hcl | 6 +++--- infra/live/prod/aws/service_worker/terragrunt.hcl | 8 ++++---- infra/live/prod/aws/task_worker/terragrunt.hcl | 4 ++-- 20 files changed, 46 insertions(+), 46 deletions(-) diff --git a/infra/live/dev/aws/database/terragrunt.hcl b/infra/live/dev/aws/database/terragrunt.hcl index bb659ce4..428ee1c2 100644 --- a/infra/live/dev/aws/database/terragrunt.hcl +++ b/infra/live/dev/aws/database/terragrunt.hcl @@ -14,7 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/frontend/terragrunt.hcl b/infra/live/dev/aws/frontend/terragrunt.hcl index 5468eac9..bc2220f9 100644 --- a/infra/live/dev/aws/frontend/terragrunt.hcl +++ b/infra/live/dev/aws/frontend/terragrunt.hcl @@ -9,7 +9,7 @@ dependency "network" { api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -23,7 +23,7 @@ dependency "cognito" { auth_readonly_group_name = "readonly" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/lambda_api/terragrunt.hcl b/infra/live/dev/aws/lambda_api/terragrunt.hcl index f8ccbd76..17e67ce6 100644 --- a/infra/live/dev/aws/lambda_api/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_api/terragrunt.hcl @@ -24,7 +24,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -46,7 +46,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -57,7 +57,7 @@ dependency "code_bucket" { bucket = "mock-code-bucket" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/lambda_worker/terragrunt.hcl b/infra/live/dev/aws/lambda_worker/terragrunt.hcl index b6a7c672..ccd15214 100644 --- a/infra/live/dev/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/dev/aws/lambda_worker/terragrunt.hcl @@ -24,7 +24,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -35,7 +35,7 @@ dependency "code_bucket" { bucket = "mock-code-bucket" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/migrations/terragrunt.hcl b/infra/live/dev/aws/migrations/terragrunt.hcl index 414bc39f..0633a506 100644 --- a/infra/live/dev/aws/migrations/terragrunt.hcl +++ b/infra/live/dev/aws/migrations/terragrunt.hcl @@ -14,7 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -29,7 +29,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -40,7 +40,7 @@ dependency "code_bucket" { bucket = "mock-code-bucket" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/network/terragrunt.hcl b/infra/live/dev/aws/network/terragrunt.hcl index 5422b42c..0824c6dc 100644 --- a/infra/live/dev/aws/network/terragrunt.hcl +++ b/infra/live/dev/aws/network/terragrunt.hcl @@ -14,7 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -30,7 +30,7 @@ dependency "cognito" { auth_issuer_url = "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_mock" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl index 62ff2df0..2b177266 100644 --- a/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/dev/aws/rds_reader_tagger/terragrunt.hcl @@ -13,7 +13,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -24,7 +24,7 @@ dependency "code_bucket" { bucket = "mock-code-bucket" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/service_api/terragrunt.hcl b/infra/live/dev/aws/service_api/terragrunt.hcl index 94cae671..f3f22b28 100644 --- a/infra/live/dev/aws/service_api/terragrunt.hcl +++ b/infra/live/dev/aws/service_api/terragrunt.hcl @@ -15,7 +15,7 @@ dependency "ecr" { repository_arn = "arn:aws:ecr:eu-west-2:111111111111:repository/mock-ecr" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -31,7 +31,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -43,7 +43,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -65,7 +65,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/service_worker/terragrunt.hcl b/infra/live/dev/aws/service_worker/terragrunt.hcl index ac2ebac3..7970bb19 100644 --- a/infra/live/dev/aws/service_worker/terragrunt.hcl +++ b/infra/live/dev/aws/service_worker/terragrunt.hcl @@ -15,7 +15,7 @@ dependency "ecr" { repository_arn = "arn:aws:ecr:eu-west-2:111111111111:repository/mock-ecr" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -31,7 +31,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -53,7 +53,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -65,7 +65,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -87,7 +87,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/dev/aws/task_worker/terragrunt.hcl b/infra/live/dev/aws/task_worker/terragrunt.hcl index c9ef3ee8..d9580640 100644 --- a/infra/live/dev/aws/task_worker/terragrunt.hcl +++ b/infra/live/dev/aws/task_worker/terragrunt.hcl @@ -24,7 +24,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -39,7 +39,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/database/terragrunt.hcl b/infra/live/prod/aws/database/terragrunt.hcl index 40ff90a9..8f8f5a6e 100644 --- a/infra/live/prod/aws/database/terragrunt.hcl +++ b/infra/live/prod/aws/database/terragrunt.hcl @@ -14,7 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/frontend/terragrunt.hcl b/infra/live/prod/aws/frontend/terragrunt.hcl index 5468eac9..bc2220f9 100644 --- a/infra/live/prod/aws/frontend/terragrunt.hcl +++ b/infra/live/prod/aws/frontend/terragrunt.hcl @@ -9,7 +9,7 @@ dependency "network" { api_invoke_url = "https://mockapi123.execute-api.eu-west-2.amazonaws.com" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -23,7 +23,7 @@ dependency "cognito" { auth_readonly_group_name = "readonly" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/lambda_api/terragrunt.hcl b/infra/live/prod/aws/lambda_api/terragrunt.hcl index ff766ad7..dfc53d7d 100644 --- a/infra/live/prod/aws/lambda_api/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_api/terragrunt.hcl @@ -24,7 +24,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -46,7 +46,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/lambda_worker/terragrunt.hcl b/infra/live/prod/aws/lambda_worker/terragrunt.hcl index dbf634dd..9ba0a4c3 100644 --- a/infra/live/prod/aws/lambda_worker/terragrunt.hcl +++ b/infra/live/prod/aws/lambda_worker/terragrunt.hcl @@ -24,7 +24,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/migrations/terragrunt.hcl b/infra/live/prod/aws/migrations/terragrunt.hcl index f0ecb4ae..857435f6 100644 --- a/infra/live/prod/aws/migrations/terragrunt.hcl +++ b/infra/live/prod/aws/migrations/terragrunt.hcl @@ -14,7 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -29,7 +29,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/network/terragrunt.hcl b/infra/live/prod/aws/network/terragrunt.hcl index 5422b42c..0824c6dc 100644 --- a/infra/live/prod/aws/network/terragrunt.hcl +++ b/infra/live/prod/aws/network/terragrunt.hcl @@ -14,7 +14,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -30,7 +30,7 @@ dependency "cognito" { auth_issuer_url = "https://cognito-idp.eu-west-2.amazonaws.com/eu-west-2_mock" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl index 5a5f452f..8eda104e 100644 --- a/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl +++ b/infra/live/prod/aws/rds_reader_tagger/terragrunt.hcl @@ -13,7 +13,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/service_api/terragrunt.hcl b/infra/live/prod/aws/service_api/terragrunt.hcl index 4c5dcdb6..7e898bda 100644 --- a/infra/live/prod/aws/service_api/terragrunt.hcl +++ b/infra/live/prod/aws/service_api/terragrunt.hcl @@ -18,7 +18,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -30,7 +30,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -52,7 +52,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/service_worker/terragrunt.hcl b/infra/live/prod/aws/service_worker/terragrunt.hcl index a62abc29..f7ec20d6 100644 --- a/infra/live/prod/aws/service_worker/terragrunt.hcl +++ b/infra/live/prod/aws/service_worker/terragrunt.hcl @@ -18,7 +18,7 @@ dependency "security" { postgres_sg = "sg-00000000000000006" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -40,7 +40,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -52,7 +52,7 @@ dependency "cluster" { cluster_name = "mock-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -74,7 +74,7 @@ dependency "network" { http_api_authorizer_id = "auth-mock123" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } diff --git a/infra/live/prod/aws/task_worker/terragrunt.hcl b/infra/live/prod/aws/task_worker/terragrunt.hcl index c9ef3ee8..d9580640 100644 --- a/infra/live/prod/aws/task_worker/terragrunt.hcl +++ b/infra/live/prod/aws/task_worker/terragrunt.hcl @@ -24,7 +24,7 @@ dependency "messaging" { ecs_worker_queue_read_policy_arn = "arn:aws:iam::111111111111:policy/mock-ecs-worker-queue-read" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } @@ -39,7 +39,7 @@ dependency "database" { database_cluster_identifier = "mock-database-cluster" } - mock_outputs_merge_strategy_with_state = "shallow" + mock_outputs_merge_strategy_with_state = "shallow" mock_outputs_allowed_terraform_commands = ["validate", "plan", "destroy", "init", "show", "graph-dependencies", "output-module-groups"] } From 026509c66f9aef46e80b2434a3fe54e6d4a4fd19 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 16:49:30 +0100 Subject: [PATCH 69/73] chore: include has changes in metadata json --- .github/actions/terragrunt/README.md | 2 +- .github/actions/terragrunt/action.yml | 8 +++++++- .github/docs/README.md | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index 9c3b66b9..97653cf3 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -40,7 +40,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - `apply` Runs `terragrunt apply -auto-approve` - `plan` - Runs `terragrunt plan -detailed-exitcode -out=/terragrunt.tfplan`. The action writes `terragrunt.plan.meta.json` into the live stack directory for every plan run and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes. + Runs `terragrunt plan -detailed-exitcode -out=/terragrunt.tfplan`. The action writes `terragrunt.plan.meta.json` into the live stack directory for every plan run, including `has_changes` and `contains_mocked_outputs`, and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes. - `apply_plan` Runs `terragrunt apply /terragrunt.tfplan`. The calling workflow is expected to download that stack's saved plan artifact into the live stack directory before invoking `apply_plan`. - `destroy` diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index 5b375ade..f2605cde 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -94,6 +94,11 @@ runs: exit 1 fi + plan_has_changes=false + if [ "$plan_exit_code" -eq 2 ]; then + plan_has_changes=true + fi + plan_contains_mocked_outputs=false if grep -Fq "mock outputs provided and returning those in dependency output" "$PLAN_LOG_PATH"; then plan_contains_mocked_outputs=true @@ -102,8 +107,9 @@ runs: jq -n \ --arg tg_directory "${{ inputs.tg_directory }}" \ + --argjson has_changes "$plan_has_changes" \ --argjson contains_mocked_outputs "$plan_contains_mocked_outputs" \ - '{tg_directory: $tg_directory, contains_mocked_outputs: $contains_mocked_outputs}' \ + '{tg_directory: $tg_directory, has_changes: $has_changes, contains_mocked_outputs: $contains_mocked_outputs}' \ > "$PLAN_META_PATH" terragrunt show -no-color "$PLAN_PATH" > "$PLAN_TEXT_PATH" diff --git a/.github/docs/README.md b/.github/docs/README.md index 216a0c8f..256363a4 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -148,7 +148,7 @@ Run these checks on every CI, workflow, or deploy-contract change. - compare every caller `with:` block against the callee `workflow_call.inputs` - compare expected outputs against actual `jobs..outputs.*` - verify optional inputs are intentionally omitted, not accidentally missing -- the repo-local `./.github/actions/terragrunt` action supports `tg_action: plan` for producing the binary plan in the live stack directory, writes `terragrunt.plan.meta.json` there for every saved plan, and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes +- the repo-local `./.github/actions/terragrunt` action supports `tg_action: plan` for producing the binary plan in the live stack directory, writes `terragrunt.plan.meta.json` there for every saved plan including `has_changes` and `contains_mocked_outputs`, and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes - `apply_plan` now expects the calling workflow job to download the matching per-stack GitHub artifact into the live stack directory before invoking Terragrunt - when a live Terragrunt `dependency` block uses `mock_outputs` for planability or destroy safety, default it to `mock_outputs_merge_strategy_with_state = "shallow"` so partial real upstream state does not suppress missing mock keys - both repo-local composite actions, `./.github/actions/just` and `./.github/actions/terragrunt`, now assume AWS credentials are already configured in the current job when they need AWS access. The repo pattern is to run `aws-actions/configure-aws-credentials` at the top of each AWS-using job and then call the local actions without extra auth inputs From 118d0f2cd9f88fa7ffe7383708fc53e60aa9a160 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 16:54:02 +0100 Subject: [PATCH 70/73] chore: check for metadata json --- .github/actions/terragrunt/README.md | 2 +- .github/actions/terragrunt/action.yml | 12 ++++++++++++ .github/docs/README.md | 2 +- 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/.github/actions/terragrunt/README.md b/.github/actions/terragrunt/README.md index 97653cf3..b3f6a2b3 100644 --- a/.github/actions/terragrunt/README.md +++ b/.github/actions/terragrunt/README.md @@ -42,7 +42,7 @@ The Terragrunt install step is kept in this repo-local action rather than hidden - `plan` Runs `terragrunt plan -detailed-exitcode -out=/terragrunt.tfplan`. The action writes `terragrunt.plan.meta.json` into the live stack directory for every plan run, including `has_changes` and `contains_mocked_outputs`, and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes. - `apply_plan` - Runs `terragrunt apply /terragrunt.tfplan`. The calling workflow is expected to download that stack's saved plan artifact into the live stack directory before invoking `apply_plan`. + Runs `terragrunt apply /terragrunt.tfplan`. The calling workflow is expected to download that stack's saved plan artifact into the live stack directory before invoking `apply_plan`. The action also requires `terragrunt.plan.meta.json` to be present there. If that metadata file is missing, or if it says `contains_mocked_outputs: true`, the action fails before apply and tells the operator to regenerate the plan from real upstream outputs. - `destroy` Runs `terragrunt destroy -auto-approve` - `init` diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index f2605cde..396d722a 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -125,6 +125,18 @@ runs: exit 1 fi + if [ ! -f "$PLAN_META_PATH" ]; then + echo "::error title=Missing saved plan metadata::Expected '${PLAN_META_PATH}' to exist before apply_plan." + exit 1 + fi + + plan_contains_mocked_outputs="$(jq -r '.contains_mocked_outputs // false' "$PLAN_META_PATH")" + + if [ "$plan_contains_mocked_outputs" = "true" ]; then + echo "::error title=Saved plan contains mocked outputs::Saved plan metadata indicates mocked outputs were used. Regenerate it after upstream real outputs exist." + exit 1 + fi + set +e APPLY_LOG_PATH="$(pwd)/terragrunt.apply.log" terragrunt apply -auto-approve "$PLAN_PATH" 2>&1 | tee "$APPLY_LOG_PATH" diff --git a/.github/docs/README.md b/.github/docs/README.md index 256363a4..e1a41817 100644 --- a/.github/docs/README.md +++ b/.github/docs/README.md @@ -149,7 +149,7 @@ Run these checks on every CI, workflow, or deploy-contract change. - compare expected outputs against actual `jobs..outputs.*` - verify optional inputs are intentionally omitted, not accidentally missing - the repo-local `./.github/actions/terragrunt` action supports `tg_action: plan` for producing the binary plan in the live stack directory, writes `terragrunt.plan.meta.json` there for every saved plan including `has_changes` and `contains_mocked_outputs`, and writes `terragrunt.plan.txt` alongside the binary plan when the plan has changes -- `apply_plan` now expects the calling workflow job to download the matching per-stack GitHub artifact into the live stack directory before invoking Terragrunt +- `apply_plan` now expects the calling workflow job to download the matching per-stack GitHub artifact into the live stack directory before invoking Terragrunt, including `terragrunt.plan.meta.json`, and it fails immediately if that metadata file is missing or says `contains_mocked_outputs: true` - when a live Terragrunt `dependency` block uses `mock_outputs` for planability or destroy safety, default it to `mock_outputs_merge_strategy_with_state = "shallow"` so partial real upstream state does not suppress missing mock keys - both repo-local composite actions, `./.github/actions/just` and `./.github/actions/terragrunt`, now assume AWS credentials are already configured in the current job when they need AWS access. The repo pattern is to run `aws-actions/configure-aws-credentials` at the top of each AWS-using job and then call the local actions without extra auth inputs - `./.github/actions/just` installs the requested `just` version through `extractions/setup-crate@v2` in the same minimal composite-action shape as `extractions/setup-just`, rather than depending on `extractions/setup-just` itself From d8de37c58b6895af4a12e20c9ec36fa9e03a2b36 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 17:06:15 +0100 Subject: [PATCH 71/73] fix: --terragrunt-non-interactive --- .github/actions/terragrunt/action.yml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index 396d722a..efa981c6 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -82,11 +82,11 @@ runs: case "${{ inputs.tg_action }}" in apply) - terragrunt apply -auto-approve -compact-warnings -var-file=override_tg_vars.tfvars.json + terragrunt --terragrunt-non-interactive apply -auto-approve -compact-warnings -var-file=override_tg_vars.tfvars.json ;; plan) set +e - terragrunt plan -input=false -lock=false -detailed-exitcode -compact-warnings -out="$PLAN_PATH" -var-file=override_tg_vars.tfvars.json 2>&1 | tee "$PLAN_LOG_PATH" + terragrunt --terragrunt-non-interactive plan -input=false -lock=false -detailed-exitcode -compact-warnings -out="$PLAN_PATH" -var-file=override_tg_vars.tfvars.json 2>&1 | tee "$PLAN_LOG_PATH" plan_exit_code=${PIPESTATUS[0]} set -e @@ -139,7 +139,7 @@ runs: set +e APPLY_LOG_PATH="$(pwd)/terragrunt.apply.log" - terragrunt apply -auto-approve "$PLAN_PATH" 2>&1 | tee "$APPLY_LOG_PATH" + terragrunt --terragrunt-non-interactive apply -auto-approve "$PLAN_PATH" 2>&1 | tee "$APPLY_LOG_PATH" apply_exit_code=${PIPESTATUS[0]} set -e @@ -165,11 +165,11 @@ runs: fi ;; destroy) - terragrunt destroy -auto-approve -compact-warnings -var-file=override_tg_vars.tfvars.json + terragrunt --terragrunt-non-interactive destroy -auto-approve -compact-warnings -var-file=override_tg_vars.tfvars.json ;; init) echo "Running init only (no infra changes)..." - terragrunt init -input=false -reconfigure + terragrunt --terragrunt-non-interactive init -input=false -reconfigure ;; graph) echo "Graph mode runs in the dedicated graph step." From edbc66502823fd7802a05ed108d2943364a27399 Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 21:03:30 +0100 Subject: [PATCH 72/73] chore: cat out meta file --- .github/actions/terragrunt/action.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/actions/terragrunt/action.yml b/.github/actions/terragrunt/action.yml index efa981c6..c0e99163 100644 --- a/.github/actions/terragrunt/action.yml +++ b/.github/actions/terragrunt/action.yml @@ -112,6 +112,9 @@ runs: '{tg_directory: $tg_directory, has_changes: $has_changes, contains_mocked_outputs: $contains_mocked_outputs}' \ > "$PLAN_META_PATH" + echo "=== terragrunt.plan.meta.json ===" + cat "$PLAN_META_PATH" + terragrunt show -no-color "$PLAN_PATH" > "$PLAN_TEXT_PATH" echo "plan_exit_code=$plan_exit_code" >> "$GITHUB_OUTPUT" From de0ec055e85d4eac89406a63c82968955d6971fd Mon Sep 17 00:00:00 2001 From: chrispsheehan Date: Fri, 22 May 2026 21:12:35 +0100 Subject: [PATCH 73/73] fix: ignore_shared_artifact_modules: true --- .github/workflows/destroy.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/destroy.yml b/.github/workflows/destroy.yml index 00730e7a..740bf3fd 100644 --- a/.github/workflows/destroy.yml +++ b/.github/workflows/destroy.yml @@ -40,7 +40,7 @@ jobs: with: environment: ${{ inputs.environment }} infra_version: ${{ github.sha }} - ignore_shared_artifact_modules: ${{ inputs.environment == 'prod' && !inputs.allow_prod_cleanup }} + ignore_shared_artifact_modules: true ignore_oidc_module: true wave_2: