Repository navigation
ci(azure): Deploy to Azure Container Apps has failed on every main run for 3 weeks — no Azure change since 2026-07-19 is deployed #1794
Description
The Azure deploy has not succeeded on main in the last 100 workflow runs, going back to at least 2026-07-19. Found while checking whether #1223 was stale.
Deploy to Azure Container Apps, branch=main, last 100 runs:
failure: 68
cancelled: 32
success: 0
oldest run in window: 2026-07-19
It fails at Terraform Apply, not at build or test:
Error: authorization.RoleAssignmentsClient#Create: Failure responding ...
Error: loading Role Definition List: could not find role 'CUDly Reservation Purchaser (custom) - ***'
Blocked resources: azurerm_role_assignment.cost_management_reader and azurerm_role_assignment.subscription_reader.
Why this went unnoticed for three weeks
CI - Build & Test is green on main and has been across every recent merge (a37e14790, d6e60f637, 726389b48). Every PR gate we look at is green. The deploy workflow is a separate run that nothing surfaces, so "CI is green" has been true and irrelevant at the same time.
This is the same shape as the multi-module gap in #1751: a check that looked comprehensive was scoped to something narrower than anyone assumed. There, ./... covered one module of six. Here, "CI green" covers build and test but not deploy.
What it means for what we have been merging
If this workflow is what ships the running Azure service, then no Azure change merged since 2026-07-19 is live. That window includes several providers/azure money-path fixes merged in the last day:
- fix(azure): surface the read error that decides purchase retry-vs-abort #1767 — purchase retry no longer abandons on a truncated "Session timed out" body
- fix(azure): pair the recommended SKU with the count in its own units #1771 — Legacy/EA recommendations no longer pair the normalized SKU with the un-normalized quantity
- sec(azure): re-check idempotency between in-process purchase retries #1793 — idempotency re-checked between in-process retries, preventing a double purchase
- sec(auth): carve execute:ri-exchange out of admin:* and seed the granting group #1758 —
execute:ri-exchangecarved out ofadmin:*
Verify that inference before acting on it. I have established the workflow fails; I have not established what it deploys, whether another path ships the Azure service, or whether the failing role assignments are the last step or an early one that aborts before anything useful happens. A partial apply that fails at the end has different consequences from one that fails before deploying the app image.
The immediate cause
A custom role definition named CUDly Reservation Purchaser (custom) - <subscription> does not exist in the target subscription, so Terraform cannot resolve it to assign it.
Establish which of these it is before fixing:
- The role was never created in this subscription (bootstrap step missed or a new subscription).
- The role was created and has since been deleted or renamed.
- The Terraform name template drifted from what the bootstrap creates (a suffix, a casing change, an interpolation that renders differently).
- The deploy identity lacks permission to read role definitions, so the role exists but the lookup returns nothing. This one looks like a missing role but is a permissions failure, and the error text does not distinguish it.
Option 4 is worth ruling out explicitly: Microsoft.Authorization/roleDefinitions/read is a separate grant from the ability to create assignments, and a caller without it sees an empty list rather than a permission error.
Verification for whatever lands
- A deploy run on
mainreaching success, not merely getting past this error to a different one. - The two blocked role assignments present in the subscription afterwards.
- Confirm the deployed image actually corresponds to current
main, since three weeks of failed applies may have left state that a single success does not fully reconcile.
Related
- Main CI red post-#1220: golangci-lint flood, npm audit, internal/config integration failures #1223 — separately confirmed stale and being closed;
CI - Build & Testis green onmain. - ci: unit and integration test jobs only run the root module — providers/*, pkg and tests/e2e are never tested #1751 — same class of blind spot: a green check whose scope was narrower than assumed.
Activity
fix(ci/azure): cover a plan-time role failure and grep each log separ…
docs(ci/azure): correct two false claims in the diagnostic's comments
fix(ci/azure): name the missing bootstrap stack when the role lookup …
cristim commentedon Aug 11, 2026
Corrected root cause: the bootstrap role is unappliable, not unapplied
This issue currently says the custom role was never created because the bootstrap step was missed. That is wrong, and the real cause has a wider blast radius. Established by execution against the live subscription, not by reading.
The actual cause
Microsoft.Capacity/reservationOrders/purchase/actiondoes not exist in Azure's operation catalog.Applying
terraform/environments/azure/ci-cd-permissionsfails with:Azure rejects the whole role definition over that one entry, so the role is never created, so the deploy stack's name-based
data.azurerm_role_definitionlookup finds nothing. The bootstrap stack was not skipped: it cannot succeed as written, and re-running it would never have helped.I validated all 12 actions in the module against the live catalog (
az provider operation show, covering both namespace-leveloperations[]andresourceTypes[].operations[]). Exactly one is invalid; the other 11 are fine.Hypothesis tested and ruled out
Microsoft.CapacityandMicrosoft.BillingBenefitswere bothNotRegisteredin subscription24d185cc. That looked like the answer, since Azure validates role actions against registered providers. I registered both and re-applied: identical error. Registration was not the cause.The two registrations are additive, reversible, and arguably correct to keep, but they were an incidental change rather than the fix, and I am flagging them rather than leaving them undisclosed.
The module's stated premise is also wrong
The module justifies its own existence with:
The built-in role's actual actions do not include
Microsoft.Capacity/reservationOrders/write,Microsoft.Capacity/calculatePrice/action, or anyMicrosoft.BillingBenefitsaction. So the custom role's purpose is legitimate, but the action added to serve that purpose is fictional. Purchasing a reservation isPUT /providers/Microsoft.Capacity/reservationOrders/{id}, which isreservationOrders/writeand was already in the list.Blast radius is wider than this issue
arm/CUDly-CrossSubscription/template.json:51carries the same invalid action. That template is customer-facing: customers deploy it to grant CUDly cross-subscription access. It should fail identically for every customer who has run it. That deserves its own triage.iac/federation/azure-target/terraformis the other customer-side consumer.Why no test caught this
scripts/testdata/role-parity/matching-arm.jsonandmatching-tf.tf.fixtureboth contain the invalid action. The ARM-vs-TF parity checker therefore passed while both sides were invalid. The fixtures prove the two representations agree; nothing proved either is a thing Azure accepts. Parity is not validity.Verification
Removing the single invalid action and applying against the live subscription created the role:
Confirmed in both directions: fails with the action present, succeeds with it removed.
Answering the open question about deployed state
The container app is running
cudly:ddeeb98-20260811004918, andddeeb98is the currentmainHEAD. The deploy job builds and ships the image, then fails later duringterraform applywhen the role lookup is resolved (plan defers it: the log showsdata.azurerm_role_definition.cudly_reservation_purchaser will be read during apply).So application code from the merged Azure money-path work is live. What has not been applying is the Terraform-managed infrastructure ordered after the failure point. That is a meaningfully better position than the worst case, but the two should not be conflated.
A rerun of the failed deploy is in flight now that the role exists; I will post the result.
Note on the CI diagnostic from #1795
The
Explain a missing bootstrap rolestep added in #1795 tells operators the bootstrap stack was never applied. Given the above, that message points at the wrong cause and is being corrected in the fix PR.