Repository navigation
fix(purchase): AZ-scoped EC2 recs: the purchase guard, idempotency key and recommendations natural key ignore the AZ #792
Description
Problem
Once go#305 lands in the platform (zonal EC2 RIs keep their AZ), three platform paths still ignore the AZ. Found by the Opus review of go#305 (F-notes). Line numbers are on origin/main 7e2a3d8.
- Purchase detail guard.
internal/api/purchase_pricing.go:320computeDetailMismatchcompares tenancy, platform, scope and memory_gb, but not the AZ. A re-matched rec whose AZ differs from the original passes the guard.
Fix: add anavailability_zonemismatch check with the same asymmetry as the other fields. When the original knows the AZ, a match whose AZ is empty or different is a mismatch. - Purchase idempotency key.
internal/api/handler_purchases.go:2901purchaseIdempotencyKeyhashes provider, service, region, resource type, engine, account, count, term and payment, but not the AZ. Two submits that differ only by AZ dedupe into one within the 2-minute window.
Fix: include the AZ in the tuple. - Recommendations natural key.
internal/config/store_postgres_recommendations.go:227ON CONFLICT (account_key, provider, service, region, resource_type, engine, term, payment_option)has no AZ, so two AZ recs for the same type and term collapse into one row. This does not buy in the wrong AZ, but it loses a rec.
Fix: add the AZ to the key, which needs a migration and an index change. - AZ names are per account (F4).
us-east-1ain one account can be a different physical zone in another. When the buying account differs from the account the rec was computed for, useAvailabilityZoneId, not the name.
Notes
Mostly money-adjacent. Item 1 is the purchase-time check. Item 2 can drop a legitimate second buy as a duplicate. Item 4 can buy in the wrong physical zone. This blocks the go provider bump (filed alongside).
Related: go#289, go#310.
Activity
fix(purchase): make zonal EC2 AZ part of the guard, identity key, ide…
cristim commentedon Oct 12, 2026
PR1 (#807: AZ-scoped EC2 recs in the purchase guard, idempotency key, recIdentityKey, rec ID, payer-account drop) merged as 3fde4ef after an Opus gate at f4b05f4. Leaving this issue open: PR2a, PR3, PR2b and PR2c remain.
cristim commentedon Oct 12, 2026
PR3 (frontend: availability_zone in purchase variant identity and cell key) landed via #809 as b197104. Issue stays open for the remaining PRs (PR2b/PR2c).
cristim commentedon Oct 12, 2026
PR2a landed via #808 (merge 631fb9c): migration 000109 adds the generated recommendations.availability_zone column and the 9-column unique index, and keeps the 8-column index. Dev applies it on deploy because auto-migrate runs at startup. Prod still needs the runbook step: dispatch database-migration.yml manually to apply 000109 before PR2b is released. Issue stays open for PR3, PR2b and PR2c.
cristim commentedon Oct 12, 2026
PR2b landed on main via #810 (merge 26470c3): the recommendations upsert now targets the 9-column recommendations_natural_key_az_idx. Dev already has migration 000109 (auto_migrate). PR2c (drop the 8-column index, plus the multi-AZ batch test) is still pending; until it lands, an AZ shift on one 8-tuple fails the dev collection write (documented limitation). Prod must follow the runbook: dispatch database-migration.yml for 000109 first, release PR2b together with PR2c, then dispatch the PR2c migration immediately after PR2b is live. Never run PR2c before PR2b. Leaving this issue open.
fix(db): drop the 8-column recommendations unique index so zonal EC2 …
cristim commentedon Oct 12, 2026
All four items have landed on main:
availability_zonecolumn and a 9-column index.The frontend variant identity was updated in fix(frontend): keep zonal EC2 AZ in purchase variant identity and cell key #809 (PR3).
Prod release must follow the runbook in #811:
Dev ran #810 (26470c3) on AWS, GCP and Azure before #811 merged and auto-migrated.
Follow-ups:
AvailabilityZoneIdfor cross-account zonal buys, as opposed to dropping those recs, has no tracking issue yet.cristim commentedon Oct 13, 2026
Prod release runbook for #792 and the #806 chain (owner item 65). Drafted by an agent from origin/main on 2026-10-14; every prod step is run by a human owner and nothing is dispatched by this comment. Verified items were checked against the workflow file and the migration files on main; UNVERIFIED items are marked in the text.
Prod release runbook: #792 (AZ-scoped EC2 recs) and #806 (platform/tenancy)
This runbook dispatches nothing. Every prod step below is run by a human owner (dispatch, approval, SQL, force). Facts were read from
origin/mainand the PRs on 2026-10-14. Anything not verified is marked UNVERIFIED. Migration paths are underinternal/database/postgres/migrations/(the workflow'sMIGRATIONS_PATH).1.
database-migration.ymlinputs (read from origin/main)cloud<cloud>-db-<env>.allruns the three concurrently. Enter only the clouds that run prod (UNVERIFIED which).environment*-db-<env>environment. Per.github/workflows/README.mdprod DB environments require approvals and are restricted tomain; the actual GitHub settings are UNVERIFIED (check Settings > Environments).directionmigrate up/migrate up N. down:migrate down N.steps0=migrate up(ALL pending files);N>0=migrate up N, i.e. apply the next N pending migration FILES in version order. It cannot target a version. down: must be >=1;migrate down Nreverts the N HIGHEST applied versions, not a named one.confirmdirection=down+prod: must be exactlyrollback-prod, otherwise validate fails. Ignored otherwise.compute_platform--ref main: the job checks out the dispatched ref and migrates from its migration files.cancel-in-progress: false. A queued run can be evicted by a later dispatch and thesummaryjob stays green: read each cloud's result, then verify with SQL (below).github-prod.tfvarssetsauto_migrate = falsefor aws, gcp and azure. Prod schema changes happen only through this workflow.stepsfrom the live prod version, never from memory. Pending files = every file with a version above prod's current version. main now holds 000109-000117, so a baresteps=0would apply all of them.2. Current state on origin/main and PR map
recommendations_availability_zoneavailability_zone+ unique indexrecommendations_natural_key_az_idx(9 cols). Keeps the 8-col index.ADD COLUMN GENERATED STOREDrewritesrecommendations; index build blocks writes. ACCESS EXCLUSIVE for the duration. Table is a regenerable cache.drop_recommendations_8col_natural_keyrecommendations_natural_key_idx.collected_at) then recreates the index. Loses AZ-only sibling rows from the cache; next collection repopulates.recommendations_platform_tenancyplatform,tenancycolumns + 11-col indexrecommendations_natural_key_az_tp_idx. Keeps the 9-col index.plan_step_cap_auto_purchase_switchpurchase_plans.max_step_commitment,global_config.auto_purchase_enabled(default false).steps=0.Code PRs:
ON CONFLICTon the 9-col index). All merged 2026-10-12.BLOCKER for #806, found on main: #821 names its migration
000112_drop_recommendations_9col_natural_key, but000112_plan_step_cap_auto_purchase_switchalready exists on main and 000113-000117 follow. Two files with version 112 make golang-migrate refuse to load the source; and any DB already past 112 would silently skip a lower-numbered file. #821 must be renumbered above the highest number on main AT MERGE TIME. 000118 is claimed by open PR #861 (000118_purchase_plans_ramp_baseline), so the next free number is 000119 or later (final number UNVERIFIED, re-readgit ls-tree origin/mainbefore merge). Below, the #806 drop is called "DROP-9COL (000119+)". #819's and #821's bodies still say 112; their runbooks are stale on the number only.3. #792 order and per-step checks
Order: (A) 000109 -> (B) code release containing #810 (and #807) -> (C) 000110. Never C before B: old code's
ON CONFLICTon the 8 cols needs that index, so every collection write fails. 000110's header says the same.Step A: migrate 000109 (no code dependency; old and new code both work, nothing reads the column yet).
SELECT version, dirty FROM schema_migrations;must be clean (dirty = f). Count pending files up to and including 109 (e.g. prod at 105 meanssteps=4; prod at 108 meanssteps=1). A prod version above 109 means A already happened.*-db-prodgate.SELECT version, dirty FROM schema_migrations;expect109, f.SELECT column_name FROM information_schema.columns WHERE table_name='recommendations' AND column_name='availability_zone';one row.SELECT indexname FROM pg_indexes WHERE tablename='recommendations' AND indexname IN ('recommendations_natural_key_idx','recommendations_natural_key_az_idx');BOTH rows present.direction=down, steps=1, confirm=rollback-prod. Safe only while 110 is not applied (it undoes the highest applied version). No data loss (derived). Code: nothing to roll back.Step B: release main (or a tag) containing #810 and #807 to prod through the normal deploy workflow (human-run).
curl -s https://<prod-host>/versionshows the expectedgit_sha;curl -s https://<prod-host>/healthis healthy. Smoke: after the next collection cycle, the recommendations list loads and the collection logs show no upsert errors (UNVERIFIED which log or alarm to watch).Step C: migrate 000110 (only after B is confirmed live on every prod instance, including the fargate platform if used).
/versionsha confirmed;SELECT version, dirty ...shows109, f(thensteps=1; if other pending files sit below 110, count them).SELECT version, dirty ...expect110, f.SELECT indexname FROM pg_indexes WHERE tablename='recommendations';showsrecommendations_natural_key_az_idx, NOTrecommendations_natural_key_idx. Smoke: trigger or wait for a collection; zonal EC2 recs for one instance type in two AZs both appear.down, steps=1, confirm=rollback-prodrecreates the 8-col index after deleting AZ-only duplicate cache rows (repopulated next cycle). Per 000110's down header this pairs with the pre-fix(config): upsert recommendations on the AZ-aware 9-column index #810 code; the 9-col arbiter in fix(config): upsert recommendations on the AZ-aware 9-column index #810 also works with both indexes present, so reverting only the migration is safe for current code.4. #806 order and per-step checks
Order: (D) 000111 -> (E) code release containing #819 (+#817, #818) -> (F) DROP-9COL (000119+, #821 renumbered). Prerequisites: #819 and #821 merged, #821 renumbered, #819 deployed to dev first (per #821 body).
Step D: migrate 000111. Same pre-checks as A. Pending count: if prod is at 110,
steps=1; the later files 112-117 stay pending on purpose unless their code ships in the same release (read their headers; whether HEAD code needs 112-117 columns is UNVERIFIED, apply them separately and earlier if so).110->111, f;SELECT column_name FROM information_schema.columns WHERE table_name='recommendations' AND column_name IN ('platform','tenancy');two rows; bothrecommendations_natural_key_az_idxandrecommendations_natural_key_az_tp_idxpresent.down, steps=1, confirm=rollback-prodONLY if 111 is the highest applied version; otherwise it reverts a different migration (e.g. 117). No data loss (derived columns).Step E: release code with #819. Verify
/versionsha and/health. Transitional defect (documented in #819): at schema 111 with both indexes, a Linux then Windows rec for the same 9-tuple raises 23505 and the recommendation cache stops refreshing until F. Keep E to F as short as possible; do F right after E is confirmed.Step F: DROP-9COL. Pre: E live everywhere;
111, f(or the highest unrelated applied version, with exactly one pending DROP-9COL file; verify withlsof the migrations dir at the dispatched ref).dirty = f;pg_indexesforrecommendationsshows onlyrecommendations_natural_key_az_tp_idx. Smoke: after a collection, Linux and Windows (and default/dedicated) recs of one type coexist.down, steps=1, confirm=rollback-proddeletes platform/tenancy-only duplicates (keeps newestcollected_at) and recreates the 9-col index; cache repopulates next cycle. To roll back fix(config): upsert recommendations on the 11-column index with platform and tenancy #819 code you must run this down FIRST (old code needs the 9-col arbiter). fix(db): drop the 9-column recommendations natural key index (#806) #821's own down is UNVERIFIED beyond its body text.5. Dirty state and recovery (human only)
schema_migrationsatversion=N, dirty=t. The app starts anyway (fail-open) and reports via/healthand the migration alarm. Auto-heal does NOT clear it silently (migrate.go,maybeAutoHealDirty).CUDLY_FORCE_MIGRATION_VERSION=<n>on the app (runsForce(n)beforeUp()), ormigrate -path <migrations> -database <url> force <n>. A wrong value permanently desyncs the version record from the schema (the 000074 class). A human owner decides the value and runs it; no agent runsforce.auto_migrate=false, so the env var path only works if someone enables it; the manualforceis the realistic route (UNVERIFIED which the team prefers).6. Hazards
upapplies only versions ABOVE the current one. A file numbered below the DB's version is skipped silently with no error. Never merge a migration numbered below main's highest, and re-check before dispatch._test.go(cosmetic).DB_AUTO_MIGRATE=trueingithub-dev.tfvars, as in staging). Dev and staging are AHEAD of prod. A green dev proves nothing about prod's pending count. Dev auto-migrate failures are non-fatal, so read dev'sschema_migrationstoo.stepscounts files,down Nhits the highest versions: astepsthat is right today is wrong once other migrations merge. Re-derive it at dispatch time from prod's live version.environmentorcloudcombination migrates the wrong database. For AWS,compute_platformselects a different RDS.7. Pre-flight checklist (owner)
SELECT version, dirty FROM schema_migrations;recorded for every prod DB (each cloud, each AWS platform). Clean and as expected.recommendationsfor the duration; size of prod table UNVERIFIED, checkpg_total_relation_size('recommendations')).*-db-prodenvironment reviewers identified and available to approve; branch restricted tomain.forceanddownonly by that owner.