Repository navigation
fix(marketplace): ambiguous AWS errors and crashes between create and persist can leave a duplicate or a stuck pending row #525
Description
Summary
Marketplace listing create and cancel can leave a duplicate listing or a row stuck in pending, and operators cannot recover a stuck row through the API.
Location (origin/main 496d9d7, internal/api/handler_marketplace.go)
reserveAndCreateListing(:221):ClientTokenisuuid.New()per call (:251), so it is not persisted and AWS cannot dedup a retry. An ambiguous AWS error (for example a timeout after AWS accepted the request) hitsreleaseMarketplaceClaim(:257), so the slot is free and a second listing is possible.- A process crash between
CreateMarketplaceListingandUpdatePurchaseHistoryListing(:266) leaves the rowpendingwith no listing id. marketplaceCancel(:336) accepts onlyactive(:360), so apendingrow cannot be recovered through the API.marketplaceCancel(:371-381): AWS cancel succeeds, the DB write fails, the row staysactive, and a retry calls AWS on an already-cancelled listing.- The success path records an empty AWS
Statusas'', which the claim treats as free while a listing is live. releaseMarketplaceClaim(:329) after a successful cancel still uses the request context.
Suggested direction
Persist the ClientToken before the AWS call and reconcile through DescribeReservedInstancesListings.
Context
Found in the #506 review (merged). Related: #335 (closed), #267, #261, #449.
Acceptance
A simulated timeout after AWS accepts the listing, and a simulated crash before persist, both recover to a single correct listing state without a duplicate, covered by tests.
Activity
cristim commentedon Oct 6, 2026
Plan for #525 (checked against current main e4c5cfc; no open PR or branch claims this; #554 and #582 touched only credential resolution and 422 term validation, not these paths).
Failure modes,
internal/api/handler_marketplace.go:reserveAndCreateListing,uuid.New()at :291): STILL PRESENT. An ambiguous AWS error releases the claim (:297) so a retry creates a second live listing.keepUncanceledListing:323).marketplaceCancelonly acceptsactive(:404): STILL PRESENT, pending rows cannot be recovered via the API.UpdatePurchaseHistoryListingwrites result.State verbatim; claim treats '' as free).AWS semantics (CreateReservedInstancesListing docs): ClientToken is a required idempotency token; a repeat with the same token returns the existing listing.
Design: derive the token deterministically from (purchase_id, previously recorded listing_id, count, price schedule) so a retry of the same attempt is idempotent on AWS's side while a deliberate re-list after cancel (listing_id changed) gets a fresh token. This needs no migration. The full fix (persist the token in a column, write the pending row before AWS, reconcile via DescribeReservedInstancesListings) needs a migration: next free number is 000026 (highest on main is 000025_admin_role_unique; no open PR adds a migration), e.g.
purchase_history.listing_client_token text NULL(no backfill; NULL for legacy rows; down drops the column).Split:
Open question for the owner: whether PR 2's reconcile runs inline on the next list/cancel request or in a background poller (the #292 status poller is referenced in comments). I will do PR 1 only now.
cristim commentedon Oct 6, 2026
Notes for PR2 (from review of #604): (1) the default price schedule is clock-derived, so a retry across a month boundary changes the derived token and AWS creates a duplicate; persisting the token before the AWS call removes the dependency. (2) row.ListingID is read before ClaimMarketplaceListingSlot, so a stale row in a concurrent request can reuse a canceled listing's token; PR2 should re-read the row after the claim or persist the token in the claim UPDATE.
fix(marketplace): record a rolled-back listing so the retry gets a ne…
fix(marketplace): make the claim race-safe and fence the release