fix(resource): atomic single-shot deprovision on DELETE — close double-deprovision TOCTOU#236
Merged
mastermanas805 merged 7 commits intoJun 3, 2026
Conversation
…e-deprovision TOCTOU Design-review follow-up to #229. The idempotent-DELETE fix used a `resource.Status == "deleted"` PRE-READ before SoftDeleteResource + deprovision — a check-then-act TOCTOU: two concurrent DELETEs both read status='active', both pass the pre-read, and BOTH fire the destructive provisioner Deprovision / storage Deprovision RPC against the same backend. That is the truehomie-db DROP incident class (an active customer DB dropped via a path that re-ran teardown). Fix: make the soft-delete the authoritative atomic guard. New models.SoftDeleteResourceIfActive does `UPDATE ... SET status='deleted' WHERE id=$1 AND status != 'deleted'` and returns RowsAffected>0. The handler deprovisions ONLY when this call won the transition (deleted==true); any racer or sequential retry gets deleted==false → idempotent {"ok":true,"already_deleted":true} with NO second deprovision. Removes the redundant pre-read (the rows-affected check subsumes it and is race-safe). SoftDeleteResource is kept unchanged for the ~18 provision-rollback callers that always operate on a just-created row. Tests: models.TestSoftDeleteResourceIfActive_Branches (sqlmock: won / no-op / error — 100% of the new fn). The existing TestResourceDelete_Idempotent_ DoubleDelete now exercises the rows-affected !deleted branch via a sequential retry (200 already_deleted, no re-deprovision). All changed handler lines covered. Verified vs real Postgres+Redis. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What (design-review follow-up to #229)
A multi-role review panel found a real TOCTOU in the merged idempotent-DELETE fix (#229). It used a
resource.Status == "deleted"pre-read beforeSoftDeleteResource+ deprovision — check-then-act: two concurrent DELETEs both readstatus='active', both pass the pre-read, and both fire the destructiveDeprovisionRPC against the same backend. That's the truehomie-db DROP incident class (an active customer DB torn down by a path that re-ran teardown). My #229 test was sequential, so it missed the concurrent case.Fix
Make the soft-delete itself the authoritative atomic guard:
The handler deprovisions only when this call won the transition (
deleted==true). Any racer or sequential retry getsdeleted==false→ idempotent{"ok":true,"already_deleted":true}with no second deprovision. The redundantstatuspre-read is removed (the rows-affected check subsumes it and is race-safe).SoftDeleteResourceis unchanged for the ~18 provision-rollback callers (always a just-created row).Coverage
Follow-up (tracked for the integration suite): a 16-way concurrent
-racehandler test asserting the provisioner receives exactly one Deprovision RPC.🤖 Generated with Claude Code