Add warning for HttpOperator deferrable with non-idempotent methods#69748
Open
Vamsi-klu wants to merge 1 commit into
Open
Add warning for HttpOperator deferrable with non-idempotent methods#69748Vamsi-klu wants to merge 1 commit into
Vamsi-klu wants to merge 1 commit into
Conversation
HttpOperator(deferrable=True) executes the HTTP request inside HttpTrigger.run() in the Triggerer. On triggerer restart, the trigger is re-created from serialize() and run() re-executes, causing duplicate POST/PATCH requests. This is silent data duplication. Add UserWarning when deferrable=True is used with POST/PATCH etc, guiding users to idempotent methods or sensor-based polling pattern like AirbyteOperator (side-effect in worker, trigger polls with GET). Long-term fix should execute non-idempotent requests in worker and only poll in trigger. Fixes: apache#67945
Contributor
Author
|
Reviewers: @ashb @potiuk @kaxil for http provider and triggerer safety. Short-term warning for non-idempotent deferrable POST, long-term fix to move request to worker. Fixes #67945 Drafted-by: Muse Spark 1.1; reviewed by @Vamsi-klu before posting |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
HttpOperator(deferrable=True) defers immediately without making request in worker. Actual POST happens inside HttpTrigger.run() in triggerer. When triggerer restarts, trigger re-instantiated from serialize() and run() called again -> duplicate POST.
Default method POST makes out-of-box vulnerable. Silent data duplication.
Code:
What I did
Short-term warning (reviewer-friendly, per issue suggestion):
Long-term fix (documented as follow-up, not implemented):
Impact
Testing
Fixes: #67945
Was generative AI tooling used to co-author this PR?
Generated-by: Muse Spark 1.1