Skip to content

Add warning for HttpOperator deferrable with non-idempotent methods#69748

Open
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:fix/http-operator-deferrable-warning-67945
Open

Add warning for HttpOperator deferrable with non-idempotent methods#69748
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:fix/http-operator-deferrable-warning-67945

Conversation

@Vamsi-klu

Copy link
Copy Markdown
Contributor

Problem

HttpOperator(deferrable=True) defers immediately without making request in worker. Actual POST happens inside HttpTrigger.run() in triggerer. When triggerer restarts, trigger re-instantiated from serialize() and run() called again -> duplicate POST.

Default method POST makes out-of-box vulnerable. Silent data duplication.

Code:

  • Operator execute_async defers HttpTrigger with method=self.method
  • Trigger run() does aiohttp request in triggerer
  • BaseTrigger docs say triggers must assume persistence, run() may be re-executed after restart

What I did

Short-term warning (reviewer-friendly, per issue suggestion):

  • Add IDEMPOTENT_METHODS = {"GET", "HEAD", "OPTIONS", "PUT", "DELETE", "TRACE"} per RFC 7231
  • In HttpOperator.execute_async(): if method.upper() not in IDEMPOTENT_METHODS, emit UserWarning referencing HttpOperator: deferrable=True may send duplicate requests on Triggerer restart #67945, guiding to idempotent methods or HttpSensor/EventSensor polling
  • Same warning in paginate_async path
  • In HttpTrigger.init(): same warning, catches direct trigger usage; HttpEventTrigger inherits via super
  • Suppress duplicate warning from trigger when operator already warned (via catch_warnings filter)

Long-term fix (documented as follow-up, not implemented):

  • Execute non-idempotent requests in worker, only poll in trigger (Airbyte pattern)
  • Or deprecate deferrable for HttpOperator, recommend HttpSensor

Impact

  • Surfaces silent data duplication risk to users
  • Default POST now warns out-of-box (good, surfaces risk)
  • No breaking change, only warning
  • Guides users to safe patterns

Testing

  • uv run --project providers-http pytest providers/http/tests/unit/http/operators/test_http.py -k async → 4 passed (now warns for POST, expected)
  • triggers → 11 passed, full http package 103 passed 1 skipped
  • Manual:
    • POST/PATCH warns with "may send duplicate"
    • GET/PUT/DELETE/HEAD/OPTIONS/TRACE no warn

Fixes: #67945

Was generative AI tooling used to co-author this PR?
  • Yes — Muse Spark 1.1
    Generated-by: Muse Spark 1.1

HttpOperator(deferrable=True) executes the HTTP request inside
HttpTrigger.run() in the Triggerer. On triggerer restart, the trigger
is re-created from serialize() and run() re-executes, causing duplicate
POST/PATCH requests. This is silent data duplication.

Add UserWarning when deferrable=True is used with POST/PATCH etc,
guiding users to idempotent methods or sensor-based polling pattern
like AirbyteOperator (side-effect in worker, trigger polls with GET).

Long-term fix should execute non-idempotent requests in worker and
only poll in trigger.

Fixes: apache#67945
@Vamsi-klu

Copy link
Copy Markdown
Contributor Author

Reviewers: @ashb @potiuk @kaxil for http provider and triggerer safety. Short-term warning for non-idempotent deferrable POST, long-term fix to move request to worker.

Fixes #67945


Drafted-by: Muse Spark 1.1; reviewed by @Vamsi-klu before posting

@Vamsi-klu Vamsi-klu marked this pull request as ready for review July 11, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HttpOperator: deferrable=True may send duplicate requests on Triggerer restart

1 participant