Skip to content

Add validate tirgger events in dbt cloud deferrable tasks#69739

Open
takayoshi-makabe wants to merge 4 commits into
apache:mainfrom
takayoshi-makabe:add-validate-trigger-events-in-dbt-cloud-deferrable-tasks
Open

Add validate tirgger events in dbt cloud deferrable tasks#69739
takayoshi-makabe wants to merge 4 commits into
apache:mainfrom
takayoshi-makabe:add-validate-trigger-events-in-dbt-cloud-deferrable-tasks

Conversation

@takayoshi-makabe

Copy link
Copy Markdown
Contributor

DbtCloudRunJobOperator.execute_complete and DbtCloudJobRunSensor.execute_complete used the trigger event without validating it:

  • event=None: happens when a deferred task resumes without a payload — a custom or subclassed trigger that yields TriggerEvent(None), or an event payload lost between the triggerer and the worker. Both methods crashed with an opaque TypeError: 'NoneType' object is not subscriptable.
  • Unrecognised status: happens on triggerer/worker version skew (a task defers under one provider version and resumes under another whose trigger emits a different status vocabulary) or with a custom trigger. DbtCloudRunJobTrigger emits success/cancelled/error/timeout, but both execute_complete methods branch deny-list style (only specific failure statuses raise, everything else succeeds), so an unknown status fell through into the success path — the task was marked SUCCESS even though the dbt Cloud job outcome was unknown.
  • The sensor additionally never checked timeout at all — it only raised on error/cancelled, so a job run that timed out while deferred was silently marked SUCCESS.

This adds a validate_execute_complete_event() helper (with a dedicated DbtCloudTriggerEventException), applied at the top of both execute_complete methods. Same fix shape as the anthropic (#69379) and OpenAI (#69506) providers: the None check mirrors the amazon provider's helper of the same name, and the status check plays the role amazon's call sites cover with their if validated_event["status"] != "success": raise allow-list branching.

The sensor's fix also replaces a bare raise AirflowException() (no message) with DbtCloudJobRunException(event["message"]), which both fixes the missing timeout handling and surfaces the trigger's actual message instead of an empty exception.

The operator keeps its existing per-status branching (cancelled/error/timeout each have distinct messages, and timeout additionally attempts job cancellation) since validate_execute_complete_event() already closes the status set to the four values above.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Claude code


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants