Fix TimeSensor dag_version churn by removing start_from_trigger support#69746
Open
Vamsi-klu wants to merge 1 commit into
Open
Fix TimeSensor dag_version churn by removing start_from_trigger support#69746Vamsi-klu wants to merge 1 commit into
Vamsi-klu wants to merge 1 commit into
Conversation
TimeSensor computed target_datetime using datetime.now() at DAG-parse time and baked it into start_trigger_args.trigger_kwargs.moment. Since that field is serialized into serialized_dag, the dag hash changed on every parse, creating a new DagVersion each scheduler loop. This removes the start_from_trigger feature from TimeSensor (consistent with TimeDeltaSensor which never had it because its target depends on runtime context) and moves target_datetime computation from __init__ to execution time (poke/execute). Wall-clock behavior is preserved, but no volatile value enters serialization. Fixes: apache#69543
Contributor
Author
|
Reviewers: @ashb (core serialization / sensors) + @potiuk @kaxil for standard provider. This fixes dag_version churn by moving target_datetime computation to execution time. Fixes #69543 Drafted-by: Muse Spark 1.1; reviewed by @Vamsi-klu before posting |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TimeSensor with start_from_trigger=True creates new SerializedDag version every parse because init computes target_datetime via datetime.now(dag.timezone) and bakes into start_trigger_args.trigger_kwargs.moment, part of serialized hash. Causes DB bloat.
What I did
Impact
Fixes dag_version churn, preserves behavior, aligns with TimeDeltaSensor.
Testing
Existing tests 10 passed, temporal triggers 19 passed. Manual serialization stability verified.
Fixes: #69543
Was generative AI tooling used to co-author this PR?
Generated-by: Muse Spark 1.1