[v3-1-test] Fix dag processor callback cleanup for versioned bundle files (#66484)#69771
Open
Vamsi-klu wants to merge 1 commit into
Open
[v3-1-test] Fix dag processor callback cleanup for versioned bundle files (#66484)#69771Vamsi-klu wants to merge 1 commit into
Vamsi-klu wants to merge 1 commit into
Conversation
…#66484) * Fix callback orphan cleanup for versioned bundle files closes: apache#66483 * Remove newsfragment for callback orphan cleanup fix * Fix versioned dag file presence checks * Preserve public signatures for versioned dag file checks * Preserve manager cleanup extension points
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #66484 ("Fix dag processor callback cleanup for versioned bundle files") to
v3-1-test, addressing #63374 (Dag-levelon_failure_callbacknever fires on a versioned/git bundle).Root cause: the Dag File Processor keyed queued callback files by a
DagFileInfowhosebundle_versionparticipates in equality, while the periodic scan re-creates the same file withbundle_version=None. On a versioned (git) bundle the purge step compared them under full equality, treated the queued versioned callback file as no longer present, and dropped it before it was processed — so the callback never fired. #66484 gates the purge/terminate/stats-cleanup checks on a version-independentpresence_keyinstead of full dataclass equality.This is a clean cherry-pick of
692728371f; the only conflict was a one-line context clash (v3-1usesStats.gauge,mainusesstats.gauge), resolved to keep thev3-1spelling. The prerequisite #61232 is already present onv3-1-test; #63795 and #69185 are separate 3.2.0/3.4.0 features and are not required for this fix. No new newsfragment — the change already carries #66484's changelog entry onmain.The five
test_manager.pyguard tests added by #66484 pass onv3-1-test, and the key one fails without thepresence_keychange (verified locally).Opened as a draft for release-manager consideration — the fix already ships in Airflow 3.3.0 (#66484's milestone); this only matters if a further 3.1.x patch is planned.
related: #63374
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines