# Add rerun_with_latest_version config hierarchy for clear/rerun behavior#63884
Open
nathadfield wants to merge 1 commit intoapache:mainfrom
Open
# Add rerun_with_latest_version config hierarchy for clear/rerun behavior#63884nathadfield wants to merge 1 commit intoapache:mainfrom
rerun_with_latest_version config hierarchy for clear/rerun behavior#63884nathadfield wants to merge 1 commit intoapache:mainfrom
Conversation
727784b to
3a0bac3
Compare
Adds a three-level configuration hierarchy controlling the default "run on latest version" checkbox when clearing DAG runs or task instances: 1. Explicit API request value (if provided) 2. DAG-level `rerun_with_latest_version` parameter 3. Global `[core] rerun_with_latest_version` config option 4. Default: False (use original bundle version) API clear endpoints now accept `run_on_latest_version: null` and resolve the default server-side using the hierarchy, so both UI and programmatic callers benefit from the configuration. closes: apache#60887
3a0bac3 to
16ca2a0
Compare
Contributor
Author
|
@ephraimbuddy @jedcunningham Here's my new PR for the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
In Airflow 2.x, bundle versioning didn't exist, so reruns always used the latest DAG code. Airflow 3.x introduced bundle versioning, which is a great improvement for reproducibility, but the default changed to running with the version as it was at parse time. For teams that deploy bug fixes and expect reruns to pick them up automatically, this means manually checking the "Run on latest version" box every time they clear a run.
This PR gives teams the ability to control which of those two behaviours is the default, both globally and per-DAG. Some teams will prefer the Airflow 2 behaviour (always latest), others will want the reproducibility that versioning provides. Rather than hardcoding one approach, this makes it configurable.
What this PR does
Adds a three-level configuration hierarchy controlling the default "Run on latest version" checkbox when clearing DAG runs or task instances:
run_on_latest_version: true/false(highest priority)rerun_with_latest_versionparameter on the DAG definition[core] rerun_with_latest_versioninairflow.cfgFalse(use original bundle version)Backend
API clear endpoints now accept
run_on_latest_version: null(meaning "not specified") and resolve the default server-side using the hierarchy. This means both UI users and programmatic API callers benefit from the configuration without needing to pass the parameter explicitly.The resolver (
_resolve_run_on_latest_version) is integrated into:clear_dag_run()endpointpost_clear_task_instances()endpointcreate_backfill()endpointFrontend
A new
useRerunWithLatestVersionhook reads the DAG-level config and global config to set the checkbox default. It handles the async nature of data loading (both config and DAG details arrive after initial render) and tracks whether the user has manually toggled the checkbox to avoid overriding their choice.Configuration
Relationship with previous PR
This is a simplified replacement for #61448, which was closed after reviewer feedback from @ephraimbuddy and @jedcunningham identified that the original implementation incorrectly modified the scheduler's new-run creation path.
This PR focuses solely on the clear/rerun path:
BundleVersionUnavailableremoved)_create_orm_dagrunor run creation logicrun_on_latest_versionAPI field name preserved (no rename)True) preserved:no breaking API changeThe naming uses
rerun_with_latest_versionfor the new config and DAG parameter (per @ephraimbuddy's suggestion to clarify this applies to reruns, not new runs), while the existing API request body field remainsrun_on_latest_version.Relationship with
disable_bundle_versioningThese two settings serve different purposes:
disable_bundle_versioning:turns off version tracking entirely (nobundle_versionrecorded on runs)rerun_with_latest_version:controls the default rerun behaviour while keeping version tracking activeThey are independent.
rerun_with_latest_versionhas no effect when versioning is disabled.related: #61448 (closed:superseded by this PR)
closes: #60887
Was generative AI tooling used to co-author this PR?