Skip to content

# Add rerun_with_latest_version config hierarchy for clear/rerun behavior#63884

Open
nathadfield wants to merge 1 commit intoapache:mainfrom
king:feat/bundle-version-default-config
Open

# Add rerun_with_latest_version config hierarchy for clear/rerun behavior#63884
nathadfield wants to merge 1 commit intoapache:mainfrom
king:feat/bundle-version-default-config

Conversation

@nathadfield
Copy link
Contributor

@nathadfield nathadfield commented Mar 18, 2026

Motivation

In Airflow 2.x, bundle versioning didn't exist, so reruns always used the latest DAG code. Airflow 3.x introduced bundle versioning, which is a great improvement for reproducibility, but the default changed to running with the version as it was at parse time. For teams that deploy bug fixes and expect reruns to pick them up automatically, this means manually checking the "Run on latest version" box every time they clear a run.

This PR gives teams the ability to control which of those two behaviours is the default, both globally and per-DAG. Some teams will prefer the Airflow 2 behaviour (always latest), others will want the reproducibility that versioning provides. Rather than hardcoding one approach, this makes it configurable.

What this PR does

Adds a three-level configuration hierarchy controlling the default "Run on latest version" checkbox when clearing DAG runs or task instances:

  1. Explicit API request:caller passes run_on_latest_version: true/false (highest priority)
  2. DAG-level:rerun_with_latest_version parameter on the DAG definition
  3. Global config:[core] rerun_with_latest_version in airflow.cfg
  4. Default:False (use original bundle version)

Backend

API clear endpoints now accept run_on_latest_version: null (meaning "not specified") and resolve the default server-side using the hierarchy. This means both UI users and programmatic API callers benefit from the configuration without needing to pass the parameter explicitly.

The resolver (_resolve_run_on_latest_version) is integrated into:

  • clear_dag_run() endpoint
  • post_clear_task_instances() endpoint
  • create_backfill() endpoint

Frontend

A new useRerunWithLatestVersion hook reads the DAG-level config and global config to set the checkbox default. It handles the async nature of data loading (both config and DAG details arrive after initial render) and tracks whether the user has manually toggled the checkbox to avoid overriding their choice.

Configuration

# airflow.cfg:set organization-wide default
[core]
rerun_with_latest_version = True
# Per-DAG override
with DAG(
    dag_id="my_dag",
    rerun_with_latest_version=True,  # or False, or None to inherit
    ...
):
    ...

Relationship with previous PR

This is a simplified replacement for #61448, which was closed after reviewer feedback from @ephraimbuddy and @jedcunningham identified that the original implementation incorrectly modified the scheduler's new-run creation path.

This PR focuses solely on the clear/rerun path:

  • No scheduler changes
  • No new exception types (BundleVersionUnavailable removed)
  • No changes to _create_orm_dagrun or run creation logic
  • Existing run_on_latest_version API field name preserved (no rename)
  • Backfill default (True) preserved:no breaking API change

The naming uses rerun_with_latest_version for the new config and DAG parameter (per @ephraimbuddy's suggestion to clarify this applies to reruns, not new runs), while the existing API request body field remains run_on_latest_version.

Relationship with disable_bundle_versioning

These two settings serve different purposes:

  • disable_bundle_versioning:turns off version tracking entirely (no bundle_version recorded on runs)
  • rerun_with_latest_version:controls the default rerun behaviour while keeping version tracking active

They are independent. rerun_with_latest_version has no effect when versioning is disabled.

related: #61448 (closed:superseded by this PR)
closes: #60887


Was generative AI tooling used to co-author this PR?
  • Yes:Claude Code (Opus 4.6)

@nathadfield nathadfield force-pushed the feat/bundle-version-default-config branch 2 times, most recently from 727784b to 3a0bac3 Compare March 18, 2026 16:40
Adds a three-level configuration hierarchy controlling the default
"run on latest version" checkbox when clearing DAG runs or task instances:

1. Explicit API request value (if provided)
2. DAG-level `rerun_with_latest_version` parameter
3. Global `[core] rerun_with_latest_version` config option
4. Default: False (use original bundle version)

API clear endpoints now accept `run_on_latest_version: null` and resolve
the default server-side using the hierarchy, so both UI and programmatic
callers benefit from the configuration.

closes: apache#60887
@nathadfield
Copy link
Contributor Author

@ephraimbuddy @jedcunningham Here's my new PR for the rerun_with_latest_version config option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add configuration option for default run_on_latest_version behavior on DAG clears/reruns

1 participant