Skip to content

Adds a new public API endpoint for backfill dag run entries - ` GET /backfills/{backfill_id}/dag_runs#67381

Open
shivaam wants to merge 1 commit into
apache:mainfrom
shivaam:pr/backfill-dag-runs-api
Open

Adds a new public API endpoint for backfill dag run entries - ` GET /backfills/{backfill_id}/dag_runs#67381
shivaam wants to merge 1 commit into
apache:mainfrom
shivaam:pr/backfill-dag-runs-api

Conversation

@shivaam
Copy link
Copy Markdown
Contributor

@shivaam shivaam commented May 23, 2026

Adds a new public API endpoint GET /backfills/{backfill_id}/dag_runs that returns
the BackfillDagRun entries for a given backfill with joined DagRun state. Users
can see what happened in a backfill: which dates ran, their states, and which slots
were skipped (with reason).

Previous PR: #64089 explored enriching BackfillResponse directly. Based on
feedback from @uranusjr, this PR takes the cleaner approach of a dedicated
sub-resource endpoint.

Changes

  • BackfillDagRunResponse / BackfillDagRunCollectionResponse Pydantic models
    with AliasPath for joined DagRun fields (matching existing BackfillResponse pattern)
  • list_backfill_dag_runs() route using joinedload(BackfillDagRun.dag_run) — LEFT OUTER JOIN
    includes skipped slots where dag_run_id is NULL
  • Pagination via limit/offset, default ordering by sort_ordinal
  • 404 when backfill doesn't exist
  • 8 unit tests: happy path, skipped slots, 404, pagination, empty backfill, ordering contract

Response shape

{
  "backfill_dag_runs": [
    {
      "id": 1,
      "backfill_id": 1,
      "dag_run_id": 42,
      "logical_date": "2024-01-01T00:00:00Z",
      "partition_key": null,
      "sort_ordinal": 1,
      "exception_reason": null,
      "dag_run_state": "success",
      "dag_run_run_id": "backfill__2024-01-01T00:00:00+00:00"
    },
    {
      "id": 2,
      "backfill_id": 1,
      "dag_run_id": null,
      "logical_date": "2024-01-02T00:00:00Z",
      "partition_key": null,
      "sort_ordinal": 2,
      "exception_reason": "already exists",
      "dag_run_state": null,
      "dag_run_run_id": null
    }
  ],
  "total_entries": 10
}

Follow-up PRs

  • UI: Progress bar on backfills table/banner + backfill detail page with Dag runs table
  • CLI: airflow backfill list and airflow backfill list-runs commands

closes: #46250


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (claude-opus-4-6)

Generated-by: Claude Code (claude-opus-4-6) following the guidelines

Adds a new public API endpoint that returns the BackfillDagRun entries
for a given backfill with joined DagRun state. Users can see what
happened in a backfill: which dates ran, their states (queued, running,
success, failed), and which slots were skipped (with reason).

- BackfillDagRunResponse / BackfillDagRunCollectionResponse models
- LEFT OUTER JOIN via joinedload includes skipped slots (null dag_run_id)
- Pagination via limit/offset, default ordering by sort_ordinal
- 404 when backfill doesn't exist
- 8 unit tests covering happy path, skipped slots, 404, pagination,
  empty backfill, and ordering contract

closes: apache#46250
@shivaam
Copy link
Copy Markdown
Contributor Author

shivaam commented May 23, 2026

@vatsrahul1001 tagging you since you mentioned you were planning to work on it.

select_stmt, total_entries = paginated_select(
statement=select(BackfillDagRun)
.where(BackfillDagRun.backfill_id == backfill_id)
.options(joinedload(BackfillDagRun.dag_run)),
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

joinedload issues a LEFT OUTER JOIN, preserving rows where dag_run_id is NULL (skipped slots).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add API endpoint that lets you see what happened in the backfill

1 participant