Skip to content

Speed up Dags list and dashboard queries on large DagRun tables#67721

Draft
yuseok89 wants to merge 3 commits into
apache:mainfrom
yuseok89:improve-dagrun-query-scalability
Draft

Speed up Dags list and dashboard queries on large DagRun tables#67721
yuseok89 wants to merge 3 commits into
apache:mainfrom
yuseok89:improve-dagrun-query-scalability

Conversation

@yuseok89
Copy link
Copy Markdown
Contributor

While reviewing #67242, we found that several existing UI queries are also slower than expected on large dag_run tables (verified with ~5M DagRuns). Adding new indexes would be a larger change (a migration), so this PR focuses on reusing the existing indexes. If there's a better direction, I'd appreciate your input.

Changes (per query)

  • Dashboard historical_metrics: window now filters on the indexed run_after instead of the unindexed start_date/end_date.
  • Dashboard dag_stats: latest run per Dag via correlated ORDER BY logical_date DESC LIMIT 1 instead of MAX(...) GROUP BY scanning every run.
  • Dags list get_dags recent runs: top-N per Dag via ORDER BY run_after DESC LIMIT N instead of rank() over all runs.

Screenshots

Before

Screenshot 2026-05-29 at 11 30 26 PM Screenshot 2026-05-29 at 11 33 52 PM

After

image image

cc @pierrejeambrun


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Opus 4.7

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added the area:API Airflow's REST/HTTP API label May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant