Skip to content

Count post-deferral scheduled tasks toward include_deferred pool slots#69770

Open
cipheraxat wants to merge 1 commit into
apache:mainfrom
cipheraxat:fix/69187-pool-count-post-deferral-scheduled
Open

Count post-deferral scheduled tasks toward include_deferred pool slots#69770
cipheraxat wants to merge 1 commit into
apache:mainfrom
cipheraxat:fix/69187-pool-count-post-deferral-scheduled

Conversation

@cipheraxat

Copy link
Copy Markdown

Summary

When a pool has include_deferred=True, deferred tasks correctly occupy slots, but after the trigger fires the task instance goes deferred → scheduled and previously stopped counting until it was queued/running again. That undercounts pool usage and weakens pools as a semaphore around external resources (the motivation for including deferred tasks in the first place).

This change counts post-deferral scheduled task instances — those with next_method set — toward pool occupancy when include_deferred is enabled:

  • Pool.occupied_slots() / Pool.slots_stats() open calculation
  • Pool.task_instance_occupies_slot() used by PoolSlotsAvailableDep so a TI that already holds a slot does not block itself

First-lifetime scheduled tasks (no next_method) are not counted, which would otherwise deadlock scheduling when open = total - scheduled.

Test plan

  • test_open_slots_including_deferred_counts_post_deferral_scheduled
  • test_open_slots_excluding_deferred_ignores_post_deferral_scheduled
  • test_post_deferral_scheduled_pooled_task_pass (pool slots available dep)
  • Existing pool / deferred occupancy tests still pass
  • prek on changed files

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Cursor following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.

When include_deferred is enabled, tasks that return to scheduled after
a trigger fires (next_method set) kept dropping out of pool occupancy,
undercounting slots until they were queued again. Count those TIs as
occupied without treating first-lifetime scheduled tasks the same way,
which would deadlock scheduling.
@cipheraxat cipheraxat requested review from XD-DENG and ashb as code owners July 11, 2026 23:28
@boring-cyborg

boring-cyborg Bot commented Jul 11, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Airflow pools do not count tasks in scheduled state

1 participant