Skip to content

Multi Team: Automatically create and assign team default pools#69768

Draft
SameerMesiah97 wants to merge 1 commit into
apache:mainfrom
SameerMesiah97:Multi-Team-Default-Team-Pool
Draft

Multi Team: Automatically create and assign team default pools#69768
SameerMesiah97 wants to merge 1 commit into
apache:mainfrom
SameerMesiah97:Multi-Team-Default-Team-Pool

Conversation

@SameerMesiah97

Copy link
Copy Markdown
Contributor

Description

This change introduces automatic default pool management for multi-team deployments.

With this implementation, a corresponding default pool named default_pool_<team_name> is automatically created on team creation. During DAG parsing, tasks that do not explicitly specify a pool are assigned to their team's default pool based on the DAG bundle association, while tasks with explicitly configured pools are left unchanged.

Rationale

In multi-team deployments, tasks without an explicit pool are currently assigned to the global default_pool, allowing one team's workload to consume capacity shared across all teams. Avoiding this requires users to manually create and configure team-specific pools and update every task to reference them.

By automatically provisioning a per-team default pool and applying it during DAG parsing, Airflow provides sensible defaults while preserving explicit pool assignments. This reduces operational overhead and helps isolate execution capacity between teams without requiring changes to existing DAGs.

Tests

Added unit tests verifying that:

  • Tasks using default_pool are assigned to the appropriate team default pool during DAG parsing.
  • Tasks with explicitly configured pools are not modified.
  • DAGs without an associated team continue to use default_pool.
  • Creating a team automatically provisions its corresponding default pool.
  • Team default pool names are generated correctly.

Documentation

Updated the multi-team documentation to describe automatic creation of team default pools and how they are applied during DAG parsing.

Backwards Compatibility

Existing behaviour is preserved when multi-team mode is disabled. Tasks with explicitly configured pools continue to use their configured pool without modification.

Closes: #69170

assign tasks without an explicit pool to the team's default pool during DAG
parsing. Add unit tests covering pool assignment and team creation, and
update the multi-team documentation to describe the new behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi Team: Add team default pools for tasks

1 participant