Skip to content

Pass the Dag bundle's team to secrets backends during Dag parsing - #72139

Open
taehwoi wants to merge 1 commit into
apache:mainfrom
taehwoi:dag-parsing-team-scoped-secrets
Open

Pass the Dag bundle's team to secrets backends during Dag parsing#72139
taehwoi wants to merge 1 commit into
apache:mainfrom
taehwoi:dag-parsing-team-scoped-secrets

Conversation

@taehwoi

@taehwoi taehwoi commented Aug 27, 2026

Copy link
Copy Markdown

Secrets backends receive team_name during task execution, but during Dag file parsing it was always None, so team-scoped connections and variables did not resolve in top-level Dag code (see #65530).

What changes:

  • The Dag processor manager gives each parsing process an in-process API client whose requests carry the bundle name (airflow-dag-bundle-name header).
  • InProcessExecutionAPI reads that header into a new TIClaims.bundle_name claim. get_team_name_dep resolves the team from DagBundleModel when the claim is set, and from the task instance otherwise, so variables and connections routes need no changes.
  • Only the in-process app reads the header. The API server takes identity from the signed token and ignores it, so a worker cannot use it to reach another team's secrets.
  • Parse-time variable writes and key listing become team-scoped as well, matching task execution.

Design note, feedback welcome: the in-process app is shared by every parsing process in the manager, so the bundle identity has to travel with each request. A header read only by the in-process auth override was the smallest carrier I found. Alternatives considered: a query parameter honored by the real server (spoofable from a task token), resolving secrets directly in the manager (bypasses the execution API), a signed token minted by the Dag processor (needs the JWT key there). If another carrier fits the AIP-92 direction better, I am glad to switch.

Not included: the triggerer has the same gap (TriggerRunnerSupervisor.team_name is known but not passed); it can reuse this mechanism through its make_client seam in a follow-up.

closes: #70495


Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code (Claude Fable 5)

Generated-by: Claude Code (Claude Fable 5) following the guidelines

@boring-cyborg

boring-cyborg Bot commented Aug 27, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@taehwoi
taehwoi force-pushed the dag-parsing-team-scoped-secrets branch 3 times, most recently from 62c8669 to 94d3571 Compare August 27, 2026 09:29
Secrets backends receive team_name during task execution (resolved from
the task instance the token identifies), but at parse time it was always
None, so team-scoped connections and variables did not resolve in
top-level Dag code.

The Dag processor now gives each parsing process an in-process API
client whose requests carry the bundle name. The in-process app puts it
in the token's new bundle_name claim and get_team_name_dep resolves the
team from DagBundleModel, the same way it resolves it from a task
instance. Only the in-process app reads the header; the API server takes
identity from the signed token and ignores it.

closes: apache#70495
@taehwoi
taehwoi force-pushed the dag-parsing-team-scoped-secrets branch from 94d3571 to d04fcec Compare August 27, 2026 09:43
@taehwoi taehwoi changed the title Pass the dag bundle's team to secrets backends during dag parsing Pass the Dag bundle's team to secrets backends during Dag parsing Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-team: pass the bundle's team_name to secrets backends during DAG parsing

1 participant