Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)"

[olegkachur-e]

This reverts commit 917abea.

• Remove hacks regarding the beam provider suspension, as they are not relevant anymore.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

---

Important

🛠️ Maintainer triage note for @olegkachur-e · by @potiuk · 2026-07-02 17:46 UTC

Some review feedback from @eladkal is waiting on you:

• 1 unresolved review thread(s) from @eladkal need a reply or a fix.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

[potiuk]

@olegkachur-e A few things need addressing before review — see our Pull Request quality criteria.

• ❌ Unit tests. See docs.
• ❌ Provider tests. See docs.
• ❌ Other failing CI checks. See docs.
• ❌ Image build. See docs.

No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[shahar1]

hi all, Beam committer here.

2. My PR in Apache beam has been merged, so we should be able to unsuspend it in version > 2.75.
   Let's keep watching until it is shipped, and then we'll be able to unsuspend it.

@shahar1 thanks for doing that.

Regarding betterproto dependency:

• the functionality where envoy-data-plane is used is an opt-in functionality of Beam that doesn't have significant usage atm
• While Betterproto doesn't have any recent releases: https://pypi.org/project/betterproto/#history, is there any concern as to this dependency having a vulnerability? It seems like we flagged it primarily due to beta in the naming, and then noticed that the codebase is old. Even if a project has recent releases, it can have codepaths that haven't been touched for years. Do we have any reason to be concerned beside the name ?
• As mentioned, Beam will drop the dependency on the envoy-data-plane in the next release, but using older version of Beam in my opinion causes more inconvenience in this case to airflow users than pulling an old dependency .

Thanks for your comment!

I reevaluated this:

• The original suspension blocker (Beam's grpcio<1.66 pin) is resolved as of Beam 2.72.
• There are no known vulnerabilities in Beam 2.72–2.75, nor in betterproto==2.0.0b6 or envoy-data-plane==1.0.3.
• envoy-data-plane==1.0.3 is part of an opt-in Beam feature with little usage.
• The dependency is already removed upstream ([Python] Drop envoy-data-plane/betterproto dependency from the Python SDK beam#39213) and ships in Beam 2.76, expected in ~1–2 months.

Given the risk is known, bounded, and time-limited, I think unsuspending now is reasonable rather than keeping users on a suspended provider for another release cycle. We already accept unmaintained transitive dependencies elsewhere in the tree on a case-by-case basis; this one at least has a concrete removal date. Of course, if there's a vunlerability found and 2.76 is yet to be released - we could resume the suspension.

@potiuk @eladkal WDYT?

@olegkachur-e Could you please resolve conflicts?

[eladkal]

Given the risk is known, bounded, and time-limited, I think unsuspending now is reasonable rather than keeping users on a suspended provider for another release cycle

I don't understand. what is the request?
Is the request to ship beam provider with a beta dependecy?

Before I write my thoughts there should be an explanation from beam mantainers why can't they just do another release? Shifting the risk to Airflow, however minimal, is a big ask. This goes way beyond just Beam. The desicion that will be taken here will be used as precedence for all the 100+ providers we have.

[eladkal]

The dependency is already removed upstream ([Python] Drop envoy-data-plane/betterproto dependency from the Python SDK beam#39213) and ships in Beam 2.76, expected in ~1–2 months.

Again, Beam needs to explain why they can't do out of release schedule to mitigate. Airflow is doing ad hoc releases when needed. I assume beam can do that too.

To clarify, this is not a desicion we can take here. This must go to mailing list for visability. For exmaple, who decide what is resonable time? Is 1 month resonable? 3? What happens if beam break the commitment... We can't really rollback. Sometimes the things we expect don't work as much. We know this very well. Releases are delayed. Releases are yanked.

[tvalentyn]

Again, Beam needs to explain why they can't do out of release schedule to mitigate. Airflow is doing ad hoc releases when needed.

An adhoc release would require you to set the upper bound to apache_beam>= 2.75.1 . Taking the best interests of Airflow users in mind, this might not be a great solution since if Beam 2.75.0 happens to be buggy, your users won't have an option to downgrade for a while.

I agree that depending on 'beta' doesn't have great optics but beyond optics per se, i don't see an actual risk (i.e. vulnerability or introducing bugs). Arguably that was Beam's fault, that ship has sailed and we learned from it, it was not Airflow's fault and you are not adding this dependency intentionally.

would it be ok for Airflow to add a constraint envoy_data_plane==1.0.3 instead of betterproto==2.0.0b6 if you are concerned with mentioning a beta dependency in your codebase?

Beam can do out-of-calendar releases but this is a nontrivial amount of work and the bar for that is high (for example an urgent dataloss issue or smth like log4j vulnerability that everyone was scrambilng to mitigate asap). During the entire project lifetime that happened perhaps 2 or 3 times .

[eladkal]

your users won't have an option to downgrade for a while.

I'd argue they are beam users as well :)
With providers we relay on upstream fixes. If beam is not willing to cut a release as was suggested here I'd suggest to work with @shahar1 on a proposal that doesn't put the risks on Airflow. If it has merit i will not object. by merit I mean actually solve our concerns, trying to create some cosmetic appears as if issue was addressed while not is not a fix. Again, I am only one voice here. There are many PMCs in the project who might think differently.

[tvalentyn]

to clarify: what is the risk for Airflow you'd like address ? If we look at the lock file, there are other dependencies that are no longer released, and whose last release was even before the pinned betterproto release. Is the word 'beta' in the name the only risk here?

[eladkal]

Is the word 'beta' in the name the only risk here?

It's not a word. It is a declaration.
https://pypi.org/project/betterproto/2.0.0b6/

The mantainers of this lib marked it as not suitable for production. No new releases since 2024.
Technically, since its not production grade the mantainers can just remove it from PyPi without further notice. It is a huge risk to take dependencies on such releases. Beam can do it from your side. Airflow isn't likely to do it. We can bring beam to dependencies but that is where the line is drawn. We will not add explicit dependency on a pre-relesse/beta package.

[tvalentyn]

We will not add explicit dependency on a pre-relesse/beta package.

We actually don't depend on this dep directly in Beam either, i am not sure how it came into the picture in this Airflow change, presumably to help with the dependency resolution, and a suggestion is to add constraints on envoy-data-plane instead if you must appease the resolver for some reason.

production grade the mantainers can just remove it from PyPi without further notice

we are going into hypotheticals here, while there are real adverse effects for users of Airflow and Beam due to suspended provider that I'd rather focus on. Left a suggestion.

[MaksYermak]

Hello @gopidesupavan @potiuk ,
This PR is in charge of me not in @olegkachur-e. I was on vacation last two weeks and could not respond on comments and fixed merge conflicts in the current PR. Yesterday I fixed merge conflicts and as I see your guys in the evening added additional changes to the code. Could you please clarify for me should I continue to work on this PR or it is not on my duty anymore?

[potiuk]

Hello @gopidesupavan @potiuk , This PR is in charge of me not in @olegkachur-e. I was on vacation last two weeks and could not respond on comments and fixed merge conflicts in the current PR. Yesterday I fixed merge conflicts and as I see your guys in the evening added additional changes to the code. Could you please clarify for me should I continue to work on this PR or it is not on my duty anymore?

Hello @MaksYermak - sorry for not responding before - I've been sick/out of action for the last week or so and slowly getting back to regular involvement.

I tried to see - similarly as @gopidesupavan to fix the dependency issue, but as @eladkal noted - fixing betterproto to 6 years old pre-release version is a bad idea. Really the issue is on beam side not ours, so it's up on them to fix it.

Luckily - they already did - it just did not make it into 2.75.0 release in July and it is planned to be released at the beginning of August. And this is the right way of solving the problem - and opens the way to un-suspend beam provider - making any workarounds on our side is just not something we should accept - especially that there is a fix in sight.

You can read more details in here, where I wrote message to the dev team of Apache Beam:

https://lists.apache.org/thread/lnm2gszq3lvqxgz70rms7zqmr757w5ol

So - at this stage, what we need to do is to wait until 2.76.0 is released by Apache beam. Once this is done (+3 days cooldown on our side) - you should just rebase/resolve issues, bump min version for it pyproject.toml of the beam provider and all issues should be solved.

I think a good idea for you might be to respond to that message if you feel that Beam team should speed it up a bit maybe - explaining how much needed it is by the composer team - and Google customers. Yes. I know it's frustrating, but it's really not something we should fix or workaround here.

[potiuk]

I think I fixed the issue - rebased and provided a fixup.