Skip to content

Support wasb:// and wasbs://#1663

Open
christophediprima wants to merge 5 commits intoapache:mainfrom
christophediprima:main
Open

Support wasb:// and wasbs://#1663
christophediprima wants to merge 5 commits intoapache:mainfrom
christophediprima:main

Conversation

@christophediprima
Copy link

@christophediprima christophediprima commented Feb 14, 2025 •
edited by kevinjqliu
Loading

Closes #2271 and #1606
This will work as soon as this is merged: fsspec/adlfs#493

@christophediprima christophediprima mentioned this pull request Feb 14, 2025
Closed
3 tasks
@Fokko Fokko changed the title support wasb and wasbs Support wasb:// and wasbs:// Feb 14, 2025
@Fokko
Copy link
Contributor

Fokko commented Feb 14, 2025

There is also an open issue on the adlfs side: fsspec/adlfs#403

Regarding fsspec/adlfs#493, is the protocol identical?

@christophediprima
Copy link
Author

christophediprima commented Feb 17, 2025

I am not sure but have been testing it with Azurite localy and it works as expected.
They have the corrrect endpoint for reaching AZBlob: https://github.com/fsspec/adlfs/blob/main/adlfs/spec.py#L488 and the java version supports it already.

I am going to try use it on the cloud.

@Fokko
Copy link
Contributor

Fokko commented Feb 18, 2025

@christophediprima Thanks for testing that, appreciate it.

We also test against azurite. Maybe we can add some simple tests as well to check the connection and make sure that we don't break it in the future.

@christophediprima
Copy link
Author

christophediprima commented Mar 10, 2025 •
edited
Loading

We have been testing it on Azure Blob Storage with my team and we had no issues. What kind of tests can you think about?

@kevinjqliu
Copy link
Contributor

kevinjqliu commented Mar 11, 2025

Looks like we have a few adls integration tests against the azurite docker

iceberg-python/tests/io/test_fsspec.py

Line 298 in b86d7d5

input_file = adls_fsspec_fileio.new_input(f"abfss://tests/{filename}")

perhaps we can extend these to include wasb and wasbs

@DerGut DerGut mentioned this pull request Jun 2, 2025
Merged
This was referenced Jun 17, 2025
Merged
Closed
kevinjqliu added a commit that referenced this pull request Jun 20, 2025
@NikitaMatskevich @kevinjqliu
Support ADLS with Pyarrow file IO (#2111)
84c91f0 
<!--
Thanks for opening a pull request!
-->

<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->

# Rationale for this change

Starting from version 20, PyArrow supports ADLS filesystem. This PR adds
Pyarrow Azure support to Pyiceberg.

PyArrow is the [default
IO](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/__init__.py#L366-L369)
for Pyiceberg catalogs. In Azure environment it handles wider spectrum
of auth strategies then Fsspec, including, for instance, [Managed
Identities](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview).
Also, prior to this PR
#1663 (that is not merged
yet) there was no support for wasb(s) with Fsspec.

See the corresponding issue for more details:
#2112

# Are these changes tested?

Tests are added under tests/io/test_pyarrow.py.

# Are there any user-facing changes?

There are no API breaking changes. Direct impact of the PR: Pyarrow
FileIO in Pyiceberg supports Azure cloud environment. Examples of impact
for final users:
- Pyiceberg is usable in services with Managed Identities auth strategy.
 - Pyiceberg is usable with wasb(s) schemes in Azure.

<!-- In the case of user-facing changes, please add the changelog label.
-->

---------

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>
amitgilad3 pushed a commit to amitgilad3/iceberg-python that referenced this pull request Jul 7, 2025
@NikitaMatskevich @kevinjqliu @amitgilad3
Support ADLS with Pyarrow file IO (apache#2111)
bb45772 
<!--
Thanks for opening a pull request!
-->

<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->

# Rationale for this change

Starting from version 20, PyArrow supports ADLS filesystem. This PR adds
Pyarrow Azure support to Pyiceberg.

PyArrow is the [default
IO](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/__init__.py#L366-L369)
for Pyiceberg catalogs. In Azure environment it handles wider spectrum
of auth strategies then Fsspec, including, for instance, [Managed
Identities](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview).
Also, prior to this PR
apache#1663 (that is not merged
yet) there was no support for wasb(s) with Fsspec.

See the corresponding issue for more details:
apache#2112

# Are these changes tested?

Tests are added under tests/io/test_pyarrow.py.

# Are there any user-facing changes?

There are no API breaking changes. Direct impact of the PR: Pyarrow
FileIO in Pyiceberg supports Azure cloud environment. Examples of impact
for final users:
- Pyiceberg is usable in services with Managed Identities auth strategy.
 - Pyiceberg is usable with wasb(s) schemes in Azure.

<!-- In the case of user-facing changes, please add the changelog label.
-->

---------

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>
@kevinjqliu kevinjqliu mentioned this pull request Aug 11, 2025
Open
@kevinjqliu
Copy link
Contributor

kevinjqliu commented Aug 11, 2025

depends on fsspec/adlfs#493

gabeiglio pushed a commit to Netflix/iceberg-python that referenced this pull request Aug 13, 2025
@NikitaMatskevich @kevinjqliu @gabeiglio
Support ADLS with Pyarrow file IO (apache#2111)
0ce2992 
<!--
Thanks for opening a pull request!
-->

<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->

# Rationale for this change

Starting from version 20, PyArrow supports ADLS filesystem. This PR adds
Pyarrow Azure support to Pyiceberg.

PyArrow is the [default
IO](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/__init__.py#L366-L369)
for Pyiceberg catalogs. In Azure environment it handles wider spectrum
of auth strategies then Fsspec, including, for instance, [Managed
Identities](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview).
Also, prior to this PR
apache#1663 (that is not merged
yet) there was no support for wasb(s) with Fsspec.

See the corresponding issue for more details:
apache#2112

# Are these changes tested?

Tests are added under tests/io/test_pyarrow.py.

# Are there any user-facing changes?

There are no API breaking changes. Direct impact of the PR: Pyarrow
FileIO in Pyiceberg supports Azure cloud environment. Examples of impact
for final users:
- Pyiceberg is usable in services with Managed Identities auth strategy.
 - Pyiceberg is usable with wasb(s) schemes in Azure.

<!-- In the case of user-facing changes, please add the changelog label.
-->

---------

Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>
@kevinjqliu kevinjqliu mentioned this pull request Aug 28, 2025
Closed
@kevinjqliu
Copy link
Contributor

kevinjqliu commented Aug 28, 2025 •
edited
Loading

i have a local change that parameterizes all the adls integration tests with abfs, abfss, wasb, and wasbs

its currently failing with, note the wrong path wasb//

self = <File-like object AzureBlobFileSystem, wasb//tests/381fddd0-11b2-49c8-8de0-1d77b5b8eb8e>, final = True, kwargs = {}, data = b'foobar', length = 6, block_id = '00000000000000000000000000000000'
commit_kw = {}, chunk = b'foobar', bc = <azure.storage.blob.aio._blob_client_async.BlobClient object at 0x11a40e600>

@kevinjqliu
Copy link
Contributor

kevinjqliu commented Aug 28, 2025

pushed the parameterized test here for reference. i changed all reference of the protocol for adls to use the adls_scheme test fixture

@kevinjqliu
Copy link
Contributor

kevinjqliu commented Aug 28, 2025

added the monkey patch solution here for reference. we can also wait for fsspec/adlfs#493 to land

@kyleknap kyleknap mentioned this pull request Sep 2, 2025
Merged
@kevinjqliu
Copy link
Contributor

kevinjqliu commented Sep 3, 2025

fsspec/adlfs#512 added the ability to override protocol

AzureBlobFileSystem.protocol = AzureBlobFileSystem.protocol + ("wasb", "wasbs")

but for older versions of adlfs, we would still need to monkey patch

@Fokko
Copy link
Contributor

Fokko commented Jan 25, 2026

Since fsspec/adlfs#512 has been merged, should we pull in the latest main?

@kevinjqliu
Copy link
Contributor

kevinjqliu commented Jan 29, 2026

ah looks like it didnt make the current latest release 2025.8.0
fsspec/adlfs@9b9437f

@kevinjqliu
Copy link
Contributor

kevinjqliu commented Jan 29, 2026

@Fokko how do u feel about the monkeypatch solution here?

@rambleraptor
Copy link
Contributor

rambleraptor commented Jan 29, 2026

Commenting here vs. the dev list:

I'm personally not a huge fan of the monkey patch solution, since it relies on us remembering to correct it after the next adlfs release.

@Fokko
Copy link
Contributor

Fokko commented Jan 29, 2026

I'd rather wait for the official release. I'm not a fan of monkey patching either as it is tied to a specific version 😞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Cannot create Iceberg Table in Azure Blob Storage

4 participants

@christophediprima @Fokko @kevinjqliu @rambleraptor