add support to configure anonymous in s3 clients#2323
add support to configure anonymous in s3 clients#2323kevinjqliu merged 5 commits intoapache:mainfrom
Conversation
There was a problem hiding this comment.
Thanks @gmweaver for adding this 🙌
Probably we want to add this to the docs as well: https://github.com/apache/iceberg-python/blob/main/mkdocs/docs/configuration.md#s3
Do you know if there is a PyArrow equialent? If so, it would be nice to add that option as well.
pyarrow has a similar config updated the PR to include. |
|
@gmweaver Thanks for being on top of this, could you run |
|
@Fokko sorry missed the |
kevinjqliu
left a comment
There was a problem hiding this comment.
LGTM! Thanks for adding this. Heres the docs for both fsspec and pyarrow
https://s3fs.readthedocs.io/en/latest/api.html
https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html
|
Thanks for the PR @gmweaver and thank you @Fokko for the review! |
Rationale for this change
Currently, it is not possible to configure
anonymousin the underlyings3fsclient (Falseby default).Resolves #2126
Are these changes tested?
Yes, unit tests added to confirm the setting is used when the property is set.
Are there any user-facing changes?
Yes, users can now set the property
s3.anonymousto True to enable. If not set, the behavior will be the same as it currently is.