Bug: UploadPart fails with Aliyun OSS due to unsupported aws-chunked encoding

[Dludora]

Apache Iceberg version

0.10.0 (latest release)

Please describe the bug 🐞

Description

When using PyIceberg with Aliyun OSS (Object Storage Service) via the S3-compatible endpoint, creating a table or appending data fails during the metadata/data upload phase. The underlying PyArrow S3 filesystem attempts to use aws-chunked encoding with x-amz-content-sha256, which is not supported by Aliyun OSS's S3 compatibility layer for certain operations.

This leads to the following error:

OSError: When uploading part for key '...' in bucket '...': 
AWS Error UNKNOWN (HTTP status 400) during UploadPart operation: 
Unable to parse ExceptionName: InvalidArgument 
Message: aws-chunked encoding is not supported with the specified x-amz-content-sha256 value.

According to Aliyun OSS Documentation, the S3-compatible API has specific constraints regarding signature versions and chunked encoding.

Steps to Reproduce

Initialize a Catalog with Aliyun OSS S3-compatible endpoint.

1. Set s3.force-virtual-addressing: "true" as required by OSS.

2. Attempt to create a table or write data.

3. Reproducible Code Snippet:

import os
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "oss_catalog",
    **{
        "type": "sql",
        "uri": "sqlite:///oss_catalog.db",
        "warehouse": "s3://your-bucket/path/",
        "s3.endpoint": "https://s3.oss-cn-wulanchabu.aliyuncs.com",
        "s3.region": "cn-wulanchabu",
        "s3.access-key-id": os.environ["OSS_AK"],
        "s3.secret-access-key": os.environ["OSS_SK"],
        "s3.force-virtual-addressing": "true",
    }
)

# This triggers the UploadPart error during metadata write
catalog.create_table("default.test_table", schema=pa_schema)

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[kevinjqliu]

Thanks for reporting the issue!

Which FileIO implementation are you using? Fsspec or Pyarrow?
Are you using the S3 Signer? The only place in the codebase i see mentions of x-amz-content-sha256

Heres the translation from the aliyun docs, via claude:

Solutions
If using aws-chunked encoding, you need to set x-amz-content-sha256 to STREAMING-AWS4-HMAC-SHA256-PAYLOAD.
If you don't need to use aws-chunked encoding, remove the Content-Encoding request header or set it to another value.
It is recommended to use OSS's standard Multipart Upload method rather than relying on the S3 SDK's aws-chunked encoding feature.
Ensure that the S3 SDK version you're using supports OSS's aws-chunked encoding implementation, and update to the latest version if necessary.

Are you able to try one of the above? Would be helpful to see which one works. I do not have an Aliyun env 😄