feat(fs): add lightweight object-store and S3 filesystem support by mrdrivingduck · Pull Request #433 · alibaba/paimon-cpp

mrdrivingduck · 2026-07-19T17:20:58Z

Background

I started this work because I wanted duckdb-paimon to query Paimon data lakes stored on S3. That required adding S3 filesystem support to paimon-cpp first.

My first implementation used Aws::S3::S3Client. It worked, but the dependency cost was much larger than I expected. Reading a Paimon table only needs a fairly small part of S3: HEAD, paginated LIST, and ranged GET. Pulling in the complete AWS S3 client and its CRT dependency stack felt disproportionate for that job, and it also made the build integration considerably more complicated.

I then looked at how DuckDB's httpfs and Iceberg extensions divide this work. DuckDB keeps the HTTP data path generic and limits the storage-specific code to things such as credentials, request signing, URL construction, and error handling. That seemed like a better direction for paimon-cpp as well.

Based on that, I rewrote the original S3 implementation around a reusable object-store filesystem layer and a small HTTP client abstraction.

Design

The implementation is still being cleaned up, and some names and file boundaries may change before it is ready to merge. But this is the architectural split I expect to keep. I am sharing it now mainly to discuss whether this direction makes sense.

The implementation is organized as follows:

FileSystem
└── ObjectStoreFileSystem
    ├── S3FileSystem
    └── [OSSFileSystem] (WIP)

Shared components
├── ObjectStoreClient
├── HttpClient
├── ObjectStoreInputStream
└── Read-ahead

ObjectStoreFileSystem, ObjectStoreClient, and ObjectStoreInputStream implement the filesystem semantics and reading behavior shared by object stores. S3FileSystem and S3ObjectStoreClient only handle S3-specific URLs, API requests, credentials, signing, and errors.

Object data is transferred by the libcurl-based HttpClient. The AWS C libraries are used only for the credential chain and SigV4 signing, so the implementation does not depend on the complete AWS S3 SDK. All of these interfaces remain internal to paimon-cpp, and the existing public FileSystem interface is unchanged.

OSS experiment

To validate the abstraction against a second provider, I also built a WIP OSS filesystem using Alibaba Cloud OSS C++ SDK v2. It reuses the common filesystem semantics, input stream, range reads, read-ahead, pagination, and error handling; the OSS-specific code mainly adapts credentials, endpoints, requests, and responses.

The OSS implementation is not part of this PR, but it could later replace or coexist with the current Jindo-based path. Jindo is a closed-source binary dependency and is no longer recommended in a number of environments, which makes it less suitable as the project's long-term OSS foundation than the open-source OSS SDK.

Scope

The S3 filesystem in this PR is read-only. It implements the operations needed to read Paimon metadata, manifests, and data files. Write and mutation operations return an explicit unsupported error.

This PR does not add a general-purpose http:// filesystem, and it does not attempt to cover HDFS. HDFS has different filesystem semantics and should remain a separate implementation if it is added later.

Testing

Unit tests cover the shared object-store behavior, reads and read-ahead, error handling, and S3 configuration and signing. I also tested metadata, manifest, and ORC reads against a real S3 Paimon warehouse and verified the integration through duckdb-paimon.

Copilot

Pull request overview

This PR adds a new internal object-store filesystem layer and a read-only S3 filesystem implementation, enabling paimon-cpp to read Paimon warehouses stored on S3 with a lightweight HTTP data path (libcurl) and minimal AWS C libraries for credential discovery and SigV4 signing.

Changes:

Introduce object_store shared components: ObjectStoreFileSystem (read-only semantics + read-ahead) and a libcurl-based HttpClient.
Add S3FileSystem + factory and an S3ObjectStoreClient that implements HEAD/LIST/Range-GET and request signing via AWS C Auth.
Extend build/CI to support an opt-in PAIMON_ENABLE_S3 option and build the minimal AWS auth dependency stack.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
third_party/versions.txt	Adds version pins/checksums for minimal AWS C auth dependency set (and s2n).
src/paimon/fs/s3/s3_file_system.h	Declares S3 filesystem options and construction helpers.
src/paimon/fs/s3/s3_file_system.cpp	Implements S3 object-store client (signing + HEAD/LIST/GET range) and S3 filesystem wrapper.
src/paimon/fs/s3/s3_file_system_test.cpp	Adds unit tests for S3 URL construction, signing, option validation, range reads, and list parsing.
src/paimon/fs/s3/s3_file_system_factory.h	Declares S3 filesystem factory for scheme registration.
src/paimon/fs/s3/s3_file_system_factory.cpp	Implements option validation and registers the S3 filesystem factory.
src/paimon/fs/s3/CMakeLists.txt	Builds the S3 filesystem library/tests when `PAIMON_ENABLE_S3` is enabled.
src/paimon/fs/object_store/object_store_file_system.h	Defines object-store client interface, read-ahead limiter, and filesystem wrapper.
src/paimon/fs/object_store/object_store_file_system.cpp	Implements read-only object-store filesystem semantics and read-ahead input stream.
src/paimon/fs/object_store/object_store_file_system_test.cpp	Adds unit tests covering object-store semantics, pagination, and read-ahead behavior.
src/paimon/fs/object_store/http_client.h	Defines internal HTTP client interface and Curl implementation.
src/paimon/fs/object_store/http_client.cpp	Implements Curl-based HTTP execution with basic retries and header parsing.
src/paimon/fs/object_store/CMakeLists.txt	Builds the shared object-store components and tests behind `PAIMON_ENABLE_S3`.
CMakeLists.txt	Adds `PAIMON_ENABLE_S3` option and wires S3/object_store subdirectories + CURL discovery.
cmake_modules/ThirdpartyToolchain.cmake	Allows `_REVISION=` entries and triggers AWS auth build when S3 is enabled.
cmake_modules/BuildAwsAuth.cmake	Adds ExternalProject build for minimal AWS C Auth dependency chain (and s2n on Linux).
cmake_modules/arrow.diff	Adjusts Arrow’s thrift build args (adds `-DWITH_OPENSSL=OFF`).
ci/scripts/build_paimon.sh	Enables `PAIMON_ENABLE_S3=ON` in CI build script.
.github/workflows/gcc8_test.yaml	Installs libcurl/OpenSSL dev packages needed for S3/AWS auth build.
.github/workflows/build_and_test.yaml	Installs libcurl/OpenSSL dev packages before building.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

lucasfang

Each path under src/paimon/fs should be an independent plugin. I suggest moving object_store to the framework side under src/paimon/common/fs/, so that future OSS FileSystem integrations can also reuse it.

mrdrivingduck · 2026-07-22T09:54:34Z

Each path under src/paimon/fs should be an independent plugin. I suggest moving object_store to the framework side under src/paimon/common/fs/, so that future OSS FileSystem integrations can also reuse it.

Agreed, done.

lucasfang · 2026-07-23T03:26:51Z

+1

mrdrivingduck · 2026-07-23T05:51:42Z

Please use int32_t instead of int

Might be good to have that in AGENTS.md 🤔

lucasfang · 2026-07-23T06:06:45Z

Please use int32_t instead of int

Might be good to have that in AGENTS.md 🤔

There is docs/code-style.md in AGENTS.md, seems that Agent miss that？

mrdrivingduck · 2026-07-23T06:11:40Z

There is docs/code-style.md in AGENTS.md, seems that Agent miss that？

Yes, I see it now. Never mind.

zjw1111

Thanks for adding the lightweight object-store and S3 implementation. I found several configuration, endpoint, pagination, and build-consistency issues that should be addressed. Details are inline.

zjw1111 · 2026-07-24T09:05:40Z

+                        URL ${S2N_URL}
+                        URL_HASH SHA256=${PAIMON_AWS_S2N_BUILD_SHA256_CHECKSUM}
+                        DOWNLOAD_EXTRACT_TIMESTAMP TRUE
+                        CMAKE_ARGS ${EP_COMMON_TOOLCHAIN}


The AWS external projects start from EP_COMMON_TOOLCHAIN instead of the project's shared EP_COMMON_CMAKE_ARGS, so they bypass the normalized C/C++ flags, fixed install layout, and CMake policy compatibility used by the rest of the third-party build. This can make these dependencies behave differently in sanitizer, customized-flag, and packaged builds. Could you start with EP_COMMON_CMAKE_ARGS here and then append the AWS-specific overrides? The same applies to paimon_add_aws_c_project below.

zjw1111 · 2026-07-24T09:05:41Z

+inline constexpr char kS3AccessKeyIdOption[] = "s3.access-key-id";
+inline constexpr char kS3SecretAccessKeyOption[] = "s3.secret-access-key";


Could we keep these option names compatible with the existing Paimon S3 configuration? Java and Python accept/document s3.access-key and s3.secret-key, while this implementation only looks for s3.access-key-id and s3.secret-access-key. Reusing a catalog configuration would silently ignore the explicit credentials and fall back to the ambient AWS provider chain, which can either fail or use an unintended principal. Please accept the established keys, optionally keeping these names as aliases, and add coverage for them.

Done. It now uses the existing Paimon option names.

zjw1111 · 2026-07-24T09:05:41Z

+        endpoint_ = ParseEndpoint(use_default_endpoint_
+                                      ? fmt::format("https://s3.{}.amazonaws.com", region_)
+                                      : endpoint->second);


The default endpoint is hard-coded to .amazonaws.com for every region. That suffix is invalid for the aws-cn partition—for example, cn-north-1 requires s3.cn-north-1.amazonaws.com.cn—so region-only configuration cannot access China buckets unless users also override the endpoint. Could you resolve the DNS suffix from the AWS partition (covering at least aws, aws-cn, and aws-us-gov), or explicitly require an endpoint for unsupported partitions?

zjw1111 · 2026-07-24T09:05:41Z

+        auto token = TagValue(body, "NextContinuationToken");
+        if (token) {
+            PAIMON_ASSIGN_OR_RAISE(result.continuation_token,
+                                   PercentDecode(*token, "NextContinuationToken"));


NextContinuationToken is opaque and is not one of the fields affected by encoding-type=url (only Delimiter, Prefix, Key, and StartAfter are). Decoding it changes tokens containing literal percent sequences: %2F becomes / and is encoded differently in the next request. Could you keep the XML-unescaped token verbatim and update TestUrlEncodedListObjects accordingly?

zjw1111

I found two places where existing project utilities could be reused; details are inline.

zjw1111 · 2026-07-24T09:27:50Z

+
+Result<bool> ParseBoolOption(const std::string& value) {
+    std::string lower = StringUtils::ToLowerCase(value);
+    if (lower == "true" || lower == "1" || lower == "yes" || lower == "on") {


Could we reuse OptionsUtils::GetValueFromMap<bool> / StringUtils::StringToValue<bool> here instead of maintaining a second boolean parser? The accepted spellings currently diverge: the shared parser supports t/y/f/n, while this one adds on/off. Is accepting on/off required by an actual S3 configuration scenario or compatibility target? If so, could we extend the shared parser with those spellings and tests, then reuse it here so boolean option semantics remain consistent across the project?

zjw1111 · 2026-07-24T09:27:50Z

+        }
+
+        AwsAuthRuntime& runtime = GetAwsAuthRuntime();
+        aws_http_message* message = aws_http_message_new_request(runtime.allocator());


Could we use the existing ScopeGuard (or another scoped owner) for message and signable here? The same manual acquire/free pattern also appears in PercentEncode/PercentDecode for the curl-allocated buffers. Reusing the common RAII utility would keep cleanup correct across all error returns and align this code with the project's resource-management convention.

mrdrivingduck · 2026-07-24T09:47:24Z

Thanks for adding the S3 integration. Have you tested it end to end against a real S3 bucket, in addition to the mocked HTTP tests? Could you share which operations were verified and whether the supported read path (HEAD, LIST, and range reads) worked as expected? The current filesystem implementation appears intentionally read-only, so could you also confirm that writes are out of scope, or, if write support is expected, whether it has been validated against real S3 as well?

Thanks for looking.

Yes, I have a WIP duckdb-paimon branch that uses this branch, and I have tested the real S3 read path end to end through DuckDB SQL. The validation covered ATTACH, paimon_scan, and paimon_snapshots against a real Paimon table on S3, including metadata access, object listing, and range reads of the data files.

The S3 filesystem is intentionally read-only in this PR; write support is out of scope. I want to make the commit small.

Review comments will be addressed later. Thanks.

Add a reusable read-only object store layer with a curl-based HTTP transport. Use the AWS C authentication components for credential resolution and request signing while keeping S3 data access independent of the full AWS SDK. Co-authored-by: GPT-5.6 Terra <codex@users.noreply.github.com>

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch 7 times, most recently from bdbb0e8 to 3137230 Compare July 20, 2026 12:45

mrdrivingduck marked this pull request as ready for review July 20, 2026 17:11

Copilot AI review requested due to automatic review settings July 20, 2026 17:11

Copilot started reviewing on behalf of mrdrivingduck July 20, 2026 17:11 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Comment thread src/paimon/fs/s3/s3_file_system.cpp Outdated

Comment thread src/paimon/fs/s3/s3_file_system.cpp

Comment thread src/paimon/fs/object_store/http_client.cpp

Comment thread cmake_modules/BuildAwsAuth.cmake Outdated

lxy-9602 requested a review from lucasfang July 21, 2026 05:31

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch from 3137230 to e52f0b9 Compare July 21, 2026 06:21

mrdrivingduck requested a review from Copilot July 21, 2026 07:20

Copilot started reviewing on behalf of mrdrivingduck July 21, 2026 07:24 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/paimon/fs/s3/s3_file_system.cpp

Comment thread src/paimon/fs/s3/s3_file_system.cpp

Comment thread src/paimon/fs/s3/s3_file_system_test.cpp Outdated

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch from e52f0b9 to 1d2f7d1 Compare July 21, 2026 07:50

mrdrivingduck requested a review from Copilot July 21, 2026 09:22

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/paimon/fs/s3/s3_file_system_test.cpp

Comment thread src/paimon/fs/s3/s3_file_system.cpp

Comment thread src/paimon/common/fs/object_store_file_system.cpp

Comment thread src/paimon/fs/object_store/object_store_file_system.cpp Outdated

Copilot started reviewing on behalf of mrdrivingduck July 21, 2026 10:08 View session

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch from 1d2f7d1 to 27ee4b7 Compare July 21, 2026 13:29

lucasfang reviewed Jul 22, 2026

View reviewed changes

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch 2 times, most recently from fe54261 to 3a106a1 Compare July 22, 2026 09:54

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch 2 times, most recently from 5cee4b4 to 9aef106 Compare July 22, 2026 15:31

lucasfang reviewed Jul 23, 2026

View reviewed changes

Comment thread src/paimon/common/fs/http_client.cpp Outdated

lucasfang reviewed Jul 23, 2026

View reviewed changes

Comment thread src/paimon/common/fs/http_client.cpp Outdated

lucasfang reviewed Jul 23, 2026

View reviewed changes

Comment thread src/paimon/common/fs/object_store_file_system.cpp Outdated

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch from 9aef106 to 7f679bb Compare July 23, 2026 05:02

lucasfang requested a review from zjw1111 July 23, 2026 06:43

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch 2 times, most recently from 9f9bad3 to b275b1b Compare July 24, 2026 04:22

zjw1111 reviewed Jul 24, 2026

View reviewed changes

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch 3 times, most recently from da745c8 to dadf021 Compare July 24, 2026 17:36

mrdrivingduck force-pushed the codex/feat_object_store_s3 branch from dadf021 to 22b46ca Compare July 24, 2026 17:36

		inline constexpr char kS3AccessKeyIdOption[] = "s3.access-key-id";
		inline constexpr char kS3SecretAccessKeyOption[] = "s3.secret-access-key";

Uh oh!

Conversation

mrdrivingduck commented Jul 19, 2026

Background

Design

OSS experiment

Scope

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lucasfang left a comment

Choose a reason for hiding this comment

Uh oh!

mrdrivingduck commented Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lucasfang commented Jul 23, 2026

Uh oh!

mrdrivingduck commented Jul 23, 2026

Uh oh!

lucasfang commented Jul 23, 2026

Uh oh!

mrdrivingduck commented Jul 23, 2026

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjw1111 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjw1111 left a comment •

edited

Loading