feat(io): support URI-aware FileIO providers

[JingsongLi]

Purpose

Closes #712.

Allow embedding applications such as Milvus to reuse their shared OpenDAL backends through the existing FileIOProvider abstraction. Providers resolve original URIs to an operator and a relative object path, while continuing to own credential refresh.

Brief change log

• Make FileIOProvider public and expose with_provider on FileIOBuilder and FileIO.
• Defer provider-backed input/output resolution to async operations, bypassing built-in storage construction and propagating provider errors without fallback.
• Validate directory path mappings and return reusable full URIs from both listing APIs, including bucket roots and operators rooted below a bucket.
• Reject object paths that OpenDAL would normalize and rename operations spanning different shared service instances.
• Add regression coverage for literal object keys, multi-bucket routing, error propagation, open-handle lifetime, cache isolation/invalidation, and existing filesystem behavior.

Tests

• Reproduced the old synchronous filesystem-resolution failure before implementing the change.
• cargo test --locked --offline -p paimon --all-targets --features fulltext,vortex — 3149 passed, 2 ignored.
• cargo test --locked --offline -p paimon --lib io:: --features storage-all,fulltext,vortex — 175 passed, including existing REST token-refresh and OSS retry tests.
• cargo clippy --locked --offline --all-targets --workspace --features fulltext,vortex -- -D warnings — passed.
• cargo check --locked --offline -p paimon --lib --no-default-features — passed, with two existing unused import/variable warnings in storage.rs.
• cargo fmt --all -- --check and git diff --check — passed.

API and Format

Adds public Rust provider injection APIs; no storage-format change. Provider-backed constructors remain synchronous and report resolution errors during subsequent async IO. Listing requires the resolved relative path to be a component suffix of the original URI. Existing readers/writers retain their operator, so credential refresh for open handles must happen inside the backend. Property-based storage configuration remains available.

Documentation

Adds a shared-backend integration example and documents URI mapping, credential lifetime, cache identity, and rename requirements.

[leaves12138]

Reviewed 80ff707033e0b350164a632b05741080902867d8 against 277213ad23c0c42394d9f47a8b355baeaf54ddcb, with particular attention to performance and API compatibility. No blocking findings; approving.

API and behavior

• This is additive at the public Rust API level: existing public method signatures remain available. FileIOProvider and the two with_provider entry points are exported and usable from an external example crate. Implementers must use a compatible opendal-core dependency and async-trait; exposing Operator introduces dependency-version coupling for users of the new extension point.
• Provider injection bypasses built-in storage construction and storage-property parsing, without requiring the corresponding built-in storage feature. It is mutually exclusive with with_fs_operator on a builder. Option-based catalog constructors are not automatically wired to this extension point.
• Provider-backed new_input/new_output defer resolution errors to the subsequent async operation. Open readers/writers retain their operator; refreshing credentials for a long-lived handle remains the backend's responsibility.
• Listing requires a component-aligned, unchanged URI suffix. Noncanonical object paths are rejected rather than silently accessing a different object. Provider-backed rename requires the exact same shared service instance, not merely equivalent bucket/endpoint settings. Provider implementations should reuse operators; even a refresh that replaces the service between the two rename resolutions can cause a safe IoUnsupported result.
• Checked cache namespace isolation/invalidation, literal object keys, listing URI round-tripping, and the existing REST-token refresh paths.

Performance validation

I compared separate base/head executables over seven interleaved rounds pinned to one CPU, using the same in-memory backend. Paimon and the benchmark example used opt-level=3; dependencies retained the same dev-profile build. These are wrapper microbenchmarks, not a fully optimized release build or an S3/OSS throughput benchmark.

Median results for the existing built-in path:

Operation	Base	PR head
Construct input	0.601 us	0.632 us
Construct output	0.590 us	0.610 us
Exists on an input handle	3.467 us	3.439 us
New input + read 4 KiB	9.398 us	9.460 us
Read 4 KiB through an open reader	4.046 us	4.146 us
List 1,000 entries	3.324 ms	3.324 ms

There is no new provider dispatch or network request in the built-in path. Allocation counts were unchanged for the measured constructors/reads; listing adds one prefix allocation per call. Handle sizes on this target decrease from 32 to 24 bytes for FileIO, and from 200 to 184 bytes for InputFile/OutputFile.

The new provider path does have measurable per-operation CPU/allocation overhead. Compared with the PR head's built-in path, a trivial shared-operator provider measured 5.099 us for exists, 10.085 us for new-input + 4-KiB read, and 4.205 ms for listing 1,000 entries. These are comparisons between backend modes, not a 26% regression in existing default listings. Deferred construction itself is cheaper (about 0.060 us).

A non-blocking optimization opportunity is the allocating normalize_path check for each provider listing entry: the probe observed approximately two additional allocations per entry. If metadata-only/local-memory listing becomes important, a non-allocating equivalent validation could reduce this cost without weakening literal-key safety.

The hot reader path is not repeatedly resolved: a counting provider observed zero calls during handle construction, one call when opening the reader, and zero additional calls for 10,000 range reads. Provider implementations should avoid rebuilding operators or fetching credentials on every resolution.

Checks

• cargo test --locked --offline -p paimon --lib: 2,812 passed, 2 ignored.
• cargo fmt --all -- --check: passed.
• cargo clippy --locked --offline -p paimon --lib -- -D warnings: passed.
• cargo check --locked --offline -p paimon --lib --no-default-features: passed; two warnings are in unchanged storage.rs.
• All 14 GitHub check runs on the reviewed head are successful.