Repository navigation
Bump iceberg-rust to pick up the upstream merge - #114
Merged
Merged
Conversation
…ilters, new REST auth) RelationalAI/iceberg-rust merged apache/iceberg-rust@main (PR #82), bringing row lineage (_row_id), bloom-filter row-group pruning, Parquet Modular Encryption plumbing, and a new REST-catalog AuthSession/AuthManager architecture, while preserving vended credentials, the custom REST authenticator, storage-credential refresh, and OverwriteAction. This required source changes here, not just a rev bump, since that merge also completed a storage-backend migration: - iceberg's old in-tree opendal storage module (OpenDalStorageFactory, OpenDalRoutingStorageFactory, the "storage-azdls" feature) was removed in favor of the standalone iceberg-storage-opendal crate (OpenDalStorageFactory now lives there, OpenDalRoutingStorageFactory renamed OpenDalResolvingStorageFactory) and iceberg-storage-refreshable (RefreshableStorageFactory). Added both as direct dependencies and updated catalog.rs/table.rs's imports and construction sites accordingly. OpenDalStorageFactory::S3 also dropped its configured_scheme field (no longer distinguishes s3 vs s3a), so that's dropped from the one call site that set it. - FileScanTask's fields all became private with accessor methods; fixed the remaining field accesses (record_count) and converted three test-only struct-literal constructions in full_pipeline.rs to the builder API. - Bumped arrow-*/parquet from 58.1 to 59.2 to match iceberg-rust's new requirement and avoid duplicate-crate-version type mismatches. Bumped iceberg_rust_ffi and the Julia package version to 0.9.5. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… storage migration
Three CI regressions surfaced by the iceberg-rust bump, all traced by running
the full Julia suite locally against this repo's own docker-compose MinIO:
1. iceberg-storage-opendal's S3 config now defaults to virtual-host-style
addressing (matching Java's actual S3FileIOProperties spec default), unlike
whatever the fork's old, now-deleted OpenDalRoutingStorageFactory did. This
repo's local MinIO can't satisfy virtual-host requests from outside the
docker network: MinIO's virtual-host mode is configured for the `minio`
domain (MINIO_DOMAIN=minio in docker-compose.yml), not `localhost`, so
`warehouse.localhost:9000` 404s with NoSuchBucket. Added
"s3.path-style-access" => "true" to get_catalog_properties() (used by every
REST/Polaris-catalog-driven test) and a new s3_path_style_properties()
helper, wired into every bare table_open(...) call in scan_tests.jl that
was relying purely on AWS_* env vars (which have no path-style equivalent).
2. OpenDalResolvingStorageFactory resolves storage by URL scheme, so it
rejects a bare local filesystem path ("/tmp/x/metadata.json") outright
with a URL-parse error ("relative URL without a base") instead of treating
it as a local file. iceberg_table_open now routes schemeless paths to
iceberg::io::LocalFsStorageFactory directly instead of the resolving
factory, restoring the ability to open local metadata files at all.
3. That LocalFsStorageFactory now reports a missing file with
ErrorKind::DataInvalid rather than the Unexpected kind the old opendal-fs
route used, which classify_data_invalid() didn't recognize (only
classify_message()'s generic fallback matched "no such file" text). Added
the same not-found text match to classify_data_invalid() so a missing
local file classifies as NOT_FOUND_METADATA regardless of which storage
backend reported it.
Verified: full Julia test suite (27950 tests) passes locally against this
repo's docker-compose stack; cargo fmt/clippy clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gbrgr
requested review from
bunjj,
hall-alex and
robertbuessow
and removed request for
robertbuessow
September 16, 2026 11:34
charnik
approved these changes
Sep 17, 2026
geokollias
approved these changes
Sep 17, 2026
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps
iceberg/iceberg-catalog-resttoRelationalAI/iceberg-rust@ec07ed0f7(main, post PR #82 — the merge ofapache/iceberg-rust@main). Brings in row lineage (_row_id), bloom-filter row-group pruning, Parquet Modular Encryption plumbing, and a new REST-catalogAuthSession/AuthManagerarchitecture, while our vended credentials, custom REST authenticator, storage-credential refresh, andOverwriteActionall carry forward unchanged in behavior.This needed real source changes, not just a rev bump, because that merge also completed a storage-backend migration on the iceberg-rust side:
iceberg's old in-tree opendal storage module (OpenDalStorageFactory,OpenDalRoutingStorageFactory, thestorage-azdlsfeature) was removed in favor of the standaloneiceberg-storage-opendalcrate (OpenDalStorageFactorynow lives there;OpenDalRoutingStorageFactoryrenamed toOpenDalResolvingStorageFactory) andiceberg-storage-refreshable(RefreshableStorageFactory). Added both as direct dependencies and updatedcatalog.rs/table.rs's imports and construction sites.OpenDalStorageFactory::S3droppedconfigured_scheme— it no longer distinguishess3vss3a— so that field is dropped from the one call site that set it.FileScanTask's fields all became private with accessor methods. Fixed the one remaining field access (record_count) and converted three test-only struct-literal constructions infull_pipeline.rsto the builder API.arrow-*/parquetfrom 58.1 to 59.2 to match iceberg-rust's new requirement — otherwise cargo resolves two incompatible versions of the same crates and every type crossing the FFI boundary fails to unify.Bumped
iceberg_rust_ffiand the Julia package version to 0.9.5.Test plan
cargo check --all-targetsclean (iceberg_rust_ffi, against the new rev)cargo clippy --all-targets -- -D warningscleancargo fmt --all -- --checkclean🤖 Generated with Claude Code