feat: expand native planning across resolved tables and scan modes - #847
Merged
JingsongLi merged 3 commits intoSep 16, 2026
Merged
Conversation
leaves12138
approved these changes
Sep 16, 2026
leaves12138
left a comment
There was a problem hiding this comment.
Reviewed efd99f5555a7f4b9e51457d17d3c1fb1c6c95b7f against base 66f655b, with Java behavior as the reference (Java checkout 475be566, particularly MergeTreeSplitGenerator, SnapshotManager, RESTCatalog, and the snapshot REST response types).
No blocking findings.
Checked the following boundaries:
- Resolved-schema construction/replacement preserves the caller's fields, field IDs and complete options. Copies retain the FileIO provider and REST environment, scope metadata to the selected branch, and discard cached time-travel resolution without mutating the original table.
- Catalog-backed latest snapshot/id lookups use REST authoritatively. Empty responses and SNAPSHOT-specific 404s do not fall back to newer filesystem state. Other errors, including HTTP 501, propagate; Java's REST
NotImplementedExceptionis not theUnsupportedOperationExceptionfallback inSnapshotManager. - Materialized DV/first-row files are packed by size across levels under the same non-L0/no-delete condition as Java. L0-containing scans retain key-overlap grouping, and first-row value filtering stays after merging.
- Python string identifiers remain supported; tuple identifiers preserve dotted components. The I/O-bearing table methods release the GIL, and malformed identifiers/schemas and query-auth-enabled native reads retain their validation guards.
Local validation on the reviewed head:
cargo test --locked --offline -p paimon --lib table::: 1,287 passed, 1 ignored.- First-row integration suite: 6 passed; REST server all-targets: 11 passed.
- REST API/catalog/object suites: 45 + 79 + 10 passed; incremental and scan-planning parity suites: 36 + 15 passed. Total: 1,489 Rust tests passed.
- Rebuilt the Python extension and ran read/table/resolved-table/catalog-GIL suites: 122 passed. Used
--auditwheel skipfor the local-only wheel becausepatchelfis unavailable; this validates runtime behavior, not portable wheel repair. cargo fmt --all -- --checkpassed. All 14 GitHub checks on this head are successful.
API/performance note: these are additive entry points, but latest-snapshot lookups for REST tables now intentionally involve a catalog request rather than filesystem discovery. Resolved-schema copies themselves do not reload schema metadata. I did not run a performance benchmark or the separate downstream PyPaimon consumer suite.
Approving. No source changes were pushed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Expand the native planning paths available to PyPaimon while preserving Java table semantics. This continues the planner alignment from #845 and uses the resolved-table constructor introduced in #846.
Catalog reloading loses externally resolved fields and removed options. REST reads also currently choose filesystem snapshots without consulting the catalog. Separately, materialized DV files can be grouped for key merging even though Java packs them as raw splits by size, which matters for files sorted by clustering columns.
Brief change log
Table.from_resolved_schema()in the Python binding, preserving Java-format schema JSON, complete options, FileIO properties and branch without another schema resolution.Table.copy_with_resolved_schema()to replace fields/options while retaining the table identity, location, FileIO provider and REST environment. Select a resolved branch without requiring its schema file, and discard any cached time-travel snapshot so the new options take effect. Expose the operation in the Python binding and release the GIL around table APIs that can perform REST I/O.(database, table)tuples inPaimonCatalog.get_table()so names containing dots do not get split into the wrong namespace. Preserve string identifiers and branch suffix parsing, and validate malformed components and system-table identifiers.Tests
cargo test --locked -p paimon --lib table::: 1,287 passed, 1 ignored.cargo test --locked -p paimon-rest-server --all-targets: 11 end-to-end tests passed, including write/commit/read and empty/latest/branch snapshots.cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings.cargo fmt --all -- --checkandgit diff --checkpassed.API and Format
Adds resolved-schema binding APIs, tuple identifiers, REST snapshot response types/API and the server's snapshot endpoint. No storage or split wire-format changes. Refreshable credentials continue to come from catalog-backed tables. This does not add clustering writes or relax create-time validation for first-row/DV tables.
Documentation
Document resolved schema/options, branch and time-travel behavior, REST snapshot semantics and the new server endpoint.