Skip to content

feat: expand native planning across resolved tables and scan modes - #847

Merged
JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-effective-table-context
Sep 16, 2026
Merged

JingsongLi merged 3 commits into
apache:mainfrom
JingsongLi:codex/native-effective-table-context

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Purpose

Expand the native planning paths available to PyPaimon while preserving Java table semantics. This continues the planner alignment from #845 and uses the resolved-table constructor introduced in #846.

Catalog reloading loses externally resolved fields and removed options. REST reads also currently choose filesystem snapshots without consulting the catalog. Separately, materialized DV files can be grouped for key merging even though Java packs them as raw splits by size, which matters for files sorted by clustering columns.

Brief change log

  • Expose Table.from_resolved_schema() in the Python binding, preserving Java-format schema JSON, complete options, FileIO properties and branch without another schema resolution.
  • Add Table.copy_with_resolved_schema() to replace fields/options while retaining the table identity, location, FileIO provider and REST environment. Select a resolved branch without requiring its schema file, and discard any cached time-travel snapshot so the new options take effect. Expose the operation in the Python binding and release the GIL around table APIs that can perform REST I/O.
  • Accept (database, table) tuples in PaimonCatalog.get_table() so names containing dots do not get split into the wrong namespace. Preserve string identifiers and branch suffix parsing, and validate malformed components and system-table identifiers.
  • Load latest snapshots and snapshot IDs through REST for catalog-backed tables, including branches. Empty responses and SNAPSHOT 404 are authoritative; permission/service errors, including HTTP 501, propagate as in Java. Implement the matching endpoint in paimon-rest-server and update the REST test server.
  • Pack materialized DV/first-row files by size across levels with raw-convertible splits. Continue grouping overlapping files for merging when L0 is present.
  • Add behavioral tests for field-ID mapping, option replacement, snapshot-cache invalidation, branch/tag/empty plans, query authorization, strict input validation, REST snapshot priority/errors/credentials, and materialized/L0 read behavior.

Tests

  • cargo test --locked -p paimon --lib table::: 1,287 passed, 1 ignored.
  • REST API/catalog integration suites: 124 passed. First-row scan suite: 6 passed, including ordinary DV merge-on-read planning of overlapping L0 first-row runs and post-merge value filtering.
  • cargo test --locked -p paimon-rest-server --all-targets: 11 end-to-end tests passed, including write/commit/read and empty/latest/branch snapshots.
  • Python binding read/table/resolved-schema suites: 121 passed with the rebuilt wheel, including dotted database/table names, main/nonempty/empty branches and invalid identifier inputs.
  • Consumer integration: 489 PyPaimon tests passed, exercising 642 native plans across append, PK, DE, dynamic buckets, cross-partition updates, incremental scans, indexes, chunk shuffle, JDBC, REST, first-row L0 and schema evolution.
  • Exact CI lint command passed: cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings. cargo fmt --all -- --check and git diff --check passed.
  • The materialized-DV regression and REST error regressions fail against the previous implementation.

API and Format

Adds resolved-schema binding APIs, tuple identifiers, REST snapshot response types/API and the server's snapshot endpoint. No storage or split wire-format changes. Refreshable credentials continue to come from catalog-backed tables. This does not add clustering writes or relax create-time validation for first-row/DV tables.

Documentation

Document resolved schema/options, branch and time-travel behavior, REST snapshot semantics and the new server endpoint.

@JingsongLi JingsongLi changed the title feat(python): expand native planning for resolved and materialized tables feat: expand native planning across resolved tables and scan modes Sep 16, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed efd99f5555a7f4b9e51457d17d3c1fb1c6c95b7f against base 66f655b, with Java behavior as the reference (Java checkout 475be566, particularly MergeTreeSplitGenerator, SnapshotManager, RESTCatalog, and the snapshot REST response types).

No blocking findings.

Checked the following boundaries:

  • Resolved-schema construction/replacement preserves the caller's fields, field IDs and complete options. Copies retain the FileIO provider and REST environment, scope metadata to the selected branch, and discard cached time-travel resolution without mutating the original table.
  • Catalog-backed latest snapshot/id lookups use REST authoritatively. Empty responses and SNAPSHOT-specific 404s do not fall back to newer filesystem state. Other errors, including HTTP 501, propagate; Java's REST NotImplementedException is not the UnsupportedOperationException fallback in SnapshotManager.
  • Materialized DV/first-row files are packed by size across levels under the same non-L0/no-delete condition as Java. L0-containing scans retain key-overlap grouping, and first-row value filtering stays after merging.
  • Python string identifiers remain supported; tuple identifiers preserve dotted components. The I/O-bearing table methods release the GIL, and malformed identifiers/schemas and query-auth-enabled native reads retain their validation guards.

Local validation on the reviewed head:

  • cargo test --locked --offline -p paimon --lib table::: 1,287 passed, 1 ignored.
  • First-row integration suite: 6 passed; REST server all-targets: 11 passed.
  • REST API/catalog/object suites: 45 + 79 + 10 passed; incremental and scan-planning parity suites: 36 + 15 passed. Total: 1,489 Rust tests passed.
  • Rebuilt the Python extension and ran read/table/resolved-table/catalog-GIL suites: 122 passed. Used --auditwheel skip for the local-only wheel because patchelf is unavailable; this validates runtime behavior, not portable wheel repair.
  • cargo fmt --all -- --check passed. All 14 GitHub checks on this head are successful.

API/performance note: these are additive entry points, but latest-snapshot lookups for REST tables now intentionally involve a catalog request rather than filesystem discovery. Resolved-schema copies themselves do not reload schema metadata. I did not run a performance benchmark or the separate downstream PyPaimon consumer suite.

Approving. No source changes were pushed.

@JingsongLi
JingsongLi merged commit c92d873 into apache:main Sep 16, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants