Skip to content

perf(manifest): write bucket-first sorted entries - #852

Open
jianguotian wants to merge 1 commit into
apache:mainfrom
jianguotian:codex/bucket-first-manifest-sort
Open

jianguotian wants to merge 1 commit into
apache:mainfrom
jianguotian:codex/bucket-first-manifest-sort

Conversation

@jianguotian

Copy link
Copy Markdown

Purpose

Keep entries for fixed-bucket and postponed-bucket tables close by bucket when
Rust writes a manifest. This produces tighter _MIN_BUCKET / _MAX_BUCKET
envelopes after rolling, so scans can skip more manifest files before fetching
and decoding their contents.

This is the write-side counterpart to the bucket metadata pruning added in
#850 and follows the bucket-first layout introduced by Apache Paimon Java in
apache/paimon#9792 and extended to non-partitioned fixed/postponed-bucket tables
in apache/paimon#9808.

Changes

  • add the Java-compatible manifest-sort.enabled table option (default false)
  • for non-data-evolution fixed/postponed-bucket tables, sort entries by:
    1. bucket
    2. the first partition field, when present
    3. file kind
    4. file name
  • apply the order to newly written delta manifests and manifests rewritten by
    the existing Rust minor/full merge paths
  • leave data files and row contents unchanged

The implementation intentionally does not claim parity with Java's explicit
compact_manifest action (apache/paimon#9802): paimon-rust does not currently
have the corresponding action/procedure API. Existing one-file manifest lists
are therefore not force-rewritten solely by this option; adding that API can be
reviewed independently.

Tests

  • cargo test -p paimon --lib (2818 passed, 2 ignored)
  • cargo clippy -p paimon --lib --tests -- -D warnings
  • cargo fmt --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant