Skip to content

feat(scan): prune manifests by bucket metadata - #850

Open
jianguotian wants to merge 2 commits into
apache:mainfrom
jianguotian:codex/manifest-bucket-pruning
Open

jianguotian wants to merge 2 commits into
apache:mainfrom
jianguotian:codex/manifest-bucket-pruning

Conversation

@jianguotian

Copy link
Copy Markdown

Purpose

Bucket predicates are currently evaluated only after each manifest file is fetched and decoded. For selective fixed-bucket queries, especially single-key lookups over bucket-sorted manifests, this leaves substantial planning I/O on the critical path.

Changes

  • Add the Java-compatible nullable _TOTAL_BUCKETS field to ManifestFileMeta.
  • Persist it only when every manifest entry has the same positive bucket count.
  • Before fetching manifest contents, combine _MIN_BUCKET, _MAX_BUCKET, and _TOTAL_BUCKETS with the existing bucket-key predicate hash calculation.
  • Apply the same pruning to full snapshot and incremental manifest-list scans.
  • Fail open for legacy, missing, invalid, mixed, or unassigned-bucket metadata.
  • Retain existing entry-level filtering as a correctness backstop.

This complements bucket-first manifest sorting: sorting tightens each manifest's bucket envelope, while this PR lets the Rust planner exploit that envelope.

_TOTAL_BUCKETS follows Java PR apache/paimon#9791. PR #848 adds the adjacent _EXTRA_FILES field independently; this branch intentionally keeps the two review units separate and will be rebased for canonical final schema order after either one lands.

Verification

  • cargo test -p paimon --lib: 2,818 passed, 2 ignored
  • cargo clippy -p paimon --lib --tests -- -D warnings
  • Regression coverage for exact bucket selection, legacy fail-open behavior, unassigned buckets, and mixed/invalid total bucket counts

Related Java work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant