Skip to content

perf(mosaic): prefetch row groups with a byte budget - #851

Open
jianguotian wants to merge 2 commits into
apache:mainfrom
jianguotian:codex/mosaic-row-group-prefetch
Open

jianguotian wants to merge 2 commits into
apache:mainfrom
jianguotian:codex/mosaic-row-group-prefetch

Conversation

@jianguotian

Copy link
Copy Markdown

Purpose

The Rust Mosaic reader currently opens row groups serially. Opening a group performs dependent range reads and decompression, so object-store latency remains on the critical path even after manifest and row-selection pruning.

Changes

  • Plan row groups after row-selection and statistics pruning, then prefetch only the survivors.
  • Open up to 8 row groups concurrently while preserving file order in emitted batches.
  • Bound decoded data held ahead with a 64 MiB estimate based on projected field types and row counts.
  • Always admit the head group so an undersized byte budget cannot stall progress.
  • Keep residual predicate evaluation and batch emission ordered on the consumer thread.
  • Join all scheduled work on cancellation, errors, and panics so no storage reads outlive the Mosaic reader.

The defaults match Java's mosaic.read.prefetch-row-groups=8 and mosaic.read.prefetch-max-bytes=64mb. Rust's format-reader factory does not yet expose format-scoped table options, so this PR establishes the bounded behavior with those defaults; option plumbing can remain an independent API change.

Verification

  • cargo test -p paimon --lib: 2,818 passed, 2 ignored
  • cargo clippy -p paimon --lib --tests -- -D warnings
  • A delayed storage fixture verifies overlapping row-group I/O and stable output order.
  • A one-byte budget regression verifies fallback to one mandatory in-flight group.

Related Java work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant