Skip to content

[core] Enable bundle writes in append-only compaction - #9680

Closed
jianguotian wants to merge 2 commits into
apache:masterfrom
jianguotian:core/append-compaction-bundle-bridge-upstream
Closed

[core] Enable bundle writes in append-only compaction#9680
jianguotian wants to merge 2 commits into
apache:masterfrom
jianguotian:core/append-compaction-bundle-bridge-upstream

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

Append-only compaction currently consumes every reader batch row by row, even when a format reader can expose the complete logical batch as BundleRecords.

This PR adds a format-neutral Core bridge for bundle-capable compaction readers:

  • Introduce BundleRecordIterator to expose all unconsumed logical rows as a borrowed bundle.
  • Let BaseAppendFileStoreWrite.compactRewrite write such bundles directly, while always releasing each batch and closing the reader on success or failure.
  • Preserve bundle iterator capabilities only through a full identity mapping. Projection/reordering, partition injection, row tracking, and ordinary iterators keep the existing row fallback.

This PR intentionally provides only the Core bridge. Format-specific bundle producers and direct writers can be added in follow-up PRs.

This complements #9660, which adds a Parquet row-group copy fast path. Both touch BaseAppendFileStoreWrite, so the PR merged second may need a small rebase.

Tests

  • JDK 8: ColumnarRowIteratorTest and BaseAppendFileStoreWriteTest (8 tests passed).
  • JDK 11: ColumnarRowIteratorTest and BaseAppendFileStoreWriteTest (8 tests passed).
  • Covered identity versus transformed mappings, direct bundle writes, borrowed-batch release after write failure, and row fallback for stateful row-tracking iterators.

@jianguotian jianguotian closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant