[core] Parallelize per-partition file listing for catalog-managed format tables#8845
Open
XiaoHongbo-Hope wants to merge 8 commits into
Open
[core] Parallelize per-partition file listing for catalog-managed format tables#8845XiaoHongbo-Hope wants to merge 8 commits into
XiaoHongbo-Hope wants to merge 8 commits into
Conversation
…mat tables Format table split planning lists every partition's files serially, which is slow for internal (catalog-managed) tables with many partitions (thousands of object store LIST calls on the coordinator). Extract the per-partition listing into a PartitionSplitPlanner strategy with two implementations selected by the existing internal/external discriminator (partitionManager != null): - SequentialSplitPlanner (external): unchanged serial path; a missing partition directory is rethrown. - ParallelSplitPlanner (internal): bounded, order-preserving fan-out via ManifestReadThreadPool; a registered partition without a directory reads as empty (matching Hive), any other listing failure fails the whole scan. Split order and per-partition sort/bin-packing are unchanged, so output is identical to the serial path. Parallelism is bounded by a new option format-table.scan.list-parallelism (default #CPU); external tables are untouched.
sundapeng
reviewed
Jul 24, 2026
| } | ||
|
|
||
| /** Lists partition files and builds splits; per-partition sort and bin-packing are shared. */ | ||
| private interface PartitionSplitPlanner { |
Member
There was a problem hiding this comment.
better to remove this interface
Inline the internal/external branch directly in splits(): a private listPartitionFilesInParallel(...) for internal (catalog-managed) tables and the serial loop for external tables. Behavior is unchanged.
XiaoHongbo-Hope
marked this pull request as ready for review
July 25, 2026 01:34
A non-positive format-table.scan.list-parallelism would create a SemaphoredDelegatingExecutor with zero/negative permits, making the listing acquire() block forever. Clamp to a minimum of 1 so a misconfigured value degrades to serial listing instead of hanging.
JingsongLi
reviewed
Jul 25, 2026
| BinaryRow partitionRow = toPartitionRow(partitionSpec); | ||
| if (partitionFilter == null || partitionFilter.test(partitionRow)) { | ||
| try { | ||
| if (partitionManager != null) { |
Contributor
There was a problem hiding this comment.
I think we could refactor the code here; the partitionManager should be renamed to SplitEnumerator so that it directly generates the corresponding Split.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.