Skip to content

Preserve bitmap-backed RowSelection in ParquetAccessPlan #23883

Description

@haohuaijin

Is your feature request related to a problem or challenge?

arrow-rs#10141 allows RowSelection to be backed by a BooleanBuffer.

However, ParquetAccessPlan::into_overall_row_selection currently calls RowSelection::iter() and converts each selection into a Vec<RowSelector>. This materializes mask-backed selections as RLE and loses the bitmap backing before it reaches the Parquet reader.

This issue is blocked until DataFusion upgrades to an Arrow/Parquet release containing arrow-rs#10141.

Describe the solution you'd like

  • Validate selection length using row_count() + skipped_row_count() instead of iter().
  • When all row-group selections are mask-backed, concatenate their BooleanBuffers directly.
  • Represent scanned row groups as all-set bitmap ranges and omit skipped row groups.
  • Keep the existing selector path as the fallback for selector-backed or mixed inputs.

Describe alternatives you've considered

Converting selectors back to a bitmap in the Parquet reader does not help because the large intermediate Vec<RowSelector> has already been allocated.

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions