Add a caller row group filter to RowReaderOptions - #26
Open
groeneai wants to merge 1 commit into
Open
Conversation
A SearchArgument compares a stored column against a literal, so a predicate whose key is wrapped in a monotonic function chain cannot be expressed in it: that would require inverting the chain. Evaluating such a predicate forward from the row group statistics is possible, but only on the caller's side, where the chain machinery lives. This adds RowReaderOptions::rowGroupFilter: a callback that receives the stripe's row group count and an accessor for the public ColumnStatistics of a (column, row group) pair, and returns one keep flag per row group. The reader intersects the result with its own search argument evaluation, so the callback can only narrow the selection and the reader remains the sole scheduler of row groups, batch boundaries and seeking. The statistics come from the row index the reader has already loaded in startNextStripe, so there is no extra IO and no second metadata traversal. They are wrapped as public ColumnStatistics via the existing convertColumnStatistics, keeping protobuf out of the public headers. The keep bit is folded into `needed` inside pickRowGroups' backward loop rather than applied to nextSkippedRows_ afterwards: that vector encodes the END of each selected run, so clearing an entry after the fact would leave the preceding run's marker spanning the skipped group and the reader would return rows the callback rejected. With a filter installed, the no-row-index early return now writes an explicit all-keep schedule. It previously relied on leaving the vector all-zero, which hasSelectedFrom and computeBatchSize read as "everything skipped". Without a filter, behaviour is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
SearchArgumentcompares a stored column against a literal, so a predicate whose key is wrappedin a monotonic function chain cannot be expressed in one: that would require inverting the chain.
Evaluating such a predicate forward from the row group statistics is possible, but only on the
caller's side, where the chain machinery lives.
This adds
RowReaderOptions::rowGroupFilter: a callback that receives the stripe's row group countand an accessor for the public
ColumnStatisticsof a (column, row group) pair, and returns onekeep flag per row group. The reader intersects the result with its own search argument evaluation,
so the callback can only narrow the selection and the reader remains the sole scheduler of row
groups, batch boundaries and seeking.
The statistics come from the row index the reader has already loaded in
startNextStripe, so thereis no extra IO and no second metadata traversal. They are wrapped as public
ColumnStatisticsviathe existing
convertColumnStatistics, keeping protobuf out of the public headers.The keep bit is folded into
neededinsidepickRowGroups' backward loop rather than applied tonextSkippedRows_afterwards: that vector encodes the END of each selected run, so clearing anentry after the fact would leave the preceding run's marker spanning the skipped group and the
reader would return rows the callback rejected.
With a filter installed, the no-row-index early return now writes an explicit all-keep schedule. It
previously relied on leaving the vector all-zero, which
hasSelectedFromandcomputeBatchSizeread as "everything skipped". Without a filter, behaviour is unchanged:
rowGroupFilter_is empty,pickRowGroupsreceivesnullptr, and every touched path takes its existing branch.Motivation is on the ClickHouse side: ORC filter pushdown there is disabled for any predicate whose
key sits under a monotonic function chain (
toUInt64(id) = 555555,id + 0 = 555555, and so on),while Parquet prunes through all of them. Details and measurements in
ClickHouse/ClickHouse#112577.