Skip to content

Add a caller row group filter to RowReaderOptions - #26

Open
groeneai wants to merge 1 commit into
ClickHouse:ClickHouse/2.3.1from
groeneai:groeneai/row-group-filter-hook
Open

Add a caller row group filter to RowReaderOptions#26
groeneai wants to merge 1 commit into
ClickHouse:ClickHouse/2.3.1from
groeneai:groeneai/row-group-filter-hook

Conversation

@groeneai

Copy link
Copy Markdown

A SearchArgument compares a stored column against a literal, so a predicate whose key is wrapped
in a monotonic function chain cannot be expressed in one: that would require inverting the chain.
Evaluating such a predicate forward from the row group statistics is possible, but only on the
caller's side, where the chain machinery lives.

This adds RowReaderOptions::rowGroupFilter: a callback that receives the stripe's row group count
and an accessor for the public ColumnStatistics of a (column, row group) pair, and returns one
keep flag per row group. The reader intersects the result with its own search argument evaluation,
so the callback can only narrow the selection and the reader remains the sole scheduler of row
groups, batch boundaries and seeking.

The statistics come from the row index the reader has already loaded in startNextStripe, so there
is no extra IO and no second metadata traversal. They are wrapped as public ColumnStatistics via
the existing convertColumnStatistics, keeping protobuf out of the public headers.

The keep bit is folded into needed inside pickRowGroups' backward loop rather than applied to
nextSkippedRows_ afterwards: that vector encodes the END of each selected run, so clearing an
entry after the fact would leave the preceding run's marker spanning the skipped group and the
reader would return rows the callback rejected.

With a filter installed, the no-row-index early return now writes an explicit all-keep schedule. It
previously relied on leaving the vector all-zero, which hasSelectedFrom and computeBatchSize
read as "everything skipped". Without a filter, behaviour is unchanged: rowGroupFilter_ is empty,
pickRowGroups receives nullptr, and every touched path takes its existing branch.

Motivation is on the ClickHouse side: ORC filter pushdown there is disabled for any predicate whose
key sits under a monotonic function chain (toUInt64(id) = 555555, id + 0 = 555555, and so on),
while Parquet prunes through all of them. Details and measurements in
ClickHouse/ClickHouse#112577.

A SearchArgument compares a stored column against a literal, so a predicate whose key is
wrapped in a monotonic function chain cannot be expressed in it: that would require inverting
the chain. Evaluating such a predicate forward from the row group statistics is possible, but
only on the caller's side, where the chain machinery lives.

This adds RowReaderOptions::rowGroupFilter: a callback that receives the stripe's row group
count and an accessor for the public ColumnStatistics of a (column, row group) pair, and
returns one keep flag per row group. The reader intersects the result with its own search
argument evaluation, so the callback can only narrow the selection and the reader remains the
sole scheduler of row groups, batch boundaries and seeking.

The statistics come from the row index the reader has already loaded in startNextStripe, so
there is no extra IO and no second metadata traversal. They are wrapped as public
ColumnStatistics via the existing convertColumnStatistics, keeping protobuf out of the public
headers.

The keep bit is folded into `needed` inside pickRowGroups' backward loop rather than applied
to nextSkippedRows_ afterwards: that vector encodes the END of each selected run, so clearing
an entry after the fact would leave the preceding run's marker spanning the skipped group and
the reader would return rows the callback rejected.

With a filter installed, the no-row-index early return now writes an explicit all-keep
schedule. It previously relied on leaving the vector all-zero, which hasSelectedFrom and
computeBatchSize read as "everything skipped". Without a filter, behaviour is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant