Repository navigation
Conversation
…ng baseline - Add InstrumentedObjectStore that correlates requested byte ranges to Parquet file structures (footers, page indexes, bloom filters, data pages) - Add test_parquet_bloom_filter_io_baseline using public SessionContext SQL APIs - Use insta snapshot to capture and compare the storage I/O trace - Documents baseline on main where all candidate column Bloom filters are loaded upfront, establishing the target diff for incremental pruning (apache#25811)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split the Bloom filter I/O test into reusable setup code in tests/parquet/io/mod.rs and the test itself in tests/parquet/io/bloom_filter_io.rs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ushdown and TopK Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Note
This PR is stacked on #26897. The new tests are in the last commit: e85a89f
Rationale for this change
#26897 adds an instrumented object store that shows the exact storage requests made by a Parquet scan. This PR uses the same test setup to show what I/O several pruning optimizations save, so changes to the parquet reader's I/O pattern are visible in review.
What changes are included in this PR?
New tests in
datafusion/core/tests/parquet/io/, all using a shared file with three row groups (three_row_group_file):row_group_pruning_io.rs: the same file with different predicates onid(none,id = 5,id > 5,id > 100), showing that pruned row groups are never fetched.filter_pushdown_io.rs:val = 7(which statistics cannot prune) withpushdown_filtersoff and on. With pushdown on,valis fetched for every row group, butidis fetched only for row group 1.topk_io.rs:ORDER BY id DESC LIMIT 2with TopK dynamic filter pushdown on and off. With it on, sort pushdown reads row group 2 first, and the dynamic filter then prunes row groups 1 and 0 before they are fetched. With it off, all three row groups are fetched.What is the testing strategy for this PR?
This PR only adds tests. I ran the TopK tests 15 times locally to check that they don't depend on timing (they use
target_partitions = 1).Are there any user-facing changes?
No