Prune hash join probe side via dynamic discrete values from the build side by gruuya · Pull Request #25292 · apache/datafusion

gruuya · 2026-09-14T12:11:23Z

Which issue does this PR close?

Closes Implement dynamic discrete pruning through a join #25291.

Rationale for this change

Avoid scanning redundant files/row-groups/pages in the probe side of hash joins, based on the values dictated by the build side.

What changes are included in this PR?

2 new configs that control the thresholds for the dynamic discrete pruning to take place
wiring up PushdownStrategy::Map/HashTableLookupExpr to carry the build side values from a hash join
extend build_predicate_expression to build the associated pruning expression from HashTableLookupExpr

What is the testing strategy for this PR?

Unit tests added covering all changes.

Also tested manually that the problem from the issue is resolved now

> select version();
+--------------------------------------------+
| version()                                  |
+--------------------------------------------+
| Apache DataFusion 55.0.0, aarch64 on macos |
+--------------------------------------------+
1 row(s) fetched.
Elapsed 0.009 seconds.

> copy (select i as k, random() as v from generate_series(0, 1999999) t(i))
to '/tmp/fact.parquet'
stored as parquet options ('format.max_row_group_size' '1000');
+---------+
| count   |
+---------+
| 2000000 |
+---------+
1 row(s) fetched.
Elapsed 0.091 seconds.

> create external table fact stored as parquet location '/tmp/fact.parquet';
0 row(s) fetched.
Elapsed 0.010 seconds.

> create table dim as
select i as k from generate_series(0, 1999999) t(i) where i % 10000 < 200;
0 row(s) fetched.
Elapsed 0.010 seconds.

> set datafusion.optimizer.hash_join_dynamic_pruning_max_distinct_values = 0; -- disabled
0 row(s) fetched.
Elapsed 0.001 seconds.

> select count(*), sum(v) from fact join dim on fact.k = dim.k; 
select count(*), sum(v) from fact join dim on fact.k = dim.k; 
select count(*), sum(v) from fact join dim on fact.k = dim.k; 
select count(*), sum(v) from fact join dim on fact.k = dim.k;
+----------+--------------------+
| count(*) | sum(fact.v)        |
+----------+--------------------+
| 40000    | 19873.614509561437 |
+----------+--------------------+
1 row(s) fetched.
Elapsed 0.116 seconds.

+----------+-------------------+
| count(*) | sum(fact.v)       |
+----------+-------------------+
| 40000    | 19873.61450956144 |
+----------+-------------------+
1 row(s) fetched.
Elapsed 0.096 seconds.

+----------+-------------------+
| count(*) | sum(fact.v)       |
+----------+-------------------+
| 40000    | 19873.61450956144 |
+----------+-------------------+
1 row(s) fetched.
Elapsed 0.096 seconds.

+----------+-------------------+
| count(*) | sum(fact.v)       |
+----------+-------------------+
| 40000    | 19873.61450956144 |
+----------+-------------------+
1 row(s) fetched.
Elapsed 0.093 seconds.

> explain analyze select count(*), sum(v) from fact join dim on fact.k = dim.k
...
|                   |           DataSourceExec: file_groups={12 groups: [[tmp/fact.parquet:0..1929814], [tmp/fact.parquet:1929814..3859628], [tmp/fact.parquet:3859628..5789442], [tmp/fact.parquet:5789442..7719256], [tmp/fact.parquet:7719256..9649070], ...]}, projection=[k, v], output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ k@0 >= 0 AND k@0 <= 1990199 AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=k_null_count@1 != row_count@2 AND k_max@0 >= 0 AND k_null_count@1 != row_count@2 AND k_min@3 <= 1990199, required_guarantees=[], metrics=[output_rows=1.99 M, elapsed_compute=551.82µs, output_bytes=248.9 MB, output_batches=1.99 K, files_ranges_pruned_statistics=12 total → 12 matched, row_groups_pruned_statistics=2.00 K total → 1.99 K matched, row_groups_pruned_bloom_filter=1.99 K total → 1.99 K matched, page_index_pages_pruned=1.99 K total → 1.99 K matched, page_index_rows_pruned=1.99 M total → 1.99 M matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, bytes_processed=22.1 MB, bytes_scanned=21.3 MB, file_open_errors=0, file_scan_errors=0, files_opened=12, files_processed=12, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_pruned_dynamic_filter=0, predicate_cache_inner_records=0, predicate_cache_records=0, bloom_filter_eval_time=75.36µs, metadata_load_time=1.25ms, page_index_eval_time=10.25ms, row_pushdown_eval_time=36ns, statistics_eval_time=1.05ms, time_elapsed_opening=17.55ms, time_elapsed_processing=106.88ms, time_elapsed_scanning_total=1.17s, time_elapsed_scanning_until_data=14.07ms, output_rows_skew=1.64%, scan_efficiency_ratio=96.46% (22.34 M/23.16 M)] |
...

> set datafusion.optimizer.hash_join_dynamic_pruning_max_distinct_values = 100000;  -- enabled, default
0 row(s) fetched.
Elapsed 0.000 seconds

> select count(*), sum(v) from fact join dim on fact.k = dim.k; 
select count(*), sum(v) from fact join dim on fact.k = dim.k; 
select count(*), sum(v) from fact join dim on fact.k = dim.k; 
select count(*), sum(v) from fact join dim on fact.k = dim.k;
+----------+--------------------+
| count(*) | sum(fact.v)        |
+----------+--------------------+
| 40000    | 19873.614509561437 |
+----------+--------------------+
1 row(s) fetched.
Elapsed 0.026 seconds.

+----------+-------------------+
| count(*) | sum(fact.v)       |
+----------+-------------------+
| 40000    | 19873.61450956144 |
+----------+-------------------+
1 row(s) fetched.
Elapsed 0.019 seconds.

+----------+-------------------+
| count(*) | sum(fact.v)       |
+----------+-------------------+
| 40000    | 19873.61450956144 |
+----------+-------------------+
1 row(s) fetched.
Elapsed 0.020 seconds.

+----------+-------------------+
| count(*) | sum(fact.v)       |
+----------+-------------------+
| 40000    | 19873.61450956144 |
+----------+-------------------+
1 row(s) fetched.
Elapsed 0.014 seconds.

> explain analyze select count(*), sum(v) from fact join dim on fact.k = dim.k;
...
|                   |           DataSourceExec: file_groups={12 groups: [[tmp/fact.parquet:0..1929814], [tmp/fact.parquet:1929814..3859628], [tmp/fact.parquet:3859628..5789442], [tmp/fact.parquet:5789442..7719256], [tmp/fact.parquet:7719256..9649070], ...]}, projection=[k, v], output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ k@0 >= 0 AND k@0 <= 1990199 AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=k_null_count@1 != row_count@2 AND k_max@0 >= 0 AND k_null_count@1 != row_count@2 AND k_min@3 <= 1990199 AND k_null_count@1 != row_count@2 AND IN_SET_INTERSECTS(k_min@3, k_max@0, 40000 values), required_guarantees=[], metrics=[output_rows=200.0 K, elapsed_compute=76.72µs, output_bytes=25.0 MB, output_batches=200, files_ranges_pruned_statistics=12 total → 12 matched, row_groups_pruned_statistics=2.00 K total → 200 matched, row_groups_pruned_bloom_filter=200 total → 200 matched, page_index_pages_pruned=200 total → 200 matched, page_index_rows_pruned=200.0 K total → 200.0 K matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, bytes_processed=22.1 MB, bytes_scanned=2.1 MB, file_open_errors=0, file_scan_errors=0, files_opened=12, files_processed=12, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_pruned_dynamic_filter=0, predicate_cache_inner_records=0, predicate_cache_records=0, bloom_filter_eval_time=64.73µs, metadata_load_time=2.78ms, page_index_eval_time=1.83ms, row_pushdown_eval_time=36ns, statistics_eval_time=773.07µs, time_elapsed_opening=77.42ms, time_elapsed_processing=91.60ms, time_elapsed_scanning_total=189.39ms, time_elapsed_scanning_until_data=11.29ms, output_rows_skew=1.63%, scan_efficiency_ratio=9.69% (2.24 M/23.16 M)] |
...

Note that the scanned rows are shrunk 10x (output_rows=1.99 M vs output_rows=200.0 K), and consequently the execution time is improved 5x (0.096 vs 0.019 seconds). It would be good to benchmark this more broadly.

Are there any user-facing changes?

Yes, the two new configs mirroring the in-list ones, as well as the construction API for HashTableLookupExpr, which now accepts an optional values arg too.

github-actions · 2026-09-14T12:16:21Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-common v55.1.0 (current)
       Built [  33.346s] (current)
     Parsing datafusion-common v55.1.0 (current)
      Parsed [   0.062s] (current)
    Building datafusion-common v55.1.0 (baseline)
       Built [  31.644s] (baseline)
     Parsing datafusion-common v55.1.0 (baseline)
      Parsed [   0.064s] (baseline)
    Checking datafusion-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.893s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field OptimizerOptions.hash_join_dynamic_pruning_max_distinct_values in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1584
  field OptimizerOptions.hash_join_dynamic_pruning_max_size in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1584

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  67.543s] datafusion-common
    Building datafusion-physical-plan v55.1.0 (current)
       Built [  36.931s] (current)
     Parsing datafusion-physical-plan v55.1.0 (current)
      Parsed [   0.154s] (current)
    Building datafusion-physical-plan v55.1.0 (baseline)
       Built [  36.549s] (baseline)
     Parsing datafusion-physical-plan v55.1.0 (baseline)
      Parsed [   0.151s] (baseline)
    Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.948s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure method_parameter_count_changed: pub method parameter count changed ---

Description:
A publicly-visible method now takes a different number of parameters, not counting the receiver (self) parameter.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/method_parameter_count_changed.ron

Failed in:
  datafusion_physical_plan::joins::HashTableLookupExpr::new takes 4 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/b13b0f0828ed599527f89b758aab6f584bd30321/datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:299, but now takes 5 parameters in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:350

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  75.861s] datafusion-physical-plan
    Building datafusion-proto v55.1.0 (current)
       Built [  54.109s] (current)
     Parsing datafusion-proto v55.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-proto v55.1.0 (baseline)
       Built [  52.215s] (baseline)
     Parsing datafusion-proto v55.1.0 (baseline)
      Parsed [   0.018s] (baseline)
    Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.134s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 107.673s] datafusion-proto
    Building datafusion-pruning v55.1.0 (current)
       Built [  38.393s] (current)
     Parsing datafusion-pruning v55.1.0 (current)
      Parsed [   0.015s] (current)
    Building datafusion-pruning v55.1.0 (baseline)
       Built [  37.264s] (baseline)
     Parsing datafusion-pruning v55.1.0 (baseline)
      Parsed [   0.016s] (baseline)
    Checking datafusion-pruning v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.097s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  76.803s] datafusion-pruning
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [  93.069s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [  91.934s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.026s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.112s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 187.978s] datafusion-sqllogictest

Exposes HashTableLookupExpr's single-column build values through the same compact sorted-domain rewrite ordinary large IN-lists already get (PrimitiveInListPruningExpr/StringInListPruningExpr), so row-group/file min/max stats alone can exclude containers - no bloom filter fetch, no LiteralGuarantee. On by default via hash_join_dynamic_pruning_max_distinct_values (default 100_000); set to 0 to disable. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

gruuya · 2026-09-14T13:30:05Z

run benchmark tpch

adriangbot · 2026-09-14T13:30:07Z

Hi @gruuya, thanks for the request (#25292 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: 2010YOUY01, Dandandan, Fokko, Jefffrey, Omega359, Rachelint, Rich-T-kid, adriangb, alamb, asubiotto, avantgardnerio, brunal, buraksenn, cetra3, codephage2020, coderfender, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jayzhan211, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mkleen, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas.

File an issue against this benchmark runner

codecov-commenter · 2026-09-14T13:30:12Z

Codecov Report

❌ Patch coverage is 88.93443% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (7b00b63) to head (e09baaa).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/pruning/src/pruning_predicate.rs	79.61%	15 Missing and 6 partials ⚠️
...tafusion/physical-plan/src/joins/hash_join/exec.rs	84.21%	1 Missing and 2 partials ⚠️
datafusion/common/src/scalar/mod.rs	95.83%	1 Missing ⚠️
...hysical-plan/src/joins/hash_join/inlist_builder.rs	96.42%	0 Missing and 1 partial ⚠️
...-plan/src/joins/hash_join/partitioned_hash_eval.rs	98.38%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #25292    +/-   ##
========================================
  Coverage   81.91%   81.91%            
========================================
  Files        1134     1134            
  Lines      425637   425866   +229     
  Branches   425637   425866   +229     
========================================
+ Hits       348654   348853   +199     
- Misses      56302    56323    +21     
- Partials    20681    20690     +9

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions Bot added documentation Improvements or additions to documentation common Related to common crate physical-plan Changes to the physical-plan crate labels Sep 14, 2026

github-actions Bot added the auto detected api change Auto detected API change label Sep 14, 2026

gruuya force-pushed the hash-join-dynamic-pruning-minmax branch from 7b174d8 to f116035 Compare September 14, 2026 12:17

gruuya changed the title ~~Prune hash join probe side via dynamic discreete values from the build side~~ Prune hash join probe side via dynamic discrete values from the build side Sep 14, 2026

gruuya force-pushed the hash-join-dynamic-pruning-minmax branch from f116035 to 542e11a Compare September 14, 2026 12:32

github-actions Bot added the proto Related to proto crate label Sep 14, 2026

gruuya force-pushed the hash-join-dynamic-pruning-minmax branch from 542e11a to 4b9a166 Compare September 14, 2026 12:58

github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Sep 14, 2026

gruuya force-pushed the hash-join-dynamic-pruning-minmax branch from 4b9a166 to e09baaa Compare September 14, 2026 13:04

gruuya mentioned this pull request Sep 14, 2026

Supress or compress the list of users that can trigger a benchmark adriangb/datafusion-benchmarking#33

Open

Navigation Menu

Uh oh!

Prune hash join probe side via dynamic discrete values from the build side - #25292

Prune hash join probe side via dynamic discrete values from the build side#25292
gruuya wants to merge 1 commit into
apache:mainfrom
splitgraph:hash-join-dynamic-pruning-minmax

gruuya commented Sep 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

gruuya commented Sep 14, 2026

Uh oh!

adriangbot commented Sep 14, 2026

Uh oh!

codecov-commenter commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

gruuya commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gruuya commented Sep 14, 2026

Uh oh!

adriangbot commented Sep 14, 2026

Uh oh!

codecov-commenter commented Sep 14, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gruuya commented Sep 14, 2026 •

edited

Loading

github-actions Bot commented Sep 14, 2026 •

edited

Loading