Repository navigation
perf: avoid rebuilding dictionary keys during group emission - #27051
Rich-T-kid wants to merge 3 commits into
Conversation
|
run benchmark tpcds env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "939ae6c37e6326f2737600e4d60ef2d3673f119a" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 939ae6c (939ae6c) to 32bd81f diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "939ae6c37e6326f2737600e4d60ef2d3673f119a"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 939ae6c (939ae6c) to 32bd81f diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "939ae6c37e6326f2737600e4d60ef2d3673f119a"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
939ae6c to
4653896
Compare
|
run benchmark dictionary_group_values baseline: |
2 similar comments
|
run benchmark dictionary_group_values baseline: |
|
run benchmark dictionary_group_values baseline: |
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#27051 (comment)).
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 4653896 (4653896) to 32bd81f diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "4653896c9a2174492c86651ae5cf28172795f28b"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 4653896 (4653896) to 32bd81f diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "4653896c9a2174492c86651ae5cf28172795f28b"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 4653896 (4653896) to 32bd81f diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "4653896c9a2174492c86651ae5cf28172795f28b"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 4653896 (4653896) to 32bd81f diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "4653896c9a2174492c86651ae5cf28172795f28b"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 4653896 (4653896) to 32bd81f diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "4653896c9a2174492c86651ae5cf28172795f28b"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 4653896 (4653896) to 32bd81f diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "32bd81fb226ccf1ae5df67ff340d77baa3e73668"
changed:
ref: "4653896c9a2174492c86651ae5cf28172795f28b"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
4653896 to
b282594
Compare
|
run benchmark dictionary_group_values baseline: |
2 similar comments
|
run benchmark dictionary_group_values baseline: |
|
run benchmark dictionary_group_values baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing b282594 (b282594) to 522287c diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "522287c20d3c09c1b2bd2a8a39bc3a164257614e"
changed:
ref: "b2825948e531b1afe34769f29909359216175b50"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing b282594 (b282594) to 522287c diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "522287c20d3c09c1b2bd2a8a39bc3a164257614e"
changed:
ref: "b2825948e531b1afe34769f29909359216175b50"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing b282594 (b282594) to 522287c diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "522287c20d3c09c1b2bd2a8a39bc3a164257614e"
changed:
ref: "b2825948e531b1afe34769f29909359216175b50"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing b282594 (b282594) to 522287c diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "522287c20d3c09c1b2bd2a8a39bc3a164257614e"
changed:
ref: "b2825948e531b1afe34769f29909359216175b50"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing b282594 (b282594) to 522287c diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "522287c20d3c09c1b2bd2a8a39bc3a164257614e"
changed:
ref: "b2825948e531b1afe34769f29909359216175b50"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing b282594 (b282594) to 522287c diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "522287c20d3c09c1b2bd2a8a39bc3a164257614e"
changed:
ref: "b2825948e531b1afe34769f29909359216175b50"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
b282594 to
5ee33fb
Compare
|
run benchmark dictionary_group_values baseline: |
1 similar comment
|
run benchmark dictionary_group_values baseline: |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
run benchmark tpch env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6" |
|
run benchmark tpcds env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6" |
|
run benchmark clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing d0d542b (d0d542b) to ea33cce diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Every group key comes from an already validated inner slot. Check the key width when allocating that slot so later key writes stay infallible and out of the per-row append path.
|
run benchmark dictionary_group_values baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d" |
|
run benchmark tpch env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d" |
|
run benchmark tpcds env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d" |
|
run benchmark clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 9d3137d (9d3137d) to ea33cce diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 9d3137d (9d3137d) to ea33cce diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 9d3137d (9d3137d) to ea33cce diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 9d3137d (9d3137d) to ea33cce diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 9d3137d (9d3137d) to ea33cce diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 9d3137d (9d3137d) to ea33cce diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 9d3137d (9d3137d) to ea33cce diff Run configurationrun benchmark dictionary_group_values
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"CPU Details (lscpu)Details
Resource Usagedictionary_group_values — base (merge-base)
dictionary_group_values — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 9d3137d (9d3137d) to ea33cce diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "ea33cce29c27ffb583e613d4a5900e9614be0fa8"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
run benchmark clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 9d3137d (9d3137d) to d0d542b diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 9d3137d (9d3137d) to d0d542b diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "d0d542bfd3f87bf266e7141b432bce2200066ae6"
changed:
ref: "9d3137dc3fcc4fbf7d0dccdbc506553a2ffb702d"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Why
DictionaryGroupValuesColumnstores each group as an index into its deduplicated values. Building the output dictionary currently walks every group again to convert those indices to dictionary keys. With nulls, it also removes the null value withtakeand remaps the keys.This makes group emission relatively expensive when Parquet RLE values stay dictionary encoded.
What changed
The group column now stores each inner slot directly in the dictionary's native key type. Nulls use an all-ones placeholder plus a validity bit, so they no longer occupy a slot in the values array. Cached equality compares native keys directly and avoids null checks for non-null batches.
Full emission moves the existing keys, validity, and values directly into the result. Partial emission still compacts the referenced values and remaps the remaining groups. Non-null batches append the validity in one operation after collecting their keys.
On 64-bit platforms an
Int32dictionary with nulls reduces the per-group mapping from an 8-byteusizeto a 4-byte key and one validity bit. Without nulls, the validity buffer remains unallocated.Results
The final rebased head was measured with the existing
dictionary_group_valuesbenchmark in three upstream runs: run 1, run 2, and run 3. High-cardinality repeated batches improve by 14%, scalar append/equality by 8–15%, and high-cardinality partial emission is flat to 7% faster. No case regressed consistently across the three runs.This change is complementary to the dictionary hash and group-mapping reuse in #26826. A current-base stack of both changes was measured in four benchmark groups:
dictionary_group_valuesThe broad runs enable
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARYon both sides.