Repository navigation
Reuse dictionary hashes in repartition and aggregation - #27032
Closed
Rich-T-kid wants to merge 5 commits into
Closed
Rich-T-kid wants to merge 5 commits into
Rich-T-kid wants to merge 5 commits into
Conversation
Reuse dictionary value hashes by column while the values Arc is unchanged, without retaining input dictionaries. Expose cache state for callers that account for memory and document the public hashing contract.
For a single non-streaming dictionary group key, the outer table already guarantees that appended values are distinct groups. Reuse the dictionary column's cached value-to-slot mapping as group indices so later batches can skip hashing and probing.
Use the caching hasher only for non-streaming group schemas that contain dictionary columns, and share cached hashes only with those columns. Streaming and non-dictionary aggregates keep the existing hashing path without per-batch cache plumbing.
github-actions
Bot
added
common
Contributor
Author
|
run benchmark tpcds env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1" |
Contributor
Author
|
run benchmark tpcds env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
baseline:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1" |
Contributor
Author
|
run benchmark tpcds env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
Contributor
Author
|
run benchmark tpcds env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true" |
Contributor
Author
|
show benchmark queue |
|
Hi @Rich-T-kid, you asked to view the benchmark queue (#27032 (comment)).
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f021807 (f021807) to f021807 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f021807 (f021807) to 1635ed9 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 1635ed9 (1635ed9) to 1635ed9 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f021807 (f021807) to f021807 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f021807 (f021807) to 1635ed9 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing f021807 (f021807) to f021807 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing f021807 (f021807) to f021807 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "f02180725bc518d0b8bce2af28706bd3a66009f1"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 1635ed9 (1635ed9) to 1635ed9 diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
ref: "1635ed94742c1c060852bc34dc6ebb2302241c7b"
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Contributor
Author
|
okay benchmark results are showing a large improvement from when we first created the flag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This stacks #26816 and #26826 so their combined effect can be measured with Parquet RLE dictionary reads enabled.
What changed
CachingArrayHasherused by both operators.RepartitionExecbatches while the valuesArcis unchanged.Validation
Focused caching-hasher, repartition, and aggregation tests pass. Formatting, scoped Clippy, and the repository pre-push checks pass.
Hosted TPC-DS SF1 results with a 32 GiB limit:
main RLE off vs on · stack RLE off vs on · main vs stack with RLE on · same-head control
This is an integration draft for #26816 and #26826; it does not supersede either PR yet.