perf: speed up non-null dictionary hash scatter by Rich-T-kid · Pull Request #27025 · apache/datafusion

Rich-T-kid · 2026-11-03T00:40:46Z

Which issue does this PR close?

Part of #26848. This is the non-null path; #26966 handles nullable keys.

Why

For a dictionary with non-null keys and values, scattering hashes overwrites every output row. with_hashes was still zeroing its reusable 8,192-row buffer first, and the scatter loop loaded and stored each row before starting the next key-indexed load.

What changed

with_hashes now keeps the initialized buffer contents when the first column is a non-null dictionary, since that column replaces every slot. The non-null scatter loop gathers eight dictionary hashes before updating the output, which gives the CPU independent loads to overlap.

Nullable keys or values continue through the existing path. Short hash buffers also keep the existing truncation behavior.

Results

Run 1 · Run 2

Skip clearing the reusable hash buffer when a dictionary with non-null keys and values will overwrite every row. Gather dictionary hashes in independent lane groups so the CPU can overlap the key-indexed loads.

Rich-T-kid · 2026-11-03T00:56:29Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

adriangbot · 2026-11-03T00:59:13Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6592499441-4263-ftkpf 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-03T01:02:29Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.02µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	95.0s
Peak memory	31.5 MiB
Avg memory	5.1 MiB
CPU user	23.7s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	90.0s
Peak memory	29.4 MiB
Avg memory	5.0 MiB
CPU user	23.5s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-11-03T01:52:34Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

adriangbot · 2026-11-03T01:52:37Z

Hi @Rich-T-kid, your benchmark configuration could not be parsed (#27025 (comment)).

Error: invalid configuration: unknown field BENCH_FILTER, expected one of env, baseline, changed, resources, shards at line 2 column 1

Usage:

run benchmark <name>           # run specific benchmark(s)
run benchmarks                 # run default suite
run benchmarks <name1> <name2> # run specific benchmarks

Any benchmark name is accepted: bench.sh suite names (e.g. tpch, clickbench_partitioned, wide_schema) and Criterion bench targets (e.g. sql_planner) are resolved automatically. A name that matches neither fails on the runner.

Per-side configuration (run benchmark tpch followed by):

env:
# shared env is inherited by BOTH the build and the run, so build
# flags go here. Builds default to no debuginfo for speed; opt back
# in for hung-job gdb dumps and cap jobs to stay within memory:
CARGO_PROFILE_RELEASE_DEBUG: "1"
CARGO_BUILD_JOBS: "1"
baseline:
ref: v45.0.0
env:
# per-side env only reaches the benchmark run, not the build
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G

Pod size and CPU architecture (each key is optional and keeps the default when absent; requests and limits are set equal):

resources:
cpu: "16"
memory: "128Gi"
arch: arm64          # arm64 or amd64

Independent benchmark shards (optional; 1–8, default 1):

run benchmark arrow_writer
shards: 4

Each shard builds both revisions, measures its assigned cases, and posts its own report. Resources and the job deadline apply per shard. DataFusion requests specifying a shard count greater than 1 fail with an error. Omit shards or use shards: 1 for DataFusion.

File an issue against this benchmark runner

Rich-T-kid · 2026-11-03T01:53:58Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

adriangbot · 2026-11-03T01:55:05Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6593384459-4264-fb8ds 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-03T01:57:21Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (8c0a4ee) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.15      3.3±0.02µs        ? ?/sec    1.00      2.8±0.06µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	65.0s
Peak memory	35.3 MiB
Avg memory	7.0 MiB
CPU user	23.4s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	60.0s
Peak memory	29.9 MiB
Avg memory	7.4 MiB
CPU user	23.5s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Existing hash utility tests cover dictionary hashing; retain a concise comment explaining the lane-gather fast path.

Rich-T-kid · 2026-11-03T03:49:08Z

@kumarUjjawal could you take a look 🙇

Rich-T-kid · 2026-11-03T03:49:30Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Rich-T-kid · 2026-11-03T03:49:31Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Rich-T-kid · 2026-11-03T03:49:33Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

kumarUjjawal

Thank you @Rich-T-kid

Left few comments please take a look.

adriangbot · 2026-11-03T04:13:00Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6594963661-4285-fbvqw 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (cb54f47) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete

File an issue against this benchmark runner

Keep the buffer-reuse and scatter dispatch checks in one place, validate the unchecked lookup invariant, and cover chunk, tail, multi-column, and buffer-reuse behavior.

adriangbot · 2026-11-03T04:15:41Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (cb54f47) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	65.0s
Peak memory	29.8 MiB
Avg memory	6.7 MiB
CPU user	21.5s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	85.0s
Peak memory	29.3 MiB
Avg memory	5.4 MiB
CPU user	23.6s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-11-03T04:18:22Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

adriangbot · 2026-11-03T04:20:10Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6594963493-4286-5r52q 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-03T04:20:12Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6594964058-4287-75g47 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete

File an issue against this benchmark runner

Rich-T-kid · 2026-11-03T04:20:36Z

thank you for the in depth review @kumarUjjawal, I addressed all your comments! added the test back in as well if you could take another look.

re-running the benchmakrs as well

adriangbot · 2026-11-03T04:22:46Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.00µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	65.0s
Peak memory	29.2 MiB
Avg memory	6.6 MiB
CPU user	21.5s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	80.0s
Peak memory	34.4 MiB
Avg memory	5.6 MiB
CPU user	25.8s
CPU sys	0.1s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-11-03T04:23:08Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.09     10.9±0.02µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	75.0s
Peak memory	29.0 MiB
Avg memory	6.1 MiB
CPU user	22.1s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	90.0s
Peak memory	29.3 MiB
Avg memory	4.9 MiB
CPU user	23.3s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-11-03T04:34:52Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6595334800-4301-245tt 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-03T04:37:08Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (c7d9b3a) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	65.0s
Peak memory	36.6 MiB
Avg memory	7.0 MiB
CPU user	24.8s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	60.0s
Peak memory	29.2 MiB
Avg memory	7.3 MiB
CPU user	23.5s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

kumarUjjawal · 2026-11-03T04:44:47Z


-        // Ensure buffer has sufficient length, clearing old values
-        buffer.clear();
+        // A dictionary with non-null keys and values overwrites every output


Skipping buffer.clear() only saves zeroing at most required_size u64s (about 64KB at batch size 8192), yet it adds a correctness coupling between the shared buffer code and one array type's scatter.

Direct A/B: restoring buffer.clear() was 1.04x slower for multiple dictionaries (10.4 vs. 10.0 µs) and 1.16x slower for a single dictionary (3.2 vs. 2.8 µs). I think that win is worth keeping now that the overwrite invariant is covered by the debug checks. Kept the skip in 90f551c9c. Run.

Use debug assertions for internal scatter invariants, avoid repeated dictionary downcasts when deciding whether to reuse the hash buffer, and document why slices and key widths share the same path.

Restore buffer initialization so the hosted benchmark can isolate the benefit of skipping it from the dictionary scatter changes.

Rich-T-kid · 2026-11-03T14:20:07Z

Restoring `buffer.clear()` to measure its impact

Direct A/B: 0a7c4da0c (skip clear) vs. b41d408fd (clear restored).

Rich-T-kid · 2026-11-03T14:20:09Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
baseline:
ref: 0a7c4da
changed:
ref: b41d408

adriangbot · 2026-11-03T14:24:02Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6606243064-4305-xk62z 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing b41d408 (b41d408) to 0a7c4da diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
baseline:
  ref: "0a7c4da0cdcea319338b4150a28108ad9c0fffad"
changed:
  ref: "b41d408fd729ac0c7397bdc0ac5118063ca534ec"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-03T14:27:19Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing b41d408 (b41d408) to 0a7c4da diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"
baseline:
  ref: "0a7c4da0cdcea319338b4150a28108ad9c0fffad"
changed:
  ref: "b41d408fd729ac0c7397bdc0ac5118063ca534ec"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.00     10.0±0.03µs        ? ?/sec    1.04     10.4±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.00      2.8±0.00µs        ? ?/sec    1.16      3.2±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	105.0s
Peak memory	31.7 MiB
Avg memory	4.5 MiB
CPU user	26.1s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	80.0s
Peak memory	27.8 MiB
Avg memory	5.1 MiB
CPU user	20.6s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

Restoring buffer clearing made the focused dictionary benchmarks 1.04x and 1.16x slower, so retain the guarded reuse path.

Rich-T-kid · 2026-11-03T19:34:42Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Rich-T-kid · 2026-11-03T19:34:47Z

run benchmark with_hashes
env:
BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

adriangbot · 2026-11-03T19:36:44Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6613579983-4344-vlrdt 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-03T19:38:17Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6613581773-4345-7l2mw 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-11-03T19:39:31Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.09     10.8±0.02µs        ? ?/sec    1.00      9.9±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.00µs        ? ?/sec    1.00      2.8±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	70.0s
Peak memory	29.6 MiB
Avg memory	6.3 MiB
CPU user	21.2s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	85.0s
Peak memory	29.3 MiB
Avg memory	5.4 MiB
CPU user	23.6s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-11-03T19:40:43Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/non-null-dictionary-hash-scatter (90f551c) to 198ab43 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dictionary_utf8_int32.*no nulls"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_non-null-dictionary-hash-scatter
-----                                        ----                                   -------------------------------------------
dictionary_utf8_int32: multiple, no nulls    1.08     10.8±0.01µs        ? ?/sec    1.00     10.0±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.16      3.2±0.01µs        ? ?/sec    1.00      2.8±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	70.0s
Peak memory	35.5 MiB
Avg memory	6.6 MiB
CPU user	23.4s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	65.0s
Peak memory	31.9 MiB
Avg memory	7.1 MiB
CPU user	24.9s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

perf: speed up non-null dictionary hash scatter

8c0a4ee

Skip clearing the reusable hash buffer when a dictionary with non-null keys and values will overwrite every row. Gather dictionary hashes in independent lane groups so the CPU can overlap the key-indexed loads.

github-actions Bot added the common Related to common crate label Nov 3, 2026

Remove redundant dictionary scatter test

cb54f47

Existing hash utility tests cover dictionary hashing; retain a concise comment explaining the lane-gather fast path.

kumarUjjawal reviewed Nov 3, 2026

View reviewed changes

Comment thread datafusion/common/src/hash_utils.rs Resolved

fix: tighten non-null dictionary hash scatter

c7d9b3a

Keep the buffer-reuse and scatter dispatch checks in one place, validate the unchecked lookup invariant, and cover chunk, tail, multi-column, and buffer-reuse behavior.

adriangbot mentioned this pull request Nov 3, 2026

Reuse dictionary hashes in repartition and aggregation #27032

Closed

kumarUjjawal reviewed Nov 3, 2026

View reviewed changes

Rich-T-kid added 2 commits November 3, 2026 09:01

fix: tighten dictionary scatter invariants

0a7c4da

Use debug assertions for internal scatter invariants, avoid repeated dictionary downcasts when deciding whether to reuse the hash buffer, and document why slices and key widths share the same path.

bench: restore hash buffer clearing

b41d408

Restore buffer initialization so the hosted benchmark can isolate the benefit of skipping it from the dictionary scatter changes.

perf: keep hash buffer reuse

90f551c

Restoring buffer clearing made the focused dictionary benchmarks 1.04x and 1.16x slower, so retain the guarded reuse path.

Conversation

Rich-T-kid commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Why

What changed

Results

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

adriangbot commented Nov 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kumarUjjawal Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

Rich-T-kid Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Rich-T-kid commented Nov 3, 2026

Restoring buffer.clear() to measure its impact

Uh oh!

Rich-T-kid commented Nov 3, 2026

Uh oh!

Rich-T-kid commented Nov 3, 2026 •

edited

Loading

Restoring `buffer.clear()` to measure its impact