fix(parquet): don't preallocate 1MiB in DeltaBitPackEncoder by cetra3 · Pull Request #11149 · apache/arrow-rs

cetra3 · 2026-09-21T08:10:17Z

Which issue does this PR close?

Closes Increased Heap Usage with Fallback Writers & Wide Schemas #11148.

Rationale for this change

DeltaBitPackEncoder::new preallocates a 1MiB buffer for its bit writer. Byte array columns eagerly construct their fallback encoder, and the DELTA_BYTE_ARRAY fallback (the default for PARQUET_2_0) holds two DeltaBitPackEncoders, so every byte array column pays 2MiB of heap up front, even when it dictionary encodes and the fallback is never used. With 100 columns that is ~200MiB.

The preallocation only ever helped the first page: the buffer is a Vec that retains its capacity across clear().

What changes are included in this PR?

DeltaBitPackEncoder starts with an empty bit writer buffer that grows on demand (removes DEFAULT_BIT_WRITER_SIZE). This also benefits DELTA_BINARY_PACKED and DELTA_LENGTH_BYTE_ARRAY.
Regression test unused_delta_fallback_does_not_preallocate in parquet/tests/arrow_writer, using the existing peak heap tracking allocator. It fails on main (peak ≈ 210MB) and passes with this change.

Are these changes tested?

Yes, with the new regression test above. The existing lib and arrow_writer tests pass.

Benchmarks (criterion, --save-baseline on main vs this branch):

arrow_writer delta byte array:

benchmark	change
small_string_shared_prefix/delta_byte_array	−3.8%
small_string_partial_prefix/delta_byte_array	−6.6%
small_string_distinct/delta_byte_array	−15.6%
large_string_shared_prefix/delta_byte_array	−3.7%
large_string_distinct/delta_byte_array	−10.4%
large_string_shared_prefix_nullable/delta_byte_array	−6.2%
large_string_shared_prefix_nullable_dense/delta_byte_array	−15.4%
large_string_shared_prefix_nullable_trailing/delta_byte_array	−1.4%
large_string_distinct_nullable/delta_byte_array	−13.0%
medium_string_shared_prefix_nullable/delta_byte_array	−3.2%
large_string_shared_prefix_list/delta_byte_array	−8.1%
string/parquet_2 (dictionary overflow → delta fallback, 3 runs)	−3.6% to −4.4%

writer_overhead:

benchmark	change
1000_cols	no change
5000_cols	−2.6%
10000_cols	−2.7%
1000_cols/repeated_batches	−0.6% (noise)
5000_cols/repeated_batches	−14.4%

The other parquet_2 / zstd_parquet_2 writer benchmarks are within noise after re-running. The one exception is int32_ree_95pct_null/parquet_2, which measured +2.6% to +5.1% across runs. However, main measured against its own baseline drifted +0.9% to +2.7% on this benchmark, and it does not exercise the delta encoder, so the difference looks like noise.

Are there any user-facing changes?

No API changes. Lower peak memory when writing many byte array columns with PARQUET_2_0 / delta encodings.

🤖 Generated with Claude Code

…1148) `DeltaBitPackEncoder::new` preallocated a 1MiB bit writer buffer. Byte array columns eagerly construct their fallback encoder, and `DELTA_BYTE_ARRAY` holds two of these, so every column paid 2MiB of heap up front even if the fallback was never used. Start with an empty buffer instead; it grows on demand and retains its capacity across pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ReVtW5wMiSAmHnPrnDxhCk

cetra3 · 2026-09-21T08:12:40Z

run benchmark arrow_writer
env:
BENCH_FILTER: parquet_2|delta_byte_array

cetra3 · 2026-09-21T08:12:41Z

run benchmark writer_overhead

adriangbot · 2026-09-21T08:15:46Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5757403436-2544-7g99r 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix-delta-bitpack-prealloc (29b3b6e) to af18bac (merge-base) diff

Run configuration

run benchmark arrow_writer
env:
  BENCH_FILTER: "parquet_2|delta_byte_array"

BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-21T08:16:19Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5757403780-2545-k9r5v 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing fix-delta-bitpack-prealloc (29b3b6e) to af18bac (merge-base) diff

Run configuration

run benchmark writer_overhead

BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench writer_overhead
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-09-21T08:19:08Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing fix-delta-bitpack-prealloc (29b3b6e) to af18bac (merge-base) diff

Run configuration

run benchmark writer_overhead

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                         fix-delta-bitpack-prealloc             main
-----                                         --------------------------             ----
writer_overhead/10000_cols                    1.00     37.8±1.56ms        ? ?/sec    1.03     39.0±0.62ms        ? ?/sec
writer_overhead/1000_cols                     1.00      3.4±0.07ms        ? ?/sec    1.00      3.4±0.04ms        ? ?/sec
writer_overhead/1000_cols/repeated_batches    1.08     12.5±0.60ms        ? ?/sec    1.00     11.6±0.59ms        ? ?/sec
writer_overhead/5000_cols                     1.00     17.8±0.45ms        ? ?/sec    1.05     18.7±0.27ms        ? ?/sec
writer_overhead/5000_cols/repeated_batches    1.00     66.2±2.79ms        ? ?/sec    1.03     68.1±2.81ms        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	60.0s
Peak memory	56.5 MiB
Avg memory	40.3 MiB
CPU user	54.8s
CPU sys	0.0s
Peak spill	0 B

branch

Metric	Value
Wall time	55.0s
Peak memory	56.5 MiB
Avg memory	39.8 MiB
CPU user	51.0s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-09-21T08:56:10Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing fix-delta-bitpack-prealloc (29b3b6e) to af18bac (merge-base) diff

Run configuration

run benchmark arrow_writer
env:
  BENCH_FILTER: "parquet_2|delta_byte_array"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                            fix-delta-bitpack-prealloc             main
-----                                                            --------------------------             ----
bool/parquet_2                                                   1.04     11.2±0.05ms    22.3 MB/sec    1.00     10.8±0.03ms    23.1 MB/sec
bool/zstd_parquet_2                                              1.03     11.6±0.06ms    21.6 MB/sec    1.00     11.2±0.06ms    22.3 MB/sec
bool_non_null/parquet_2                                          1.02      4.7±0.01ms    26.4 MB/sec    1.00      4.6±0.02ms    26.9 MB/sec
bool_non_null/zstd_parquet_2                                     1.02      5.2±0.01ms    24.3 MB/sec    1.00      5.0±0.01ms    24.8 MB/sec
bool_ree/parquet_2                                               1.02     26.4±0.10ms     8.1 MB/sec    1.00     26.0±0.07ms     8.2 MB/sec
bool_ree/zstd_parquet_2                                          1.02     26.7±0.10ms     8.0 MB/sec    1.00     26.3±0.06ms     8.1 MB/sec
decimal/parquet_2                                                1.00    168.4±2.36ms   356.3 MB/sec    1.00    168.9±2.49ms   355.3 MB/sec
decimal/zstd_parquet_2                                           1.00    188.4±2.41ms   318.4 MB/sec    1.00    188.5±2.01ms   318.3 MB/sec
fixed_size_binary_ree/parquet_2                                  1.01     62.8±0.36ms    16.0 MB/sec    1.00     62.5±0.69ms    16.1 MB/sec
fixed_size_binary_ree/zstd_parquet_2                             1.02     64.3±0.70ms    15.7 MB/sec    1.00     63.3±0.67ms    15.9 MB/sec
float_with_nans/parquet_2                                        1.00     70.0±0.31ms   200.0 MB/sec    1.00     70.0±0.35ms   200.1 MB/sec
float_with_nans/zstd_parquet_2                                   1.01    107.2±0.46ms   130.5 MB/sec    1.00    106.6±0.34ms   131.3 MB/sec
fsb/parquet_2                                                    1.01     12.9±0.11ms  1249.6 MB/sec    1.00     12.8±0.09ms  1257.5 MB/sec
fsb/zstd_parquet_2                                               1.00     13.8±0.08ms  1171.0 MB/sec    1.00     13.8±0.12ms  1172.0 MB/sec
int32_ree/parquet_2                                              1.00     34.1±0.12ms    11.9 MB/sec    1.00     34.0±0.13ms    12.0 MB/sec
int32_ree/zstd_parquet_2                                         1.00     34.5±0.12ms    11.8 MB/sec    1.00     34.4±0.19ms    11.8 MB/sec
int32_ree_95pct_null/parquet_2                                   1.00     21.0±0.03ms    19.3 MB/sec    1.00     21.1±0.08ms    19.3 MB/sec
int32_ree_95pct_null/zstd_parquet_2                              1.00     21.2±0.06ms    19.2 MB/sec    1.00     21.3±0.08ms    19.1 MB/sec
large_string_distinct/delta_byte_array                           1.02     60.0±0.62ms     4.2 GB/sec    1.00     59.0±0.46ms     4.2 GB/sec
large_string_distinct_nullable/delta_byte_array                  1.00     56.1±0.39ms     4.5 GB/sec    1.00     55.9±0.52ms     4.5 GB/sec
large_string_non_null/parquet_2                                  1.01     51.7±0.10ms     4.8 GB/sec    1.00     51.4±0.10ms     4.9 GB/sec
large_string_non_null/zstd_parquet_2                             1.01     51.8±0.11ms     4.8 GB/sec    1.00     51.1±0.09ms     4.9 GB/sec
large_string_shared_prefix/delta_byte_array                      1.02     43.0±0.12ms     5.8 GB/sec    1.00     42.4±0.15ms     5.9 GB/sec
large_string_shared_prefix_list/delta_byte_array                 1.00     54.3±0.32ms     4.6 GB/sec    1.03     55.9±1.96ms     4.5 GB/sec
large_string_shared_prefix_nullable/delta_byte_array             1.01     40.5±0.15ms     6.2 GB/sec    1.00     39.9±0.12ms     6.3 GB/sec
large_string_shared_prefix_nullable_dense/delta_byte_array       1.01     21.9±0.22ms     5.7 GB/sec    1.00     21.6±0.08ms     5.8 GB/sec
large_string_shared_prefix_nullable_trailing/delta_byte_array    1.01     40.4±0.08ms     6.2 GB/sec    1.00     40.1±0.22ms     6.2 GB/sec
list_nested/parquet_2                                            1.00    128.9±0.37ms   223.8 MB/sec    1.00    129.2±0.34ms   223.3 MB/sec
list_nested/zstd_parquet_2                                       1.00    141.6±0.35ms   203.7 MB/sec    1.00    142.1±0.44ms   203.1 MB/sec
list_primitive/parquet_2                                         1.01    202.8±0.33ms     2.6 GB/sec    1.00    201.5±0.32ms     2.6 GB/sec
list_primitive/zstd_parquet_2                                    1.00    425.9±0.71ms  1280.6 MB/sec    1.00    424.3±0.31ms  1285.5 MB/sec
list_primitive_non_null/parquet_2                                1.00   222.8±10.65ms     2.4 GB/sec    1.17    261.0±0.88ms     2.0 GB/sec
list_primitive_non_null/zstd_parquet_2                           1.00   621.8±13.76ms   875.2 MB/sec    1.03    642.5±7.41ms   847.1 MB/sec
list_primitive_sparse_99pct_null/parquet_2                       1.00     12.3±0.05ms     3.0 GB/sec    1.00     12.4±0.04ms     3.0 GB/sec
list_primitive_sparse_99pct_null/zstd_parquet_2                  1.00     12.5±0.03ms     2.9 GB/sec    1.00     12.5±0.04ms     2.9 GB/sec
list_struct_with_list/parquet_2                                  1.00    244.4±0.54ms   186.6 MB/sec    1.00    243.6±0.69ms   187.1 MB/sec
list_struct_with_list/zstd_parquet_2                             1.00    267.6±0.55ms   170.4 MB/sec    1.00    267.1±0.90ms   170.7 MB/sec
medium_string_shared_prefix_nullable/delta_byte_array            1.01     35.3±0.06ms     7.1 GB/sec    1.00     35.0±0.07ms     7.2 GB/sec
primitive/parquet_2                                              1.01    116.5±0.61ms   385.0 MB/sec    1.00    115.1±0.81ms   389.9 MB/sec
primitive/zstd_parquet_2                                         1.01    149.0±0.78ms   301.2 MB/sec    1.00    147.4±0.31ms   304.4 MB/sec
primitive_all_null/parquet_2                                     1.00    273.2±2.34µs   160.4 GB/sec    1.01    276.3±2.22µs   158.6 GB/sec
primitive_all_null/zstd_parquet_2                                1.00    366.2±2.37µs   119.7 GB/sec    1.02    372.0±2.14µs   117.8 GB/sec
primitive_non_null/parquet_2                                     1.00     71.2±0.26ms   617.7 MB/sec    1.00     71.2±0.24ms   618.1 MB/sec
primitive_non_null/zstd_parquet_2                                1.00    104.5±0.28ms   421.2 MB/sec    1.00    104.7±0.60ms   420.1 MB/sec
primitive_sparse_99pct_null/parquet_2                            1.00     10.5±0.05ms     4.2 GB/sec    1.00     10.5±0.05ms     4.2 GB/sec
primitive_sparse_99pct_null/zstd_parquet_2                       1.00     12.5±0.14ms     3.5 GB/sec    1.00     12.5±0.10ms     3.5 GB/sec
short_string_non_null/parquet_2                                  1.00     24.4±0.15ms   491.7 MB/sec    1.00     24.4±0.07ms   491.3 MB/sec
short_string_non_null/zstd_parquet_2                             1.00     27.2±0.04ms   440.7 MB/sec    1.00     27.4±0.10ms   438.7 MB/sec
small_string_distinct/delta_byte_array                           1.00    963.2±8.04µs     8.1 GB/sec    1.00   958.9±10.66µs     8.2 GB/sec
small_string_partial_prefix/delta_byte_array                     1.02   1006.7±6.88µs     7.8 GB/sec    1.00    990.0±3.41µs     7.9 GB/sec
small_string_shared_prefix/delta_byte_array                      1.01    977.2±1.88µs     8.0 GB/sec    1.00    963.9±1.59µs     8.1 GB/sec
string/parquet_2                                                 1.00     98.3±5.71ms     5.2 GB/sec    1.77    173.9±0.60ms     2.9 GB/sec
string/zstd_parquet_2                                            1.00    385.1±6.05ms  1361.5 MB/sec    1.01    387.4±6.95ms  1353.1 MB/sec
string_and_binary_view/parquet_2                                 1.01     58.0±0.25ms   556.2 MB/sec    1.00     57.4±0.11ms   561.6 MB/sec
string_and_binary_view/zstd_parquet_2                            1.00     71.6±0.13ms   450.4 MB/sec    1.00     71.4±0.14ms   451.6 MB/sec
string_dictionary/parquet_2                                      1.01     47.2±0.15ms     5.5 GB/sec    1.00     46.6±0.18ms     5.5 GB/sec
string_dictionary/zstd_parquet_2                                 1.00    192.4±0.63ms  1372.9 MB/sec    1.00    192.0±2.89ms  1375.5 MB/sec
string_dictionary_low_cardinality_100/parquet_2                  1.00     21.4±0.05ms   186.8 MB/sec    1.00     21.5±0.05ms   186.0 MB/sec
string_dictionary_low_cardinality_100/zstd_parquet_2             1.00     21.8±0.04ms   184.1 MB/sec    1.00     21.9±0.04ms   183.2 MB/sec
string_dictionary_low_cardinality_20/parquet_2                   1.00     21.7±0.06ms   184.8 MB/sec    1.00     21.7±0.06ms   184.2 MB/sec
string_dictionary_low_cardinality_20/zstd_parquet_2              1.00     21.8±0.07ms   183.5 MB/sec    1.00     21.8±0.06ms   183.3 MB/sec
string_dictionary_low_cardinality_400/parquet_2                  1.00     22.1±0.11ms   181.9 MB/sec    1.00     22.0±0.15ms   182.5 MB/sec
string_dictionary_low_cardinality_400/zstd_parquet_2             1.00     22.4±0.10ms   178.8 MB/sec    1.00     22.4±0.09ms   179.0 MB/sec
string_non_null/parquet_2                                        1.00    108.3±9.52ms     4.7 GB/sec    1.08   117.1±10.71ms     4.4 GB/sec
string_non_null/zstd_parquet_2                                   1.01    491.5±5.30ms  1066.2 MB/sec    1.00    487.9±6.88ms  1074.1 MB/sec
string_ree/parquet_2                                             1.02    124.9±0.47ms   131.3 MB/sec    1.00    122.8±0.42ms   133.6 MB/sec
string_ree/zstd_parquet_2                                        1.03    141.3±1.03ms   116.1 MB/sec    1.00    137.7±0.63ms   119.1 MB/sec
string_ree_95pct_null/parquet_2                                  1.02     24.5±0.23ms    57.3 MB/sec    1.00     24.2±0.12ms    58.2 MB/sec
string_ree_95pct_null/zstd_parquet_2                             1.00     25.1±0.18ms    56.1 MB/sec    1.00     25.0±0.12ms    56.2 MB/sec
struct_all_null/parquet_2                                        1.00    117.4±1.13µs   134.1 GB/sec    1.02    120.0±0.98µs   131.2 GB/sec
struct_all_null/zstd_parquet_2                                   1.00    158.3±0.96µs    99.5 GB/sec    1.01    160.6±0.88µs    98.1 GB/sec
struct_non_null/parquet_2                                        1.00     32.7±0.07ms   489.7 MB/sec    1.00     32.6±0.16ms   491.5 MB/sec
struct_non_null/zstd_parquet_2                                   1.00     46.4±0.07ms   344.7 MB/sec    1.00     46.3±0.17ms   345.5 MB/sec
struct_sparse_99pct_null/parquet_2                               1.00      5.9±0.02ms     2.7 GB/sec    1.03      6.1±0.11ms     2.6 GB/sec
struct_sparse_99pct_null/zstd_parquet_2                          1.00      6.7±0.03ms     2.4 GB/sec    1.00      6.6±0.03ms     2.4 GB/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	1180.3s
Peak memory	2.8 GiB
Avg memory	2.5 GiB
CPU user	1135.2s
CPU sys	38.0s
Peak spill	0 B

branch

Metric	Value
Wall time	1165.3s
Peak memory	2.9 GiB
Avg memory	2.6 GiB
CPU user	1139.4s
CPU sys	20.0s
Peak spill	0 B

File an issue against this benchmark runner

github-actions Bot added the parquet Changes to the parquet crate label Sep 21, 2026

Navigation Menu

fix(parquet): don't preallocate 1MiB in DeltaBitPackEncoder - #11149

fix(parquet): don't preallocate 1MiB in DeltaBitPackEncoder#11149
cetra3 wants to merge 1 commit into
apache:mainfrom
pydantic:fix-delta-bitpack-prealloc

cetra3 commented Sep 21, 2026

Uh oh!

cetra3 commented Sep 21, 2026

Uh oh!

cetra3 commented Sep 21, 2026

Uh oh!

adriangbot commented Sep 21, 2026

Uh oh!

adriangbot commented Sep 21, 2026

Uh oh!

adriangbot commented Sep 21, 2026

Uh oh!

adriangbot commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cetra3 commented Sep 21, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

cetra3 commented Sep 21, 2026

Uh oh!

cetra3 commented Sep 21, 2026

Uh oh!

adriangbot commented Sep 21, 2026

Uh oh!

adriangbot commented Sep 21, 2026

Uh oh!

adriangbot commented Sep 21, 2026

Uh oh!

adriangbot commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants