Skip to content

perf(parquet): decode levels directly into int16 batches - #1176

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-definition-level-int16
Open

perf(parquet): decode levels directly into int16 batches#1176
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-definition-level-int16

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What changed

  • decode RLE and bit-packed levels directly into []int16
  • count max definition levels during decoding
  • remove the intermediate uint64 level batch and its second conversion pass
  • add benchmarks for all-defined, mostly-defined, and alternating levels
  • check partial value counts when a literal run is truncated

Benchmarks

Apple M1 Pro, -benchtime=300ms -count=5:

RLE workload Before After Change
1,024 all defined 2,137 ns/op 1,495 ns/op -30%
1,024 mostly defined 5,740 ns/op 5,329 ns/op -7%
65,536 all defined 61,213 ns/op 24,601 ns/op -60%
65,536 alternating 120,994 ns/op 89,543 ns/op -26%

Allocations were unchanged.

@fallintoplace
fallintoplace marked this pull request as draft August 11, 2026 21:46
@fallintoplace
fallintoplace marked this pull request as ready for review August 18, 2026 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant