perf(arrow/array): append fixed-width nulls in bulk - #1173
Open
fallintoplace wants to merge 4 commits into
Open
Conversation
fallintoplace
force-pushed
the
perf/arrow-fixed-width-append-nulls
branch
from
August 11, 2026 21:27
95fa3a5 to
e52e43e
Compare
fallintoplace
marked this pull request as ready for review
August 12, 2026 18:41
fallintoplace
marked this pull request as draft
August 12, 2026 18:53
fallintoplace
marked this pull request as ready for review
August 12, 2026 20:51
zeroshade
pushed a commit
that referenced
this pull request
Aug 14, 2026
…#1183) ## What does this PR do? - reserves fixed-size list parent slots once in `AppendNulls` and `AppendEmptyValues` - updates the parent validity bitmap in bulk - appends all child null or empty values with one bulk builder call The child append count is `listSize * n`, so the parent and child lengths stay in sync. ## Why? Both methods currently loop over every parent value. Each parent append also loops over every child value. This removes the parent loop and lets child builders use their bulk append paths. The change is independent, but it also composes with #1173 and #1180. ## Benchmarks Apple M1 Pro, 65,536 `FixedSizeList<Int32>` rows, `GOMAXPROCS=1`. Standalone on `main`: | list width | nulls before | nulls after | change | empty before | empty after | change | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 4 | 1.10 ms | 0.86 ms | -22% | 1.55 ms | 1.07 ms | -31% | | 16 | 3.76 ms | 2.74 ms | -27% | 5.43 ms | 3.90 ms | -28% | | 64 | 13.03 ms | 11.07 ms | -15% | 19.35 ms | 14.86 ms | -23% | With #1173 and #1180 applied underneath: | list width | nulls before | nulls after | change | empty before | empty after | change | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 4 | 1.56 ms | 0.18 ms | -88% | 2.01 ms | 0.20 ms | -90% | | 16 | 4.69 ms | 0.33 ms | -93% | 6.89 ms | 0.42 ms | -94% | | 64 | 16.27 ms | 0.74 ms | -95% | 22.18 ms | 0.92 ms | -96% | ```text GOMAXPROCS=1 go test ./arrow/array -run '^$' -bench '^BenchmarkFixedSizeListBuilderBulkAppend$/(nulls|empty)/rows=65536/width=(4|16|64)$' -benchmem -benchtime=300ms -count=5 ``` ## Tests - added coverage for unaligned parent validity bits - verifies parent validity and null counts - verifies child length, null slots, and zero-valued empty slots - ran `go test ./arrow/...` - ran `go test -race ./arrow/array` - ran `go vet ./arrow/array`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Fixed-width builders currently implement
AppendNullsby callingAppendNullfor every value. Large batches repeatedly reserve capacity, update the validity bitmap, and update builder counters.What changes are included in this PR?
This only changes
AppendNulls. Empty-value appends are separate because their value bytes must also be zeroed.Apple M1 Pro results for 65,536 nulls with
-cpu=1:Are these changes tested?
Yes.
go test ./arrow/...go test -race ./arrow/arraygo vet -composites=false ./arrow/arrayAre there any user-facing changes?
No.