Skip to content

perf(arrow/array): bulk append fixed-size list nulls and empty values - #1183

Merged
zeroshade merged 2 commits into
apache:mainfrom
fallintoplace:perf/arrow-fixed-size-list-bulk-appends
Aug 14, 2026
Merged

perf(arrow/array): bulk append fixed-size list nulls and empty values#1183
zeroshade merged 2 commits into
apache:mainfrom
fallintoplace:perf/arrow-fixed-size-list-bulk-appends

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What does this PR do?

  • reserves fixed-size list parent slots once in AppendNulls and AppendEmptyValues
  • updates the parent validity bitmap in bulk
  • appends all child null or empty values with one bulk builder call

The child append count is listSize * n, so the parent and child lengths stay in sync.

Why?

Both methods currently loop over every parent value. Each parent append also loops over every child value.

This removes the parent loop and lets child builders use their bulk append paths. The change is independent, but it also composes with #1173 and #1180.

Benchmarks

Apple M1 Pro, 65,536 FixedSizeList<Int32> rows, GOMAXPROCS=1.

Standalone on main:

list width nulls before nulls after change empty before empty after change
4 1.10 ms 0.86 ms -22% 1.55 ms 1.07 ms -31%
16 3.76 ms 2.74 ms -27% 5.43 ms 3.90 ms -28%
64 13.03 ms 11.07 ms -15% 19.35 ms 14.86 ms -23%

With #1173 and #1180 applied underneath:

list width nulls before nulls after change empty before empty after change
4 1.56 ms 0.18 ms -88% 2.01 ms 0.20 ms -90%
16 4.69 ms 0.33 ms -93% 6.89 ms 0.42 ms -94%
64 16.27 ms 0.74 ms -95% 22.18 ms 0.92 ms -96%
GOMAXPROCS=1 go test ./arrow/array -run '^$' -bench '^BenchmarkFixedSizeListBuilderBulkAppend$/(nulls|empty)/rows=65536/width=(4|16|64)$' -benchmem -benchtime=300ms -count=5

Tests

  • added coverage for unaligned parent validity bits
  • verifies parent validity and null counts
  • verifies child length, null slots, and zero-valued empty slots
  • ran go test ./arrow/...
  • ran go test -race ./arrow/array
  • ran go vet ./arrow/array

@fallintoplace
fallintoplace marked this pull request as draft August 12, 2026 12:18
@fallintoplace
fallintoplace marked this pull request as ready for review August 12, 2026 20:34

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parent/child length synchronization is right in both paths, and delegating to the child builders' bulk appends is the correct layering — this composes cleanly with #1173/#1180. The nested ValidateFull test and slot-boundary assertions cover the tricky cases.


This review was drafted with the help of an AI-assisted tool and may contain mistakes; it was reviewed and confirmed by an Apache Arrow maintainer before posting. See the contributing guide for what the project considers a maintainer review.

@zeroshade
zeroshade merged commit 80bfd31 into apache:main Aug 14, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants