Skip to content

perf(arrow/array): compare fixed-width values in bulk - #1171

Open
fallintoplace wants to merge 4 commits into
apache:mainfrom
fallintoplace:perf/arrow-fixed-width-equality
Open

perf(arrow/array): compare fixed-width values in bulk#1171
fallintoplace wants to merge 4 commits into
apache:mainfrom
fallintoplace:perf/arrow-fixed-width-equality

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Rationale for this change

Fixed-width array equality currently checks validity and values one element at a time. The values are already contiguous, so larger arrays can be compared much more efficiently in bulk.

What changes are included in this PR?

  • Compare validity bitmaps in bulk.
  • Compare null-free integer and temporal arrays with bytes.Equal.
  • Walk contiguous valid runs for nullable arrays, ignoring bytes under null slots.
  • Keep floating-point arrays on the scalar path to preserve NaN and signed-zero behavior.
  • Keep arrays shorter than 8 values on the scalar path.
  • Add benchmarks for int32/int64 arrays, several null patterns, and mismatch positions.
  • Add tests for null-slot and floating-point semantics.

On an M1 Pro with GOMAXPROCS=1, 65,536-value equality benchmarks improved as follows:

Case Before After
int32, all valid 505.9 µs 6.5 µs
int32, 1% null 516.2 µs 14.2 µs
int32, alternating null 432.2 µs 247.9 µs
int64, all valid 520.0 µs 13.2 µs
int64, 1% null 523.0 µs 20.3 µs
int64, alternating null 439.0 µs 245.9 µs

Allocations remain at zero.

Are these changes tested?

  • go test ./arrow/...
  • go test -race ./arrow/array
  • go vet -composites=false ./arrow/array

The full Arrow test run used the checked-out Parquet test data.

Are there any user-facing changes?

No.

@fallintoplace
fallintoplace marked this pull request as ready for review August 18, 2026 18:27

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bulk equality implementation is correct for valid arrays. Differential testing across affected types, independent slice offsets, bitmap boundaries, null patterns, and nested/chunked wrappers found no valid-input divergence. The full array suite, race tests, vet, and diff checks pass.

I am requesting changes for two points noted inline:

  1. The run reader introduces a new panic on an IPC-reachable malformed array with a nonzero null count and an empty validity buffer.
  2. The new nullable fixed-width regression test is shorter than the bulk-path threshold, so it only tests the unchanged scalar path.

Nonblocking observation: validityBitmapEqual now treats NullN() == 0 as authoritative without examining an attached validity bitmap. That changes behavior for inconsistent C Data metadata. It does not affect valid arrays, but documenting that policy—or preserving bitmap comparison when a bitmap exists—would make the malformed-input behavior explicit.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/array/numeric_generic.go
Comment thread arrow/array/compare_test.go Outdated

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both previously blocking findings are addressed:

  • empty validity buffers now fall back safely without reaching the run reader;
  • the fixed-width equality tests now exercise the bulk path with independent non-byte-aligned offsets, null-slot differences, valid-value differences, boundary-spanning runs, and Timestamp coverage.

The array, IPC, and C Data tests pass, as does the array race suite. I left one nonblocking observation inline about consistency when malformed imported metadata supplies an attached validity bitmap that disagrees with NullN().


This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.

Comment thread arrow/array/numeric_generic.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants