perf(arrow/array): compare fixed-width values in bulk - #1171
perf(arrow/array): compare fixed-width values in bulk#1171fallintoplace wants to merge 4 commits into
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
The bulk equality implementation is correct for valid arrays. Differential testing across affected types, independent slice offsets, bitmap boundaries, null patterns, and nested/chunked wrappers found no valid-input divergence. The full array suite, race tests, vet, and diff checks pass.
I am requesting changes for two points noted inline:
- The run reader introduces a new panic on an IPC-reachable malformed array with a nonzero null count and an empty validity buffer.
- The new nullable fixed-width regression test is shorter than the bulk-path threshold, so it only tests the unchanged scalar path.
Nonblocking observation: validityBitmapEqual now treats NullN() == 0 as authoritative without examining an attached validity bitmap. That changes behavior for inconsistent C Data metadata. It does not affect valid arrays, but documenting that policy—or preserving bitmap comparison when a bitmap exists—would make the malformed-input behavior explicit.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
zeroshade
left a comment
There was a problem hiding this comment.
Both previously blocking findings are addressed:
- empty validity buffers now fall back safely without reaching the run reader;
- the fixed-width equality tests now exercise the bulk path with independent non-byte-aligned offsets, null-slot differences, valid-value differences, boundary-spanning runs, and
Timestampcoverage.
The array, IPC, and C Data tests pass, as does the array race suite. I left one nonblocking observation inline about consistency when malformed imported metadata supplies an attached validity bitmap that disagrees with NullN().
This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow Go maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow Go
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Arrow Go handles maintainer review:
CONTRIBUTING.md.
Rationale for this change
Fixed-width array equality currently checks validity and values one element at a time. The values are already contiguous, so larger arrays can be compared much more efficiently in bulk.
What changes are included in this PR?
bytes.Equal.On an M1 Pro with GOMAXPROCS=1, 65,536-value equality benchmarks improved as follows:
Allocations remain at zero.
Are these changes tested?
go test ./arrow/...go test -race ./arrow/arraygo vet -composites=false ./arrow/arrayThe full Arrow test run used the checked-out Parquet test data.
Are there any user-facing changes?
No.