Skip to content

perf: Optimize array set ops on sliced arrays#20693

Open
neilconway wants to merge 3 commits intoapache:mainfrom
neilconway:neilc/optimize-sliced-arrays
Open

perf: Optimize array set ops on sliced arrays#20693
neilconway wants to merge 3 commits intoapache:mainfrom
neilconway:neilc/optimize-sliced-arrays

Conversation

@neilconway
Copy link
Contributor

Which issue does this PR close?

N/A

Rationale for this change

Several array set operations (array_union, array_intersect, array_distinct, array_except) operate on all values in the underlying values buffer of a ListArray when doing batched row conversion. For sliced ListArrays, values() returns the full underlying buffer, which means we end up doing row conversion for rows that aren't in the visible slice. This was not a correctness issue but it is inefficient.

What changes are included in this PR?

  • Change array set ops to do row conversion on the visible slice, not the full values buffer
  • Add unit tests for array set ops on sliced ListArrays. These tests pass with or without this PR, but it seems wise to have more test coverage for sliced ListArrays
  • Add benchmarks for array set ops on sliced ListArrays.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

AI usage

Multiple AI tools were used to iterate on this PR. I have reviewed and understand the resulting code.

@github-actions github-actions bot added the functions Changes to functions implementation label Mar 4, 2026
@neilconway
Copy link
Contributor Author

#20677 is similar, but in that case operating on the full values array leads to a bug.

@neilconway
Copy link
Contributor Author

Benchmarks:

  group                         base                                   target
  -----                         ----                                   ------
  array_distinct_sliced/10      1.72    588.8±8.31µs        ? ?/sec    1.00    342.8±8.83µs        ? ?/sec
  array_distinct_sliced/100     1.92      5.6±0.09ms        ? ?/sec    1.00      2.9±0.10ms        ? ?/sec
  array_distinct_sliced/50      1.93      2.8±0.06ms        ? ?/sec    1.00  1469.1±44.20µs        ? ?/sec
  array_except_sliced/10        1.75  1110.5±10.75µs        ? ?/sec    1.00    634.5±8.48µs        ? ?/sec
  array_except_sliced/100       4.06     23.2±0.29ms        ? ?/sec    1.00      5.7±0.13ms        ? ?/sec
  array_except_sliced/50        1.95      5.6±0.09ms        ? ?/sec    1.00      2.9±0.03ms        ? ?/sec
  array_intersect_sliced/10     1.80  1122.6±10.06µs        ? ?/sec    1.00   623.3±18.21µs        ? ?/sec
  array_intersect_sliced/100    3.93     24.1±0.23ms        ? ?/sec    1.00      6.1±0.12ms        ? ?/sec
  array_intersect_sliced/50     1.93      5.9±0.06ms        ? ?/sec    1.00      3.1±0.05ms        ? ?/sec
  array_union_sliced/10         1.65  1175.6±11.55µs        ? ?/sec    1.00   710.9±18.95µs        ? ?/sec
  array_union_sliced/100        3.65     24.9±0.29ms        ? ?/sec    1.00      6.8±0.15ms        ? ?/sec
  array_union_sliced/50         3.60     12.2±0.38ms        ? ?/sec    1.00      3.4±0.11ms        ? ?/sec

These benchmarks use a ratio of 10:1 hidden:visible rows. The reason the benchmark results don't show a ~10x speedup is that previously, we did row conversion on the hidden rows but only applied the set operation to the visible rows, so the latter part was not inefficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant