Skip to content

feat(compute): add list_element - #1138

Open
fallintoplace wants to merge 21 commits into
apache:mainfrom
fallintoplace:feat/compute-list-element
Open

feat(compute): add list_element#1138
fallintoplace wants to merge 21 commits into
apache:mainfrom
fallintoplace:feat/compute-list-element

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Arrow-Go supports the list, large-list, list-view, and fixed-size-list array types, but callers currently need to read offsets and child arrays themselves to select one element from every row. A list_element kernel provides the common nested-data operation directly through compute.

What changes are included in this PR?

  • Add list_element for List, LargeList, ListView, LargeListView, and FixedSizeList.
  • Accept signed or unsigned integer indexes, including a one-element integer array used as the index.
  • Return an array of the list element type while preserving parent nulls and child nulls.
  • Return errors for null or negative indexes, empty lists, and indexes outside the length of a valid list.
  • Use the existing take kernels for primitive, binary, and fixed-width child values to avoid one array allocation per input row. Keep the generic concatenate path for nested child values.
  • Handle list-view offsets and sizes without requiring the input to be materialized first.

Are these changes tested?

  • go test ./arrow/compute/... -count=1
  • Added coverage for sliced List and FixedSizeList inputs, non-contiguous ListView offsets, unsigned indexes, string and nested-list children, parent and child nulls, and invalid indexes.
  • Added BenchmarkListElement cases for 1K, 100K, and 1M rows with allocation reporting.

Are there any user-facing changes?

Yes. This adds the public compute.ListElement function. Existing functions and APIs are unchanged.

@fallintoplace
fallintoplace marked this pull request as ready for review August 7, 2026 23:46
@fallintoplace
fallintoplace marked this pull request as draft August 7, 2026 23:48
@fallintoplace
fallintoplace marked this pull request as ready for review August 8, 2026 22:56
@fallintoplace
fallintoplace marked this pull request as draft August 8, 2026 23:08
@fallintoplace
fallintoplace marked this pull request as ready for review August 9, 2026 00:36
@fallintoplace
fallintoplace marked this pull request as draft August 9, 2026 00:37
@fallintoplace
fallintoplace force-pushed the feat/compute-list-element branch from 693b7a0 to a592af7 Compare August 24, 2026 10:22
@fallintoplace
fallintoplace marked this pull request as ready for review August 24, 2026 10:24

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list_element implementation has broad test coverage and the compute, array, and targeted race suites pass. However, I reproduced a blocking public API panic for an unsupported scalar first argument; this should return a dispatch/type error instead.

Comment thread arrow/compute/scalar_nested.go Outdated
return nil, fmt.Errorf("%w: list_element scalar output type %s is not supported", arrow.ErrNotImplemented, listType.Elem())
}

listValue := args[0].(*ScalarDatum).Value.(scalar.ListScalar)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: This unchecked assertion makes unsupported scalar inputs panic instead of returning the normal dispatch/type error. For example, ListElement(ctx, ScalarDatum(Int32(7)), ScalarDatum(Int64(0))) panics with *scalar.Int32 is not scalar.ListScalar; the equivalent unsupported array input returns an error. Please guard the list-like/type assertion and either fall through to normal kernel dispatch or return arrow.ErrType, with a regression test for a non-list scalar argument.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants