fix: Fix Spark slice function Null type to GenericListArray casting issue#20469
Open
erenavsarogullari wants to merge 3 commits intoapache:mainfrom
Open
fix: Fix Spark slice function Null type to GenericListArray casting issue#20469erenavsarogullari wants to merge 3 commits intoapache:mainfrom
slice function Null type to GenericListArray casting issue#20469erenavsarogullari wants to merge 3 commits intoapache:mainfrom
Conversation
Contributor
|
Does Spark return null (void) or an array of null (void)? I tested on PySpark 4.1.1 >>> spark.sql("select slice(NULL, 1, 2) as a").printSchema()
root
|-- a: array (nullable = true)
| |-- element: void (containsNull = true)
|
Member
Author
|
Spark returns Case2: input array has null element ( |
Contributor
|
This seems to suggest return type should be list of nulls instead of just null |
martin-g
reviewed
Feb 23, 2026
7d4efce to
dcaa80c
Compare
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
dcaa80c to
2d066d7
Compare
Member
Author
Yes, latest fix aims to address this by returning |
2d066d7 to
f8cb205
Compare
f8cb205 to
f5832c4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
slicefunctionNulltype toGenericListArraycasting issue #20466.Rationale for this change
Currently, Spark
slicefunction accepts Null Arrays and returnNullfor this particular queries. DataFusion-Sparkslicefunction also needs to returnNULLwhen Null Array is set.Spark Behavior (tested with latest Spark master):
DF Behaviour:
Current:
New:
What changes are included in this PR?
Explained under first section.
Are these changes tested?
Added new UT cases for both
slice.rsandslice.slt.Are there any user-facing changes?
Yes, currently,
slicefunction returns error message forNullArray inputs, however, expected behavior is to be returnedNULLso end-user will get expected result instead of error message.