[SPARK-57964][PYSPARK] Fix DataFrame.dropna VALUE_NOT_ANY_OR_ALL parameter name mismatch#57135
Closed
marcuslin123 wants to merge 2 commits into
Closed
[SPARK-57964][PYSPARK] Fix DataFrame.dropna VALUE_NOT_ANY_OR_ALL parameter name mismatch#57135marcuslin123 wants to merge 2 commits into
marcuslin123 wants to merge 2 commits into
Conversation
added 2 commits
July 8, 2026 14:29
zhengruifeng
approved these changes
Jul 9, 2026
zhengruifeng
reviewed
Jul 9, 2026
| }, | ||
| ) | ||
|
|
||
| # Regression test: invalid 'how' should raise PySparkValueError, not AssertionError |
Contributor
There was a problem hiding this comment.
what does the 'not AssertionError' here mean?
Contributor
Author
There was a problem hiding this comment.
Before this fix, invalid 'how' values would raise an AssertionError instead of the intended PySparkValueError.
HyukjinKwon
approved these changes
Jul 9, 2026
zhengruifeng
pushed a commit
that referenced
this pull request
Jul 10, 2026
…eter name mismatch
### What changes were proposed in this pull request?
Fix the `messageParameters` key in `DataFrame.dropna` validation from `arg_type` to `arg_value`, matching the `VALUE_NOT_ANY_OR_ALL` error template which interpolates `<arg_value>`.
Also fix the Spark Connect `dropna` implementation which used the wrong error class (`CANNOT_BE_EMPTY` instead of `VALUE_NOT_ANY_OR_ALL`) with mismatched parameters, making both classic and Connect paths consistent.
### Why are the changes needed?
`df.dropna(how="foo")` is meant to raise a clear `PySparkValueError` saying "Value for `how` must be 'any' or 'all', got 'foo'." Instead, the mismatch between the template placeholder (`<arg_value>`) and the provided parameter key (`arg_type`) causes an internal assertion failure, surfacing an opaque `AssertionError` to the user.
The Connect path had a similar issue — it used `CANNOT_BE_EMPTY` (which expects `<item>`) but passed `{"arg_name": ..., "arg_value": ...}`, also resulting in an `AssertionError`.
### Does this PR introduce _any_ user-facing change?
Yes. Users who pass an invalid `how` argument to `DataFrame.dropna` will now see (in both classic and Connect modes):
```
PySparkValueError: [VALUE_NOT_ANY_OR_ALL] Value for `how` must be 'any' or 'all', got 'foo'.
```
Instead of the previous opaque `AssertionError`.
### How was this patch tested?
Added a regression test in `test_stat.py` within the existing `test_dropna` method that verifies `dropna(how="foo")` raises `PySparkValueError` with the correct error class and parameters.
### Was this patch authored or co-authored using generative AI tooling?
Generative AI tooling (Claude Code) was used as an assistive tool for implementation guidance.
Closes #57135 from marcuslin123/SPARK-57964-fix-dropna-error.
Lead-authored-by: Marcus Lin <193166030+marcuslin123@users.noreply.github.com>
Co-authored-by: marcus <marcuslin@berkeley.edu>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
(cherry picked from commit 8e13ddb)
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix the
messageParameterskey inDataFrame.dropnavalidation fromarg_typetoarg_value, matching theVALUE_NOT_ANY_OR_ALLerror template which interpolates<arg_value>.Also fix the Spark Connect
dropnaimplementation which used the wrong error class (CANNOT_BE_EMPTYinstead ofVALUE_NOT_ANY_OR_ALL) with mismatched parameters, making both classic and Connect paths consistent.Why are the changes needed?
df.dropna(how="foo")is meant to raise a clearPySparkValueErrorsaying "Value forhowmust be 'any' or 'all', got 'foo'." Instead, the mismatch between the template placeholder (<arg_value>) and the provided parameter key (arg_type) causes an internal assertion failure, surfacing an opaqueAssertionErrorto the user.The Connect path had a similar issue — it used
CANNOT_BE_EMPTY(which expects<item>) but passed{"arg_name": ..., "arg_value": ...}, also resulting in anAssertionError.Does this PR introduce any user-facing change?
Yes. Users who pass an invalid
howargument toDataFrame.dropnawill now see (in both classic and Connect modes):Instead of the previous opaque
AssertionError.How was this patch tested?
Added a regression test in
test_stat.pywithin the existingtest_dropnamethod that verifiesdropna(how="foo")raisesPySparkValueErrorwith the correct error class and parameters.Was this patch authored or co-authored using generative AI tooling?
Generative AI tooling (Claude Code) was used as an assistive tool for implementation guidance.