Skip to content

[SPARK-58947][CORE] Handle Char in createArray and add tests for SparkCollectionUtils - #58227

Open
uros-b wants to merge 3 commits into
apache:masterfrom
uros-b:test-sparkcollectionutils
Open

[SPARK-58947][CORE] Handle Char in createArray and add tests for SparkCollectionUtils#58227
uros-b wants to merge 3 commits into
apache:masterfrom
uros-b:test-sparkcollectionutils

Conversation

@uros-b

@uros-b uros-b commented Aug 23, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Adds a unit test suite for SparkCollectionUtils (common/utils, org.apache.spark.util), which had no test coverage, and fixes a gap the tests surfaced.

Tests cover all four methods: toMapWithIndex (empty input, duplicate keys, equivalence to zipWithIndex.toMap); isEmpty / isNotEmpty (null, empty, non-empty java.util.Map); and createArray for every primitive type plus reference types and size == 0.

Adding the createArray coverage exposed a latent bug: it dispatched on 7 of the 8 primitive classOf types (Boolean/Byte/Short/Int/Long/Float/Double) but not Char, so a Char argument fell through to the Array[AnyRef] cast and threw ClassCastException (a char[] is not an Object[]). This adds the missing Char case so createArray[Char] fills the array like the other primitives.

Why are the changes needed?

SparkCollectionUtils is used in production (e.g. StructType field indexing via toMapWithIndex) but had no tests. The Char gap is currently unreachable (no caller passes Char), but it is a latent crash for any future Char caller; adding the case makes the dispatch complete and symmetric with the other primitives.

Does this PR introduce any user-facing change?

No. SparkCollectionUtils is private[spark]; this adds tests and a missing internal Char case.

How was this patch tested?

New SparkCollectionUtilsSuite (extends AnyFunSuite, the base used by the sibling common/utils suites), covering all four methods including createArray for all eight primitive types, reference types, and size == 0.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

@dongjoon-hyun dongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for filling this gap. CI is green, the suite matches the AnyFunSuite style used by SparkStringUtilsSuite in the same module, and I confirmed none of these methods had prior coverage anywhere.

One substantive comment inline about the Char case, plus two nits:

  • Line 62: createArray(0, 7) is an Int, i.e. a primitive case, but it sits in the test named createArray fills reference-typed arrays and returns empty for size 0. Moving it up to the primitive test (or renaming) would keep the two tests cleanly split.
  • Optional: ScalaTest's === compares arrays structurally and prints both sides on failure, so assert(SparkCollectionUtils.createArray(3, 7) === Array(7, 7, 7)) debugs a bit more easily than sameElements. Not a blocker.

Comment thread common/utils/src/test/scala/org/apache/spark/util/SparkCollectionUtilsSuite.scala Outdated
@uros-b uros-b changed the title [SPARK-58947][CORE][TEST] Add tests for SparkCollectionUtils [SPARK-58947][CORE] Add tests for SparkCollectionUtils and handle Char in createArray Aug 23, 2026
@uros-b

uros-b commented Aug 23, 2026

Copy link
Copy Markdown
Member Author

Thank you @dongjoon-hyun for the careful review. I addressed all comments/concerns.

@uros-b
uros-b requested a review from dongjoon-hyun August 23, 2026 21:28
@uros-b uros-b changed the title [SPARK-58947][CORE] Add tests for SparkCollectionUtils and handle Char in createArray [SPARK-58947][CORE] Handle Char in createArray and add tests for SparkCollectionUtils Aug 23, 2026

@dongjoon-hyun dongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants