Skip to content

Spark 4.2: Comet native collect_set does not normalize NaN / -0.0 (SPARK-57298) #4966

Description

@andygrove

Spark 4.2 (SPARK-57298) makes collect_set normalize NaN and -0.0 for floating-point types. Comet's native collect_set aggregate does not perform this normalization, so results differ from Spark.

Failing Spark SQL test (currently marked IgnoreComet in dev/diffs/4.2.0.diff):

  • DataFrameAggregateSuite: "SPARK-57298: collect_set normalizes NaN and -0.0 for floating-point types"

Note: the array-expression normalization path (array_distinct / array_except) is already handled; this issue tracks the aggregate collect_set path.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions