Skip to content

[Bug] mod returns NaN instead of raising for a zero floating point divisor in ANSI mode, and ignores -0.0 divisors #23894

Description

@andygrove

Describe the bug

datafusion-spark's mod shares the try_rem helper in
datafusion/spark/src/function/math/modulus.rs with pmod. That helper has two
gaps in its zero divisor handling. Both are visible through mod.

  1. ANSI mode, floating point divisor. try_rem delegates to Arrow's rem
    kernel when datafusion.execution.enable_ansi_mode is true. Arrow's rem
    only reports division by zero for integer and decimal types. Floating point
    divisors follow IEEE 754 and quietly produce NaN. Spark raises
    REMAINDER_BY_ZERO for a zero divisor of any numeric type.

  2. -0.0 divisor, both modes. The legacy path nulls out zero divisors by
    comparing with eq(right, 0). Arrow's comparison kernels order floating
    point values totally, so -0.0 is not equal to 0.0 and a -0.0 divisor is
    not recognised. Spark's isZero check is a numeric comparison, so -0.0
    counts as zero.

pmod was fixed for both cases as part of an audit of that function and no
longer routes through try_rem. mod still does.

To Reproduce

Spark 4.2.0 (verified with pyspark==4.2.0):

SET spark.sql.ansi.enabled = true;
SELECT MOD(CAST(10.5 AS DOUBLE), CAST(0.0 AS DOUBLE));
-- [REMAINDER_BY_ZERO] Remainder by zero. ... SQLSTATE: 22012
SELECT MOD(CAST(10.5 AS DOUBLE), CAST('-0.0' AS DOUBLE));
-- [REMAINDER_BY_ZERO] Remainder by zero. ... SQLSTATE: 22012

SET spark.sql.ansi.enabled = false;
SELECT MOD(CAST(10.5 AS DOUBLE), CAST(0.0 AS DOUBLE));   -- NULL
SELECT MOD(CAST(10.5 AS DOUBLE), CAST('-0.0' AS DOUBLE)); -- NULL

DataFusion:

set datafusion.execution.enable_ansi_mode = true;
SELECT mod(10.5::float8, 0.0::float8);   -- NaN, expected an error
SELECT mod(10.5::float8, -0.0::float8);  -- NaN, expected an error

set datafusion.execution.enable_ansi_mode = false;
SELECT mod(10.5::float8, -0.0::float8);  -- NaN, expected NULL

Expected behavior

mod raises for a zero divisor of any numeric type in ANSI mode and returns
NULL in legacy mode, treating -0.0 as zero in both.

Affected versions

Spark 3.5.8, 4.0.4, 4.1.3 and 4.2.0 all behave the same way here. The behaviour
difference across those versions is only the error class name, which changed
from DIVIDE_BY_ZERO to REMAINDER_BY_ZERO in Spark 4.1.

Relevant code

datafusion/spark/src/function/math/modulus.rs, try_rem.

spark_pmod in the same file shows one possible shape for the fix. Its
is_zero helper treats -0.0 as zero and its ANSI branch masks the check by
the validity of the dividend, because Spark's remainder expressions are null
intolerant and a NULL dividend short circuits to NULL before the divisor is
validated.

Surfaced by the audit-datafusion-spark-expression skill.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingspark

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions