Describe the bug
datafusion-spark's mod shares the try_rem helper in
datafusion/spark/src/function/math/modulus.rs with pmod. That helper has two
gaps in its zero divisor handling. Both are visible through mod.
-
ANSI mode, floating point divisor. try_rem delegates to Arrow's rem
kernel when datafusion.execution.enable_ansi_mode is true. Arrow's rem
only reports division by zero for integer and decimal types. Floating point
divisors follow IEEE 754 and quietly produce NaN. Spark raises
REMAINDER_BY_ZERO for a zero divisor of any numeric type.
-
-0.0 divisor, both modes. The legacy path nulls out zero divisors by
comparing with eq(right, 0). Arrow's comparison kernels order floating
point values totally, so -0.0 is not equal to 0.0 and a -0.0 divisor is
not recognised. Spark's isZero check is a numeric comparison, so -0.0
counts as zero.
pmod was fixed for both cases as part of an audit of that function and no
longer routes through try_rem. mod still does.
To Reproduce
Spark 4.2.0 (verified with pyspark==4.2.0):
SET spark.sql.ansi.enabled = true;
SELECT MOD(CAST(10.5 AS DOUBLE), CAST(0.0 AS DOUBLE));
-- [REMAINDER_BY_ZERO] Remainder by zero. ... SQLSTATE: 22012
SELECT MOD(CAST(10.5 AS DOUBLE), CAST('-0.0' AS DOUBLE));
-- [REMAINDER_BY_ZERO] Remainder by zero. ... SQLSTATE: 22012
SET spark.sql.ansi.enabled = false;
SELECT MOD(CAST(10.5 AS DOUBLE), CAST(0.0 AS DOUBLE)); -- NULL
SELECT MOD(CAST(10.5 AS DOUBLE), CAST('-0.0' AS DOUBLE)); -- NULL
DataFusion:
set datafusion.execution.enable_ansi_mode = true;
SELECT mod(10.5::float8, 0.0::float8); -- NaN, expected an error
SELECT mod(10.5::float8, -0.0::float8); -- NaN, expected an error
set datafusion.execution.enable_ansi_mode = false;
SELECT mod(10.5::float8, -0.0::float8); -- NaN, expected NULL
Expected behavior
mod raises for a zero divisor of any numeric type in ANSI mode and returns
NULL in legacy mode, treating -0.0 as zero in both.
Affected versions
Spark 3.5.8, 4.0.4, 4.1.3 and 4.2.0 all behave the same way here. The behaviour
difference across those versions is only the error class name, which changed
from DIVIDE_BY_ZERO to REMAINDER_BY_ZERO in Spark 4.1.
Relevant code
datafusion/spark/src/function/math/modulus.rs, try_rem.
spark_pmod in the same file shows one possible shape for the fix. Its
is_zero helper treats -0.0 as zero and its ANSI branch masks the check by
the validity of the dividend, because Spark's remainder expressions are null
intolerant and a NULL dividend short circuits to NULL before the divisor is
validated.
Surfaced by the audit-datafusion-spark-expression skill.
Describe the bug
datafusion-spark'smodshares thetry_remhelper indatafusion/spark/src/function/math/modulus.rswithpmod. That helper has twogaps in its zero divisor handling. Both are visible through
mod.ANSI mode, floating point divisor.
try_remdelegates to Arrow'sremkernel when
datafusion.execution.enable_ansi_modeistrue. Arrow'sremonly reports division by zero for integer and decimal types. Floating point
divisors follow IEEE 754 and quietly produce
NaN. Spark raisesREMAINDER_BY_ZEROfor a zero divisor of any numeric type.-0.0divisor, both modes. The legacy path nulls out zero divisors bycomparing with
eq(right, 0). Arrow's comparison kernels order floatingpoint values totally, so
-0.0is not equal to0.0and a-0.0divisor isnot recognised. Spark's
isZerocheck is a numeric comparison, so-0.0counts as zero.
pmodwas fixed for both cases as part of an audit of that function and nolonger routes through
try_rem.modstill does.To Reproduce
Spark 4.2.0 (verified with
pyspark==4.2.0):DataFusion:
Expected behavior
modraises for a zero divisor of any numeric type in ANSI mode and returnsNULLin legacy mode, treating-0.0as zero in both.Affected versions
Spark 3.5.8, 4.0.4, 4.1.3 and 4.2.0 all behave the same way here. The behaviour
difference across those versions is only the error class name, which changed
from
DIVIDE_BY_ZEROtoREMAINDER_BY_ZEROin Spark 4.1.Relevant code
datafusion/spark/src/function/math/modulus.rs,try_rem.spark_pmodin the same file shows one possible shape for the fix. Itsis_zerohelper treats-0.0as zero and its ANSI branch masks the check bythe validity of the dividend, because Spark's remainder expressions are null
intolerant and a NULL dividend short circuits to NULL before the divisor is
validated.
Surfaced by the audit-datafusion-spark-expression skill.