Describe the bug
Under datafusion.execution.enable_ansi_mode = true, datafusion-spark's
pmod and mod correctly raise on a zero divisor, but the message is Arrow's:
DataFusion error: Arrow error: Divide by zero error
Spark 4.2.0 raises:
[REMAINDER_BY_ZERO] Remainder by zero. Use `try_mod` to tolerate divisor being 0
and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to
bypass this error. SQLSTATE: 22012
The condition detected is the same. Only the text differs. DataFusion does not
model Spark's error classes or SQLSTATE values, so an exact reproduction is not
possible, but the message body is expressible.
Cross-version note
The error class name is itself version dependent:
| Spark |
Error class |
| 3.5.8 |
DIVIDE_BY_ZERO |
| 4.0.4 |
DIVIDE_BY_ZERO |
| 4.1.3 |
REMAINDER_BY_ZERO |
| 4.2.0 |
REMAINDER_BY_ZERO |
Pmod.eval calls QueryExecutionErrors.divideByZeroError in 3.5.8 and 4.0.4
and QueryExecutionErrors.remainderByZeroError in 4.1.3 and later. DataFusion
has no mechanism for version-specific expectations, tracked by
#23887.
To Reproduce
set datafusion.execution.enable_ansi_mode = true;
SELECT pmod(10::int, 0::int);
-- DataFusion error: Arrow error: Divide by zero error
Expected behavior
A message matching Spark 4.2.0's REMAINDER_BY_ZERO text.
Scope
This is a repository-wide question rather than a pmod question. The in-repo
precedent, datafusion/spark/src/function/math/abs.rs and
datafusion/sqllogictest/test_files/spark/math/abs.slt, asserts Arrow's native
overflow message rather than Spark's ARITHMETIC_OVERFLOW text, so changing
pmod alone would make the crate less consistent. A decision on whether
datafusion-spark should reproduce Spark's ANSI error text in general would be
useful before individual functions are changed.
Relevant code
datafusion/spark/src/function/math/modulus.rs, spark_pmod and try_rem.
Surfaced by the audit-datafusion-spark-expression skill.
Describe the bug
Under
datafusion.execution.enable_ansi_mode = true,datafusion-spark'spmodandmodcorrectly raise on a zero divisor, but the message is Arrow's:Spark 4.2.0 raises:
The condition detected is the same. Only the text differs. DataFusion does not
model Spark's error classes or SQLSTATE values, so an exact reproduction is not
possible, but the message body is expressible.
Cross-version note
The error class name is itself version dependent:
DIVIDE_BY_ZERODIVIDE_BY_ZEROREMAINDER_BY_ZEROREMAINDER_BY_ZEROPmod.evalcallsQueryExecutionErrors.divideByZeroErrorin 3.5.8 and 4.0.4and
QueryExecutionErrors.remainderByZeroErrorin 4.1.3 and later. DataFusionhas no mechanism for version-specific expectations, tracked by
#23887.
To Reproduce
Expected behavior
A message matching Spark 4.2.0's
REMAINDER_BY_ZEROtext.Scope
This is a repository-wide question rather than a
pmodquestion. The in-repoprecedent,
datafusion/spark/src/function/math/abs.rsanddatafusion/sqllogictest/test_files/spark/math/abs.slt, asserts Arrow's nativeoverflow message rather than Spark's
ARITHMETIC_OVERFLOWtext, so changingpmodalone would make the crate less consistent. A decision on whetherdatafusion-sparkshould reproduce Spark's ANSI error text in general would beuseful before individual functions are changed.
Relevant code
datafusion/spark/src/function/math/modulus.rs,spark_pmodandtry_rem.Surfaced by the audit-datafusion-spark-expression skill.