Describe the bug
Spark's Pmod declares inputType = NumericType and relies on the analyzer's
string promotion rule, so a string argument is implicitly cast. DataFusion's
SparkPmod uses Signature::numeric(2, Volatility::Immutable), which rejects
strings at planning time.
To Reproduce
Spark 4.2.0 (verified with pyspark==4.2.0), and covered by Spark's own golden
file sql/core/src/test/resources/sql-tests/results/typeCoercion/native/promoteStrings.sql.out:
SELECT pmod('1', CAST(1 AS TINYINT)); -- 0, result type bigint
SELECT pmod('10', CAST(3 AS INT)); -- ansi on: 1 (int); ansi off: 1.0 (double)
SELECT pmod('1', '1'); -- DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE
Note that Spark's promotion target differs by mode: with
spark.sql.ansi.enabled = true the string is cast to the other argument's type,
and with it disabled the pair is promoted to double. Two strings are rejected
in both modes.
DataFusion:
SELECT pmod('10'::string, 3::int);
-- Error during planning: For function 'pmod' Utf8 and Int32 are not coercible
-- to a common numeric type.
Expected behavior
A string argument is accepted and coerced the way Spark coerces it.
Scope
This is not specific to pmod. Every datafusion-spark function that uses
Signature::numeric inherits it, and reproducing Spark's mode-dependent
promotion target needs a decision about how far datafusion-spark should go in
emulating Spark's analyzer. Filing it so the gap is visible rather than
proposing a pmod-local workaround.
A commented-out query recording the Spark 4.2.0 result is in
datafusion/sqllogictest/test_files/spark/math/pmod.slt.
Relevant code
datafusion/spark/src/function/math/modulus.rs, SparkPmod::new.
Surfaced by the audit-datafusion-spark-expression skill.
Describe the bug
Spark's
PmoddeclaresinputType = NumericTypeand relies on the analyzer'sstring promotion rule, so a string argument is implicitly cast. DataFusion's
SparkPmodusesSignature::numeric(2, Volatility::Immutable), which rejectsstrings at planning time.
To Reproduce
Spark 4.2.0 (verified with
pyspark==4.2.0), and covered by Spark's own goldenfile
sql/core/src/test/resources/sql-tests/results/typeCoercion/native/promoteStrings.sql.out:Note that Spark's promotion target differs by mode: with
spark.sql.ansi.enabled = truethe string is cast to the other argument's type,and with it disabled the pair is promoted to
double. Two strings are rejectedin both modes.
DataFusion:
Expected behavior
A string argument is accepted and coerced the way Spark coerces it.
Scope
This is not specific to
pmod. Everydatafusion-sparkfunction that usesSignature::numericinherits it, and reproducing Spark's mode-dependentpromotion target needs a decision about how far
datafusion-sparkshould go inemulating Spark's analyzer. Filing it so the gap is visible rather than
proposing a
pmod-local workaround.A commented-out query recording the Spark 4.2.0 result is in
datafusion/sqllogictest/test_files/spark/math/pmod.slt.Relevant code
datafusion/spark/src/function/math/modulus.rs,SparkPmod::new.Surfaced by the audit-datafusion-spark-expression skill.