Describe the bug
SparkPmod::return_type returns arg_types[0] unchanged. For decimal inputs
Spark instead derives the result type with Pmod.resultDecimalType, which
follows the Remainder rule:
val resultScale = max(s1, s2)
val resultPrecision = min(p1 - s1, p2 - s2) + resultScale
Because DataFusion's Signature::numeric first coerces both arguments to a
common decimal type, the two precisions it sees are already equal and the rule
collapses to the input precision. Spark applies the rule to the declared
argument types, so it produces a narrower type.
The computed values agree. Only the reported precision differs.
To Reproduce
Spark 4.2.0 (verified with pyspark==4.2.0):
SELECT typeof(pmod(CAST(2.5 AS DECIMAL(3,1)), CAST(1.2 AS DECIMAL(2,1))));
-- decimal(2,1)
DataFusion:
SELECT arrow_typeof(pmod(2.5::decimal(3,1), 1.2::decimal(2,1)));
-- Decimal128(3, 1)
Expected behavior
The result type follows Spark's resultDecimalType, including the
spark.sql.decimalOperations.allowPrecisionLoss branch that calls
DecimalType.adjustPrecisionScale.
Impact
Low. The values are identical and no test in the tree depends on the narrower
type. It matters for schema-sensitive consumers and for any downstream overflow
check that is driven by the declared precision.
Relevant code
datafusion/spark/src/function/math/modulus.rs, SparkPmod::return_type.
SparkMod::return_type has the same shape and Spark's Remainder uses the same
rule, so a fix should probably cover both.
Surfaced by the audit-datafusion-spark-expression skill.
Describe the bug
SparkPmod::return_typereturnsarg_types[0]unchanged. For decimal inputsSpark instead derives the result type with
Pmod.resultDecimalType, whichfollows the
Remainderrule:Because DataFusion's
Signature::numericfirst coerces both arguments to acommon decimal type, the two precisions it sees are already equal and the rule
collapses to the input precision. Spark applies the rule to the declared
argument types, so it produces a narrower type.
The computed values agree. Only the reported precision differs.
To Reproduce
Spark 4.2.0 (verified with
pyspark==4.2.0):DataFusion:
Expected behavior
The result type follows Spark's
resultDecimalType, including thespark.sql.decimalOperations.allowPrecisionLossbranch that callsDecimalType.adjustPrecisionScale.Impact
Low. The values are identical and no test in the tree depends on the narrower
type. It matters for schema-sensitive consumers and for any downstream overflow
check that is driven by the declared precision.
Relevant code
datafusion/spark/src/function/math/modulus.rs,SparkPmod::return_type.SparkMod::return_typehas the same shape and Spark'sRemainderuses the samerule, so a fix should probably cover both.
Surfaced by the audit-datafusion-spark-expression skill.