Skip to content

[Bug] pmod returns a wider decimal type than Spark #23895

Description

@andygrove

Describe the bug

SparkPmod::return_type returns arg_types[0] unchanged. For decimal inputs
Spark instead derives the result type with Pmod.resultDecimalType, which
follows the Remainder rule:

val resultScale = max(s1, s2)
val resultPrecision = min(p1 - s1, p2 - s2) + resultScale

Because DataFusion's Signature::numeric first coerces both arguments to a
common decimal type, the two precisions it sees are already equal and the rule
collapses to the input precision. Spark applies the rule to the declared
argument types, so it produces a narrower type.

The computed values agree. Only the reported precision differs.

To Reproduce

Spark 4.2.0 (verified with pyspark==4.2.0):

SELECT typeof(pmod(CAST(2.5 AS DECIMAL(3,1)), CAST(1.2 AS DECIMAL(2,1))));
-- decimal(2,1)

DataFusion:

SELECT arrow_typeof(pmod(2.5::decimal(3,1), 1.2::decimal(2,1)));
-- Decimal128(3, 1)

Expected behavior

The result type follows Spark's resultDecimalType, including the
spark.sql.decimalOperations.allowPrecisionLoss branch that calls
DecimalType.adjustPrecisionScale.

Impact

Low. The values are identical and no test in the tree depends on the narrower
type. It matters for schema-sensitive consumers and for any downstream overflow
check that is driven by the declared precision.

Relevant code

datafusion/spark/src/function/math/modulus.rs, SparkPmod::return_type.
SparkMod::return_type has the same shape and Spark's Remainder uses the same
rule, so a fix should probably cover both.

Surfaced by the audit-datafusion-spark-expression skill.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingspark

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions