Skip to content

[SPARK-58965][SQL][PYTHON] Add gcd and lcm math functions - #58242

Open
Vivek1106-04 wants to merge 1 commit into
apache:masterfrom
Vivek1106-04:SPARK-58965-gcd-lcm
Open

[SPARK-58965][SQL][PYTHON] Add gcd and lcm math functions#58242
Vivek1106-04 wants to merge 1 commit into
apache:masterfrom
Vivek1106-04:SPARK-58965-gcd-lcm

Conversation

@Vivek1106-04

Copy link
Copy Markdown

What changes were proposed in this pull request?

This PR adds two new built-in math functions, gcd and lcm:

gcd(expr1, expr2) -- returns BIGINT
lcm(expr1, expr2) -- returns BIGINT

Both arguments are implicitly cast to BIGINT and the result is BIGINT, matching the existing
factorial expression, which likewise takes an integral argument and returns BIGINT.

Semantics:

Case gcd lcm
either argument is NULL NULL NULL
both arguments are 0 0 0
one argument is 0 abs(other) 0
negative arguments result is non-negative result is non-negative
result not representable as BIGINT ARITHMETIC_OVERFLOW under ANSI mode, NULL otherwise ARITHMETIC_OVERFLOW under ANSI mode, NULL otherwise

The greatest common divisor is computed with the Euclidean algorithm. The least common multiple
divides by the greatest common divisor before multiplying, so a representable result never
overflows on the way there — for example lcm(4611686018427387904, 2) returns 4611686018427387904
even though the naive product 4611686018427387904 * 2 would not fit.

Overflow arises in exactly two places, and in both PostgreSQL raises as well:

  • gcd where the result would be -Long.MinValue, which is not representable — that is, the
    input pairs (0, x), (x, 0) and (x, x) for x = Long.MinValue.
  • lcm where abs(a) / gcd(a, b) * abs(b) exceeds Long.MaxValue.

Both are reported with the existing ARITHMETIC_OVERFLOW error condition, so no new error
condition is introduced. Overflow is gated on ANSI mode as it is elsewhere in Spark — conv in the
same file takes the same approach — raising under ANSI mode and returning NULL otherwise.

Both helpers live in MathUtils and are shared by the interpreted and codegen paths so the two
cannot diverge.

The functions are exposed through SQL, the Scala/Java functions API, PySpark, and Spark Connect.

Why are the changes needed?

Spark SQL currently has no way to compute either value. There is no expression for it, and no
combination of existing built-ins produces the result, so users have to fall back to a UDF — which
for PySpark means a Python round trip per row and no whole-stage codegen.

Both functions are standard in comparable engines:

  • PostgreSQL 13+: gcd(a, b), lcm(a, b) for integer, bigint and numeric
  • DuckDB: gcd(a, b), lcm(a, b) (aliases greatest_common_divisor, least_common_multiple)

Common uses include reducing fractions and ratios to lowest terms, aligning batch or partition
sizes, computing the repeat period of overlapping schedules, and normalizing denominators before
aggregation.

Does this PR introduce any user-facing change?

Yes. Two new built-in functions are available in SQL, the Scala/Java functions API, PySpark and
Spark Connect. Previously gcd and lcm were unresolved function names:

-- Before
spark-sql> SELECT gcd(24, 36);
[UNRESOLVED_ROUTINE] Cannot resolve routine `gcd` on search path
[`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`]. SQLSTATE: 42883

-- After
spark-sql> SELECT gcd(24, 36);
12
spark-sql> SELECT lcm(4, 6);
12

No existing behavior changes; the change is purely additive.

How was this patch tested?

New tests:

  • MathExpressionsSuite — unit tests for both expressions covering ordinary values, negative
    inputs, zero, NULL, the boundary values around Long.MinValue / Long.MaxValue, and overflow in
    both ANSI and non-ANSI mode, plus interpreted-vs-codegen consistency checks.
  • math.sql golden-file tests, regenerated for the ANSI and non-ANSI results and the analyzer
    results.
  • PlanGenerationTestSuite function tests, with the regenerated Spark Connect plan and explain
    golden files.
  • PySpark doctests for gcd and lcm.

Existing suites re-run and passing: MathExpressionsSuite, SQLQueryTestSuite (math.sql),
ExpressionsSchemaSuite, PlanGenerationTestSuite, ProtoToParsedPlanTestSuite.

dev/lint-scala and dev/lint-python both pass.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

Spark SQL has no expression, and no combination of existing built-ins,
that computes the greatest common divisor or the least common multiple
of two integers, so users have to fall back to a UDF.

This adds `gcd(expr1, expr2)` and `lcm(expr1, expr2)`. Both arguments are
implicitly cast to BIGINT and the result is BIGINT, matching the existing
`factorial` expression. The result is never negative, `gcd(0, 0)` and
`lcm(0, 0)` are 0, and a NULL argument yields NULL.

The greatest common divisor is computed with the Euclidean algorithm.
The least common multiple divides by it before multiplying, so a result
that is representable does not overflow on the way there. Results that
are genuinely unrepresentable raise ARITHMETIC_OVERFLOW under ANSI mode
and return NULL otherwise, as elsewhere in Spark. Both helpers live in
MathUtils and are shared by the interpreted and codegen paths.

PostgreSQL 13+ and DuckDB provide the same two functions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant