Skip to content

Backport ANSI skipped-operand fix for arithmetic and concat to branch-1.2 #7542

Open

Description

@tlfua

Describe the bug

branch-1.2 has the ANSI bug from #7246 for arithmetic and concat: native Comet evaluates an operand on rows where Spark skips it, so a query Spark answers fails with CAST_INVALID_INPUT. For example, with spark.sql.ansi.enabled=true:

CREATE TABLE t(n int, txt string) USING parquet;
INSERT INTO t VALUES (NULL, 'bad'), (3, '6');
SELECT n + CAST(txt AS INT) FROM t;  -- Spark: NULL, 9. Comet: CAST_INVALID_INPUT

The fix on main is #7456 (not merged yet). It cannot be cherry-picked as is: it builds on #7326 (LazyLogical, speculative evaluation), which branch-1.2 does not have. branch-1.2 does have NullShortCircuit (#6716), so a backport could wrap the operands without speculation.

This issue tracks whether, and how, to backport the fix to branch-1.2, as docs/source/contributor-guide/backporting.md asks for a correctness fix.

Steps to reproduce

Run the query above on branch-1.2 with ANSI mode on.

Expected behavior

Comet returns the same result as Spark: NULL where n is NULL, without evaluating CAST(txt AS INT) there.

Additional context

Willingness to contribute

I can contribute a fix for this bug independently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions