You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Backport ANSI skipped-operand fix for arithmetic and concat to branch-1.2 #7542
branch-1.2 has the ANSI bug from #7246 for arithmetic and concat: native Comet evaluates an operand on rows where Spark skips it, so a query Spark answers fails with CAST_INVALID_INPUT. For example, with spark.sql.ansi.enabled=true:
CREATETABLEt(n int, txt string) USING parquet;
INSERT INTO t VALUES (NULL, 'bad'), (3, '6');
SELECT n + CAST(txt ASINT) FROM t; -- Spark: NULL, 9. Comet: CAST_INVALID_INPUT
The fix on main is #7456 (not merged yet). It cannot be cherry-picked as is: it builds on #7326 (LazyLogical, speculative evaluation), which branch-1.2 does not have. branch-1.2 does have NullShortCircuit (#6716), so a backport could wrap the operands without speculation.
This issue tracks whether, and how, to backport the fix to branch-1.2, as docs/source/contributor-guide/backporting.md asks for a correctness fix.
Steps to reproduce
Run the query above on branch-1.2 with ANSI mode on.
Expected behavior
Comet returns the same result as Spark: NULL where n is NULL, without evaluating CAST(txt AS INT) there.
Additional context
Without speculation, the wrapped form is up to about 2x slower than eager evaluation for cheap operands such as a * (b - c) on batches with NULLs (benchmark in fix: skip arithmetic and concat operands that Spark skips under ANSI mode #7456). Maintainers may prefer to backport with speculation included, or to accept the slowdown for correctness.
Activity