fix: prevent unsafe integer interval propagation - #25234
haohuaijin wants to merge 10 commits into
Conversation
| let source_type = self.data_type(); | ||
| // An unbounded integer endpoint still has a finite limit imposed by its | ||
| // type. Preserve that limit when widening so subsequent arithmetic can | ||
| // prove that it does not overflow the destination type. | ||
| use DataType::{Int8, Int16, Int32, Int64, UInt8, UInt16, UInt32, UInt64}; | ||
| let widening_integer_cast = matches!( | ||
| (&source_type, data_type), | ||
| (Int8, Int16 | Int32 | Int64) | ||
| | (Int16, Int32 | Int64) | ||
| | (Int32, Int64) | ||
| | (UInt8, UInt16 | UInt32 | UInt64 | Int16 | Int32 | Int64) | ||
| | (UInt16, UInt32 | UInt64 | Int32 | Int64) | ||
| | (UInt32, UInt64 | Int64) | ||
| ); | ||
| let lower = if widening_integer_cast && self.lower.is_null() { | ||
| get_extreme_value!( | ||
| MIN, | ||
| MIN_DECIMAL128_FOR_EACH_PRECISION, | ||
| MIN_DECIMAL256_FOR_EACH_PRECISION, | ||
| &source_type | ||
| ) | ||
| } else { | ||
| self.lower.clone() | ||
| }; | ||
| let upper = if widening_integer_cast && self.upper.is_null() { | ||
| get_extreme_value!( | ||
| MAX, | ||
| MAX_DECIMAL128_FOR_EACH_PRECISION, | ||
| MAX_DECIMAL256_FOR_EACH_PRECISION, | ||
| &source_type | ||
| ) | ||
| } else { | ||
| self.upper.clone() | ||
| }; |
There was a problem hiding this comment.
Preserve source type bounds during integer widening casts (e.g. UInt32 to Int64), allowing overflow checks to recognize safe arithmetic and retain valid optimizations. like the test case in https://github.com/apache/datafusion/pull/25234/changes#diff-4e8dde04785dd86931aa4ccbab316d31d7a66eb6f1126721fcf15906919a8f98R240-R286
There was a problem hiding this comment.
This addresses the CI timeout: lost bounds blocked join pruning, leaving the FIFO test waiting for output.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25234 +/- ##
==========================================
- Coverage 81.93% 81.93% -0.01%
==========================================
Files 1133 1134 +1
Lines 423529 426309 +2780
Branches 423529 426309 +2780
==========================================
+ Hits 347032 349302 +2270
- Misses 55907 56299 +392
- Partials 20590 20708 +118 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Hi @kosiew would you mind reviewing this fix? |
kosiew
left a comment
There was a problem hiding this comment.
@haohuaijin, thanks for working on this. The changes look good to me. I like that this addresses both wrapping integer arithmetic and truncating division while keeping the interval and sort-property inference conservative. The regression coverage around ordering and statistics is also helpful.
I left one non-blocking suggestion for some additional coercion-path coverage.
| use DataType::{Int8, Int16, Int32, Int64, UInt8, UInt16, UInt32, UInt64}; | ||
| use arrow::compute::CastOptions; | ||
|
|
||
| let types = [ |
There was a problem hiding this comment.
Nice coverage here. One additional test that might be useful is exercising this through a mixed signed/unsigned coercion path, since coerce_operands and coerce_for_comparison also invoke this cast when unifying operand types. For example, we could check that an unbounded UInt8 is coerced to the bounded 0..255 range before a widening operation with Int16, perhaps through mul/div or contains/intersect. Not a blocker since the direct cast matrix already covers the underlying behavior.
Which issue does this PR close?
Rationale for this change
Arithmetic filters can incorrectly identify a column as constant and eliminate a required
SortExec, returning rows in the wrong order. Integer multiplication can wrap, and integer division truncates, so their mathematical inverses do not necessarily describe all valid inputs.Both multiplication inputs evaluate to
2; both division inputs evaluate to1. Nevertheless, interval inference narrowsato a singleton and removes the sort. Without the filters, the sorts are retained and these queries return the correct order. WithORDER BY ... LIMIT, incorrect sort elimination can also change which rows are returned.What changes are included in this PR?
What is the testing strategy for this PR?
filter_without_sort_exec.sltcover both reproductions and sorting a wrapped expression; verified failing before the fix and passing afterward.Are there any user-facing changes?
Affected queries retain necessary sorting. Unsafe interval optimizations are skipped; SQL arithmetic semantics and public APIs are unchanged.