Skip to content

fix: prevent unsafe integer interval propagation - #25234

Open
haohuaijin wants to merge 10 commits into
apache:mainfrom
haohuaijin:fix/integer-interval-propagation
Open

haohuaijin wants to merge 10 commits into
apache:mainfrom
haohuaijin:fix/integer-interval-propagation

Conversation

@haohuaijin

@haohuaijin haohuaijin commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Arithmetic filters can incorrectly identify a column as constant and eliminate a required SortExec, returning rows in the wrong order. Integer multiplication can wrap, and integer division truncates, so their mathematical inverses do not necessarily describe all valid inputs.

CREATE TABLE wrap_test (a INT) AS VALUES (-2147483647), (1);
SELECT a FROM wrap_test
WHERE a * 2::INT = 2::INT ORDER BY a DESC;
-- Expected: 1, -2147483647
-- Actual before this fix: -2147483647, 1

CREATE TABLE division_test (a INT) AS VALUES (2), (3);
SELECT a FROM division_test
WHERE a / 2::INT = 1::INT ORDER BY a DESC;
-- Expected: 3, 2
-- Actual before this fix: 2, 3

Both multiplication inputs evaluate to 2; both division inputs evaluate to 1. Nevertheless, interval inference narrows a to a singleton and removes the sort. Without the filters, the sorts are retained and these queries return the correct order. With ORDER BY ... LIMIT, incorrect sort elimination can also change which rows are returned.

What changes are included in this PR?

  • Return unbounded output intervals when unchecked integer addition, subtraction or multiplication may wrap.
  • Skip inverse propagation for integer division, potentially wrapping arithmetic, and zero products where an operand may be zero.
  • Retain interval narrowing for supported arithmetic proven safe from these cases.
  • Correct a Filter statistics test whose previous lower bound excluded a valid wrapping-subtraction input.

What is the testing strategy for this PR?

  • SQL regressions in filter_without_sort_exec.slt cover both reproductions and sorting a wrapped expression; verified failing before the fix and passing afterward.
  • Unit tests enumerate selected Int8 domains under checked and wrapping arithmetic, ensuring forward and propagated intervals retain valid runtime values. Additional tests cover unsigned subtraction underflow and safe multiplication narrowing.
  • Relevant interval, analysis and Filter tests, the complete affected SLT file, Clippy and repository lint checks pass locally.

Are there any user-facing changes?

Affected queries retain necessary sorting. Unsafe interval optimizations are skipped; SQL arithmetic semantics and public APIs are unchanged.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Sep 12, 2026
@haohuaijin haohuaijin changed the title Fix/integer interval propagation fix: prevent unsafe integer interval propagation Sep 12, 2026
@github-actions github-actions Bot added the logical-expr Logical plan and expressions label Sep 13, 2026
Comment on lines +425 to +458
let source_type = self.data_type();
// An unbounded integer endpoint still has a finite limit imposed by its
// type. Preserve that limit when widening so subsequent arithmetic can
// prove that it does not overflow the destination type.
use DataType::{Int8, Int16, Int32, Int64, UInt8, UInt16, UInt32, UInt64};
let widening_integer_cast = matches!(
(&source_type, data_type),
(Int8, Int16 | Int32 | Int64)
| (Int16, Int32 | Int64)
| (Int32, Int64)
| (UInt8, UInt16 | UInt32 | UInt64 | Int16 | Int32 | Int64)
| (UInt16, UInt32 | UInt64 | Int32 | Int64)
| (UInt32, UInt64 | Int64)
);
let lower = if widening_integer_cast && self.lower.is_null() {
get_extreme_value!(
MIN,
MIN_DECIMAL128_FOR_EACH_PRECISION,
MIN_DECIMAL256_FOR_EACH_PRECISION,
&source_type
)
} else {
self.lower.clone()
};
let upper = if widening_integer_cast && self.upper.is_null() {
get_extreme_value!(
MAX,
MAX_DECIMAL128_FOR_EACH_PRECISION,
MAX_DECIMAL256_FOR_EACH_PRECISION,
&source_type
)
} else {
self.upper.clone()
};

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserve source type bounds during integer widening casts (e.g. UInt32 to Int64), allowing overflow checks to recognize safe arithmetic and retain valid optimizations. like the test case in https://github.com/apache/datafusion/pull/25234/changes#diff-4e8dde04785dd86931aa4ccbab316d31d7a66eb6f1126721fcf15906919a8f98R240-R286

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This addresses the CI timeout: lost bounds blocked join pruning, leaving the FIFO test waiting for output.

@haohuaijin
haohuaijin marked this pull request as ready for review September 13, 2026 09:40
@codecov-commenter

codecov-commenter commented Sep 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.60317% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.93%. Comparing base (bb21f51) to head (99eaf75).
⚠️ Report is 37 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/expr-common/src/interval_arithmetic.rs 83.87% 0 Missing and 15 partials ⚠️
datafusion/physical-expr/src/expressions/binary.rs 99.06% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25234      +/-   ##
==========================================
- Coverage   81.93%   81.93%   -0.01%     
==========================================
  Files        1133     1134       +1     
  Lines      423529   426309    +2780     
  Branches   423529   426309    +2780     
==========================================
+ Hits       347032   349302    +2270     
- Misses      55907    56299     +392     
- Partials    20590    20708     +118     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@haohuaijin

haohuaijin commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Hi @kosiew would you mind reviewing this fix?

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@haohuaijin, thanks for working on this. The changes look good to me. I like that this addresses both wrapping integer arithmetic and truncating division while keeping the interval and sort-property inference conservative. The regression coverage around ordering and statistics is also helpful.

I left one non-blocking suggestion for some additional coercion-path coverage.

use DataType::{Int8, Int16, Int32, Int64, UInt8, UInt16, UInt32, UInt64};
use arrow::compute::CastOptions;

let types = [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice coverage here. One additional test that might be useful is exercising this through a mixed signed/unsigned coercion path, since coerce_operands and coerce_for_comparison also invoke this cast when unifying operand types. For example, we could check that an unbounded UInt8 is coerced to the bounded 0..255 range before a widening operation with Int16, perhaps through mul/div or contains/intersect. Not a blocker since the direct cast matrix already covers the underlying behavior.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added in 99eaf75

@haohuaijin

Copy link
Copy Markdown
Contributor Author

Thanks @kosiew , i apply suggestion in 99eaf75

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integer interval inference can incorrectly remove ORDER BY

3 participants