feat(file-index): read range bitmap indexes - #849
Conversation
| PredicateOperator::NotEq if literals.len() == 1 => { | ||
| let value = self.codec.value(&literals[0])?; | ||
| if value.is_nan() { | ||
| self.bsi.existing.clone() | ||
| } else { | ||
| self.not(&self.eq(&value)) |
There was a problem hiding this comment.
eq() matches both signed zeros, but the Arrow row filter treats them as distinct. For f != +0.0, this drops -0.0 even though the row filter would keep it. I reproduced this with JAVA_FLOAT_V1 and evaluate_exact_leaf_predicate.
NOT IN and strict comparisons have the same issue (f < +0.0 loses -0.0). Could we keep these rows, or return Remain, and add a test against the row filter?
| PredicateOperator::Gt if literals.len() == 1 => { | ||
| self.gt(&self.codec.value(&literals[0])?) |
There was a problem hiding this comment.
NaN ordering still causes false negatives. With JAVA_FLOAT_V1, f > f32::from_bits(0xffc00000) matches all four non-null rows in evaluate_exact_leaf_predicate, but the index returns an empty selection because the literal is normalized to positive NaN.
Negative NaN values in the data can also be lost for f < 0.0. Could we keep pruning conservative for these cases and add regression coverage?
3 participants
Purpose
Paimon Java can write
range-bitmapfile indexes, but the Rust reader currently ignores them. As a result, selective predicates cannot turn those indexes into row selections, including for Mosaic row-group scheduling.Changes
FileIndexResult::Selection.This PR is query-side only: it reads Java-written range-bitmap indexes. A Rust writer can be added independently.
Verification
cargo test -p paimon --lib: 2,821 passed, 2 ignoredcargo clippy -p paimon --lib --tests -- -D warningsRelated Java work