fix(btree): include null pages in non-IsNull queries for correct thre…

[wkalt]

…e-valued logic

BTree index search only consulted pages whose value range matched the query. When a queried value (e.g. 0) didn't exist in any page, no pages were searched and the null row set was empty. This caused NOT(x = 0) to produce a BlockList with no nulls tracked, incorrectly allowing NULL rows to pass the filter.

The fix adds null pages (both partially-null and all-null) to every non-IsNull query's page list as Matches::Some, so FlatIndex::search() evaluates the predicate and correctly marks those rows as NULL. This preserves three-valued logic for compound expressions like (x != 0) OR (x < 5).

[github-actions]

PR Review

Solid bug fix with good analysis. The three-valued logic gap when queried values don't appear in any BTree page is a real correctness issue, and the fix is well-targeted.

No blocking issues found

The approach of adding null pages as Matches::Some (not Matches::All) is correct — it forces FlatIndex::search() to evaluate the predicate, which properly classifies null rows as NULL rather than TRUE. The HashSet deduplication avoids double-counting pages that already appear in the results. Both unit and integration tests cover the regression path well.

Minor note (non-blocking)

The project guidelines in CLAUDE.md recommend RoaringBitmap over HashSet<u32>. In this case, page counts are small enough that HashSet is perfectly fine (and possibly faster), so this is not actionable — just flagging for awareness.

LGTM

[fenfeng9]

This PR appears to address the issue #5918.

[wkalt]

@fenfeng9 awesome! Thanks for the pointer. This bug was found with a generative testing framework I have been hacking on.

[codecov]

Codecov Report

❌ Patch coverage is 94.28571% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/btree.rs	94.28%	3 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

Thank you for working on this!