fix(index): use writer-side IVF shuffle offsets instead of re-decoding by XuQianJin-Stars · Pull Request #8941 · lance-format/lance

XuQianJin-Stars · 2026-09-02T12:31:29Z

Summary

Keep IVF two-file shuffle partition ranges from flush-time prefix sums instead of re-decoding shuffle_offsets.lance, which could disagree with in-memory counts (127 vs 126) and fail create_index.
Reject null __ivf_part_id values instead of treating Arrow's dummy 0 as partition 0.
Derive flush row counts from interleaved rows and fail if they do not match the partition histogram.

Fixes dataset::scanner::test::test_ann_with_deletion with LanceFileVersion::Stable and stable_row_ids=true.

Test plan

cargo test -p lance-index --lib vector::v3::shuffler::tests
cargo test -p lance --lib dataset::scanner::test::test_ann_with_deletion
CI lance tests including test_ann_with_deletion::...::stable_row_ids_2_true

Avoid treating null partition IDs as partition 0, and keep flush-time prefix sums so create_index cannot fail when the offsets sidecar round-trip disagrees with in-memory partition counts.

Drop writer-side prefix sums once they would exceed the resident-memory cap and fall back to the sidecar on-demand path instead of always preloading.

Session construction was eagerly calling tempfile::tempdir(), which panics on Windows CI with PermissionDenied under parallel test load.

Vec::extend doubles allocation, so a length check can still grow past the 256 MiB resident-memory cap. Reserve exactly and drop the in-memory copy if allocated bytes would exceed the ceiling.

…ader Writer-side capacity drops were treated as a generic sidecar reopen, so reader construction could retry Vec::with_capacity from logical length and exceed the 256 MiB cap. Carry ForcedOnDemand explicitly and make sidecar preload fallible and capacity-checked.

A single k=10 query sat on the 0.5 bar and failed at 0.4 on Windows CI. Average recall over 10 queries, match the search metric to ground truth, and raise 4-bit refine so mean recall stays above the contract.

lance-gatekeeper

✅ Gate recommendation: approve.

The test-only follow-up preserves the accepted shuffle fix and strengthens the ANN regression coverage: indexed and exact searches use the same metric, recall is averaged across multiple fixed queries without lowering the 0.5 contract, and the affected cases remain lightweight.

fix(index): use writer-side IVF shuffle offsets instead of re-decoding

a690c36

Avoid treating null partition IDs as partition 0, and keep flush-time prefix sums so create_index cannot fail when the offsets sidecar round-trip disagrees with in-memory partition counts.

github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Sep 2, 2026