Skip to content

fix(index): report corrupt IVF metadata instead of aborting - #9006

Open
LuciferYang wants to merge 1 commit into
lance-format:mainfrom
LuciferYang:fix/ivf-rq-ivf-proto-corrupt-error
Open

fix(index): report corrupt IVF metadata instead of aborting#9006
LuciferYang wants to merge 1 commit into
lance-format:mainfrom
LuciferYang:fix/ivf-rq-ivf-proto-corrupt-error

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Problem

TryFrom<PbIvf> for IvfModel had two ways to abort on a damaged index file. It asserted that the offsets and lengths vectors matched in length, and assert_eq! is compiled into release builds. The v1 centroid branch derived the centroid dimension by dividing the value count by the number of partition lengths, which divides by zero when a file carries centroids and no lengths; integer division by zero cannot be compiled out at all.

Fixes #9005.

What this changes

Return corrupt_file from both, with the counts. rust/AGENTS.md maps format and integrity failures to that variant and reserves assert! for conditions that prevent data corruption.

These are read-side guards. Nothing in Lance writes either shape: add_partition grows both vectors together and the serializer copies both, so the trigger is damaged or foreign input.

Test plan

test_ivf_model_rejects_mismatched_proto_lengths parses a proto with 2 offsets and 3 lengths; with the assert_eq! restored it fails on the assertion instead. test_ivf_model_rejects_v1_centroids_without_lengths parses v1 centroids with no lengths; without the guard it fails with attempt to divide by zero. test_ivf_model_derives_offsets_from_lengths pins the normal shape the mismatch check has to leave alone, where offsets are empty and derived by scanning the lengths.

  • cargo test -p lance-index --lib 1233 passed, 3 ignored
  • cargo clippy --all --tests --benches -- -D warnings clean
  • cargo fmt --all --check clean

Not in this change

IvfModel::find_partitions still does self.centroids.clone().unwrap() (storage.rs:113), and row_range and centroid index by partition without a bound. Those are post-parse, in different functions, and worth a separate look.

TryFrom<PbIvf> for IvfModel had two ways to abort on a damaged file. It asserted that the offsets and lengths vectors matched in length, and assert_eq! is compiled into release builds. The v1 centroid branch derived the centroid dimension by dividing the value count by the number of partition lengths, which divides by zero when a file carries centroids and no lengths; integer division by zero cannot be compiled out at all.

Return corrupt_file from both, with the counts. Nothing in Lance writes either shape: add_partition grows both vectors together and the serializer copies both, so these are read-side guards for damaged or foreign input.
@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Sep 4, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The shared IVF metadata conversion now returns recoverable corruption errors for both malformed shapes while preserving valid legacy centroid loading and derived offsets.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: corrupt IVF metadata aborts the process instead of being reported

1 participant