fix(index): report corrupt IVF metadata instead of aborting - #9006
Open
LuciferYang wants to merge 1 commit into
Open
fix(index): report corrupt IVF metadata instead of aborting#9006LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
TryFrom<PbIvf> for IvfModel had two ways to abort on a damaged file. It asserted that the offsets and lengths vectors matched in length, and assert_eq! is compiled into release builds. The v1 centroid branch derived the centroid dimension by dividing the value count by the number of partition lengths, which divides by zero when a file carries centroids and no lengths; integer division by zero cannot be compiled out at all. Return corrupt_file from both, with the counts. Nothing in Lance writes either shape: add_partition grows both vectors together and the serializer copies both, so these are read-side guards for damaged or foreign input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TryFrom<PbIvf> for IvfModelhad two ways to abort on a damaged index file. It asserted that the offsets and lengths vectors matched in length, andassert_eq!is compiled into release builds. The v1 centroid branch derived the centroid dimension by dividing the value count by the number of partition lengths, which divides by zero when a file carries centroids and no lengths; integer division by zero cannot be compiled out at all.Fixes #9005.
What this changes
Return
corrupt_filefrom both, with the counts.rust/AGENTS.mdmaps format and integrity failures to that variant and reservesassert!for conditions that prevent data corruption.These are read-side guards. Nothing in Lance writes either shape:
add_partitiongrows both vectors together and the serializer copies both, so the trigger is damaged or foreign input.Test plan
test_ivf_model_rejects_mismatched_proto_lengthsparses a proto with 2 offsets and 3 lengths; with theassert_eq!restored it fails on the assertion instead.test_ivf_model_rejects_v1_centroids_without_lengthsparses v1 centroids with no lengths; without the guard it fails withattempt to divide by zero.test_ivf_model_derives_offsets_from_lengthspins the normal shape the mismatch check has to leave alone, where offsets are empty and derived by scanning the lengths.cargo test -p lance-index --lib1233 passed, 3 ignoredcargo clippy --all --tests --benches -- -D warningscleancargo fmt --all --checkcleanNot in this change
IvfModel::find_partitionsstill doesself.centroids.clone().unwrap()(storage.rs:113), androw_rangeandcentroidindex by partition without a bound. Those are post-parse, in different functions, and worth a separate look.