fix(python): reject a num_partitions that disagrees with ivf_centroids_file - #9009
Open
LuciferYang wants to merge 1 commit into
Open
fix(python): reject a num_partitions that disagrees with ivf_centroids_file#9009LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
…s_file The ivf_centroids_file branch overwrote num_partitions with the file's cluster count, unconditionally and with no warning, so create_index(num_partitions=8, ivf_centroids_file=<4-cluster .npy>) built a 4-partition index. The only mismatch guard, on the numpy-array path, compares the already-overwritten value against itself and can never fire for the file path. Compare before overwriting, and name both counts.
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The change restores the existing partition-count/centroid-count invariant at the Python boundary: explicit disagreements fail with both values, while omitted counts continue to derive from the file. The focused regression covers both paths, and the Rust core independently enforces the same invariant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
ivf_centroids_filebranch overwrotenum_partitionswith the file's cluster count, unconditionally and with no warning, socreate_index(num_partitions=8, ivf_centroids_file=<4-cluster .npy>)built a 4-partition index. The numpy-path mismatch guard runs after the overwrite, so it compares the value against itself and can never fire here.Fixes #9008.
What this changes
Compare before overwriting, and name both counts. Rejecting rather than warning matches the surrounding code: the same function already rejects a numpy centroid count that disagrees with
num_partitions, and rootAGENTS.mdsays to reject invalid values with descriptive errors rather than silently adjust.The check now sits after the
num_partitionstype normalization, so a non-int still gets theTypeErrorrather than a mismatch message computed from an unnormalized value.ivf_centroids_fileis a public parameter with no docstring entry; it gets one, saying the file's row count sets the partition count.Worth noting for review: the Rust core already rejects this pair (
rust/lance/src/index/vector.rs:563-570), and the overwrite is the only reason its check never fires. Deleting the overwrite instead would also work, but only on top of #9001, which is what stops the numpy-path guard from rejecting centroids supplied withoutnum_partitions. Checking here keeps the two changes independent and fails one FFI hop earlier.Test plan
test_ivf_centroids_file_num_partitions_mismatchsaves a 4 cluster.npy, asserts thatnum_partitions=8against it raises with both counts, then builds withnum_partitionsomitted and asserts the index has 4 partitions. Omitting it is the case that shows the file's count is what sets the partition count; passing 4 would satisfy the assertion either way.Without the fix the first half fails with
DID NOT RAISE <class 'ValueError'>.uv run pytest python/tests/test_vector_index.py93 passed, 11 skipped, 1 deselectedtest_create_index_progress_callback_error_before_completion_propagates, which fails the same way with upstream'sdataset.pyon this machine.uv run make lintclean