Skip to content

fix(python): reject a num_partitions that disagrees with ivf_centroids_file - #9009

Open
LuciferYang wants to merge 1 commit into
lance-format:mainfrom
LuciferYang:fix/ivf-rq-centroids-file-num-partitions
Open

fix(python): reject a num_partitions that disagrees with ivf_centroids_file#9009
LuciferYang wants to merge 1 commit into
lance-format:mainfrom
LuciferYang:fix/ivf-rq-centroids-file-num-partitions

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Problem

The ivf_centroids_file branch overwrote num_partitions with the file's cluster count, unconditionally and with no warning, so create_index(num_partitions=8, ivf_centroids_file=<4-cluster .npy>) built a 4-partition index. The numpy-path mismatch guard runs after the overwrite, so it compares the value against itself and can never fire here.

Fixes #9008.

What this changes

Compare before overwriting, and name both counts. Rejecting rather than warning matches the surrounding code: the same function already rejects a numpy centroid count that disagrees with num_partitions, and root AGENTS.md says to reject invalid values with descriptive errors rather than silently adjust.

The check now sits after the num_partitions type normalization, so a non-int still gets the TypeError rather than a mismatch message computed from an unnormalized value.

ivf_centroids_file is a public parameter with no docstring entry; it gets one, saying the file's row count sets the partition count.

Worth noting for review: the Rust core already rejects this pair (rust/lance/src/index/vector.rs:563-570), and the overwrite is the only reason its check never fires. Deleting the overwrite instead would also work, but only on top of #9001, which is what stops the numpy-path guard from rejecting centroids supplied without num_partitions. Checking here keeps the two changes independent and fails one FFI hop earlier.

Test plan

test_ivf_centroids_file_num_partitions_mismatch saves a 4 cluster .npy, asserts that num_partitions=8 against it raises with both counts, then builds with num_partitions omitted and asserts the index has 4 partitions. Omitting it is the case that shows the file's count is what sets the partition count; passing 4 would satisfy the assertion either way.

Without the fix the first half fails with DID NOT RAISE <class 'ValueError'>.

  • uv run pytest python/tests/test_vector_index.py 93 passed, 11 skipped, 1 deselected
  • The deselected one is test_create_index_progress_callback_error_before_completion_propagates, which fails the same way with upstream's dataset.py on this machine.
  • uv run make lint clean

…s_file

The ivf_centroids_file branch overwrote num_partitions with the file's cluster count, unconditionally and with no warning, so create_index(num_partitions=8, ivf_centroids_file=<4-cluster .npy>) built a 4-partition index. The only mismatch guard, on the numpy-array path, compares the already-overwritten value against itself and can never fire for the file path.

Compare before overwriting, and name both counts.
@github-actions github-actions Bot added A-python Python bindings bug Something isn't working labels Sep 4, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The change restores the existing partition-count/centroid-count invariant at the Python boundary: explicit disagreements fail with both values, while omitted counts continue to derive from the file. The focused regression covers both paths, and the Rust core independently enforces the same invariant.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: create_index silently ignores num_partitions when ivf_centroids_file is given

1 participant