feat: add IVF_HNSW_RQ vector index

[ddupg]

Summary

• add IVF_HNSW_RQ across the Rust, Python, Java, JNI, namespace, metadata, and distributed-index surfaces
• build HNSW with transient exact rotated residual vectors while persisting only the graph and RaBitQ storage
• rebuild exact graphs for incremental updates, row-id remapping, and logical segment merging
• support 1-bit and multi-bit RaBitQ; reject non-L2 metrics at the build boundary

Exact build vectors are removed from the storage batch before the auxiliary index file is written. Reopen and query paths therefore depend only on persisted HNSW and RaBitQ data and do not use vector reconstruction fallback.

Benchmark

Controlled synthetic benchmark: 100,000 rows, 32 dimensions, 32 queries, k=10, 4 IVF partitions, all partitions probed, 5-bit RaBitQ, shared centroids and rotation.

Index	ef	Recall@10	Mean query	P95	Speedup	Build	Size
IVF_RQ	-	90.31%	3.715 ms	4.062 ms	1.00x	0.152 s	5.78 MB
IVF_HNSW_RQ	20	78.75%	2.558 ms	2.931 ms	1.45x	2.247 s	24.34 MB
IVF_HNSW_RQ	50	89.69%	2.701 ms	3.059 ms	1.38x	2.247 s	24.34 MB
IVF_HNSW_RQ	100	90.31%	3.151 ms	3.440 ms	1.18x	2.247 s	24.34 MB

At equal recall (ef=100), IVF_HNSW_RQ reduced mean latency by 15.2% (1.179x speedup). At ef=50, it lost 0.625 recall percentage points and reduced mean latency by 27.3% (1.376x speedup). In this small controlled workload, graph construction was about 14.7x slower and the index was about 4.21x larger. These are microbenchmark results, not production throughput claims.

Command:

cargo test --release -p lance --features slow_tests test_ivf_hnsw_rq_benchmark -- --nocapture

Testing

• cargo test -p lance ivf_hnsw_rq -- --nocapture (8 passed)
• cargo test -p lance test_merge_existing_hnsw_segments_rebuilds_graph -- --nocapture (4 passed: FLAT/PQ/SQ/RQ)
• cargo check -p lance-index --tests
• cargo check -p lance --tests
• cargo check -p lance-namespace-impls --tests
• cargo clippy --all --tests --benches -- -D warnings
• cargo fmt --all -- --check
• uv run pytest python/tests/test_vector_index.py::test_create_reopen_and_query_ivf_hnsw_rq_index
• uv run make lint
• Java JNITest#testIvfHnswRqIndexParams and VectorIndexTest#testCreateIvfHnswRqIndex
• Java Spotless, Checkstyle, and JNI Clippy

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 07115427-7ab2-4c49-b3d4-2f0a8fd3627c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}