[Feature](lance) Support batch vector search with per-query Top-K

[Gabriel39]

Motivation

Track Batch Search for Lance vector retrieval under #66340. A batch contains independent query vectors, each producing its own Top-K results. For example, three queries with top_k=10 return up to 30 rows, with a query index identifying which query produced each result.

This is distinct from Multi-vector Search, where several subvectors form one logical query and produce one ranked result set. It is also distinct from issuing multiple concurrent SQL statements or changing the scanner's output batch size.

Current status and upstream dependencies

• vector_search() accepts one logical query. TSearchVector.num_vectors represents the subvectors of one multi-vector query, not independent queries.
• The BE reader calls lance_scanner_nearest() or lance_scanner_nearest_multivector(). Batch input and a query-index result column are not exposed through Doris/lance-c.
• The planner inserts one global Top-N above the search TVF to merge split results. Batch Search requires an independent Top-N for each query.
• Lance Rust already supports batch queries through lance-format/lance#6828, including in the Lance v11 dependency used by the current Doris patch chain. Basic support does not require the newer shared-scan optimization.
• lance-format/lance#7640 adds shared IVF partition scans. Its optimized path has eligibility restrictions; HNSW, refinement, adaptive nprobes, and other unsupported combinations fall back to per-query execution. Exposing batch input alone does not guarantee shared index computation.

The parent issue's statement that Lance Scanner accepts only one independent query vector is therefore outdated. The remaining work includes the C binding and Doris integration.

Proposed initial contract

• Search one fixed snapshot, vector column, and dataset per batch.
• Accept a matrix of independent single-vector queries using an explicit batch input, tentatively query_vectors, mutually exclusive with query_vector. Final naming is subject to API review.
• Share metric, filter, search options, top_k, and offset across the batch initially.
• Return a zero-based query identifier, tentatively _query_index, alongside the existing projected columns and _distance.
• Apply top_k and offset independently to each query. A query may return fewer than K rows. Repeated query vectors retain distinct query identifiers.
• Preserve existing single-vector and multi-vector behavior. Define filter placement explicitly and retain the distinction between the search filter and an outer SQL WHERE.
• Batch input must not imply an unspecified SQL output order. Document how callers can order results by query identifier and distance.

Implementation work

• FE API and analysis: validate batch shape, dimensions, supported element types, null/non-finite values, reserved-column conflicts, and input size; support prepared-statement binding without reusing stale query data.
• Execution protocol: represent independent query count and payload explicitly, separately from multi-vector subvector count; define compatibility behavior for unsupported FE/BE combinations.
• lance-c: add a batch-nearest C/C++ API that builds the appropriate Arrow query array for Lance and preserves the query-index output. Define buffer ownership, validation, error handling, and cancellation behavior.
• BE reader: pass batch requests, map the query-index virtual column, preserve the pinned snapshot and fragment/index-segment scope, and propagate stream failures and cancellation.
• Planner and merge: produce up to top_k + offset candidates per query per split, merge by query identifier, and apply per-query offset/Top-K. Do not replace this with a global LIMIT batch_size * top_k. Audit filter, projection, and limit rewrites around the search TVF.
• Resource control: bound batch size, payload bytes, candidate expansion across queries/splits/refinement, internal concurrency, and buffered results; account for memory and release resources on cancellation or errors.
• Dependency integration: update the Doris lance-c dependency/patch chain. Evaluate the shared-IVF-scan optimization separately from basic batch support, including upgrade/backport compatibility and fallback behavior.
• EXPLAIN/Profile: expose query count, execution mode (shared batch or per-query fallback), prefilter construction count/time, unique index partitions loaded where measurable, search time, and result materialization time. Document counter scope and aggregation.
• Documentation and benchmark client: document semantics and limitations, and add a client path that actually submits batches instead of independent SQL requests.

Acceptance criteria

• FE/BE unit tests and regression tests cover batch size one and multiple queries, duplicate query vectors, empty results, invalid input, prepared-statement reuse, and resource limits.
• Validate each query against independent execution: exact-search results match under a defined tie policy; ANN tests use controlled inputs/options and validate the documented approximate-search semantics and recall.
• Cover multiple fragments, index segments, and BE splits, including offset, deletion visibility, indexed/unindexed data, filters, and projections. No query may lose results because another query consumed a global limit.
• Verify supported index/metric/type combinations and explicit fallback behavior, including HNSW and refinement. Existing single-query and multi-vector tests continue to pass.
• Cancellation and failure tests demonstrate that batch tasks, streams, and memory are released.
• Benchmark several batch sizes and concurrency levels with the same dataset, search options, and comparable recall. Report query-vector throughput, batch latency, amortized time per vector, and peak memory separately; do not interpret batch latency divided by batch size as individual request latency.
• Document measured benefits and limitations. In particular, do not claim the shared IVF scan speedup for HNSW's per-query fallback.