fix(index): sort search top-k results by relevance when not truncating#614
Open
XiaoHongbo-Hope wants to merge 3 commits into
Open
fix(index): sort search top-k results by relevance when not truncating#614XiaoHongbo-Hope wants to merge 3 commits into
XiaoHongbo-Hope wants to merge 3 commits into
Conversation
`SearchResult::top_k` only sorted results by score on the truncation path (candidates > k). When `k >= candidate count`, it returned rows in the unordered scored-map / insertion order instead of by relevance. Because the row order was previously discarded downstream (row-range scans re-read in file order), this went unnoticed — but consumers that honor the row order (e.g. the DataFusion `vector_search` scan) then get unranked results whenever `limit >= number of matched rows`. This affects `vector_search`, `full_text_search`, and `hybrid_search`. Sort the no-truncation branch by relevance rank too, and update the unit tests that asserted the old insertion-order behavior.
XiaoHongbo-Hope
force-pushed
the
fix-vector-search-topk-ordering
branch
from
July 26, 2026 14:02
4c1418e to
5f605ba
Compare
full_text's SearchResult::top_k had the same bug as the vector one: when `k >= candidate count` it returned rows in input/shard order instead of by relevance score. Drop the no-truncation early-return so both paths sort best-first, and update the test that asserted the old order.
Use total_cmp (NaN-safe) and an explicit smaller-row-id tie-break in full_text's top_k, matching vector_search's ordering. This keeps the result deterministic regardless of input/shard order rather than relying on a stable sort over the input sequence.
XiaoHongbo-Hope
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SearchResult::top_konly sorted by score on the truncation path (candidates > k).When
k >= candidate count, it returned rows in the unordered scored-map / insertionorder instead of by relevance.
This went unnoticed because the row order was previously discarded downstream (row-range
scans re-read in file order); but consumers that honor the row order — e.g. the
DataFusion
vector_searchscan (#613) — then get unranked results wheneverlimit >= number of matched rows. The same bug existed in full_text's separateSearchResult::top_k. (hybrid_searchre-ranks its results itself, so it is notaffected.)
Fix: sort the no-truncation path by relevance rank in both the vector and full_text
SearchResult::top_k, with a consistent NaN-safe comparator and smaller-row-idtie-break; update the unit tests that asserted the old insertion-order behavior.