Distance threshold query silently fails when using bit quantization

[rolf-moz]

This query seemed to ignore the 'v.distance <= :distanceThreshold' condition

      FROM vec_history v
      JOIN vec_history_mapping m USING (rowid)
      WHERE v.embedding MATCH :vector
        AND k = :limit
        AND v.distance <= :distanceThreshold
      ORDER BY v.distance
    )

Table was created with

      CREATE VIRTUAL TABLE vec_history USING vec0(
        embedding FLOAT[${this.#embeddingSize}]
        distance_metric=cosine
            INDEXED BY rescore(
          quantizer=bit,
          oversample=50
        )
      );

[realgetOff]

I dug into this and can confirm it, with a wider blast radius than the title suggests: it isn't specific to bit quantization, and it isn't specific to rescore.

Root cause

vec0BestIndex accepts the distance constraint and unconditionally sets omit = 1:

pIdxInfo->aConstraintUsage[i].argvIndex = argvIndex++;
pIdxInfo->aConstraintUsage[i].omit = 1;

That is a promise to SQLite that the vtab applies the constraint itself, so SQLite drops the WHERE term from the query plan and never re-checks it. But the constraint is only ever read back out of idxStr in the FLAT chunk scan (vec0Filter_knn_chunks_iter). grep -n VEC0_IDXSTR_KIND_KNN_DISTANCE_CONSTRAINT sqlite-vec.c returns only that one function. rescore_knn, vec0Filter_knn_diskann and ivf_query_knn never look at it, so for those column types the predicate silently evaporates.

It looks like a regression rather than an oversight: #166 landed the distance constraints on 2026-02-13, and the new ANN backends (#276 rescore, #278 DiskANN, #277 IVF) were merged on 2026-03-31 without wiring them up.

Reproduction

200 random float[8] rows, same query vector, k = 20, AND distance <= 0.3:

[flat]     20 candidates -> 1 row returned,  0 violating   (correct)
[rescore]  20 candidates -> 20 rows returned, 20 violating  (all of them)
[diskann]  20 candidates -> 20 rows returned, 9-11 violating

DiskANN matters most in practice here, since it is enabled by default (SQLITE_VEC_ENABLE_DISKANN 1) whereas IVF is opt-in. All four operators (<, <=, >, >=) are affected, and both quantizer=bit and quantizer=int8 on the rescore path — so the issue title's "bit quantization" is incidental.

An impossible constraint is also not honored: AND distance < -1.0 returns a full page of rows instead of none.

Fix

I have a patch ready and will open a PR referencing this issue. The shape of it:

• one shared predicate helper, so omit = 1 has a single point of truth that every backend routes through;
• FLAT keeps its existing pre-filter semantics — its four nested switch cases collapse into one loop calling the helper, which is a net -44 lines and leaves all 204 existing snapshots byte-identical;
• rescore applies it to the rescored float distances before the top-k truncation (the coarse quantized distances aren't comparable to a user threshold), so a lower-bound constraint still fills k;
• DiskANN/IVF filter their final result sets.

One honest caveat I'll document rather than paper over: on DiskANN a distance > constraint can return fewer than k rows, because the filter runs after the graph search rather than over the candidate pool. Matching FLAT there would mean reaching into diskann_search, which felt out of scope for a correctness fix.

Verified with 4 new parametrized tests over flat / rescore-bit / rescore-int8 / diskann: 17 failed before the patch, 0 after, full suite 262 passed, and clean under ASan + UBSan.