feat: FTS: add OSS OpenSearch backend, semantic FTS recall, filtered FTS cases

[jamesgao-jpg]

Summary

This PR adds OSS OpenSearch and the backend support needed for both unfiltered and filtered full-text search benchmarking. It also publishes validated permuted filtered-concurrency results for Elasticsearch, OSS OpenSearch, Zilliz Cloud, and Turbopuffer, with a dedicated Filtered QPS frontend view. The frontend keeps filtered results in a dedicated view, while each backend's standard and filtered records are consolidated into one published JSON artifact.

PR #794 introduced the preceding full-text search benchmark foundation used by this work. This PR supersedes #814 with a reduced publication scope. PR #818 introduces shared CLI run-context notes; its commit is incorporated here so FTS results preserve deployment and constraint context.

Changes

• add the OSS OpenSearch backend client, configuration, CLI registration, and focused tests
• expose OSS OpenSearch FTS cases in the Run Test frontend
• construct OSS OpenSearch FTS CLI configuration without vector-only HNSW parameters
• default OSS OpenSearch to zero replicas, preserve the force-merge flag, and bound green-health readiness to 30 minutes with diagnostics
• require Streamlit 1.61 for the stateful tab and stretch-width APIs used by the FTS results page
• retain unfiltered semantic FTS metrics and controls
• restore filtered FTS backend cases, dataset preparation, runner orchestration, metrics, and CLI --fts-filter-rate control
• distribute FTS filter_id values with a deterministic bijection so exact-selectivity filters are scattered across corpus insertion order rather than selecting a contiguous suffix
• use one deterministic affine permutation for FTS filter IDs across case construction, dataset preparation, qrel filtering, and result metadata
• restore filtered FTS request translation for Milvus, Elastic Cloud, OSS OpenSearch, TurboPuffer, and Vespa
• add a Filtered QPS frontend tab with dataset and backend filters
• show filtered concurrency QPS by backend, filter rate, and concurrency
• preserve separately measured serial provenance for Elasticsearch, OSS OpenSearch, and Turbopuffer, while using direct same-run serial metrics for refreshed Zilliz Cloud records
• publish 40 validated permuted result records, consolidated into four backend-level JSONs
• cover HotpotQA Large and MS MARCO Large at filter rates 0.50, 0.75, 0.90, 0.95, and 0.99; refreshed Zilliz Cloud records use concurrency 40, 60, and 80, while the other backends retain 60 and 80
• remove the obsolete Zilliz filtered artifact and the published Vespa FTS result while preserving both backend implementations
• report OSS OpenSearch bulk item failures with the actual successful-document count instead of treating partial writes as full success
• add shared --note and --note-file CLI options while preserving Pinecone compatibility
• preserve current upstream backend and compatibility changes by incorporating the latest main

Result Provenance

The Elasticsearch and OSS OpenSearch filtered results come from the approved 2026-07-16 total-series benchmark executed with VDBBench commit e24110b (feat(fts): make filter id distribution configurable). The Turbopuffer results come from the completed 2026-08-02 run executed with commit 07bcefd (feat: publish permuted filtered FTS results). The 16 Zilliz Cloud records were refreshed from the approved 2026-08-07 Task 15919 series executed from clean detached commit 078d524 and published in commit 0e17679.

Elasticsearch, OSS OpenSearch, and Turbopuffer retain serial P95/P99, recall, NDCG, and MRR composed from their approved serial runs; those records carry additional_parameters.serial_measurement with source provenance. The refreshed Zilliz Cloud records measured serial and concurrent search directly in the same run and therefore do not require composed-source provenance.

The ten Turbopuffer records preserve the measured result content with only db_config.api_key cleared before publication. All 40 filtered records were validated for:

• historical composed records may retain custom_case.filter_id_distribution = "permuted"; refreshed Zilliz Cloud records omit the removed selector
• additional_parameters.fts_filter.filter_id_distribution = "affine_permutation_v1"
• valid affine multiplier and offset values forming a bijection for the dataset cardinality
• concurrency levels [60, 80] for historical records and [40, 60, 80] for refreshed Zilliz Cloud records
• positive serial P95/P99, recall, NDCG, and MRR values
• either serial provenance identifying the separately measured source run or direct same-run serial and concurrent stages
• embedded backend run context with published credentials empty

The standard and filtered records remain consolidated into one JSON per backend. The refreshed bundle contains 76 entries: 36 standard and 40 filtered. For Zilliz Cloud, source comparison confirmed that all 16 measured records remained unchanged except for the documented db_config.note evidence text.

The concurrent measurements use 30 seconds per concurrency level, k=100, and IDs-only payloads. OpenSearch and Elasticsearch ran as isolated single-node Docker deployments. Refreshed Zilliz Cloud results used Task 15919 with an 8 CU performance instance, one replica, and one persistent segment per benchmark collection; small and medium datasets used insert batch size 1,000 and load concurrency 4, while the large datasets reused validated loaded collections for search-only runs. Turbopuffer used the managed service in aws-us-west-2.

Turbopuffer Run Caveat

The Turbopuffer run observed service-side HTTP 429 responses at several concurrency points, including 63 responses for HotpotQA Large at 50% matched documents. The published QPS values count successful request throughput under that service limit; they should not be interpreted as an error-free saturation curve.

Deferred Scope

• session and documentation-only migration files

Compatibility

FTS collections loaded before commit bf85cf7 contain legacy filter_id values. Reload those collections before running filtered FTS search so stored IDs and semantic qrel filtering use the same affine permutation.

Verification

• Level 1 RED/GREEN regression checks were run independently for each review fix
• 47 focused case, CLI, dataset, runner, and frontend tests passed for removal of the distribution switch
• OSS OpenSearch Run Test selection, minimal FTS CLI construction, replica defaults, force-merge propagation, and bounded health diagnostics passed focused tests
• the Streamlit dependency-floor contract passed, and a fresh editable install resolved Streamlit 1.61.0 and imported the FTS page
• filtered chart transformation covers four backends at concurrency 60 and 80, plus concurrency 40 for refreshed Zilliz Cloud records
• 40 filtered result records remain validated for metadata, affine-bijection, matrix coverage, direct-or-composed serial semantics, and empty published credentials
• four consolidated backend-level JSONs retain 76 result entries: 36 standard and 40 filtered
• DCO verification passed for every outgoing commit
• no live backend probe was run for these review fixes; OpenSearch readiness behavior was verified with deterministic mocks and the official Cluster Health contract
• exact production restoration remains recorded in commit 659f739, followed by isolated current-lint compatibility commit a352b73
• deterministic exact-selectivity FTS filter distribution was added in commit bf85cf7
• affine permutation is the only supported FTS filter-ID algorithm

New Zilliz Cloud and Turbopuffer provider runs are included in this update; Elasticsearch and OSS OpenSearch continue to use the previously validated total-series evidence.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jamesgao-jpg
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[XuanYang-cn]

Requesting changes for the standards/maintainability issue below. Reviewed at 1b0a0bb97904fa259c3c0d350a394022d39383cd.

1. The filtered-FTS document-to-filter-ID mapping has two independent implementations. FtsDatasetManager._build_qrel_filter_ids() replays the corpus and derives IDs from selected_doc_ids, emitted_count, and FtsFilterIdPermutation, while FtsDocumentIterator.__next__() separately derives the IDs used for inserted documents. These paths already differ in error handling: the iterator skips malformed documents, but the qrel mapping loop does not. Any future drift in selection, skipping, or ordering can make the filter IDs used to prune qrels disagree with the IDs stored in the backend, silently producing invalid filtered recall. Please make filter-ID assignment a single shared primitive used by both paths, and add a regression test proving qrel IDs equal emitted document IDs when documents are skipped or the selected set is sparse.

Lower-severity cleanup:

• MilvusConfig(...) construction is repeated across 18 CLI commands in vectordb_bench/backend/clients/milvus/cli.py. A shared builder/helper would prevent command-specific defaults from drifting.
• OSSOpenSearchFtsConfig duplicates the FTS text mapping and BM25 similarity construction from ElasticCloudFtsConfig. Please consider extracting the shared mapping/settings logic so the two Elasticsearch-compatible backends stay aligned.

[jamesgao-jpg]

Requesting changes for the standards/maintainability issue below. Reviewed at 1b0a0bb97904fa259c3c0d350a394022d39383cd.

1. The filtered-FTS document-to-filter-ID mapping has two independent implementations. FtsDatasetManager._build_qrel_filter_ids() replays the corpus and derives IDs from selected_doc_ids, emitted_count, and FtsFilterIdPermutation, while FtsDocumentIterator.__next__() separately derives the IDs used for inserted documents. These paths already differ in error handling: the iterator skips malformed documents, but the qrel mapping loop does not. Any future drift in selection, skipping, or ordering can make the filter IDs used to prune qrels disagree with the IDs stored in the backend, silently producing invalid filtered recall. Please make filter-ID assignment a single shared primitive used by both paths, and add a regression test proving qrel IDs equal emitted document IDs when documents are skipped or the selected set is sparse.

Lower-severity cleanup:

• MilvusConfig(...) construction is repeated across 18 CLI commands in vectordb_bench/backend/clients/milvus/cli.py. A shared builder/helper would prevent command-specific defaults from drifting.
• OSSOpenSearchFtsConfig duplicates the FTS text mapping and BM25 similarity construction from ElasticCloudFtsConfig. Please consider extracting the shared mapping/settings logic so the two Elasticsearch-compatible backends stay aligned.

I addressed the primary issue and both lower-severity cleanup items in separate commits:

1. Use one source of truth for FTS document filter-ID assignment: c0ea1f4

   • Added _iter_selected_documents_with_filter_ids() as the shared primitive for document selection, malformed-document skipping, emitted ordering, and permutation assignment.
   • Both _build_qrel_filter_ids() and FtsDocumentIterator now consume that primitive, preventing inserted IDs and qrel-pruning IDs from drifting.
   • Added a regression covering sparse selection with a skipped malformed document. VERIFIED: 35 focused tests passed.

2. Share Milvus CLI connection configuration: 1b25713

   • Added _build_milvus_config() and replaced repeated MilvusConfig(...) construction across all 18 Milvus CLI commands.
   • VERIFIED: 3 focused Milvus CLI tests passed.

3. Share Elasticsearch-compatible FTS mapping and BM25 settings: 078d524

   • Extracted the common text/filter mapping and BM25 similarity construction into elasticsearch_compatible.py.
   • ElasticCloudFtsConfig and OSSOpenSearchFtsConfig now delegate to the same helpers, while backend-specific lifecycle settings such as force merge remain separate.
   • VERIFIED: 3 focused Elasticsearch/OpenSearch configuration tests passed.

All three commits include the required DCO sign-off. Please take another look when convenient.