Vector Database Benchmark

This repository contains a comprehensive benchmark tool designed to evaluate the quality and performance of various vector search engines. It measures insertion speed, search latency (diverse, sequential, filtered, and bulk), recall accuracy, and memory usage using the SIFT1M dataset.

Supported Engines

The benchmark supports both widely available open-source engines and specialized custom implementations:

Standard Engines

• ChromaDB: Popular open-source vector database.
• LanceDB: Serverless vector database based on the Lance data format.
• Qdrant and Qdrant-Edge: High-performance vector search engine with an advanced filtering system.
• FAISS: Efficient similarity search and clustering of dense vectors.
• USearch: A smaller, faster HNSW implementation.

Custom Engines (Internal/Private)

• MeMo (pymemo): A custom embedded database, optimized for local vector storage and retrieval.
• mSEARCH: A modular HNSW vector search implementation.

Important

The custom engines (MeMo and mSEARCH) are in-development and now available for Windows (Python 3.13).

• Release: Precompiled binaries are ready in the latest repository release.
• Custom Requests: For other platforms or specific builds, please contact me on GitHub.

Setup and Usage

Prerequisites

Ensure you have the required dependencies installed:

pip install -r vector_bench_requirements.txt

Running the Benchmark

You can run the benchmark using the main script. By default, it uses 1,000 documents across all supported datasets. Use --full for a deep-dive across multiple scales.

python vector_bench.py --docs 10000 --datasets sift,glove-25

CLI Options

Flag	Type	Default	Description
--docs	int	1000	Number of documents to index/search.
--full	flag	False	Run the full suite with scales of 1k, 10k, and 100k docs.
--datasets	str	all	Comma-separated list of datasets (e.g., sift,glove-25) or all.
--output	str	benchmark_results.md	Name of the output Markdown file for report generation.

Loading Custom Engines

To enable the custom engines, place their respective libraries/modules in a local directory named numerable at the root of the benchmark directory:

.
├── vector_bench.py
├── numerable/      <-- Place pymemo and msearch libraries here
└── data/           <-- Automatically downloaded datasets (SIFT, GloVe, etc.)

The script will automatically detect and include these engines in the benchmark if found.

Benchmark Results

The latest benchmark results, including detailed metrics for each engine across various datasets and scales, can be found in the results directory:

• 📊 Detailed Markdown Report
• 🌐 Visual Comparison (HTML)

Methodology

• Datasets: Automatically downloaded from ANN-Benchmarks. Supported datasets include:
  • sift (128d, L2)
  • glove-25 (25d, Cosine)
  • glove-100 (100d, Cosine)
  • nytimes (256d, Cosine)
  • deep1b (96d, Cosine)
  • fashion-mnist (784d, L2)
• Isolation: Each engine is executed in a fresh subprocess to ensure accurate memory (RSS) and resource tracking.
• Recall Calculation: Calculated dynamically against the current subset of the dataset (Recalculated Ground Truth).
• Latency: Measured for single queries, sequential repeats, filtered searches, and bulk batches.