streaming (incremental) MESSI

CONTINUITY.md

@@ -65,10 +65,13 @@ These algorithms do not all share the same dependency profile or feature set.
 
 - Top-k search is the baseline capability.
 - Range search is modeled through `SearchConfig` and is not universally implemented.
-- Streaming insert is implemented by `BruteForceSearch`, `LbBruteforce`, and `Coconut`;
+- Streaming insert is implemented by `BruteForceSearch`, `LbBruteforce`, `Messi`, and `Coconut`;
   the base implementation still throws for algorithms that do not support it.
 - Bruteforce streaming grows the owned in-memory database incrementally. LbBruteforce also
   computes SAX summaries incrementally, using the breakpoint set fixed by the initial build.
+- MESSI uses those fixed breakpoints to route stable SAX/position records into its live iSAX
+  tree, including new-root creation and leaf splitting. Callers must serialize inserts and
+  queries; simultaneous update/search is not supported.
 - `setNormalized(bool)` is a declaration about the input data, not a preprocessing step.
 - Data can come from in-memory arrays or file-backed sources via `DataSource`.
 

README.md

@@ -48,7 +48,7 @@ The following table summarizes the key features of each algorithm:
 |-----------|-------------|
 | **Bruteforce** | Naive parallel similarity search implementation with incremental streaming inserts |
 | **Lower Bound Bruteforce** | Optimized bruteforce with lower bounding and incremental streaming inserts |
-| **[MESSI](https://helios2.mi.parisdescartes.fr/~themisp/messi/)** | In-memory parallel similarity search |
+| **[MESSI](https://helios2.mi.parisdescartes.fr/~themisp/messi/)** | In-memory parallel similarity search with incremental iSAX-tree inserts |
 | **[PARIS](https://helios2.mi.parisdescartes.fr/~themisp/paris/)** | Disk-based parallel similarity search |
 | **[SING](https://helios2.mi.parisdescartes.fr/~themisp/sing/)** | GPU-accelerated in-memory parallel similarity search |
 | **[Odyssey](https://helios2.mi.parisdescartes.fr/~themisp/odyssey/)** | Distributed and parallel in-memory similarity search |
@@ -60,20 +60,43 @@ The following table summarizes the key features of each algorithm:
 
 ### Incremental streaming inserts
 
-`BruteForceSearch`, `LbBruteforce`, and `Coconut` implement the common streaming API. Build
+`BruteForceSearch`, `LbBruteforce`, `Messi`, and `Coconut` implement the common streaming API. Build
 the initial index once, then append one series or a contiguous batch without rebuilding:
 
 ```cpp
-daisy::BruteForceSearch search(daisy::DistanceType::L2_SQUARED);
+daisy::Messi search(daisy::DistanceType::L2_SQUARED);
 search.buildIndex(initial_data, initial_size, dim);
 search.insert(one_series);
 search.insertBatch(batch_data, batch_size);
 ```
 
 Inserted series receive consecutive IDs beginning at the size of the initial database and are
-immediately visible to top-k and range searches. `LbBruteforce` computes a SAX summary for each
-insert using the breakpoints established during the initial build. Inserts can reallocate the
-owned database, so callers should not retain a pointer returned by `getDatabase()` across them.
+immediately visible to supported top-k and range searches. `LbBruteforce` and `Messi` compute a
+SAX summary for each insert using the breakpoints established during the initial build. MESSI
+routes each new summary into the live iSAX tree and splits full leaves without rebuilding the
+index. Its first insert copies a borrowed initial in-memory database into owned growable storage.
+
+Streaming updates are not concurrent with queries. Inserts can reallocate the owned database,
+so callers should not retain a pointer returned by `getDatabase()` across them.
+
+### Range (distance-r) queries
+
+Every algorithm except Coconut's streaming-only paths answers range queries through
+`SearchConfig`. Instead of a fixed `k`, each query returns however many series fall within the
+radius, so the results come back as one vector per query:
+
+```cpp
+daisy::SearchConfig config;
+config.type = daisy::QueryType::RANGE;
+config.r = radius;                 // squared L2 distance
+
+std::vector<std::vector<daisy::idx_t>> I;
+std::vector<std::vector<float>> D;
+search.searchIndex(query, n_query, config, I, D);
+```
+
+`demos/demo_<Algorithm>_Range.cpp` shows this for each algorithm and cross-checks the returned
+sets against brute force.
 
 
 
@@ -195,6 +218,7 @@ cd build
 ./benchmark/bm_bruteforce_L2Square
 ./benchmark/bm_LbBruteforce_L2Square
 ./benchmark/bm_Messi_L2Square
+./benchmark/bm_Messi_Streaming
 
 # Advanced algorithms (if available)
 ./benchmark/bm_Odyssey_L2Square    # MPI required
@@ -221,5 +245,3 @@ For questions and suggestions through mail, you can contact us at [manos.chatzak
 
 
 
-
-

benchmark/CMakeLists.txt

@@ -209,6 +209,18 @@ if(DEBUG_MSG)
     message(STATUS "Include directories added for bm_Messi_L2Square.")
 endif()
 
+add_executable(bm_Messi_Streaming bm_Messi_Streaming.cpp)
+target_link_libraries(bm_Messi_Streaming
+    PRIVATE
+    benchmark::benchmark
+    benchmark::benchmark_main
+    dino_lib
+)
+target_include_directories(bm_Messi_Streaming
+    PRIVATE
+    ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+)
+
 # ////// FAISS FLAT //////
 if(DEBUG_MSG)
     message(STATUS "---")

benchmark/bm_Messi_Streaming.cpp

@@ -0,0 +1,59 @@
+#include <benchmark/benchmark.h>
+
+#include "../lib/algos/Messi.hpp"
+
+#include <cmath>
+#include <vector>
+
+namespace
+{
+    constexpr int DIM = 96;
+    constexpr int INITIAL = 4096;
+
+    std::vector<float> makeSeries(int n, int phase_offset)
+    {
+        std::vector<float> data(static_cast<size_t>(n) * DIM);
+        for (int i = 0; i < n; ++i)
+        {
+            float *series = data.data() + static_cast<size_t>(i) * DIM;
+            const float phase = static_cast<float>(phase_offset + i) * 0.013f;
+            for (int j = 0; j < DIM; ++j)
+                series[j] = std::sin(0.07f * j + phase) +
+                            0.5f * std::cos(0.19f * j - phase);
+        }
+        return data;
+    }
+}
+
+static void BM_Messi_InsertBatch(benchmark::State &state)
+{
+    state.PauseTiming();
+    const daisy::idx_t batch_size = static_cast<daisy::idx_t>(state.range(0));
+    auto initial = makeSeries(INITIAL, 0);
+    auto batch = makeSeries(static_cast<int>(batch_size), INITIAL);
+
+    daisy::MessiConfig config;
+    config.index_workers = 2;
+    config.search_workers = 2;
+    daisy::Messi search(daisy::DistanceType::L2_SQUARED, config);
+    search.buildIndex(initial.data(), INITIAL, DIM);
+    state.ResumeTiming();
+
+    for (auto _ : state)
+    {
+        search.insertBatch(batch.data(), batch_size);
+        benchmark::ClobberMemory();
+    }
+
+    state.SetItemsProcessed(
+        static_cast<int64_t>(state.iterations()) * static_cast<int64_t>(batch_size));
+}
+
+BENCHMARK(BM_Messi_InsertBatch)
+    ->Arg(1)
+    ->Arg(64)
+    ->Arg(1024)
+    ->Iterations(20)
+    ->Unit(benchmark::kMicrosecond);
+
+BENCHMARK_MAIN();

demos/CMakeLists.txt

@@ -44,6 +44,13 @@ if(BUILD_DEMO)
         ${CMAKE_CURRENT_SOURCE_DIR}/../commons
     )
 
+    add_executable(demo_Bruteforce_Range demo_Bruteforce_Range.cpp)
+    target_link_libraries(demo_Bruteforce_Range PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_Bruteforce_Range PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
     # ////// COCONUT (static + streaming) //////
     if(BUILD_COCONUT)
         if(DEBUG_MSG)
@@ -61,6 +68,13 @@ if(BUILD_DEMO)
             ${CMAKE_CURRENT_SOURCE_DIR}/../lib
             ${CMAKE_CURRENT_SOURCE_DIR}/../commons
         )
+
+        add_executable(demo_Coconut_Range demo_Coconut_Range.cpp)
+        target_link_libraries(demo_Coconut_Range PRIVATE dino_lib commons_lib)
+        target_include_directories(demo_Coconut_Range PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+            ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+        )
     elseif(DEBUG_MSG)
         message(STATUS "BUILD_COCONUT is OFF. Skipping Coconut demos.")
     endif()
@@ -138,6 +152,13 @@ if(BUILD_DEMO)
         ${CMAKE_CURRENT_SOURCE_DIR}/../commons
     )
 
+    add_executable(demo_LbBruteforce_Range demo_LbBruteforce_Range.cpp)
+    target_link_libraries(demo_LbBruteforce_Range PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_LbBruteforce_Range PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
     # ////// LBBRUTEFORCE DTW //////
     if(DEBUG_MSG)
         message(STATUS "---")
@@ -204,6 +225,20 @@ if(BUILD_DEMO)
         message(STATUS "Include directories added for demo_Messi_L2Square.")
     endif()
 
+    add_executable(demo_Messi_Streaming demo_Messi_Streaming.cpp)
+    target_link_libraries(demo_Messi_Streaming PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_Messi_Streaming PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
+    add_executable(demo_Messi_Range demo_Messi_Range.cpp)
+    target_link_libraries(demo_Messi_Range PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_Messi_Range PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
     # ////// MESSI DTW //////
     if(DEBUG_MSG)
         message(STATUS "---")
@@ -307,6 +342,13 @@ if(BUILD_DEMO)
             ${CMAKE_CURRENT_SOURCE_DIR}/../lib
             ${CMAKE_CURRENT_SOURCE_DIR}/../commons
         )
+
+        add_executable(demo_Odyssey_Range demo_Odyssey_Range.cpp)
+        target_link_libraries(demo_Odyssey_Range PRIVATE dino_lib commons_lib MPI::MPI_CXX)
+        target_include_directories(demo_Odyssey_Range PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+            ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+        )
     else()
         if(DEBUG_MSG)
             message(STATUS "BUILD_ODYSSEY_AVAILABLE is FALSE. demo_Odyssey_L2Square (MPI-dependent) will NOT be built.")
@@ -379,6 +421,13 @@ if(BUILD_DEMO)
         message(STATUS "Include directories added for demo_ParIS_DTW.")
     endif()
 
+    add_executable(demo_ParIS_Range demo_ParIS_Range.cpp)
+    target_link_libraries(demo_ParIS_Range PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_ParIS_Range PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
     # ////// SING L2Square //////
     if(DEBUG_MSG)
         message(STATUS "---")
@@ -396,6 +445,13 @@ if(BUILD_DEMO)
             ${CMAKE_CURRENT_SOURCE_DIR}/../lib
             ${CMAKE_CURRENT_SOURCE_DIR}/../commons
         )
+
+        add_executable(demo_Sing_Range demo_Sing_Range.cpp)
+        target_link_libraries(demo_Sing_Range PRIVATE dino_lib commons_lib)
+        target_include_directories(demo_Sing_Range PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+            ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+        )
     else()
         if(DEBUG_MSG)
             message(STATUS "BUILD_SING_AVAILABLE is FALSE. demo_Sing_L2Square will NOT be built.")
@@ -435,6 +491,13 @@ if(BUILD_DEMO)
         if(DEBUG_MSG)
             message(STATUS "Include directories added for demo_Sofa_L2Square.")
         endif()
+
+        add_executable(demo_Sofa_Range demo_Sofa_Range.cpp)
+        target_link_libraries(demo_Sofa_Range PRIVATE dino_lib commons_lib)
+        target_include_directories(demo_Sofa_Range PRIVATE
+            ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+            ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+        )
     else()
         if(DEBUG_MSG)
             message(STATUS "BUILD_SOFA_AVAILABLE is FALSE. demo_Sofa_L2Square (FFTW3-dependent) will NOT be built.")
@@ -471,6 +534,13 @@ if(BUILD_DEMO)
         message(STATUS "Include directories added for demo_Hercules_L2Square.")
     endif()
 
+    add_executable(demo_Hercules_Range demo_Hercules_Range.cpp)
+    target_link_libraries(demo_Hercules_Range PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_Hercules_Range PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
     # ////// DUMPYOS L2Square //////
     if(DEBUG_MSG)
         message(STATUS "---")
@@ -531,6 +601,13 @@ if(BUILD_DEMO)
         message(STATUS "Include directories added for demo_DumpyOS_DTW.")
     endif()
 
+    add_executable(demo_DumpyOS_Range demo_DumpyOS_Range.cpp)
+    target_link_libraries(demo_DumpyOS_Range PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_DumpyOS_Range PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
     # ////// FRESH L2Square //////
     if(DEBUG_MSG)
         message(STATUS "---")
@@ -591,6 +668,13 @@ if(BUILD_DEMO)
         message(STATUS "Include directories added for demo_Fresh_DTW.")
     endif()
 
+    add_executable(demo_Fresh_Range demo_Fresh_Range.cpp)
+    target_link_libraries(demo_Fresh_Range PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_Fresh_Range PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
 
 else()
     if(DEBUG_MSG)

demos/demo_Bruteforce_Range.cpp

@@ -0,0 +1,62 @@
+// Bruteforce range search: report every series within squared-L2 distance r of a query.
+// Brute force is the exact baseline that every other range demo checks itself against.
+
+#include "../commons/dataloaders.hpp"
+#include "../lib/daisy.hpp"
+
+#include <algorithm>
+#include <cstdio>
+#include <set>
+#include <vector>
+
+static void reportRange(const char *name, float r,
+                        const std::vector<std::vector<daisy::idx_t>> &I,
+                        const std::vector<std::vector<float>> &D)
+{
+    printf("=== %s range search (squared L2 <= %.1f) ===\n", name, r);
+    for (size_t qi = 0; qi < I.size(); qi++)
+    {
+        printf("Query %zu: %zu hits", qi, I[qi].size());
+        if (!I[qi].empty())
+        {
+            const auto minmax = std::minmax_element(D[qi].begin(), D[qi].end());
+            printf("  [closest=%.4f, farthest=%.4f]", *minmax.first, *minmax.second);
+        }
+        printf("\n");
+    }
+}
+
+int main()
+{
+    daisy::idx_t n_database = 200000;
+    unsigned long long dim = 96;
+    unsigned long long n_query = 10;
+
+    // Independent z-normalized series sit around 2*dim apart, so a radius below that
+    // keeps the result sets small without leaving every query empty.
+    const float r = 0.60f * 2.0f * dim;
+
+    float *database = loadRandomData(n_database, dim, 100, true);
+    float *query = loadRandomData(n_query, dim, 50, true);
+
+    printf("Loaded %llu database points and %llu query points with dimension %llu\n",
+           n_database, n_query, dim);
+
+    daisy::SearchConfig config;
+    config.type = daisy::QueryType::RANGE;
+    config.r = r;
+
+    daisy::BruteForceSearch bf_search(daisy::DistanceType::L2_SQUARED);
+    bf_search.setNumThreads(4);
+    bf_search.buildIndex(database, n_database, dim);
+
+    std::vector<std::vector<daisy::idx_t>> I;
+    std::vector<std::vector<float>> D;
+    bf_search.searchIndex(query, n_query, config, I, D);
+    reportRange("Bruteforce", r, I, D);
+
+    delete[] database;
+    delete[] query;
+
+    return 0;
+}