Add: Repeated Levenshtein dictionary search by grouville · Pull Request #10 · ashvardanian/StringWars

grouville · 2026-08-16T06:55:34Z

This PR adds a benchmark for exact bounded Levenshtein search over a dictionary that is built once and searched many times.

It accompanies StringZilla #336. The existing similarities/ benchmark measures a full distance matrix. This benchmark covers a different operation: build one dictionary index, submit repeated queries, and return every dictionary entry inside an inclusive edit bound.

What this adds

The benchmark has one deterministic query generator and public adapters for StringZilla, RapidFuzz, SymSpell, Rust fst, Tantivy, and Lucene. Byte and UTF-8 correctness use separate references.

One driver now builds the shared low-bound track, validates every adapter, and only then records timings. Each repetition starts fresh processes in a seeded shuffled order. It keeps the runners' original key-value logs and writes a JSON manifest with input hashes, revisions, commands, execution order, wall time, and peak process memory.

The mixed workload gives equal weight to exact queries, substitutions, insertions, deletions, several two-edit combinations, adjacent swaps, and longer extensions. A label only records how a query was made from one source word. It does not assume that no other dictionary word matches.

Contract before speed

The directly comparable result is:

(query ID, original dictionary ID, exact plain Levenshtein distance)

Every match inside the bound must be returned. Duplicate dictionary strings keep separate IDs. An adjacent swap costs two unit edits. Direct timings are accepted only after canonical binary result files match.

This matters because the available tools do not all answer the same question:

Runner	Reuses a dictionary	Timed result	Direct speed comparison
StringZilla public C API	Yes	Every original ID and exact distance	Reference implementation
RapidFuzz materialized scan	No index	The same complete result	Yes, but primarily a correctness oracle
SymSpell exact compatibility	Yes	The same result on unique lowercase dictionaries	Yes on that restricted track
SymSpell native	Yes	Native suggestions and distances	No, result counts differ
Rust `fst`	Yes	Matching IDs for unique ASCII keys	No exact distances
Tantivy	Yes	Matching document IDs	No exact distances
Lucene exact automaton	Yes	Match count	No IDs or exact distances

The benchmark keeps the last four because they explain how the new operation stands against real ecosystem tools. It does not turn less output into an artificial direct speedup claim.

Timing behavior

Dictionary construction is always reported separately. StringZilla exposes four query modes because one warm number cannot represent every deployment:

cold_end_to_end uses a fresh reader and includes output sizing, allocation, retry, and materialization.
warm_presized measures a repeated batch after reader memory and exact result capacity are available.
steady_growable reuses a grow-only service buffer and includes a resize and retry when needed.
single_query_latency reports p50, p95, and p99 for individual queries.

Threshold-specialized runs build one index per maximum bound. Shared-index runs build once and query the same index at several smaller bounds. Larger-bound sweeps remain separate because several ecosystem tools stop at k=2 and because a high-result workload can measure output bandwidth more than search.

Current comparison

This preliminary run used 213,557 unique lowercase English words and 10,000 deterministic mixed queries on one pinned Intel Core i5-9300H AVX2 core. Each line is the median of five repetitions, with no cache eviction. StringZilla was rebuilt from the exact reviewed head of PR #336.

Complete-output timings:

Runner	`k=1`	`k=2`	Returned matches
StringZilla cold end-to-end	13.37 ms	162.72 ms	7,742 / 89,980
StringZilla warm pre-sized	6.50 ms	87.91 ms	7,742 / 89,980
StringZilla steady growable	6.55 ms	158.49 ms	7,742 / 89,980
SymSpell exact compatibility	65.44 ms	718.72 ms	7,742 / 89,980
RapidFuzz materialized scan	51.92 s	67.88 s	7,742 / 89,980

StringZilla, RapidFuzz, and exact SymSpell produced byte-identical canonical result files at both bounds. SymSpell is the closest reusable index in this table. On these medians, StringZilla's warm path is 10.1x faster at k=1 and 8.2x faster at k=2. The growable k=2 run retried because its initial capacity was intentionally small.

RapidFuzz scans every dictionary word for every query. Its result is essential for correctness, but its large time is not presented as the headline indexed comparison.

Ecosystem context:

Runner	`k=1`	`k=2`	What differs
SymSpell native	60.82 ms	693.79 ms	Different distance and duplicate behavior; 8,788 / 92,156 results
Rust `fst`	2.049 s	14.941 s	IDs only, unique ASCII keys
Tantivy	808.11 ms	6.387 s	Document IDs only
Lucene exact automaton	5.715 s	47.765 s	Count only

FST, Tantivy, and Lucene returned the expected 7,742 and 89,980 match totals. Tantivy does not promise result iteration order, so its checksum is now order-independent and stays stable across repeats. Lucene's first JVM repetition was visibly slower; the table uses the five-run median and keeps it out of direct ratios.

The corresponding construction times were 0.220 / 1.443 seconds for the threshold-specific StringZilla indexes, 0.474 / 1.723 seconds for SymSpell, 0.094 seconds for FST, 0.278 seconds for Tantivy, and 3.374 seconds for Lucene.

Reproducibility

Final reviewed head: dde76292117f3e9c35394e864ad8986f12217719.

The dictionary SHA-256 is bb04a5beebdd2b1bf84fa0ae0e6445043f0af6d3c9a289f4a98b2a889a7952c3. The query SHA-256 is 74347f357ac188cd78d7699015ab721e409e4498b52ed7add67bc1361cdfde6e with seed 243.

The run used GCC 13.3.0, Rust 1.97.1, OpenJDK 21.0.12, Lucene 10.3.1, and the revisions pinned by this branch. Rust adapters build in release mode, the Lucene adapter builds with Maven, and the exact result artifacts match as described above.

The source tree contains the benchmark, its driver, and a small CI dictionary. Machine-specific manifests and logs remain generated artifacts. They will be archived with the final run and linked from this draft rather than committed as a permanent table for one host.

Existing boundaries

The current English comparison is byte-oriented, unique, and lowercase because that is the common subset supported by every low-bound adapter.
The UTF-8 track checks codepoint distance separately and does not make every ecosystem adapter Unicode-equivalent.
Lucene and Tantivy stop at k=2 in this comparison. Wide-bound evidence is a separate StringZilla and RapidFuzz exact-output sweep.
Native ranking, top-N suggestions, weighted distance, normalized similarity, and mutable dictionaries are different contracts.
The benchmark does not claim that a warm pre-sized batch is cold request latency.

Why this PR is still a draft

The adapters now expose the relevant semantic differences, but the current numbers are not publication evidence yet:

The new CI job builds all six adapters on a small deterministic workload. Exact artifacts must match across StringZilla, RapidFuzz, and SymSpell, while FST, Tantivy, and Lucene must return the same totals. GitHub currently marks its first run as requiring maintainer approval because the workflow was introduced from a fork. This is a correctness check, not a hosted-runner performance claim.
The current table predates the new shuffled driver. It came from a shared local host with unrelated load, and StringZilla-only k=2 repetitions varied materially afterward.
The final run still needs the driver's 20 repetitions on quiet pinned CPUs, plus frequency and NUMA metadata, retained index size, and reader scratch in the archived machine record.
AVX2 and AVX-512 should be reported separately. The algorithm is not AVX-512-only, but the final architecture claim should be measured rather than inferred.
Duplicate-heavy, longer-text, and natural UTF-8 workloads still need their own result tables before making a broader claim.

Until those points are closed, the current result supports continued review of StringZilla #336. It does not establish a broad state-of-the-art claim.

Suggested review order

Commits 1 through 3 add repeatable queries, the public StringZilla runner, and the RapidFuzz exact-result check.
Commits 4 through 8 add SymSpell, FST, Tantivy, Lucene, and UTF-8 references one at a time.
Commits 9 through 11 correct the edit distribution and separate cold, warm, growable, count-only, and materialized work.
Commit 12 removes avoidable allocation work from the exact SymSpell adapter.
Commit 13 documents the contract and final-run protocol.
Commit 14 makes Tantivy checksums independent of its unspecified result order.
Commit 15 gives Lucene the same configurable repetition count as the other runners.
Commit 16 adds the reproducible build, validation, shuffled execution, raw logs, and manifest.
Commit 17 runs the same validation against every adapter in CI.

Every implementation needs to receive the same queries. Add a small generator whose output is fixed by the dictionary, query count, mode, and seed. Mixed files evenly interleave exact words, one-edit changes, two-edit changes, and five-character extensions. Those labels only describe how a query was made. They are not treated as expected answers because another dictionary entry may still match. Byte and UTF-8 modes follow the same rules. UTF-8 edits operate on decoded characters, so one generated edit remains one edit. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Add the StringZilla side of repeated fuzzy search over a dictionary that is built once and then left unchanged. The runner calls the public C batch API rather than the internal C++ index used while the algorithm was being developed. Dictionary construction and one output-sizing pass stay outside the query timer. The three sparse result arrays are then reused, while finding and writing every query and dictionary match remains inside the timer. Result sorting and file output stay outside it. The same runner covers byte and UTF-8 behavior, one or several CPU cores, cache eviction, repeated batches, and bounds one through four. It validates every returned query ID, dictionary ID, and distance before writing the shared result format. On the local Intel AVX2 run with 213,557 words and 10,000 queries, one core took 7.45 ms at k=1 and 66.60 ms at k=2. Four physical cores took 2.11 and 17.99 ms. The complete serial and parallel files matched RapidFuzz. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Use RapidFuzz as the reference for the byte benchmark. It receives the same dictionary, queries, and inclusive distance bounds as StringZilla. It scans the complete dictionary for every query, so it checks correctness rather than serving as the closest indexed performance comparison. Both runners write the same file format. Comparing those files checks every dictionary ID and exact distance for every query, not only the final match count. The complete results matched at k=1 and k=2 on the recorded English run. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Add the closest reusable index found for this workload, pinned to the official SymSpell-Rust repository. SymSpell normally lowercases words, counts an adjacent swap as one edit, and cannot preserve duplicate IDs. The runner therefore uses unique lowercase input and checks every suggestion again with plain Levenshtein distance. Sorting is only needed for the result file and stays outside the timer. On the recorded English run, StringZilla took 1.948 ms at k=1 and 41.555 ms at k=2. SymSpell took 20.931 and 455.857 ms. Both complete result files matched RapidFuzz. The comparison is part of the normal StringWars Cargo target and dependency lock instead of a separate Rust project. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Add Rust fst 0.4.7 to the shared dictionary-search benchmark. fst returns dictionary IDs without exact distances and cannot keep duplicate keys. The comparable run therefore uses unique ASCII input, where its character behavior and StringZilla's byte behavior agree. On the English run, fst took 0.866, 5.469, 62.284, and 226.019 seconds at bounds one through four. StringZilla took 1.948 ms, 41.555 ms, 5.461 seconds, and 16.223 seconds. Match counts agreed, but fst does less output work, so that difference stays visible. The optional Unicode path first checks three one-character examples. The pinned fst version misses valid substitutions there, so the runner stops instead of timing answers that do not match. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Add Tantivy 0.26.1 through its public fuzzy term query, with adjacent swaps disabled. The runner builds one in-memory index, waits for its indexing work to finish, then collects every matching document address. Build and query time are reported separately. Tantivy returns IDs without exact distances and supports bounds one and two, so the output states those differences. On the English run, Tantivy took 485.1 ms at k=1 and 4.183 seconds at k=2. StringZilla took 1.948 and 41.555 ms. Most of this commit is the generated Cargo lock update for Tantivy's search and indexing dependencies. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Add Lucene 10.3.1 on OpenJDK 21 through its public fuzzy query and Levenshtein automaton. The normal fuzzy query applies a shorter-term rule that plain Levenshtein search does not have. Its different match count is reported but not used for a speed ratio. The automaton mode disables adjacent swaps and returns the expected totals, so that is the comparable mode. Lucene returns hit counts without dictionary IDs or exact distances. On the English run, its exact automaton took 1.676 seconds at k=1 and 14.965 seconds at k=2. StringZilla took 1.948 and 41.555 ms. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Add the RapidFuzz reference for Unicode search. It validates and decodes UTF-8 before timing, then measures distance between Unicode characters. The helper extracts 348,980 unique lowercase terms from the Simplified Chinese dictionary without changing their text. StringZilla and RapidFuzz produced identical complete result files on both the small non-ASCII test and the natural Chinese data. The Chinese run returned 2,219,220 matches at k=1 and 343,237,926 at k=2. StringZilla took 16.017 ms and 5.149 seconds. The very large k=2 output is an important limit on broader speed claims. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

The first mixed workload changed sampled words mostly through substitutions. That was deterministic, but it did not exercise the insertion and deletion behavior that a Levenshtein index is designed to handle. Generate exact queries, each one-edit operation, several two-edit combinations, adjacent swaps, and a five-symbol extension in equal proportions. A swap still costs two plain Levenshtein edits. The mode name describes only how the query was made from its sampled source. It makes no claim about distance from other dictionary entries. The complete-result oracle remains responsible for the answer. Use the same selection and mutation rules for bytes and decoded UTF-8 codepoints. Skip source words that cannot support a requested operation instead of silently turning it into a different edit. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

The StringZilla runner used one untimed search to discover the exact result capacity, then timed the same batch with warm reader memory and perfect output sizing. That is a useful steady-state measurement, but it should not stand in for every query cost. Report four named modes instead: a fresh reader with sizing and retry, a warm pre-sized batch, a reusable growable service buffer, and single-query p50, p95, and p99 latency. Keep dictionary construction separate and print the work included by each line. A threshold-specialized run still builds the low-bound indexes independently. A shared run builds one index at the requested maximum and queries every smaller bound, which matches a deployed service that accepts different thresholds. Allow the exact StringZilla and RapidFuzz track to sweep through distance 254. The ecosystem adapters keep their smaller limits, so large-bound crossover results remain a separate complete-output comparison rather than being mixed into the low-bound table. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Counting matches and returning every query ID, dictionary ID, and distance are different amounts of work. Report them as separate RapidFuzz modes so a count-only result is never compared directly with StringZilla full output. The materialized mode keeps one result vector across repeats, computes the same checksum fields as the StringZilla runner, and continues to write the common binary artifact used for exact result comparison. Allow the byte and UTF-8 correctness runners through distance 254. This supports the separate large-bound crossover sweep without changing the smaller limits of tools whose own APIs stop earlier. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

SymSpell does not return the same contract as StringZilla. Its native suggestions may use different distance behavior, and the adapter still needs to recover every exact plain-Levenshtein ID before a direct timing comparison is valid. The first compatibility adapter decoded strings and allocated dynamic-programming rows while processing every suggestion. That measured avoidable harness work rather than the cost of the lookup. Decode the dictionary and queries once, then reuse the verifier rows and match buffer. Report native SymSpell output and exact compatibility output as separate modes. The exact mode remains restricted to unique lowercase dictionaries because those are real limits of this adapter. The benchmark target compiles with the pinned SymSpell revision, and its tiny complete-output artifact matches the StringZilla runner. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Repeated dictionary retrieval is not the same workload as the existing dense distance matrices, so give it one focused page and link that page from the root benchmark catalogue. Document the exact output contract before listing tools. StringZilla, RapidFuzz, and the exact SymSpell mode can be compared only when they return the same original IDs and distances. Native suggestions, unique terms, and counts remain useful context but do not support a direct speedup claim. Describe the generated edit shapes, cold and warm timing modes, threshold-specialized and shared indexes, required machine metadata, and the common binary result check. Keep the larger-bound crossover separate because several ecosystem tools stop at small distances. Results and raw machine logs are intentionally absent. They belong in the draft PR and archived run artifacts after the protocol passes, not as permanent numbers in the benchmark source tree. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

Tantivy returns the same document set on every repeat, but DocSetCollector does not promise one iteration order. Feeding those addresses into an order-sensitive checksum made identical results print different checksums and weakened the benchmark evidence. Combine document addresses with a commutative checksum instead. Match counts and timed work are unchanged, while five repeated runs now report one checksum at both k=1 and k=2. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

The Lucene runner always measured three repetitions while the benchmark protocol requires at least twenty for final evidence. That made it too easy to publish a Lucene number collected under a different method from the other runners. Read the shared STRINGWARS_REPEATS setting, keep three as the convenient local default, and reject invalid values. This also let the current comparison use five repetitions consistently across every available adapter. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

The individual adapters already print the work they perform, but reproducing the full comparison still required a long sequence of manual build, query generation, validation, and timing commands. That made it easy to omit one adapter, reuse a favorable order, or publish timings before checking the complete answers. Add one focused driver for the shared k=1 and k=2 track. It builds the existing C++, Rust, and Lucene runners, generates the mixed workload, and refuses to time invalid results. StringZilla, RapidFuzz, and exact SymSpell must produce byte-identical artifacts. FST, Tantivy, and Lucene must return the same match totals. Each measured repetition starts fresh processes in a seeded shuffled order. The driver preserves every runner log and writes a small manifest with input hashes, repository and dependency revisions, commands, environment settings, wall time, and peak process memory. Host-specific controls and published machine logs remain outside the source tree. The six-adapter validation passed on the included small dictionary, and a fresh source build reached the same checks without prepared binaries. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

The benchmark branch had no automated check, so a runner could stop compiling or drift to a different result contract without making the PR fail. Add one small unique lowercase dictionary and run the same public driver in validation-only mode. CI builds all six adapters against the reviewed StringZilla and RapidFuzz revisions, compares every exact artifact at k=1 and k=2, checks the ID-only and count-only totals, and uploads the manifest and raw logs. This is deliberately a correctness job, not a hosted-runner performance claim. It uses 100 deterministic queries and one repetition, while controlled benchmark machines keep the separate 20-run protocol. Signed-off-by: Guillaume de Rouville <guillaume.derouville@gmail.com>

grouville · 2026-09-01T16:23:39Z

Closing this together with the related StringZilla work, since we’re not planning to pursue the underlying implementation further for now.

The benchmarking methodology and results are still useful context, and can be picked back up if we revisit this area.

Related implementation: StringZilla #336

grouville mentioned this pull request Aug 16, 2026

Add: Exact Levenshtein search for immutable dictionaries ashvardanian/StringZilla#336

Closed

grouville marked this pull request as draft August 16, 2026 07:43

grouville force-pushed the levenshtein-index-benchmarks branch from 47fc7f0 to 3597d94 Compare August 16, 2026 08:48

grouville added 13 commits August 17, 2026 14:04

grouville force-pushed the levenshtein-index-benchmarks branch from 76d0717 to 56e5489 Compare August 17, 2026 21:07

grouville added 4 commits August 17, 2026 14:56

grouville closed this Sep 1, 2026

Navigation Menu

Add: Repeated Levenshtein dictionary search - #10

Add: Repeated Levenshtein dictionary search#10
grouville wants to merge 17 commits into
ashvardanian:mainfrom
grouville:levenshtein-index-benchmarks

grouville commented Aug 16, 2026 •

edited

Loading

Uh oh!

grouville commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

grouville commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

Contract before speed

Timing behavior

Current comparison

Reproducibility

Existing boundaries

Why this PR is still a draft

Suggested review order

Uh oh!

grouville commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

grouville commented Aug 16, 2026 •

edited

Loading