research(nightly): streaming WAL-ANN with coherence-gated merge (ADR-272)

[ruvnet]

Summary

• Introduces crates/ruvector-wal-ann: a streaming vector index crate implementing a three-tier WAL → coherence gate → navigable graph architecture (ADR-272 / SWAL-ANN).
• All inserted vectors are immediately searchable via WAL linear scan; coherence scoring drives adaptive merge timing rather than size-only policies used by every production vector database.
• Ships with a benchmark binary covering three gate strategies (EagerMerge, LazyMerge, CoherenceGatedMerge); all variants pass the acceptance criteria of recall@10 ≥ 0.70 and mean query latency < 5 ms on 3 000 × 64-dim f32 Gaussian data.

Changes

New crate: crates/ruvector-wal-ann

File	Purpose
src/wal.rs	VectorWal bounded in-memory buffer with brute-force linear-scan search
src/gate.rs	MergeGate trait + EagerGate, LazyGate, CoherenceGate
src/graph.rs	NavGraph incremental NSW with beam search, back-edge pruning, long-jump edges
src/lib.rs	WalAnnIndex<G> generic streaming index; sampled coherence scoring (O(1024·D))
src/bin/benchmark.rs	Three-variant benchmark binary

15 unit tests, 0 warnings, 0 unsafe.

New documents

• docs/adr/ADR-272-streaming-wal-ann.md — Architecture Decision Record
• docs/research/nightly/2026-07-14-streaming-wal-ann/README.md — Full research document
• docs/research/nightly/2026-07-14-streaming-wal-ann/gist.md — SEO-optimized public article

Workspace

• Cargo.toml — added crates/ruvector-wal-ann to workspace members

Benchmark Results

Dataset: 3 000 × 64-dim f32 Normal(0,1) · Queries: 100 × k=10

Variant	Merges	Insert (ms)	Vecs/sec	Recall@10	Mean (µs)	p95 (µs)	QPS
EagerMerge	94	319.6	9 388	0.716	110.1	150.8	9 084
LazyMerge	6	316.7	9 472	0.716	113.5	150.6	8 811
CoherenceGatedMerge	12	354.7	8 458	0.716	105.5	132.8	9 477

All variants PASS (recall@10 ≥ 0.70, mean latency < 5 ms). Memory: 1 156 KB per index at 3 K × 64-dim.

Architecture

insert(v) ──► VectorWal ──► CoherenceGate ──► NavGraph
                │                  │               │
            linear scan      quality check    beam search
           (always live)    (every 8 inserts)  (post-merge)

CoherenceGate fires a merge when coherence < 0.08 OR wal_size >= 256. Coherence is approximated in O(1024·D) via 16 WAL samples × 64 graph samples.

Test Plan

• cargo test -p ruvector-wal-ann — 15 unit tests pass
• cargo run --release -p ruvector-wal-ann --bin benchmark — all 3 variants PASS acceptance criteria
• cargo fmt --package ruvector-wal-ann — zero formatting diffs
• No secrets or credentials committed
• Phase 2: replace single-layer NSW with multi-layer HNSW backend (target recall ≥ 0.90)
• Phase 2: durable mmap WAL + MPSC multi-writer
• Phase 3: MCP tool surface + ruFlo integration + WASM target

Related

• ADR-254 (coherence-gated HNSW search)
• ADR-264 (LSM-ANN)
• ADR-268 (capability-gated ANN)
• Research: docs/research/nightly/2026-07-14-streaming-wal-ann/

🤖 Generated with claude-flow

https://claude.ai/code/session_019c8AvD6CVnA7GZyqzxg3X9

---

Generated by Claude Code