Hash Memory Engine

The Git-style memory layer for AI — store knowledge once, retrieve it efficiently, use it with any LLM.

A compressed, content-addressed memory & retrieval engine for RAG and LLM applications. It cuts vector-database storage by ~50% while keeping 96%+ of baseline retrieval quality — and it runs fully offline.

---

What is the Hash Memory Engine?

The Hash Memory Engine (HME) is an open-source, content-addressed memory layer for Retrieval-Augmented Generation (RAG). It is not a large language model and not a full vector database replacement — it is an optimisation layer that sits under your RAG stack and makes the memory it uses dramatically cheaper to store and faster to search, without sacrificing answer quality.

It combines five ideas that usually live in separate tools into one engine:

• Content-addressed storage — every text chunk is identified by the BLAKE3 hash of its normalised content (like Git commits), so identical chunks are stored exactly once.
• Exact deduplication — byte-identical content across documents collapses to a single physical copy.
• Compression — chunk text is stored with Zstandard.
• Compact semantic retrieval — float32, int8, and binary vector representations, with a two-stage binary shortlist → quantised rerank search.
• Plug-and-play LLM integration — works with any model; the LLM only ever receives original text, never hashes.

In one line: Store knowledge once, retrieve it efficiently, and use it with any LLM.

---

Why it matters (the headline result)

Measured on the full SciFact BEIR benchmark (5,183 docs → 12,084 chunks, 300 queries) with a real semantic embedder (all-MiniLM-L6-v2):

System	Recall@10	NDCG@10	Query latency	Total storage	vs. baseline
Float32 baseline (standard RAG)	0.811	0.656	25.7 ms	29.2 MB	—
Hash Memory Engine (INT8)	0.783	0.645	14.9 ms	13.9 MB	52% smaller · 42% faster

HME-INT8 keeps 96.5% of baseline Recall@10 and 98.4% of NDCG@10, while being ~42% faster and ~52% smaller. The float32 baseline of 0.811 / 0.656 matches the published BEIR numbers for this model, confirming the comparison is fair.

---

How it works

flowchart TD
    A[Documents] --> B[Normalise + Chunk]
    B --> C[BLAKE3 content hash]
    C -->|duplicate| D[Reuse existing chunk]
    C -->|new| E[Zstandard compress]
    E --> F[Embed → float / int8 / binary]
    F --> G[Binary index + rerank store]
    G --> H[(SQLite metadata + blobs)]

    Q[User question] --> R[Embed query]
    R --> S[Binary Hamming shortlist ~100]
    S --> T[Quantised float rerank → top-k]
    T --> U[Decompress original text]
    U --> V[LLM answer with citations]

Loading

Ingest: normalise → chunk → BLAKE3 hash → dedup → compress → embed → index. Query: embed → binary shortlist → quantised rerank → decompress → grounded, cited answer.

Full design: docs/architecture.md · docs/storage-format.md.

---

Key results (all measured, all reproducible)

Storage: where the savings come from

The float32 vectors are the entire storage bottleneck in classic RAG. Replacing them with an int8 rerank store (built from a binary shortlist) shrinks that component ~4× and cuts total system storage in half — with almost no quality loss.

Generalisation: not tuned to one dataset

One fixed configuration, run across four BEIR datasets from different domains:

Dataset	Domain	Recall@10 retention	Storage cut
SciFact	scientific claims	96.5%	52%
NFCorpus	medical	90.5%	52%
ArguAna	argument retrieval	99.2%	53%
Quora	duplicate questions	100.0%	68%

Scale: stable and predictable to 1,000,000 chunks

Storage grows linearly (~0.65 MB per 1k chunks), retrieval quality stays flat, and the engine is stable at 1M chunks. For low-latency serving at that scale, an optional faiss ANN backend brings query latency from ~54 ms down to ~2 ms. See docs/scale.md.

Multi-agent shared memory

Used as the shared memory for a multi-agent system — where agents communicate by writing findings and retrieving each other's — HME matches a standard shared vector store's cross-agent recall while using a fraction of the memory, and makes concurrent identical writes idempotent (one physical copy, N references).

Shared-memory design	Storage	Cross-agent recall@10
Naive (agents can't see each other)	32.9 MB	0.13
Shared vector DB (float32 — the usual approach)	32.9 MB	0.85
Hash Memory Engine	4.5 MB	0.86

Same communication quality as a vector-DB memory, ~86% less storage — and the win grows with agent redundancy (66% → 97%). Full method and honest caveats: docs/multi-agent.md.

---

Install

git clone https://github.com/shameerrahman1995/Hash_Memory_Engine.git
cd Hash_Memory_Engine
python3 -m venv .venv && source .venv/bin/activate
pip install -e .                    # core (numpy + blake3 + zstandard) — fully offline

# optional extras
pip install -e ".[embeddings]"      # sentence-transformers for real semantic quality
pip install -e ".[faiss]"           # faiss-cpu ANN backend for million-scale latency
pip install -e ".[dev]"             # pytest

Quickstart

from hme import HashMemoryEngine, Config

# Fully offline: deterministic hashing embedder + in-memory store.
engine = HashMemoryEngine(Config(embedding_backend="hashing"))

engine.ingest_text("Paris is the capital of France.", source="geo", document_id="geo")
engine.ingest_text("Python was created by Guido van Rossum.", source="py", document_id="py")
engine.finalize()

# Two-stage binary → quantised rerank is the default mode.
results = engine.search("What is the capital of France?", top_k=3)
for r in results:
    print(r.score, r.source, r.text)

# Grounded, cited answer (offline EchoLLM by default; plug in any LLM).
answer = engine.answer("What is the capital of France?", top_k=3)
print(answer.answer, answer.sources)

For production-quality semantics, use Config(embedding_backend="sentence-transformers", rerank_backend="int8").

Command line

python scripts/ingest.py    --store ./store --path ./docs
python scripts/query.py     --store ./store --query "how do I reset my password?" --answer
python scripts/evaluate.py  --dataset scifact --download --backend sentence-transformers
python scripts/verify.py    --store ./store          # content-addressed integrity check

---

Frequently asked questions

Is the Hash Memory Engine a vector database?

No. It is a memory/optimisation layer for RAG that you put in front of or instead of a vector store when storage and memory cost matter. It provides retrieval (float, int8, and binary), deduplication, and compression, and it can act as an optimisation layer alongside an existing vector database.

How does it reduce RAG storage cost?

Three independent mechanisms: (1) exact content-addressed deduplication removes byte-identical chunks; (2) Zstandard compression shrinks stored text; (3) quantised/binary vectors (int8 or binary codes) replace bulky float32 embeddings. Measured storage reduction is ~50% at equal quality, and up to ~79% on duplicate-heavy corpora.

Does compression or quantisation hurt retrieval quality?

Barely. The two-stage design — a fast binary Hamming shortlist followed by an int8 rerank — retains 96.5% of the float32 baseline Recall@10 on SciFact and ≥90% across four datasets. Text compression is lossless (byte-exact round-trip).

Which embedding models does it support?

Any. It ships with a deterministic hashing embedder (so it runs fully offline with zero downloads) and an optional sentence-transformers backend (e.g. all-MiniLM-L6-v2). The engine is model-independent.

Can it scale to millions of vectors?

Yes — tested stable and linear to 1,000,000 chunks. The exact numpy index is O(n); an optional faiss HNSW backend gives ~2 ms queries at 1M for low-latency production serving.

Is it safe to run in production?

It includes reference-counted deletion with garbage collection, an hme verify integrity checker, crash-safe atomic writes, storage-format versioning, thread-safe concurrent reads, and input/resource limits (decompression-bomb, path-traversal, SQL-injection, and oversized-input guards). See docs/reliability.md.

How is this different from just using FAISS or a vector DB?

FAISS/vector DBs index float vectors; they don't deduplicate content, compress text, or offer a content-addressed store. HME adds the storage and memory optimisation layer around retrieval — and can use FAISS underneath as its ANN backend.

---

Documentation

Doc	Contents
architecture.md	Data-flow diagrams, module map, technology choices
storage-format.md	SQLite schema, content addressing, index formats
benchmark-methodology.md	Hypothesis, baselines A–F, metrics
results.md	Full measured results (quantisation, generalisation)
scale.md	Scale sweep to 1M, faiss comparison, concurrency
reliability.md	Deletion/GC, hme verify, crash-safety, security
multi-agent.md	Multi-agent shared-memory benchmark vs vector-DB memory
testing.md	How to run the evaluation harness on BEIR datasets
roadmap.md	Phase-by-phase status

Regenerate the charts above with python scripts/make_charts.py (reads results/*.json).

---

Success criteria (from the development plan)

The prototype targets — and meets — the following: ≥50% storage reduction, ≥90–95% of baseline retrieval quality, exact duplicate elimination, stable operation at 100k+ chunks, and no more than 10% latency overhead (in fact it is faster). See docs/results.md for the measured pass/fail against every criterion.

Limitations (honest)

• The offline default embedder is a lexical hashing fallback — install the embeddings extra for full semantic quality.
• The default index is exact numpy brute-force (fine to ~100k; use the optional faiss backend for low-latency 1M+).
• Factual-correctness / hallucination answer metrics require a real LLM (scripts/answer_eval.py --llm openai); the offline EchoLLM is extractive.
• Learned/LSH binary codes and a REST API are on the roadmap.

Contributing

Contributions are welcome — new encoders, storage backends, framework adapters, benchmark datasets, and docs. See CONTRIBUTING.md and CODE_OF_CONDUCT.md. Good first issues are listed in the contributing guide.

License

Apache License 2.0 — free for commercial and open-source use.

---

_{Keywords: RAG memory · retrieval-augmented generation · vector database optimisation · embedding compression · binary embeddings · int8 quantisation · content-addressed storage · BLAKE3 deduplication · Zstandard · semantic search · LLM memory layer · storage cost reduction.}