diff --git a/concepts/protocols.mdx b/concepts/protocols.mdx index 40dec54..9d45482 100644 --- a/concepts/protocols.mdx +++ b/concepts/protocols.mdx @@ -32,7 +32,7 @@ gives you endpoints and connections. From there, you choose: **Use existing protocols** - Pick from protocols built by the iroh community: - [`iroh-blobs`](/protocols/blobs) - Content-addressed blob storage and transfer -- [`iroh-docs`](/protocols/kv-crdts) - Collaborative key-value documents with CRDTs +- [`iroh-docs`](/protocols/documents) - Collaborative key-value documents with CRDTs - [`iroh-gossip`](/connecting/gossip) - Topic-based message broadcasting in swarms - [`iroh-automerge`](https://github.com/n0-computer/iroh-experiments/tree/main/iroh-automerge) - Automerge document sync (experimental) @@ -76,7 +76,7 @@ Ready to start building with protocols? **Using existing protocols:** - [Blob storage with iroh-blobs](/protocols/blobs) -- [Document collaboration with iroh-docs](/protocols/kv-crdts) +- [Document collaboration with iroh-docs](/protocols/documents) - [Message broadcasting with iroh-gossip](/connecting/gossip) **Building your own:** diff --git a/docs.json b/docs.json index 1733978..a98e75f 100644 --- a/docs.json +++ b/docs.json @@ -57,7 +57,7 @@ "group": "Sending Data", "expanded": false, "pages": [ - "protocols/kv-crdts", + "protocols/documents", "protocols/blobs", "protocols/rpc", "protocols/automerge", @@ -178,6 +178,10 @@ { "source": "/examples/examples", "destination": "/examples" + }, + { + "source": "/protocols/kv-crdts", + "destination": "/protocols/documents" } ] } diff --git a/index.mdx b/index.mdx index bd1547b..a5923c0 100644 --- a/index.mdx +++ b/index.mdx @@ -72,11 +72,11 @@ composable, so you can pick and choose what you need. Store and transfer large binary files. - Collaborative key-value store with conflict-free replication. + Collaborative key-value documents with conflict-free replication. anyhow::Result<()> { + // create an iroh endpoint that includes the standard address lookup + // mechanisms we've built at number0 + let endpoint = Endpoint::bind(presets::N0).await?; + + // build the blobs store (in-memory here; use FsStore::load for persistence) + let blobs = MemStore::default(); + + // build the gossip protocol + let gossip = Gossip::builder().spawn(endpoint.clone()); + + // build the docs protocol + // use Docs::persistent(path) for on-disk storage instead + let docs = Docs::memory() + .spawn(endpoint.clone(), (*blobs).clone(), gossip.clone()) + .await?; + + // register all three protocols on the router + let _router = Router::builder(endpoint.clone()) + .accept(BLOBS_ALPN, BlobsProtocol::new(&blobs, None)) + .accept(GOSSIP_ALPN, gossip) + .accept(DOCS_ALPN, docs.clone()) + .spawn(); + + // docs is ready — see the next sections for how to use it + Ok(()) +} +``` + + +`Docs::memory()` keeps everything in RAM and is perfect for experimenting. +For real use, pair `Docs::persistent(path)` with `FsStore::load(path)` from +`iroh-blobs` so both metadata and content survive restarts. + + +## Creating and sharing a document + +Once `docs` is spawned, you need two things before you can write: an **author** +to sign your entries, and a **document** to write into. + +```rust +// create an author (or load one you saved previously) +let author = docs.author_create().await?; + +// create a new, empty document +let doc = docs.create().await?; + +// generate a ticket that grants write access to peers +use iroh_docs::api::protocol::ShareMode; +let ticket = doc.share(ShareMode::Write, Default::default()).await?; +println!("share this ticket with a peer: {ticket}"); +``` + +On the other side, a peer imports the ticket to join the same document: + +```rust +use std::str::FromStr; +use iroh_docs::DocTicket; + +let ticket = DocTicket::from_str(&ticket_str)?; +let doc = docs.import(ticket).await?; +``` + +## Writing and reading entries + +Entries are `(key, value)` pairs signed by an author. The value is stored as a +blob, so writing an entry hashes your bytes into the blobs store and records +the hash in the document. + +```rust +// write an entry +doc.set_bytes(author, b"greeting".to_vec(), "hello, world".into()).await?; + +// read one entry back +use iroh_docs::store::Query; + +if let Some(entry) = doc.get_one(Query::single_latest_per_key().key_exact("greeting")).await? { + // the entry holds the hash; fetch the actual bytes from the blobs store + let bytes = blobs.blobs().get_bytes(entry.content_hash()).await?; + println!("{}", std::str::from_utf8(&bytes)?); +} + +// iterate all entries, latest write per key +use n0_future::StreamExt; + +let mut entries = doc.get_many(Query::single_latest_per_key()).await?; +while let Some(entry) = entries.next().await { + let entry = entry?; + let bytes = blobs.blobs().get_bytes(entry.content_hash()).await?; + println!("{:?} = {:?}", entry.key(), bytes); +} +``` + + +The document only stores the hash of each value. If you want the content, +fetch it from the blobs store. When two peers sync, docs exchanges the entry +metadata and blobs transfers any content the other side is missing. + + +## Reacting to sync + +`doc.subscribe()` returns a stream of live events — new entries from peers, +sync progress, content download completions — which is usually how UIs stay +up to date: + +```rust +use iroh_docs::engine::LiveEvent; + +let mut events = doc.subscribe().await?; +while let Some(event) = events.next().await { + match event? { + LiveEvent::InsertRemote { entry, .. } => { + println!("peer inserted {:?}", entry.key()); + } + LiveEvent::ContentReady { hash } => { + println!("content {hash} is now available locally"); + } + _ => {} + } +} +``` + +## How sync works + +Peers converge by exchanging a small number of messages using **range-based +set reconciliation** — recursively partitioning their entry sets and comparing +fingerprints of the partitions to detect where they disagree. The algorithm is +described in [Meyer 2022](https://arxiv.org/abs/2212.13567); the key property +is that fully-in-sync peers only need to exchange a single fingerprint to +confirm it. + +The crate exposes a generic storage interface with in-memory and persistent +file-based implementations. The persistent one uses +[redb](https://github.com/cberner/redb), an embedded key-value store, and +persists the whole store with all replicas to a single file. + +## Full examples + +- [iroh-docs setup example](https://github.com/n0-computer/iroh-docs/tree/main/examples) — the minimal setup shown above as a runnable file. +- [tauri-todos](https://github.com/n0-computer/iroh-examples/tree/main/tauri-todos) — a desktop todo app that uses `iroh-docs` end-to-end: persistent storage, ticket-based sharing, live sync, and an application model layered over entries. \ No newline at end of file diff --git a/protocols/kv-crdts.mdx b/protocols/kv-crdts.mdx deleted file mode 100644 index 656ff70..0000000 --- a/protocols/kv-crdts.mdx +++ /dev/null @@ -1,85 +0,0 @@ ---- -title: "Key-value Store" ---- - - -A key-value store for multi-dimensional documents, built on CRDTs, with an -efficient synchronization protocol. `iroh-docs` provides a protocol handler for -the iroh networking stack, enabling storage and synchronization of documents over -peer-to-peer connections. - -## When would I use this? - -You would use `iroh-docs` when you need a distributed key-value store that can -handle concurrent updates from multiple peers, ensuring eventual consistency -without conflicts. This is particularly useful for collaborative applications, -distributed configuration management, and any scenario where data needs to be -shared and synchronized across multiple devices or users. - - -## Installation - -``` -cargo add iroh-docs -``` - -- [API documentation](https://docs.rs/iroh-docs/latest/iroh_docs/) - -## Example - - -```rust -use iroh::{protocol::Router, Endpoint}; -use iroh_blobs::{BlobsProtocol, store::mem::MemStore, ALPN as BLOBS_ALPN}; -use iroh_docs::{protocol::Docs, ALPN as DOCS_ALPN}; -use iroh_gossip::{net::Gossip, ALPN as GOSSIP_ALPN}; - -#[tokio::main] -async fn main() -> anyhow::Result<()> { - // create an iroh endpoint that includes the standard discovery mechanisms - // we've built at number0 - let endpoint = Endpoint::bind(presets::N0).await?; - - // build the gossip protocol - let gossip = Gossip::builder().spawn(endpoint.clone()); - - // build the docs protocol - // use Docs::persistent to use local storage - let docs = Docs::memory() - .spawn(endpoint.clone(), (*blobs).clone(), gossip.clone()) - .await?; - - // create a router builder, we will add the - // protocols to this builder and then spawn - // the router - let builder = Router::builder(endpoint.clone()); - - // setup router - let _router = builder - .accept(GOSSIP_ALPN, gossip) - .accept(DOCS_ALPN, docs) - .spawn(); - - // do fun stuff with docs! - Ok(()) -} - -``` - -## How it works - -The crate operates on Replicas. A replica contains an unlimited number of -Entries. Each entry is identified by a key, its author, and the replica's -namespace. Its value is the 32-byte BLAKE3 hash of the entry's content data, the -size of this content data, and a timestamp. The content data itself is not -stored or transferred through a replica. - -Range-based set reconciliation is a simple approach to efficiently compute the -union of two sets over a network, based on recursively partitioning the sets and -comparing fingerprints of the partitions to probabilistically detect whether a -partition requires further work. - -The crate exposes a generic storage interface with in-memory and persistent -file-based implementations. The latter makes use of [redb], an embedded -key-value store, and persists the whole store with all replicas to a single -file. \ No newline at end of file diff --git a/what-is-iroh.mdx b/what-is-iroh.mdx index b7910b9..f58f2be 100644 --- a/what-is-iroh.mdx +++ b/what-is-iroh.mdx @@ -24,8 +24,8 @@ integrity. building applications that can operate without reliance on servers. - **Files & blobs**: With protocols like [iroh-blobs](/protocols/blobs), iroh enables efficient file transfer. -- **Structured data**: iroh's support for flexible data protocols like [KV -CRDTs](/protocols/kv-crdts) and [Automerge](/protocols/automerge) allows +- **Structured data**: iroh's support for flexible data protocols like +[Documents](/protocols/documents) and [Automerge](/protocols/automerge) allows developers to build applications that support real-time collaborative editing and data synchronization. Any kind of CRDT or OT sync protocol can be integrated. - **Real-time communication**: Build [chat applications](/examples/chat), [RPC @@ -96,7 +96,7 @@ this node, routed by their ALPN. ## Getting started To get started with iroh, check out the [quickstart guide](/quickstart) or explore the -[protocols documentation](/protocols/kv-crdts) to see what protocols are available and +[protocols documentation](/protocols/documents) to see what protocols are available and how to use them in your applications. Read the [how it works documentation](/concepts/endpoints) to understand the underlying