Sync collection registration + core API catch-up (lite→Origin round-trip)#5
Merged
Conversation
Replace the single monolithic trait_impl.rs (564 lines) with a trait_impl/ directory containing document.rs, graph.rs, vector.rs, dispatch.rs, sql_lifecycle.rs, and mod.rs. Each file covers one logical slice of the NodeDb trait implementation. Array CRDT sync state (array_replica, array_schemas, array_outbound, array_inbound, array_catchup) is now gated behind `#[cfg(not(target_arch = "wasm32"))]` to allow the same codebase to compile for native and WASM targets. Fix CsrIndex checkpoint deserialization to use the fallible `from_checkpoint` path (returns `Result<Option<_>>`) instead of the deprecated infallible unwrap variant.
All graph methods now require an explicit collection name: graph_insert_edge, graph_delete_edge, graph_traverse, graph_shortest_path, and graph_stats. This aligns the graph API with the rest of the NodeDb trait where every operation is scoped to a named collection. The change propagates through all binding layers: - C FFI (nodedb_lite.h, ffi_graph.rs, jni_bridge) - WASM bindings (graphInsertEdge, graphTraverse, graphDeleteEdge, graphShortestPath, graphStats) - Integration and e2e test suites NodeId construction switches from the infallible NodeId::new to the validated NodeId::try_new / NodeId::from_validated path. EdgeId parsing switches from the infallible EdgeId::new to EdgeId::from_str with proper error propagation. Add graphStats to the WASM API surface and add graph_stats_parity test to verify Lite output matches expected structure.
SyncFrame::encode_or_empty silently swallowed serialization errors by returning an empty frame. Replace every call site with try_encode, which returns Option<SyncFrame>. In production paths (transport.rs, maintenance.rs) encoding failures are logged and the operation is skipped rather than sending corrupt data. The handshake path treats an encode failure as a hard error and propagates it. Test helpers use .expect() since a failure there indicates a broken test invariant. Replace the hardcoded wire_version literal 1 with the WIRE_FORMAT_VERSION constant from nodedb_types throughout transport, handshake, and examples.
Strict engine: ColumnType::Decimal is now a struct variant
(Decimal { precision, scale }) and requires an exhaustive match arm.
version_column_counts keys are u16; cast u32 schema.version when
inserting. tuple_version lookups cast to u16 consistently.
Columnar store: segment IDs are now u64 in the mutation API;
cast u32 segment_id values at all call sites. on_memtable_flushed now
returns a Result and is propagated. write_segment and the compaction
path accept additional Option parameters.
Spatial index: checkpoint_to_bytes and from_checkpoint now take an
Option<_> hints argument.
Array segments/retention: writer.finish() now takes an Option argument.
FTS manager: add a wildcard arm to the query mode match so new variants
do not cause a compile error.
Catalog: CollectionInfo gains primary and vector_primary fields;
populate them at every construction site. SqlCatalog::collection_info
gains a DatabaseId parameter. Decimal pattern updated to Decimal{..}.
column_type_to_sql adds a catch-all bytes arm.
DDL KV: add a catch-all arm to the KV type match.
Extend the WASM CI workflow to run headless Chrome browser tests via wasm-pack after the existing Node.js test step. Add an npm-publish job that triggers on version tags (v*) and performs a dry-run publish of the @nodedb/lite package to npm with Apache-2.0 license metadata and repository link written into package.json. Ignore the wasm-pack output directory (nodedb-lite-wasm/pkg/) in .gitignore — it is a generated artifact and must not be committed. Update nodedb-lite-wasm/README.md to document the current build instructions, Node.js and browser usage examples, and the public API surface.
Bump the workspace version from 0.0.6 to 0.1.0-beta.1 in Cargo.toml. Dependency version constraints on nodedb-types and other shared crates are relaxed to '*' so the workspace resolves the latest compatible versions from the local patch. Update README and docs/lite.md to reflect beta status: accurate API examples using the real NodeDbLite constructor, platform support matrix with iOS in-progress note, and corrected package installation instructions. Add: - CHANGELOG.md documenting the 0.1.0-beta.1 release - CONTRIBUTING.md with contribution guidelines - CODE_OF_CONDUCT.md (Contributor Covenant) - SECURITY.md with vulnerability disclosure policy - docs/lite-sql-support.md covering SQL plan support surface - docs/lite-support-matrix.md covering platform, engine, and sync capability matrix
…odules Extract test helpers from common/mod.rs into dedicated modules: clock, harness, ops, origin, schema, and sql. Each module owns a single responsibility, making it easier to locate and extend test utilities without navigating a 260-line omnibus file.
…difiers Introduce a dedicated LiteError::Unsupported variant so callers can distinguish "feature not available in this beta" from internal query errors. Apply the guard in the Scan plan handler: ORDER BY, LIMIT, window functions, and WHERE predicates all return Unsupported rather than silently producing wrong results. The catch-all arm also switches from LiteError::Query to LiteError::Unsupported for unimplemented plan variants.
Add real-transport interop tests for the full sync protocol: handshake, live delta delivery, delta acknowledgement, reconnect, resync, shape negotiation, CRDT semantics, and compensation. Corresponding array sync and definition sync interop files provide placeholder tests (all #[ignore]) for the transport paths not yet gate-tested against a live Origin node. Update array_sync_* module docs to clearly label them as edge-side simulations that bypass WebSocket transport, and cross-reference the interop placeholders for the real-transport coverage gap. Add crdt_semantics suite (policy resolution, rejection paths) and semantics suite (clock, compat, fork).
sql_matrix.rs gates all 44 SqlPlan variants — 8 supported paths and 36 that must return LiteError::Unsupported — ensuring no regressions as the engine evolves. sql_parity/ tests SQL behavior across schemaless, strict, columnar, timeseries, and negative cases, verifying engine-level consistency. sync_load.rs exercises the sync stack under concurrent write pressure. Add an FFI test asserting that nodedb_execute_sql returns valid JSON for a constant-expression query.
Add docs/lite-sync-protocol.md documenting the full sync wire protocol, message types, session lifecycle, and promotion criteria for EXPERIMENTAL features (definition sync, array sync). Add nodedb-lite/docs/lite-sql-support.md with the per-variant SQL matrix, file:line anchors for each supported plan, and known gaps for the 36 unsupported variants. Update docs/lite-support-matrix.md to reflect the correct experimental status of array sync and definition sync, with precise citations of the missing Lite receive path and the gate-test requirement. Update README to accurately describe the 8 supported SqlPlan variants and reference the new per-variant matrix.
…d changelog entries Add tokio-postgres to workspace and nodedb-lite dependencies to support Origin harness connectivity in interop tests. Extend nextest.toml to assign sync_interop, sync_load, and sql_parity test binaries to the heavy group, ensuring they run with full thread allocation and do not interfere with the lighter unit test pass. Record array sync and definition sync experimental status in CHANGELOG.
Add `scan_range_bounded_sync` to the `StorageEngineSync` trait, with a redb implementation that computes namespace-prefixed start/end keys and performs a half-open byte-range scan. Results are ordered lexicographically and capped by an optional limit. Required by the KV engine's TTL range eviction and future range-query paths.
Prefix every stored KV value with an 8-byte little-endian u64 deadline (milliseconds since Unix epoch, 0 = no expiry). Encode/decode is transparent to callers — all public methods apply the prefix automatically. Keys are treated as raw bytes internally so binary keys are supported without UTF-8 restrictions. Expired entries are filtered on read and cleaned up lazily via range scans during flush.
Replace the single shared `CsrIndex` with a `HashMap<String, CsrIndex>`
so graph edges are scoped to the collection they were inserted into.
Traversals, neighbor lookups, health stats, and batch edge inserts all
operate on the per-collection index.
Edges are stored in CRDT under a `__edges__{collection}` namespace to
match the index isolation. The legacy single-CSR checkpoint key is
detected on open and silently removed; per-collection checkpoints are
restored separately.
Add checkpoint modules for the FTS collection manager and spatial index manager. On flush the full in-memory state — posting memtable, surrogate maps, R-trees, and doc-to-entry mappings — is serialized to the respective storage namespace. On open the indices are restored from these checkpoints without re-tokenizing source documents or rebuilding R-trees. The spatial checkpoint also persists the doc_id → entry_id mapping, preventing duplicate R-tree entries on upsert/delete after a cold open.
…nd timeseries Introduce the `sync::outbound` module with per-engine outbound queues (`ColumnarOutbound`, `VectorOutbound`, `FtsOutbound`, `SpatialOutbound`, `TimeseriesOutbound`). Each queue buffers pending sync frames until the transport loop drains them. Wire `ColumnarEngine` with optional `outbound` and `timeseries_outbound` fields so inserts are automatically enqueued when sync is enabled. The timeseries outbound uses a distinct `TimeseriesPush` frame type to match Origin's separate timeseries engine path.
Replace the single `transport.rs` file with a `transport/` directory: `connect`, `delegate`, `dispatch`, and `push/` submodules. Each module has a single responsibility — connection setup, inbound frame dispatch, sync delegate trait, and outbound push loop respectively. Add `pending_array_ack` to `SyncClient` state so successfully applied `ArrayDelta` and `ArrayDeltaBatch` frames generate an `ArrayAckMsg` that the push loop transmits to advance Origin's GC frontier. Expose `replica_id()` on `ArrayInbound` to support ack construction without holding the inbound lock.
Enqueue vector insert and delete operations to `VectorOutbound` when sync is enabled, covering both the primary embedding path and named field embeddings. Extend the sync delegate with `handle_array_delta` and `handle_array_delta_batch` handlers. Both decode the incoming op payload to extract the HLC, invoke the inbound apply path, and return an `ArrayAckMsg` on success. Expose `array_schema_hlc` on `NodeDbLite` for tests that need to construct `ArrayOpHeader` values matching the locally registered schema.
Convert `execute_plan` and all sub-executors to async so they can call async engine methods directly without blocking threads. Extract SQL-value coercion into `query::coerce` (single source of truth for `ColumnType` mapping), columnar DML into `query::columnar_dml`, and strict DML into `query::strict_dml`. The query engine now routes INSERT, UPDATE, DELETE, and UPSERT to the correct engine-specific dispatcher based on the `EngineType` in the plan, rather than defaulting to document semantics for all mutations. Add `StrictEngine::list_rows` to expose full-table scans for the strict query path.
…overage Add engine gate suites for KV (including TTL and range scan), graph (per-collection isolation, traversal scoping), and spatial (R-tree CRUD, geospatial queries). Add FTS persistence tests verifying that index state survives a simulated restart without document re-indexing. Add real-transport array sync interop tests covering multi-op sequences against a live Origin instance. Extend SQL parity suites for strict, columnar, and timeseries DML, negative cases, and UPDATE/DELETE for engine-aware dispatch paths. Update definition sync interop, graph stats parity, HTAP, integration, and existing SQL parity tests to reflect per-collection graph isolation and async query engine signatures.
Add sync interop tests for vector insert/delete round-trips and timeseries batch push to Origin. Add a vector engine gate suite covering HNSW insert, nearest-neighbor search, and delete across collections. Expand the sync protocol spec with outbound queue semantics, per-engine frame types, and array ack flow. Update the SQL support matrix to reflect columnar, strict, and timeseries DML coverage in the query engine. Register new nextest test group for real-transport interop tests.
nodedb/core.rs (1 271 lines) is replaced by a core/ directory with separate files for types, open/close, flush, and utility ops. engine/vector/search.rs is likewise replaced by a search/ directory that isolates lazy-load logic from the hot search path. No behaviour is changed; this is a pure file-layout refactor.
Introduces engine/vector/sidecar/ with lazy install and async persist helpers. VectorState gains codec_sidecars and per_index_config maps so each collection can carry an independent quantization sidecar. ensure_hnsw is promoted to a free function and gains a VectorStorageDtype argument; callers in batch insert and vector trait impl now look up the per-collection dtype before creating a new index. On insert the sidecar is lazily installed and the vector is encoded in-place; on delete the sidecar slot is tombstoned and the sidecar is persisted so restarts do not re-surface deleted entries. Encode/persist failures degrade gracefully to FP32 rerank rather than aborting the write.
vector_op.rs handles VectorOp::Search (and future ops) through a single execute_vector_op entry point; vector_write.rs owns the write path. The physical visitor adapter delegates to these modules, removing the inline search implementation that was previously embedded in adapter.rs. Unsupported ArrayOp variants (Project, Aggregate, Elementwise, Compact) are converted from unimplemented! panics to proper LiteError::Unsupported returns, giving callers a recoverable error instead of a process abort. unsupported.rs gains the missing _storage_dtype parameter on the vector primary write stub to match the updated visitor trait signature.
Extract shared cell-value conversion and timestamp helpers into a dedicated ops/util module, then implement the four array operations that previously returned Unsupported errors. Wire each op through the physical visitor adapter using the new ops sub-crate structure.
Introduce four focused op-family modules under src/query/ — document_ops, kv_ops, crdt_ops, and meta_ops — each broken into sub-files by concern (reads, writes, indexes, etc.). Add value_utils for shared JSON/Loro conversion helpers used across the handlers. Wires the new modules into query/mod.rs so the rest of the crate can resolve them.
…dules Replace the monolithic adapter.rs with an adapter/ directory containing dedicated files for each op family (array, crdt, document, kv, meta). The unsupported-stub macro is updated to reflect the narrower set of still-unimplemented variants now that document, kv, crdt, and meta are handled. Each dispatch module calls into the corresponding *_ops handler, keeping the visitor impl free of business logic.
…f purge CrdtEngine gains LoroMovableList helpers (list_insert, list_move, list_delete, list_set_field) via a shared with_delta_capture envelope that snapshots the vv before mutation, exports the resulting delta, and enqueues it as a PendingDelta. Also adds export_delta_from and compact_at_version for version-history management. RetentionState gains purge_before_ms(cutoff_ms) for callers that supply an explicit cutoff rather than deriving it from the configured retention period.
Wires the graph_pagerank trait method through to the CSR engine. Standard PageRank uses uniform initial ranks; Personalized PageRank biases toward caller-supplied seed nodes (values are normalized before the power iteration). Empty collections return an empty Vec rather than an error — a graph with no edges has no ranks to report, which is not a failure condition. Five integration tests cover: standard convergence, personalized bias, damping override, empty-graph short-circuit, and the iteration cap.
Mechanical lint fixes across nodedb-lite and nodedb-lite-wasm: collapse nested if-let chains, remove redundant guard patterns, and other clippy-clean-pass items. One substantive bug: sync/array/schema_registry.rs::put_schema was holding a RwLock guard across an .await point, which can deadlock when the executor parks the task. Drop the guard explicitly before awaiting.
…arts
Documents in bitemporal collections survived flush but their FTS and
graph indexes were rebuilt only from current-row data, silently dropping
all history. open.rs now runs a second-pass scan over DocumentHistory so
every version's text and edges are re-indexed into the in-memory
structures after reopen.
Graph tombstone sentinel mismatch: record_edge_delete was checking
u64::MAX while record_edge_insert wrote i64::MAX as u64, making delete
tombstones a silent no-op. Introduce SYSTEM_TO_CURRENT constant and use
it consistently.
Vector id_map was never written to the checkpoint, so vector_search
returned raw HNSW integer strings instead of document IDs after restart.
flush.rs now writes the id_map blob alongside the HNSW checkpoint;
open.rs restores it.
Rebuild logic extracted from ops.rs into core/rebuild/{text,graph}.rs
for separation of concerns.
Add `execute_sql_with_params` to `LiteQueryEngine` and wire it through `execute_sql_impl` in the trait implementation. The public `execute_sql` now delegates to the new method with an empty params slice so all call sites remain unchanged. $N placeholders are resolved at the AST level via `nodedb_sql::plan_sql_with_params` before the plan is executed. Supported Value variants: Null, Bool, Integer, Float, String, Uuid. Tests in `sql_matrix.rs` cover string equality, integer equality, and multi-placeholder substitution.
CREATE COLLECTION now calls `CrdtEngine::register_collection` immediately so the SQL catalog can resolve the collection name for SELECT even before any document has been inserted. On open, the registration set is rebuilt by scanning the `document_bitemporal:` prefix in the Meta namespace so the fix survives process restart. `execute_scan` for schemaless-document collections checks the bitemporal flag and, when set, reads live state directly from DocumentHistory rather than the Loro in-memory snapshot (which is only updated on explicit flush). `scan_live_documents` in `history::ops` provides the authoritative live-ID set by walking the history prefix and filtering to the latest non-tombstoned version per document. Test `select_after_create` covers the cold-start path: CREATE COLLECTION, INSERT, SELECT — all without an intervening flush.
…illis_i64 Remove four duplicate `now_ms()` / `now_ms_u64()` local functions scattered across the array engine, HTAP bridge, event stream, KV collection, and value_utils modules. All call sites now use the platform-independent `crate::runtime::now_millis()` (u64) and the new `now_millis_i64()` (i64) helper, which works on native and WASM without `SystemTime` / `UNIX_EPOCH` import repetition. Also fixes the `#[allow(clippy::await_holding_lock)]` suppression in `kv_put_with_deadline` by scoping the mutex guard so it is dropped before the await point.
Replace the monolithic `history/ops.rs` (flags, reads, writes all in one file) with a proper module directory: ops/flags.rs — bitemporal flag read/write in Namespace::Meta ops/read.rs — versioned_get_current, versioned_get_as_of, scan_live_documents ops/write.rs — versioned_put, versioned_tombstone ops/backfill.rs — LatestVersion index rebuild for legacy databases Add `latest_version_key` to `history/key.rs` for the O(1) current-version pointer layout. On open, `NodeDbLite` now runs `backfill_latest_version` for every bitemporal collection found in Meta storage. The backfill is idempotent (safe on fresh databases) and logs a warning rather than failing open if it errors, so databases written before the index was introduced continue to work via the prefix-scan fallback.
Introduce a `SyncGate` trait that hosts can install at startup to decide,
per document write, whether that document should be pushed to Origin.
Documents the gate rejects are still written to local CRDT state and indexed
locally (so local reads, FTS, and vector search work), but are excluded from:
- the outbound CRDT delta stream (CrdtEngine::drop_pending)
- the FTS sync outbound queue
- vector insert sync (wired in the follow-up document_put_with_vector path)
API:
pub trait SyncGate: Send + Sync {
fn should_sync(&self, collection: &str, fields: &HashMap<String, Value>) -> bool;
}
NodeDbLite::set_sync_gate(gate: Arc<dyn SyncGate>)
The gate is held in an RwLock (one writer at startup, many readers on every
write) and re-exported from the crate root alongside NodeDbLite.
Also adds FtsCollectionManager::search_with_allowed (over-fetch 8× then filter
by ID set) and threads an `allowed_ids` parameter through run_text_search so
the gate can scope FTS results to a local haystack.
Includes an integration test (tests/sync_gate.rs) that verifies a
`PublicOnlyGate` withholds non-public documents from pending CRDT deltas while
keeping them readable locally.
…arch Both NodeDb trait methods now accept an optional &HashSet<String> of document IDs that gates results: vector_search(..., allowed_ids: Option<&HashSet<String>>) text_search(..., allowed_ids: Option<&HashSet<String>>) For vector search, the set is translated to a RoaringBitmap of HNSW internal surrogates and passed as a prefilter, so the ANN graph only visits allowed candidates. For text search the existing search_with_allowed over-fetch path (8× top_k) is used. All call sites updated: FFI (C API, JNI), WASM bindings, and all integration tests. Passing None preserves the prior behavior exactly. New test: vector_search_allowed_ids_filters_to_set verifies that a document outside the allowed set is excluded even when its embedding is identical to the query vector.
…h_vector_batch_impl
Add BatchItem and document_put_with_vector_batch_impl to NodeDbLite.
The batch path acquires the CRDT lock exactly once regardless of batch size,
runs all per-item Loro mutations under that single lock, then calls
export(updates_since(version_before)) once to produce one pending delta for
the entire batch. FTS indexing, bitemporal history writes, and HNSW inserts
run after the CRDT lock is released in the same relative order as the
single-item path.
BatchItem and the impl are exported from the crate root so callers can write:
use nodedb_lite::{BatchItem, NodeDbLite};
Includes an integration test (tests/document_batch_ingest.rs) covering:
- 100-document batch fully queryable via document_get
- vector search returns correct nearest neighbors for batched embeddings
- the entire batch produces exactly one pending CRDT delta
…dules Remove the last per-module now_ms() / now_ms_u64() local functions in: - engine/array/ops (aggregate, compact, elementwise, project) - engine/columnar/store - engine/strict/crud - nodedb/collection/ddl - query/ddl/kv - query/graph_ops/edges - query/kv_ops (reads, sorted/query, writes) - query/meta_ops (continuous_agg, temporal) - query/physical_visitor (adapter, text_op) - query/timeseries_ops/writes - query/visitor/array/dml - storage/pagedb_storage All call sites now use crate::runtime::now_millis() (u64) or now_millis_i64() (i64). Also threads allowed_ids: None through the graph_rag, fusion, and text_op callers updated by the allowed_ids feature. No behavior changes.
Introduce `Encryption` enum with three variants: - `Plaintext`: explicit opt-out (all-zero key) - `Passphrase`: Argon2id key derivation with a persisted 16-byte salt sidecar at `<path>.salt` (mode 0o600 on Unix) - `RawKey`: caller-supplied 32-byte key, no sidecar written `PagedbStorage::open` now takes an `Encryption` argument and uses `resolve_kek_native` to derive the 32-byte pagedb page-encryption key before opening the database. Corruption recovery reuses the same `Encryption` variant when recreating the database. Remove the hardcoded all-zero KEK placeholder and the related documentation references to internal gap tracking documents.
…cap config Add `LiteConfig::auto_flush_ms` (default 1000 ms) that drives a new `NodeDbLite::start_auto_flush` background task. The task calls `flush()` on each tick, bounding the data-loss window uniformly across all engines (KV buffer, CRDT deltas, HNSW id-map, CSR graph, FTS, spatial). Add `LiteConfig::outbound_queue_cap` (default 100 000) that caps the number of pending entries in each durable outbound sync queue. Writes return `LiteError::Backpressure` when the cap is reached. Both settings are also configurable via environment variables: `NODEDB_LITE_AUTO_FLUSH_MS` and `NODEDB_LITE_OUTBOUND_QUEUE_CAP`.
…cked queues Rewrite all five outbound queues (columnar, vector, FTS, spatial, timeseries) to persist every entry in their respective storage namespace before returning from `enqueue`. Entries survive process restarts and disconnects; the sync transport re-drains them on reconnect while Origin deduplicates via its idempotent-producer gate. Key changes: - Add `DurableOutboundQueue<S>`: generic storage-backed bounded queue with configurable cap. Returns `LiteError::Backpressure` at capacity. - Add `StreamSeqTracker<S>`: per-stream monotonic sequence frontier persisted in `Namespace::Meta`. Tracks `last_assigned` and `last_acked` per stream so frame numbering resumes after restart. - Add `constants::PUSH_DRAIN_LIMIT` (256) to bound each push cycle. - Add `reconcile_outbound_enqueue` helper for uniform backpressure error logging at enqueue sites. - FTS and spatial queues gain a staging buffer that is flushed to durable storage by the periodic flush path, keeping synchronous write paths free of async storage I/O. - All outbound queue types are now generic over `S: StorageEngine`.
Move the monolithic `nodedb/sync_delegate.rs` into a `nodedb/sync_delegate/` directory with three focused modules: - `mod.rs`: core SyncDelegate trait impl (producer state persistence, stream seq coordination, engine outbound drain/ack methods) - `array_handlers.rs`: inbound array delta application - `definition_apply.rs`: inbound collection definition application The trait surface itself gains async drain/ack methods with explicit durable-key tracking to support the delete-on-ack model introduced by the durable outbound queues. Idempotent-producer state (producer_id, accepted_epoch) is persisted to storage and reloaded on reconnect.
…meout metrics SyncClient / FlowController enhancements: - Add `fenced` flag to SyncClient; set on `AckStatus::Fenced`. The push loop halts all outbound frames when fenced and waits for reconnect (which clears the flag). - Add `producer_id` and `accepted_epoch` fields populated from the handshake ack; stamped on every outbound frame for idempotent-producer gating by Origin. - Add per-array-ack coalescing: `pending_array_ack` is now keyed by array name so the highest-HLC ack for each array is retained and multiple arrays can be acked in one push tick. - Add token refresh exponential backoff: `token_last_attempt_ms` and `token_refresh_backoff_ms` (initial 5 s, max 5 min) prevent hammering a failing token provider; `is_refresh_backoff_elapsed` gates each attempt. - Add 30-second timeout on the token provider call so a hung provider cannot block the sync task. - Add `stale_timeouts` counter to `SyncMetrics` and `FlowController::cleanup_stale_and_record` to evict timed-out in-flight entries and increment the metric in one step.
…tamping Transport layer changes for the durable outbound queue model: - Add `dispatch_acks` module: each ack type is handled in its own function that applies `AckStatus` (Applied/Duplicate → delete durable entry; Fenced → halt push; Gap → warn) and advances the stream seq frontier via `record_ack`. - `dispatch_frame`: route all engine ack types to `dispatch_acks`; handle `AckStatus` on `DeltaAck` (fenced halt, gap warning). - `connect_and_run`: reload persisted producer state before reconnect, clear engine in-flight maps so durable entries are eligible for re-drain, persist server-assigned producer state after handshake. - Push loop: stamp every outbound frame with `producer_id`, `epoch`, and a stable per-collection `seq` (assigned once, reused on re-sends). Halt all push when fenced. Run stale in-flight cleanup with AIMD multiplicative decrease every tick. - `SyncDelegate` trait: all pending/ack methods are now async; add `next_stream_seq`, `set_pending_delta_seq`, `persist_*_seq` for durable seq stamping; add `clear_engine_in_flight`, `load/persist _producer_state`.
…e outbound queues Write-path changes across document, vector, batch, and query layers: - Reject writes with `LiteError::Backpressure` when the memory governor reports `PressureLevel::Critical` in `document_put_impl`, `vector_insert_impl`, `document_put_with_vector_batch_impl`, and the batch vector ingest path. - Wire all outbound enqueue calls through `reconcile_outbound_enqueue` so backpressure errors are uniformly logged and propagated. - FTS and spatial enqueue calls switched from direct durable enqueue to the new staging path (`stage_index`, `stage_delete`, `stage_insert`); the flush task spills staged entries to durable storage periodically. - `NodeDbLite::open_inner` wires `StreamSeqTracker`, loads the Lite identity (`lite_id` + `epoch`) for the handshake, opens all outbound queues with the configured cap, and wires FTS/spatial outbound queues into the query engine for SQL-path writes. - `flush()` now drains the KV write buffer first and spills FTS/spatial staging buffers to durable queues on each flush tick. - Add `LiteError::Backpressure` variant; fix typo in SPDX identifiers.
…ocumentation - Wire `wasm_bindgen_futures::spawn_local` into `runtime::spawn` (was a compile-gate placeholder). - Wire `gloo_timers::future::sleep` into `runtime::sleep` (was a placeholder). - Add `gloo-timers`, `wasm-bindgen-futures`, and `js-sys` as unconditional dependencies in `nodedb-lite/Cargo.toml` so the WASM target compiles without feature flags. - Add crate-level documentation to `lib.rs` with a quick-start snippet, durability contract, and `Encryption` re-export. - Minor cleanups in columnar store, CRDT engine, FTS checkpoint/manager, timeseries sync, and vector lazy-load (comment accuracy, import hygiene).
… script for sync tests New test files: - `encryption_roundtrip.rs`: verifies passphrase and raw-key encryption round-trip across reopen. - `auto_flush.rs`: verifies that the background flush task persists writes within the configured interval. - `semantics/dedup.rs`: verifies that reconnect re-sends are deduplicated by Origin (idempotent-producer gate). Nextest configuration: - Add `scripts/ensure-origin.sh` setup script that builds the Origin binary before sync-interop and SQL parity test runs. - Wire the setup script into both the `default` and `ci` profiles for all `sync_interop_*`, `sync_load`, `definition_sync_interop`, and `sql_parity` test binaries. Existing integration and sync-interop tests updated throughout to match the new async delegate API, durable outbound queue signatures, and `Encryption` parameter on `PagedbStorage::open`.
…versions FFI (nodedb-lite-ffi): - Add `handle_registry.rs`: opaque-handle registry used by C and JNI APIs to avoid raw pointer exposure. - Add `jni_bridge/document.rs`: JNI document engine bridge. - Update C header (`nodedb_lite.h`) and all FFI source files to match renamed storage types and new `Encryption` parameter on `open`. - Update JNI bridge (`array`, `core`, `mod`) to use handle registry. WASM (nodedb-lite-wasm): - Update `lib.rs` to use `PagedbStorageMem` and the new `vector_search` signature with `allowed_ids` parameter. - Update WASM README to reflect pagedb backend and new API surface. Docs: - README: update storage backend name from redb to pagedb throughout; update `vector_search` call sites to include the `allowed_ids` argument; add `nodedb-array` and `nodedb-physical` to dev-patch example. - `docs/lite-support-matrix.md`: remove reference to internal protocol notes directory. - `SECURITY.md`: minor copy cleanup. Workspace: - `Cargo.toml`: pin all nodedb-* dependency versions to `0.3`. Examples: - `live_sync.rs`, `load_test.rs`: update to new API signatures.
… API - Add `producer_id`, `epoch`, and `seq` fields (all zero) to `ArrayDeltaMsg` constructions in tests for inbound delta handling. - Add `applied_seq` and `AckStatus::Applied` to `ArrayAckMsg` construction in `ack_sender`. - Update internal test calls to `PagedbStorageDefault::open` to pass `Encryption::Plaintext` to match the new storage open signature.
Update Lite to compile against the latest nodedb-crdt, nodedb-fts, and query-planner APIs: carry structured `Violation` lists on `PolicyResolution::Escalate`/`Deferred` via a wire-to-validator CompensationHint mapping, adjust FTS checkpoint/manager call sites, wire the new continuous-aggregate and array-sync fields, and rework the physical-plan dispatcher for the renamed/consolidated QueryOp and MetaOp variants (dropping the now-impossible `RawResponse` handler and returning `LiteError::Unsupported` for distributed-only ops that can never reach the single-node planner).
…atalog Persist a full CollectionDescriptor plus a bitemporal flag on collection metadata, add an import_collection_schema sync delegate handler and wire transport for inbound schema announcements, and have the SQL catalog prefer persisted metadata over engine-based detection. Synced and DDL-created collections now report their real engine, bitemporal flag, and column schema instead of hardcoded defaults, with engine-based detection kept as a fallback for implicit CRDT-only collections.
Sends a CollectionSchema (opcode 0x13) frame ahead of a collection's first pending CRDT delta in each sync session so Origin materializes the collection before its data arrives, mirroring Origin's own announce-before-shape-snapshot ordering. Announces are deduped per session via a new announced_collections set on SyncClient, cleared on handshake so a fresh session re-announces everything.
Announced collection descriptors were tagged with database_id=1, but inbound deltas apply under DatabaseId::DEFAULT (0), leaving synced collections registered but unqueryable on Origin. Add an interop test covering lite-to-Origin collection registration end to end, plus a non-panicking query helper for bounded-retry polling while the collection becomes visible.
Adds a proof step verifying that a text-match-free DocumentScan observes the 3 synced documents, confirming CRDT deltas materialize into the sparse document store (shape-servable), not just the FTS index.
The redundant second assertion block was collapsed since it was identical to the existing check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the accumulated
litedevelopment branch intomain. Highlights of this session's work (on top of priorlitecommits):Sync collection registration (closes the lite side of nodedb#146)
PlanVisitor,QueryOp/GraphOp/CrdtOp/MetaOp,PolicyResolution,CollectionInfo, etc.).CollectionSchema(0x13) materializes a local collection; the SQL catalog now surfaces the real engine, bitemporal flag, and columns (previously hardcoded) — fixing "table not found" for synced/bitemporal document collections.CollectionSchemaframe before the first CRDT delta for a collection (per-session dedup,Hlc::ZERO), so Origin registers the collection create-only before its data arrives.pg_class) and serves its documents viatext_matchand a plainSELECT id.Verification
-ffiand-wasmcompile clean (incl.wasm32-unknown-unknown); array sync tests green.Note: the matching Origin-side materialization fix landed separately in nodedb#146.