Skip to content

feat(desktop): add the library storage worker (P2.6) - #125

Merged
22 commits merged into
mainfrom
claude/feat-p2-6-storage-worker
Nov 17, 2026
Merged

22 commits merged into
mainfrom
claude/feat-p2-6-storage-worker

Conversation

@HMarzban

@HMarzban HMarzban commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Desktop plan P2.6: the SQLite engine behind the desktop library, and the storage worker that runs it inside Electron main (D-A1). Folders, tags and documents for the files sidebar (P2.7a/b) live here. Nothing in the web app or its build imports it yet. P2.5's shell will start the worker.

Based on main (with #119 and #121 merged), so the diff is P2.6 only. Implements DocStore through createRpcDocStore. compact resolves { foreign: null }.

Engine (desktop/storage/engine/, no node:, bun: or Electron imports)

  • Format-1 schema. The Local release needs no later migration:
    • docs: lineage; link_key (NOT NULL, minted at insert); kind; the §6.3 list columns.
    • Also folders, tags, doc_tags, doc_view(doc_id, width, place, remote_images) and import_sources.
    • snapshots, and updates with an AUTOINCREMENT seq.
    • Purging a document cascades its rows, snapshot, tags, view and import source.
  • Open order.
    1. Refuse a cloud-synced folder.
    2. Take library.lock (with a pid liveness check).
    3. Set WAL and synchronous=FULL; on macOS also fullfsync and checkpoint_fullfsync.
    4. Check format_version before any write. A newer library opens read-only, byte-identical (T6).
    5. Run quick_check if unclean_exit was set.
    6. Run migrations: a VACUUM INTO pre-copy first (keeps 2), then one transaction per step. A failed step leaves the library read-only.
  • Content.
    • Append and compaction each run in one transaction.
    • A stale compaction is a no-op. An upToSeq past the document's rows is refused.
  • One window per document.
    • already_open for a second window.
    • forbidden for writes from a window that doesn't hold the document.
    • document_open for main's own writes (purge, for now).
    • claim lets main replay its registry after a worker restart.
  • Index. Documents, folders and tags; Trash (emptied after 30 days); the sidebar snapshot, which reads no BLOBs.
  • Change feed. Every change the sidebar shows moves rev, appends included, since they carry the title. After each request or idle pass, the worker sends index_changed with { rev, changes: [{ kind, id }] }. The IndexChange shape is provisional until P2.3's DocumentIndex feed settles it.
  • Daily VACUUM INTO. Runs after 30 s idle, keeps 7, and can restore one document from a given day.
  • Quit flush. Shutdown checkpoints the WAL, clears unclean_exit and releases the lock.

Around it

  • worker.ts and client.ts:
    • The worker handles one request at a time, so flush and shutdown follow every earlier write.
    • Replies are Result envelopes, and bytes are transferred.
    • worker_restarting is returned for pending calls if the worker dies.
  • lib/doc-migrations/:
    • The open-time migration hook list (empty, DOC_SCHEMA 0).
    • checkImport, the import guard: it refuses imports over 64 MiB, non-Yjs bytes, and unknown nodes, marks or roots. Legacy link and code are allowed.
  • lib/platform/desktop/rpc-doc-store.ts: the DocStore adapter that IpcDocStore will wrap.
  • tsconfig.desktop-node.json (arch §3.2) is now part of bun run typecheck.
  • The import-boundary test covers the storage rows of arch §3.3.

Decisions to check

  1. lib/schema-names.ts is now on the §3.3 allowlist for main and storage. The import guard needs the schema's name lists. The alternative is to copy those lists.

  2. The worker enforces the registry too. It requires the writer's hold, not just "no other holder", and adds claim for restarts. Arch §7 puts this check in main; the worker checks again.

  3. load of a never-created document is not_found. Documents are created through the index, which mints lineage and link_key. I asked P2.1 whether its contract suite expects an empty document instead.

  4. T7 (slow-hydrate overwrite) is the web library's. It tests the session layer, not storage, so this soak doesn't run it.

  5. A daily copy can outlast the quit budget. Needs a follow-up. A quit flush queues behind a running VACUUM INTO. On the hosted runners a copy of the year-end 238 MB library blocked the worker for:

    • macOS: median 0.8 s, max 1.7 s.
    • Linux: median 1.0 s, max 1.9 s.
    • Windows: median 1.8 s, max 9.0 s.

    The Windows maximum is past the quit budget (3 s flush, then 2 s shutdown). A quit that lands inside such a copy would drop the final batch, which breaks D-A2's "a clean quit loses nothing". The copy runs only after 30 s without a write, so this needs a quit in that window.

    The proposed fix: run the copy on its own read-only connection in a second worker_thread. In WAL, a reader never blocks the single writer, so appends keep committing during the copy. The other option is for shutdown to wait for a running job. Not changed in this PR.

  6. import_sources keeps both shapes, pending the owner's answer from arch §5.3: relative_path and source_app, plus source_hash and imported_at.

Soak (T1–T8) and the 1,000-document timing

desktop-storage-soak.yml runs on macos-26, windows-latest and ubuntu-24.04. The worker runs under Electron 45.3.2 on node:sqlite, and the verifier reopens every library on bun:sqlite. Pull requests that touch storage run 20 kills per case. Nightly runs, and PRs labelled storage-soak-full, run 200 per case. This PR carries the label.

Run 44055069967 (commit 1390edb) is tabled below: all 12 jobs green, 200 kills per kill case. A second run on the merged head, 44060367140 (5c55dbf), was green on all 12 jobs too. Its sidebar times from process start were 374 ms (macOS), 226 ms (Windows) and 199 ms (Linux), and its daily-copy maximum on Windows was 8.4 s.

Case macos-26 windows-latest ubuntu-24.04
Contract (P2.1's suite on node:sqlite in the worker) 16/16 16/16 16/16
T1 SIGKILL during append: kills / inside a txn / failures 200 / 3 / 0 200 / 9 / 0 200 / 1 / 0
T1 dense (5 ms / 20 ms batching) 100 / 4 / 0 100 / 12 / 0 100 / 2 / 0
T2 SIGKILL during compaction (5 MB doc) 200 / 200 / 0 200 / 200 / 0 200 / 178 / 0
T3 pragmas as set; 4 KB commit p50 fullfsync on vs off; synchronous=OFF control ok; 2.66 vs 1.03 ms; 50 kills, 0 failures ok; 3.60 vs 3.78 ms (no fullfsync on Windows); 0 failures ok; 0.48 vs 0.45 ms; 0 failures
T4 quit with edits in flight 20/20 clean, every edit stored 20/20 20/20
T5 SIGKILL during a 3-step migration: stopped at v1/v2/v3/v4 7/7/4/22, all resumed to v4 5/11/4/20 9/9/4/18
T6 newer library read-only, writes refused, file byte-identical same same
T7 N/A here: P2.1's local-session tests own it
T8 year: 1,000 docs, 365 days, 178,481 appends, 7,300 compactions 197 → 238 MB, 7 copies, 0 rows left same same
Worker ready, from process start (§10: 150 ms Mac / 300 ms Win) 415 ms 175 ms 132 ms
Sidebar rows, 1,000 docs, from process start (§10: 600 ms Mac / 1.0 s Win) 444 ms 188 ms 145 ms
Open after an unclean exit, quick_check included (≤ 2 s) 542 ms 196 ms 54 ms
Daily VACUUM INTO, median / max 845 / 1,714 ms 1,825 / 9,044 ms 1,049 / 1,924 ms

The sidebar time is from Electron process start until the rows reach main, the median of 5 runs. It excludes the renderer's IPC hop and paint, which P2.5 and P2.7 add.

These are hosted-runner numbers, not the §10 verdict machines (an M1, and a Windows laptop with Defender on). The macOS VM's "worker ready" time is over its 150 ms row, mostly Electron's own boot before whenReady.

T3 only shows that the pragmas read back as set and how long commits take. Power-loss durability needs owner hardware.

Test plan

  • bun test tests/unit/desktop tests/unit/lib/doc-migrations tests/unit/ci: engine, import guard, migrations, boundaries
  • bun run typecheck (now four projects), bun run check, bun run lint
  • T8 year simulation locally on bun:sqlite (1,000 docs × 365 days): 197 → 237 MB, 7 daily copies, 0 rows left uncompacted
  • desktop-storage-soak.yml green on all three OSes, 200 kills per case (runs 44055069967 and 44060367140)
  • CI green on 5c55dbf
  • P2.1's shared contract suite on both drivers: bun test on bun:sqlite, and on node:sqlite in the Electron worker

tsconfig.desktop-node.json covers desktop/{storage,main,build} with Node
types and no Bun types, so a Bun-only API fails the typecheck in code that
runs inside Electron (desktop architecture §3.2). The root project excludes
those folders.
The SQLite engine behind the desktop library (plan P2.6), behind a
SqliteDriver seam so the same code runs on node:sqlite in the Electron
storage worker and on bun:sqlite in tests:

- format-1 schema: docs (lineage, link_key minted at insert, kind and the
  list columns), folders, tags, doc_tags, doc_view, import_sources,
  snapshots and updates with an AUTOINCREMENT seq
- open order: cloud-folder refusal, library.lock, WAL and
  synchronous=FULL (fullfsync on macOS), then the format_version guard
  before any write; quick_check after an unclean exit
- format migrations with a pre-migration VACUUM INTO (keeps 2), one
  transaction per step, read-only on failure
- append and renderer-driven compaction in one transaction each, with
  stale and future upToSeq guards
- the one-window registry: already_open, forbidden, document_open
- documents, folders, tags, Trash and the sidebar snapshot, with rev
- daily VACUUM INTO copies after 30 s idle, keeping 7
- shutdown: checkpoint, clear unclean_exit, release the lock
lib/doc-migrations/ holds the pure Yjs forward migrations that
local-session runs after hydrate and before the editor binds, and that
import runs on .tinydoc and web downloads (desktop architecture §5.1,
D-A5). The list starts empty, so DOC_SCHEMA is 0.

checkImport refuses a state that is too large, isn't Yjs, or carries a
node, mark or root this build doesn't know, before it reaches the
library. The legacy link and code marks pass: the import migrations
rename them.
The worker accepted appends and compactions from any window when no
window held the document. It now requires the writer's own hold, as
desktop architecture §7 requires of main, and adds claim() so main can
replay its registry after a worker restart. Compaction is bounded by the
document's own last row instead of the library-wide sequence.
worker.ts runs the engine on node:sqlite in a worker_thread (D-A1) and
answers one request at a time, so flush and shutdown run after every
write sent before them. Replies are Result envelopes and bytes are
transferred. Idle jobs run between requests, and index writes emit
index_changed.

StorageClient starts the worker, waits for ready, rejects pending calls
with worker_restarting if the worker dies, and ends with the quit flush's
shutdown (terminated after 2 s).

createRpcDocStore turns those replies into the DocStore port, for
IpcDocStore and the tests. Engine DocStore cases run in-process on
bun:sqlite and, framework-free, in Electron.
The first rows of desktop architecture §3.3's import-boundaries test: the
engine imports no electron, node:*, bun:*, React or Tiptap and stays in
its folder; the worker reaches lib/ only through the allowlist and never
bun:*; lib/platform/desktop never imports desktop/{main,preload,storage}.
Copy diagnostics and soak T3 read back journal_mode, synchronous,
fullfsync, checkpoint_fullfsync and foreign_keys as the library runs.
tests/load/desktop-durability-soak.ts launches the real storage worker
inside a windowless Electron (desktop/storage-harness, its own lockfile
on Electron 45.3.2), kills it during appends, compactions and format
migrations, quits it with edits in flight, opens a newer library with
it, and runs a year of editing over 1,000 documents before timing
startup to sidebar rows. Every library is verified by reopening it with
the engine on bun:sqlite. T7 is the web library's.

desktop-storage-soak.yml runs it on macOS, Windows and Linux: 20 kills
per case on pull requests that touch storage, 200 nightly and on
demand.
A pull request labelled storage-soak-full runs 200 kills per case, as
the nightly does; workflow_dispatch only works once the file is on main.
The architecture spec now lists where the engine, client and soak harness
live, the open order as built, the worker-side registry check and claim,
the compaction guards, the import guard, the lib/schema-names.ts boundary
allowance, and that T7 belongs to the web library.
@HMarzban HMarzban added the storage-soak-full Run the full 200-kill desktop storage soak on this PR label Nov 17, 2026
A person whose user name is box or dropbox had their library refused as
a cloud folder. The check now skips the home folder's own segments.
Appends change a document's title, snippet and modified time, but did not
move rev, and opening a document or the idle Trash purge moved rev
without an event. The library now records each changed row, appends bump
rev in their transaction, and the worker sends index_changed with the
rev and the changed rows after every request and idle pass (architecture
§7, §8.1). A failed daily copy is kept for stats instead of dropped.
T8 now gates sidebar rows against architecture §10 from process start,
reports worker ready against its own row, runs one startup after an
unclean exit to time quick_check, sends realistic snippets, and records
how long each daily VACUUM INTO blocks the worker. Temp-folder cleanup
retries and never fails a case.
Bun's close() leaves a connection whose statements are still live open as
a zombie, holding the library's file handles. On Windows the soak could
not delete a library the verifier had closed (EBUSY). The driver now
finalizes its cached statements first.
…into claude/feat-p2-6-storage-worker

# Conflicts:
#	CHANGELOG.md
createDoc takes the document's schema (DOC_SCHEMA for a new document, the
file's on import) and accepts any id that is safe in SQL, paths and logs.
Main's IPC guard still lets only UUIDs through; tests and imports use
other ids.
The shared contract now runs on the engine over bun:sqlite under bun
test, and over node:sqlite in the real storage worker in the soak's
contract case. It replaces the interim engine cases.
@HMarzban
HMarzban changed the base branch from claude/feat-p2-1-doc-session-1ef6a3 to claude/feat-p2-1-local-session November 17, 2026 09:23
… takes it

DOC_MIGRATIONS now plugs into LocalSessionOptions without an adapter.
runDocMigrations stays for import, where a file migrates all or nothing.
@HMarzban
HMarzban changed the base branch from claude/feat-p2-1-local-session to main November 17, 2026 09:25
@HMarzban HMarzban removed the storage-soak-full Run the full 200-kill desktop storage soak on this PR label Nov 17, 2026
@HMarzban HMarzban added the storage-soak-full Run the full 200-kill desktop storage soak on this PR label Nov 17, 2026
@HMarzban HMarzban closed this pull request by merging all changes into main in 29f6faf Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

storage-soak-full Run the full 200-kill desktop storage soak on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant