Repository navigation
feat(desktop): add the library storage worker (P2.6) - #125
Merged
22 commits merged intoNov 17, 2026
Merged
22 commits merged into
22 commits merged into
Conversation
tsconfig.desktop-node.json covers desktop/{storage,main,build} with Node
types and no Bun types, so a Bun-only API fails the typecheck in code that
runs inside Electron (desktop architecture §3.2). The root project excludes
those folders.
The SQLite engine behind the desktop library (plan P2.6), behind a SqliteDriver seam so the same code runs on node:sqlite in the Electron storage worker and on bun:sqlite in tests: - format-1 schema: docs (lineage, link_key minted at insert, kind and the list columns), folders, tags, doc_tags, doc_view, import_sources, snapshots and updates with an AUTOINCREMENT seq - open order: cloud-folder refusal, library.lock, WAL and synchronous=FULL (fullfsync on macOS), then the format_version guard before any write; quick_check after an unclean exit - format migrations with a pre-migration VACUUM INTO (keeps 2), one transaction per step, read-only on failure - append and renderer-driven compaction in one transaction each, with stale and future upToSeq guards - the one-window registry: already_open, forbidden, document_open - documents, folders, tags, Trash and the sidebar snapshot, with rev - daily VACUUM INTO copies after 30 s idle, keeping 7 - shutdown: checkpoint, clear unclean_exit, release the lock
lib/doc-migrations/ holds the pure Yjs forward migrations that local-session runs after hydrate and before the editor binds, and that import runs on .tinydoc and web downloads (desktop architecture §5.1, D-A5). The list starts empty, so DOC_SCHEMA is 0. checkImport refuses a state that is too large, isn't Yjs, or carries a node, mark or root this build doesn't know, before it reaches the library. The legacy link and code marks pass: the import migrations rename them.
The worker accepted appends and compactions from any window when no window held the document. It now requires the writer's own hold, as desktop architecture §7 requires of main, and adds claim() so main can replay its registry after a worker restart. Compaction is bounded by the document's own last row instead of the library-wide sequence.
worker.ts runs the engine on node:sqlite in a worker_thread (D-A1) and answers one request at a time, so flush and shutdown run after every write sent before them. Replies are Result envelopes and bytes are transferred. Idle jobs run between requests, and index writes emit index_changed. StorageClient starts the worker, waits for ready, rejects pending calls with worker_restarting if the worker dies, and ends with the quit flush's shutdown (terminated after 2 s). createRpcDocStore turns those replies into the DocStore port, for IpcDocStore and the tests. Engine DocStore cases run in-process on bun:sqlite and, framework-free, in Electron.
The first rows of desktop architecture §3.3's import-boundaries test: the
engine imports no electron, node:*, bun:*, React or Tiptap and stays in
its folder; the worker reaches lib/ only through the allowlist and never
bun:*; lib/platform/desktop never imports desktop/{main,preload,storage}.
Copy diagnostics and soak T3 read back journal_mode, synchronous, fullfsync, checkpoint_fullfsync and foreign_keys as the library runs.
tests/load/desktop-durability-soak.ts launches the real storage worker inside a windowless Electron (desktop/storage-harness, its own lockfile on Electron 45.3.2), kills it during appends, compactions and format migrations, quits it with edits in flight, opens a newer library with it, and runs a year of editing over 1,000 documents before timing startup to sidebar rows. Every library is verified by reopening it with the engine on bun:sqlite. T7 is the web library's. desktop-storage-soak.yml runs it on macOS, Windows and Linux: 20 kills per case on pull requests that touch storage, 200 nightly and on demand.
A pull request labelled storage-soak-full runs 200 kills per case, as the nightly does; workflow_dispatch only works once the file is on main.
The architecture spec now lists where the engine, client and soak harness live, the open order as built, the worker-side registry check and claim, the compaction guards, the import guard, the lib/schema-names.ts boundary allowance, and that T7 belongs to the web library.
A person whose user name is box or dropbox had their library refused as a cloud folder. The check now skips the home folder's own segments.
Appends change a document's title, snippet and modified time, but did not move rev, and opening a document or the idle Trash purge moved rev without an event. The library now records each changed row, appends bump rev in their transaction, and the worker sends index_changed with the rev and the changed rows after every request and idle pass (architecture §7, §8.1). A failed daily copy is kept for stats instead of dropped.
T8 now gates sidebar rows against architecture §10 from process start, reports worker ready against its own row, runs one startup after an unclean exit to time quick_check, sends realistic snippets, and records how long each daily VACUUM INTO blocks the worker. Temp-folder cleanup retries and never fails a case.
Bun's close() leaves a connection whose statements are still live open as a zombie, holding the library's file handles. On Windows the soak could not delete a library the verifier had closed (EBUSY). The driver now finalizes its cached statements first.
…into claude/feat-p2-6-storage-worker # Conflicts: # CHANGELOG.md
createDoc takes the document's schema (DOC_SCHEMA for a new document, the file's on import) and accepts any id that is safe in SQL, paths and logs. Main's IPC guard still lets only UUIDs through; tests and imports use other ids.
The shared contract now runs on the engine over bun:sqlite under bun test, and over node:sqlite in the real storage worker in the soak's contract case. It replaces the interim engine cases.
HMarzban
changed the base branch from
claude/feat-p2-1-doc-session-1ef6a3
to
claude/feat-p2-1-local-session
November 17, 2026 09:23
… takes it DOC_MIGRATIONS now plugs into LocalSessionOptions without an adapter. runDocMigrations stays for import, where a file migrates all or nothing.
HMarzban
changed the base branch from
claude/feat-p2-1-local-session
to
main
November 17, 2026 09:25
…into claude/feat-p2-6-storage-worker
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Desktop plan P2.6: the SQLite engine behind the desktop library, and the storage worker that runs it inside Electron main (D-A1). Folders, tags and documents for the files sidebar (P2.7a/b) live here. Nothing in the web app or its build imports it yet. P2.5's shell will start the worker.
Based on
main(with #119 and #121 merged), so the diff is P2.6 only. ImplementsDocStorethroughcreateRpcDocStore.compactresolves{ foreign: null }.Engine (
desktop/storage/engine/, nonode:,bun:or Electron imports)docs:lineage;link_key(NOT NULL, minted at insert);kind; the §6.3 list columns.folders,tags,doc_tags,doc_view(doc_id, width, place, remote_images)andimport_sources.snapshots, andupdateswith anAUTOINCREMENTseq.library.lock(with a pid liveness check).synchronous=FULL; on macOS alsofullfsyncandcheckpoint_fullfsync.format_versionbefore any write. A newer library opens read-only, byte-identical (T6).quick_checkifunclean_exitwas set.VACUUM INTOpre-copy first (keeps 2), then one transaction per step. A failed step leaves the library read-only.upToSeqpast the document's rows is refused.already_openfor a second window.forbiddenfor writes from a window that doesn't hold the document.document_openfor main's own writes (purge, for now).claimlets main replay its registry after a worker restart.rev, appends included, since they carry the title. After each request or idle pass, the worker sendsindex_changedwith{ rev, changes: [{ kind, id }] }. TheIndexChangeshape is provisional until P2.3'sDocumentIndexfeed settles it.VACUUM INTO. Runs after 30 s idle, keeps 7, and can restore one document from a given day.unclean_exitand releases the lock.Around it
worker.tsandclient.ts:flushandshutdownfollow every earlier write.Resultenvelopes, and bytes are transferred.worker_restartingis returned for pending calls if the worker dies.lib/doc-migrations/:DOC_SCHEMA0).checkImport, the import guard: it refuses imports over 64 MiB, non-Yjs bytes, and unknown nodes, marks or roots. Legacylinkandcodeare allowed.lib/platform/desktop/rpc-doc-store.ts: theDocStoreadapter thatIpcDocStorewill wrap.tsconfig.desktop-node.json(arch §3.2) is now part ofbun run typecheck.Decisions to check
lib/schema-names.tsis now on the §3.3 allowlist for main and storage. The import guard needs the schema's name lists. The alternative is to copy those lists.The worker enforces the registry too. It requires the writer's hold, not just "no other holder", and adds
claimfor restarts. Arch §7 puts this check in main; the worker checks again.loadof a never-created document isnot_found. Documents are created through the index, which mints lineage andlink_key. I asked P2.1 whether its contract suite expects an empty document instead.T7 (slow-hydrate overwrite) is the web library's. It tests the session layer, not storage, so this soak doesn't run it.
A daily copy can outlast the quit budget. Needs a follow-up. A quit flush queues behind a running
VACUUM INTO. On the hosted runners a copy of the year-end 238 MB library blocked the worker for:The Windows maximum is past the quit budget (3 s flush, then 2 s shutdown). A quit that lands inside such a copy would drop the final batch, which breaks D-A2's "a clean quit loses nothing". The copy runs only after 30 s without a write, so this needs a quit in that window.
The proposed fix: run the copy on its own read-only connection in a second
worker_thread. In WAL, a reader never blocks the single writer, so appends keep committing during the copy. The other option is forshutdownto wait for a running job. Not changed in this PR.import_sourceskeeps both shapes, pending the owner's answer from arch §5.3:relative_pathandsource_app, plussource_hashandimported_at.Soak (T1–T8) and the 1,000-document timing
desktop-storage-soak.ymlruns onmacos-26,windows-latestandubuntu-24.04. The worker runs under Electron 45.3.2 onnode:sqlite, and the verifier reopens every library onbun:sqlite. Pull requests that touch storage run 20 kills per case. Nightly runs, and PRs labelledstorage-soak-full, run 200 per case. This PR carries the label.Run 44055069967 (commit 1390edb) is tabled below: all 12 jobs green, 200 kills per kill case. A second run on the merged head, 44060367140 (5c55dbf), was green on all 12 jobs too. Its sidebar times from process start were 374 ms (macOS), 226 ms (Windows) and 199 ms (Linux), and its daily-copy maximum on Windows was 8.4 s.
The sidebar time is from Electron process start until the rows reach main, the median of 5 runs. It excludes the renderer's IPC hop and paint, which P2.5 and P2.7 add.
These are hosted-runner numbers, not the §10 verdict machines (an M1, and a Windows laptop with Defender on). The macOS VM's "worker ready" time is over its 150 ms row, mostly Electron's own boot before
whenReady.T3 only shows that the pragmas read back as set and how long commits take. Power-loss durability needs owner hardware.
Test plan
bun test tests/unit/desktop tests/unit/lib/doc-migrations tests/unit/ci: engine, import guard, migrations, boundariesbun run typecheck(now four projects),bun run check,bun run lintbun:sqlite(1,000 docs × 365 days): 197 → 237 MB, 7 daily copies, 0 rows left uncompacteddesktop-storage-soak.ymlgreen on all three OSes, 200 kills per case (runs 44055069967 and 44060367140)bun teston bun:sqlite, and on node:sqlite in the Electron worker