Skip to content

feat(desktop): make daily copies in a thread that quit can stop (P2.6 decision 5) - #128

Open
HMarzban wants to merge 8 commits into
mainfrom
claude/feat-p2-6-daily-copy-thread
Open

HMarzban wants to merge 8 commits into
mainfrom
claude/feat-p2-6-daily-copy-thread

Conversation

@HMarzban

Copy link
Copy Markdown
Contributor

Summary

P2.6's decision 5, approved by the coordinator as option A. Daily copies no longer hold up a save or a quit.

Before, a daily VACUUM INTO ran on the storage worker, so every append queued behind it, for up to 9 s on windows-latest. That included the quit flush, so a quit during a copy could drop the final batch. That breaks "never drop a Yjs update".

Moving VACUUM INTO to a second worker_thread isn't enough on its own:

  • terminate() can't interrupt a native SQLite call, and Node joins worker threads on exit. A hidden Electron 45.3.2 probe exited 9.3 s after app.exit(), which was called at 0.3 s.
  • node:sqlite has no interrupt().
  • backup() restarts on every write from the writer.

So the copy is now made in batches, and the thread can be stopped between them.

Stacked on #125. It targets main; #125's commits show here until #125 merges, then I'll merge origin/main in.

The copy (desktop/storage/engine/copy-library.ts)

  1. One consistent snapshot. The copy's own connection attaches the library read-only (file:…?mode=ro, URI-escaped for Windows paths, spaces and #). The whole copy is one transaction on that connection. Batches are INSERT INTO main.t SELECT * FROM lib.t by rowid range. WITHOUT ROWID tables are paged in primary-key order. Batches never commit separately.
  2. The schema, verbatim. Tables come from lib.sqlite_master first, then the rows, then indexes and triggers. sqlite_sequence is copied after the rows, so AUTOINCREMENT never reuses a seq after a restore. user_version and application_id are copied too; format_version lives in meta, a table.
  3. Verified before it's kept. Row counts per table are checked against the snapshot, and PRAGMA integrity_check must pass, both inside the transaction. Any mismatch rolls back. Only a verified file is renamed over the day's copy, and then a passive WAL checkpoint runs. A failure keeps the previous copies, records daily_copy_failed in stats(), and retries on the next idle pass.
  4. Batch size is at most 256 rows or about 4 MiB, measured with LENGTH() from the record headers. One batch is one native statement, so this bounds how long a stop has to wait.

The thread and quit

  • The storage worker starts copy-worker.ts (its own bundle, copy-worker.cjs, 11 KB) for each copy, so appends never wait on it.
  • shutdown terminates the copy thread, waits for it (a 500 ms guard; measured 6–9 ms), lets the engine delete the partial file, keeps the previous copy, and then runs the usual checkpoint and close. Opening the library sweeps any .partial a killed process left.
  • Architecture §6.3 now records the quit budget (3 s flush, then 2 s worker shutdown) and that a daily copy never counts against it.
  • §5.4 records utilityProcess + VACUUM INTO (option B) as the upgrade path once P2.5 exists.

Tests

  • Unit, on bun:sqlite:
    • the copy is identical to the library (sqlite_master, row counts, sqlite_sequence, meta, pragmas, integrity_check);
    • rows the writer commits between batches aren't in the copy;
    • a short copy is refused with nothing committed;
    • a stop between batches releases every file handle;
    • batches are bounded by bytes;
    • paths with spaces and # work;
    • a restore round trip: the copy opens through Library.open, with an identical sidebar and identical document states, and the next seq is past the original's;
    • P2.1's DocStore contract runs on a library that is a restored daily copy.
  • Soak T4-copy (all three OSes, 10 runs each, on a ~200 MB library): the library idles until the copy thread is running, the harness types through the copy, then quits. Every typed edit must be stored, the shutdown clean, no .partial left, and the previous copy byte-identical. Locally (M5 Max, hidden Electron): the copy stopped in 6–8 ms, shutdown took 16–19 ms, the process was gone 14 ms later, and the slowest append during a copy took 6 ms.
  • T8 now times the batched copy on node:sqlite in its thread and compares its size with VACUUM INTO on the year-end 1,000-document library. Results per OS are added below once the soak finishes.

For P2.7a

Unchanged from #125: the worker sends { type: "index_changed", rev, changes: [{ kind: "doc" | "folder" | "tag", id }] } after every request and idle pass that changes what the sidebar shows, appends included. rev grows with every change, as P2.3's DocumentIndex contract expects.

Test plan

  • bun run test (1,831 pass), bun run typecheck, bun run check, bun run lint
  • T4-copy and a small T8 locally on Electron 45.3.2 (hidden)
  • desktop-storage-soak.yml green on macOS, Windows and Linux at 200 kills, with the copy size and time on windows-latest

The copy now reads through its own read-only connection, so it can run
off the writer: in WAL a reader never blocks the writer. beginDailyCopy
hands out one job when the library is idle, completeDailyCopy publishes
it and runs a passive checkpoint, and failDailyCopy drops it, keeping
older copies for a retry. Shutdown abandons a running copy and opening
the library sweeps partial copies a killed process left.
The daily copy no longer uses VACUUM INTO, one native call that a thread
can't be stopped in. copyLibraryBatched attaches the library read-only to
the copy's connection and copies it in one transaction: the schema from
sqlite_master verbatim, the rows in batches of at most 256 rows or about
4 MiB (yielding between them), then indexes and triggers, sqlite_sequence,
user_version and application_id. Row counts and integrity_check are
checked before it commits; only then is the copy renamed into place.
runIdleJobs and takeDailyCopyNow become async.
The storage worker starts copy-worker.ts for each daily copy, so appends
and the quit flush never wait on it. Shutdown terminates the copy thread
(it yields between batches, so this takes milliseconds), waits for it,
deletes the partial file and keeps the previous copy.
T4-copy quits the storage worker while a daily copy runs in its thread
and checks that every typed edit is stored, the shutdown is clean, no
partial copy is left and the previous copy is byte-identical. T8 times
the batched copy on node:sqlite and compares its size with VACUUM INTO's
on the year-end library.
@HMarzban HMarzban added the storage-soak-full Run the full 200-kill desktop storage soak on this PR label Nov 17, 2026
The daily copy attaches the library as file:…?mode=ro. Bun's bundled
SQLite on Linux and Windows treats that as a plain file name unless the
connection is opened with SQLITE_OPEN_URI, so every copy failed with
io_error there; macOS's system SQLite and node:sqlite accept URIs.
…up clearly

The year's daily copies are dated ahead of the runner's clock, so the
real-clock copy sorted oldest and was pruned before T8 could measure it.
T4-copy now stops with the copy error if its setup copy fails.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

storage-soak-full Run the full 200-kill desktop storage soak on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant