Repository navigation
Conversation
The copy now reads through its own read-only connection, so it can run off the writer: in WAL a reader never blocks the writer. beginDailyCopy hands out one job when the library is idle, completeDailyCopy publishes it and runs a passive checkpoint, and failDailyCopy drops it, keeping older copies for a retry. Shutdown abandons a running copy and opening the library sweeps partial copies a killed process left.
The daily copy no longer uses VACUUM INTO, one native call that a thread can't be stopped in. copyLibraryBatched attaches the library read-only to the copy's connection and copies it in one transaction: the schema from sqlite_master verbatim, the rows in batches of at most 256 rows or about 4 MiB (yielding between them), then indexes and triggers, sqlite_sequence, user_version and application_id. Row counts and integrity_check are checked before it commits; only then is the copy renamed into place. runIdleJobs and takeDailyCopyNow become async.
The storage worker starts copy-worker.ts for each daily copy, so appends and the quit flush never wait on it. Shutdown terminates the copy thread (it yields between batches, so this takes milliseconds), waits for it, deletes the partial file and keeps the previous copy.
T4-copy quits the storage worker while a daily copy runs in its thread and checks that every typed edit is stored, the shutdown is clean, no partial copy is left and the previous copy is byte-identical. T8 times the batched copy on node:sqlite and compares its size with VACUUM INTO's on the year-end library.
The daily copy attaches the library as file:…?mode=ro. Bun's bundled SQLite on Linux and Windows treats that as a plain file name unless the connection is opened with SQLITE_OPEN_URI, so every copy failed with io_error there; macOS's system SQLite and node:sqlite accept URIs.
…up clearly The year's daily copies are dated ahead of the runner's clock, so the real-clock copy sorted oldest and was pruned before T8 could measure it. T4-copy now stops with the copy error if its setup copy fails.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
P2.6's decision 5, approved by the coordinator as option A. Daily copies no longer hold up a save or a quit.
Before, a daily
VACUUM INTOran on the storage worker, so every append queued behind it, for up to 9 s onwindows-latest. That included the quit flush, so a quit during a copy could drop the final batch. That breaks "never drop a Yjs update".Moving
VACUUM INTOto a second worker_thread isn't enough on its own:terminate()can't interrupt a native SQLite call, and Node joins worker threads on exit. A hidden Electron 45.3.2 probe exited 9.3 s afterapp.exit(), which was called at 0.3 s.node:sqlitehas nointerrupt().backup()restarts on every write from the writer.So the copy is now made in batches, and the thread can be stopped between them.
Stacked on #125. It targets
main; #125's commits show here until #125 merges, then I'll mergeorigin/mainin.The copy (
desktop/storage/engine/copy-library.ts)file:…?mode=ro, URI-escaped for Windows paths, spaces and#). The whole copy is one transaction on that connection. Batches areINSERT INTO main.t SELECT * FROM lib.tby rowid range.WITHOUT ROWIDtables are paged in primary-key order. Batches never commit separately.lib.sqlite_masterfirst, then the rows, then indexes and triggers.sqlite_sequenceis copied after the rows, soAUTOINCREMENTnever reuses a seq after a restore.user_versionandapplication_idare copied too;format_versionlives inmeta, a table.PRAGMA integrity_checkmust pass, both inside the transaction. Any mismatch rolls back. Only a verified file is renamed over the day's copy, and then a passive WAL checkpoint runs. A failure keeps the previous copies, recordsdaily_copy_failedinstats(), and retries on the next idle pass.LENGTH()from the record headers. One batch is one native statement, so this bounds how long a stop has to wait.The thread and quit
copy-worker.ts(its own bundle,copy-worker.cjs, 11 KB) for each copy, so appends never wait on it.shutdownterminates the copy thread, waits for it (a 500 ms guard; measured 6–9 ms), lets the engine delete the partial file, keeps the previous copy, and then runs the usual checkpoint and close. Opening the library sweeps any.partiala killed process left.utilityProcess+VACUUM INTO(option B) as the upgrade path once P2.5 exists.Tests
sqlite_master, row counts,sqlite_sequence,meta, pragmas,integrity_check);#work;Library.open, with an identical sidebar and identical document states, and the next seq is past the original's;DocStorecontract runs on a library that is a restored daily copy.T4-copy(all three OSes, 10 runs each, on a ~200 MB library): the library idles until the copy thread is running, the harness types through the copy, then quits. Every typed edit must be stored, the shutdown clean, no.partialleft, and the previous copy byte-identical. Locally (M5 Max, hidden Electron): the copy stopped in 6–8 ms, shutdown took 16–19 ms, the process was gone 14 ms later, and the slowest append during a copy took 6 ms.VACUUM INTOon the year-end 1,000-document library. Results per OS are added below once the soak finishes.For P2.7a
Unchanged from #125: the worker sends
{ type: "index_changed", rev, changes: [{ kind: "doc" | "folder" | "tag", id }] }after every request and idle pass that changes what the sidebar shows, appends included.revgrows with every change, as P2.3'sDocumentIndexcontract expects.Test plan
bun run test(1,831 pass),bun run typecheck,bun run check,bun run lintT4-copyand a small T8 locally on Electron 45.3.2 (hidden)desktop-storage-soak.ymlgreen on macOS, Windows and Linux at 200 kills, with the copy size and time onwindows-latest