Repository navigation
Conversation
…r collection Layout F keeps a collection's records out of leaf bodies, in packs written once and found through a sorted index keyed by (record id, record hash); rewritten leaf nodes are stored as deltas against an earlier full node. Each collection's public and private records are kept apart (runs/<collection>/<set>/: packs, index runs, a manifest as objects, its own maintain jobs), so a public-only mirror copies files that hold nothing private. Version hashes are the same as today's layout's. Chosen with the scale bench (record-storage-layout.md, scale-testing.md): at 12M records a daily version commits in 68-93 s against today's 294-371 s and writes 165-414 MB against 16 GB; 20,000 collections load flat. - A layout per collection: collections.record_layout (migration 0025; existing collections are `today`), new collections get PlatformStorage.newCollections (default `today` for now); forks take their parent's layout and reference its files; stores open each collection's repository in its layout. - F's jobs: layout.maintain and layout.merge per set, within a Worker's storage-call budget; reclaim per set (not started yet: needs the write fence). - Commits: units hold their runs, the assembly names each tree's as one pending object; a type changing privacy commits serially and copies its records to the other set. - Mirrors copy the sets the placement holds; sync packs carry bodies built from F's records and full nodes, so receivers see today's format; fsck reads F. - Cleanup drops a deleted F collection's files after its grace, keeping what a live fork still reads. - Compaction writes a joined run index once (the last two pieces can both join: a write R2 refuses). - The local file store lists from the prefix's folder and reads small objects whole in get. The server suite runs once a layout (vitest projects `today` and `f`), and a test runs today's and F side by side: same versions, same records, both pass fsck.
…cords; every record read names its set
… a 20 s default timeout for the doubled server suite)
isTravis
marked this pull request as ready for review
October 20, 2026 16:44
isTravis
added a commit
that referenced
this pull request
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layout F, the record layout the scale bench chose (record-storage-layout.md, scale-testing.md, plan: layout-f-pr.md). Draft: nothing changes on deploy until a collection is created on F.
What F is
A collection's records live in packs written once, found through a sorted index keyed by (record id, record hash); leaves hold entries only, and rewritten leaf nodes are stored as deltas. Everything is per collection and per set:
runs/<collection>/public/andruns/<collection>/private/each hold their own packs, index runs, manifest (objects, no database) and maintain jobs, so a public-only mirror copies files with nothing private in them. Version hashes are identical to today's layout's.At 12M records on the bench, a daily version commits in 68–93 s against today's 294–371 s and writes 165–414 MB against 16 GB; 20,000 collections in one location load flat at 5.1 min per 1,000. The cost: scans of the latest version are slower (a type stream 433 s against 104 s at 12M).
In this PR
collections.record_layout(migration0025_record_layout; existing collections aretoday). New collections getPlatformStorage.newCollections, defaulttoday: set it tofto create F collections (staging first). Forks take their parent's layout; an F fork references its parent's files per set.layout.maintain/layout.mergeper set, each job within Workers' subrequests (a storage-call budget, the next job queued when it stops). Reclaim per set is in, but nothing starts it yet: records only go dead when versions are thinned, and it needs the write fence first.KeyedLayout.dropCollection). Today's sweep never touchesruns/, and out-of-line records stay marked through leaf entries.get; a gunzip stream no longer leaks a listener on backpressure.Not in it
The bench (
testing/), the other layouts (A, C, D, E), the D1 manifest rows, today's read-ahead experiments. The conversion of existing collections to F is the next PR (planned in layout-f-pr.md). A size cap on export chunks and splitting the reference index by collection size were bench findings too, for their own PRs.Tests
todayandf): 556 tests.test/layouts.test.ts: today's and F side by side in one location, the same pushes (serial and parallel): identical versions, identical records, both pass fsck, each reading nodes the other wrote.Deploying
Migration
0025adds three columns. Nothing else changes untilnewCollectionsisf: then new collections are created on F. Known pre-existing typecheck error inpackages/server/src/app.ts(on main too).