Skip to content

Record layout F: records in packs per collection and set - #61

Merged
isTravis merged 3 commits into
mainfrom
layout-f
Oct 20, 2026
Merged

isTravis merged 3 commits into
mainfrom
layout-f

Conversation

@isTravis

Copy link
Copy Markdown
Member

Layout F, the record layout the scale bench chose (record-storage-layout.md, scale-testing.md, plan: layout-f-pr.md). Draft: nothing changes on deploy until a collection is created on F.

What F is

A collection's records live in packs written once, found through a sorted index keyed by (record id, record hash); leaves hold entries only, and rewritten leaf nodes are stored as deltas. Everything is per collection and per set: runs/<collection>/public/ and runs/<collection>/private/ each hold their own packs, index runs, manifest (objects, no database) and maintain jobs, so a public-only mirror copies files with nothing private in them. Version hashes are identical to today's layout's.

At 12M records on the bench, a daily version commits in 68–93 s against today's 294–371 s and writes 165–414 MB against 16 GB; 20,000 collections in one location load flat at 5.1 min per 1,000. The cost: scans of the latest version are slower (a type stream 433 s against 104 s at 12M).

In this PR

  • A layout per collection. collections.record_layout (migration 0025_record_layout; existing collections are today). New collections get PlatformStorage.newCollections, default today: set it to f to create F collections (staging first). Forks take their parent's layout; an F fork references its parent's files per set.
  • F's jobs. layout.maintain / layout.merge per set, each job within Workers' subrequests (a storage-call budget, the next job queued when it stops). Reclaim per set is in, but nothing starts it yet: records only go dead when versions are thinned, and it needs the write fence first.
  • Commits. Units hold their index runs and packs; the assembly names each tree's as one pending object (a 30M push left ~3,300 before). A type changing privacy commits serially on F and copies its records into the other set's packs.
  • Mirrors copy the sets the placement holds (a public-only mirror gets the public set alone). Sync packs carry leaf bodies built from F's records and full nodes, so receivers see today's format. fsck reopens a repository in its layout and reads by set.
  • Cleanup: a deleted F collection's files go after the tombstone's grace, keeping what a live fork still reads (KeyedLayout.dropCollection). Today's sweep never touches runs/, and out-of-line records stay marked through leaf entries.
  • Fixes found on the way: a compaction's joined run index is written once (the last two pieces can both join, a second write R2 refuses); the local file store lists from the prefix's folder (it walked the whole bucket) and reads small objects whole in get; a gunzip stream no longer leaks a listener on backpressure.

Not in it

The bench (testing/), the other layouts (A, C, D, E), the D1 manifest rows, today's read-ahead experiments. The conversion of existing collections to F is the next PR (planned in layout-f-pr.md). A size cap on export chunks and splitting the reference index by collection size were bench findings too, for their own PRs.

Tests

  • The server suite runs twice, once a layout (vitest projects today and f): 556 tests.
  • test/layouts.test.ts: today's and F side by side in one location, the same pushes (serial and parallel): identical versions, identical records, both pass fsck, each reading nodes the other wrote.
  • Protocol: F's merge, maintain, fork and manifest tests (246), with new ones for the storage-call budget, the manifest head, and pending runs ordered within a millisecond; server tests for split-set mirrors, privacy moves on F, one pending object a tree, and the compaction index.

Deploying

Migration 0025 adds three columns. Nothing else changes until newCollections is f: then new collections are created on F. Known pre-existing typecheck error in packages/server/src/app.ts (on main too).

…r collection

Layout F keeps a collection's records out of leaf bodies, in packs written once
and found through a sorted index keyed by (record id, record hash); rewritten
leaf nodes are stored as deltas against an earlier full node. Each collection's
public and private records are kept apart (runs/<collection>/<set>/: packs,
index runs, a manifest as objects, its own maintain jobs), so a public-only
mirror copies files that hold nothing private. Version hashes are the same as
today's layout's.

Chosen with the scale bench (record-storage-layout.md, scale-testing.md): at
12M records a daily version commits in 68-93 s against today's 294-371 s and
writes 165-414 MB against 16 GB; 20,000 collections load flat.

- A layout per collection: collections.record_layout (migration 0025; existing
  collections are `today`), new collections get PlatformStorage.newCollections
  (default `today` for now); forks take their parent's layout and reference its
  files; stores open each collection's repository in its layout.
- F's jobs: layout.maintain and layout.merge per set, within a Worker's
  storage-call budget; reclaim per set (not started yet: needs the write fence).
- Commits: units hold their runs, the assembly names each tree's as one pending
  object; a type changing privacy commits serially and copies its records to
  the other set.
- Mirrors copy the sets the placement holds; sync packs carry bodies built from
  F's records and full nodes, so receivers see today's format; fsck reads F.
- Cleanup drops a deleted F collection's files after its grace, keeping what a
  live fork still reads.
- Compaction writes a joined run index once (the last two pieces can both
  join: a write R2 refuses).
- The local file store lists from the prefix's folder and reads small objects
  whole in get.

The server suite runs once a layout (vitest projects `today` and `f`), and a
test runs today's and F side by side: same versions, same records, both pass fsck.
… a 20 s default timeout for the doubled server suite)
@isTravis
isTravis marked this pull request as ready for review October 20, 2026 16:44
@isTravis
isTravis merged commit 2cc48dd into main Oct 20, 2026
1 check passed
@isTravis
isTravis deleted the layout-f branch October 20, 2026 16:46
isTravis added a commit that referenced this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant