[isTravis]

PR 2 of layout F (planning: layout-f-pr.md, "PR 2"): a job that converts an existing collection from today's layout (leaf bodies) to F (packs and an index per collection and set).

How it works

POST /api/admin/convert {collection} (stewards) starts a layout.convert job; GET /api/admin/convert?collection= reads its state.

• Version by version. The job walks the collection's versions in order. For each version it compares each set's tree of each type with the version before (newNodes), reads every new leaf's body (today's layout), and writes the records F's index doesn't yet hold into the set's packs, as stored. An out-of-line record stays a pointer. (convertTree and versionTrees in @underlay/protocol.)
• Nothing that hashes changes. Nodes don't change, so neither do roots or version hashes.
• Each record written once. However many versions hold a record, it's written once, and a job that runs again writes nothing twice (KeyedLayout.missing reads the newest manifest).
• Bounded jobs. Each job takes 2,000 leaves or 4 minutes, saves a cursor (version, tree, key), and queues the next. The F index's merges run as it goes: layout.maintain with layout: 'f' opens F whatever the collection's layout.
• Pushes continue meanwhile. The collection keeps reading and committing in today's layout, and versions committed during the conversion are converted in turn.

The switch

When the head is converted, one statement moves record_layout to f, but only if the head is still the last version converted and no push of the collection is committing. Otherwise the job tries again in 30 s.

So that no version is ever published in a layout the collection isn't in:

• A commit keeps one layout. It reads the collection's layout fresh when it starts (stores.forget) and keeps it. A parallel commit records it in its plan, and its units, assembly and validation open the collection in that layout (stores.forCollectionAs).
• Publishing checks it. publishVersion publishes only if the collection is still in the commit's layout (recordLayout). A commit that finds it changed fails with 503, push again (LayoutChangedError).
• Commits wait after the switch. Isolates note a collection's layout for a minute, so commits wait 90 s after the switch (LayoutSwitchingError, 503). Until then no version only F can read appears while an isolate may still read in today's layout.

Not here

• Old leaf bodies stay. Nothing reads them once a collection is on F, but storage cleanup's mark still covers them: marking a node keeps its body whatever the collection's layout. A follow-up marks bodies only for collections on today's layout, so the cleanup can drop them after its grace period.
• Mirrors. A mirrored collection's placements get F's files from the next version they copy; not tried on a mirror.
• Forks. A fork on today's layout of a converted parent keeps reading the shared bodies. Converting a fork gives it its own packs.

Tests

test/convert.test.ts:

• Every version reads the same. Four versions: serial and parallel commits, private records, a record over the out-of-line size, deletes, and one committed after the conversion started. Each reads the same from F as from its leaf bodies, with the same hashes. fsck passes, and the records written equal the distinct records exactly. It then commits on F, serially and in parallel.
• The switch waits while a push is committing.
• Commits wait out the switch with a 503.
• A version written in one layout isn't published once the collection has left that layout.

Protocol 258 and server 574 (both layouts) pass.

Deploy

No migration. Both Workers. Before crossref-demo/works (110.7M records), size its bodies by prefix and try it on a smaller collection on staging.