[tefkah]

On layout F, an upload now writes its records straight into the collection's packs, where they stay. Each run entry carries the record's place in its pack (l) instead of its body (b). Compaction then moves entries of about 150 bytes instead of whole records, and commit units index records without reading or writing them again.

Why: a 30M-record Crossref session spent about 10.5 h compacting 206 GB on staging (2026-10-19, after #57 and #58). F leaves upload staging and compaction as they were, and then its units copy every body once more into packs. With this change a record's bytes are written once, at upload.

Measured locally

Same harness, an F collection, 30,000 records of about 5.6 KB (167 MB raw), 30 uploads of 1,000, compaction forced (commitRuns: 4), parallel commit. Sizes are bytes put to the store.

	main (F)	this branch
Session runs written at upload (gzip)	97.6 MB	3.3 MB
Compacted runs (raw)	162.1 MB	4.7 MB
Record packs	94.4 MB, at commit	94.4 MB, at upload
Upload + compaction	9.9 s	5.8 s
Commit	2.5 s	0.7 s
Full read of the version back (cold)	505 GETs	5,298 GETs; 537 with uploads in key order

Compaction moves about 34× fewer bytes here. At 30M records it rewrites three levels, so the saving per session is about 200 GB × 3.

The cost is read locality. A pack holds one upload batch, so records uploaded in random key order end up spread across packs: a full read of the version costs about 10× the GETs. Uploaded in key order, reads are as before. The Crossref loader should send records in DOI order (a loader change, not made here). Repacking in key order after a commit could come later if unsorted bulk loads matter.

How

Upload (push/packs.ts, push/delta.ts), F collections only:

• The batch's last entry of each key goes into one pack per set (KeyedLayout.writePacks, in key order). Large records go in too; nothing is staged out of line on F.
• The run entry gets l (its place) and r (the files it references, read at upload) and loses b.
• Pack ids are recorded in a new push_packs table before the packs are written, and their stats go in the upload's existing run batch. A dead upload never leaves a pack that cleanup can't find.

Commit:

• toRecordEntry carries packed and fileRefs. RepoSink indexes a packed entry where it lies (RecordWriter.addPlaced) and writes no pack. FileRefDelta counts fileRefs for entries without bodies.
• commitVersion names the session's packs (adoptSessionPacks) just before publish, for the serial and parallel paths alike.
• Uploads of 1 MB raw or more (UPLOAD_PACK_FINAL_RAW) go in the registry with a fixed key, and are never merged: merging them was the external sort again, on the set's one maintain lane. Smaller ones, from small pushes, go in the manifest and merge as a commit's do.
• Naming happens after the commit's index runs are appended, never before. A pack merge re-points its records in a newer layer, and a commit layer folded in after it would point back into the swept pack.
• Retries name nothing twice: the manifest's new adopted list (last 1,000 commits), plus the fixed registry key. PackRegistry.read also de-duplicates by id.

Cleanup (cleanup/internal.ts, settleSessionPacks), once a session is finished and past its grace:

• A version names the session: nothing to do.
• No commit ever started (new push_sessions.commit_started, never cleared, unlike finalize_started_at): the packs are deleted.
• A commit started and never published: the packs are listed, not deleted. Its index may already name their records: one identical to a record the head holds has its newest place in such a pack. Reclaim drops what no version uses.

Tests

• Server suite, both layouts: 573 passed, 9 skipped. test/upload-packs.test.ts (F only, 8 tests) covers:
  • runs carry places
  • commit writes no packs
  • large records in packs, nothing staged
  • private records in the private set
  • file references counted from the run entry (a missing file is refused)
  • parallel commit after compaction gives the same version hash and records as today's layout
  • registry and manifest listing and merges
  • cleanup deletes packs of a never-committed push, keeps a published one's, and lists, not deletes, the packs of a push whose commit started.
• Protocol: 256 passed (+3: indexed where they lie and re-pointed by merges, listed once across retries even after a merge took them, dropped only when never listed).
• Two existing tests are now layout-aware:
  • The cleanup test about staged large records runs on today's layout only.
  • run_bytes is smaller than the records on F.
• pnpm fmt:check, pnpm lint, pnpm typecheck clean.

Since review (2026-10-20)

• Rebased onto Layout F: a commit's writer holds a run at most #63 and Layout F: index keys lead with the type; reads and merges bounded #65 (merged in with a merge commit, no force-push). Upload packs are keyed (type, id, hash). RecordWriter.addPlaced(type, entry, place) adds to the same entries as add, so drain writes a run every writerRunEntries (test: 350 placed entries at a limit of 100 → four index runs, no packs).
• Key-ordered sessions ("in_key_order": true at open, F collections only; the response says whether it was taken):
  • Each records request must be in (type, id) order, or it fails with 422 at the first line out of order.
  • Records stream into packs as they arrive (keyOrderPlacer, KeyedLayout.uploadWriter), so packs fill to 16 MB with neighbouring records, and no batch is held in memory.
  • Requests can be 64 MiB / 50,000 lines (batchLimits, advertised in limits).
• overlapping_uploads on every session: uploads whose key range overlapped an earlier run's when they landed (checked after the upload's run is recorded, so concurrent uploads aren't missed). Zero means the load was sorted. It's in the session status alongside in_key_order.
• Pack rows come from write hooks (PackHooks.before / after): a row before each pack is written, its stats once it is. Unsorted batches use the same hooks.
• Cleanup:
  • A pack without stats never reached a recorded run, so it's always deleted.
  • dropPacks deletes only packs the collection doesn't list (manifest, retired, or the session's registry object).
• Docs: docs/protocol-v2.md §11.4 covers in_key_order, the ordering rule and its 422, the status fields, and the limits.
• Tests: protocol 263 (streaming into whole packs, a new pack at each step down, write hooks), server 576 / 12 skipped on both layouts. The F upload tests (11) add key-ordered streaming over 8 MB, the out-of-order refusal and its cleanup, and overlap counting.

Strategy checked (2026-10-21)

The open review question (a server-side key-order repack for clients that can't sort) and the larger one behind it (does F hold up once Crossref takes daily updates for years) were simulated against a profile of the whole dump and its measured change rate. Write-up: Layout F Daily Updates; in meta, planning/kf/underlay/daily-updates-layout.md.

• This PR stands. Writing records once at upload is right whatever maintenance runs after it.
• Bulk loads go in key order (in_key_order). No server repack: the Crossref loader sorts, and a full-dump repack isn't worth building.
• Merging unsorted uploads' small packs doesn't help. Layout F: uploads write records straight into packs; runs carry their places #64 plus that fix reads as badly as main at three years (leaf median/p99 272/3,953 GETs against main's 255/2,556).
• F's tiered merges don't hold up under daily updates, on main too. With the measured workload (97.6k changed records a day, skewed to recent works) a cold leaf read grows from 23 to 255 GETs at the median over three years (607 at 800k a day). Thinning reclaims nothing and old-version dumps read at random. That's a follow-up, not a change here: leveled maintenance with current versions leveled down to the base, history in monthly runs, and the index split into a current index and a leveled history index ("c3x": 19 GETs median and 97 at p99, flat over three years; ~$0.60 a month more). Packs should also close at type boundaries; on its own that brings main to 72 median, still growing.

Conflicts

None: #63 and #65 are merged in.

Deploy

Migration 0026_push_packs (table push_packs; columns push_sessions.commit_started, in_key_order, overlapping_uploads) with or before the deploy. Both Workers:

• Main Worker: uploads write packs.
• Bulk Worker: units index packed entries; assembly publishes and names packs.

Metric: the next large session on F compacts in minutes, not hours; its push_runs.bytes is about 1/40 of the records; commit units write no packs/ objects.

Rollback: redeploy both Workers at their previous versions. Published versions read the same: F reads go through the index, whatever wrote the pack. Sessions still open with packed uploads can't be committed by the old code (entries without bodies): upload them again.

[isTravis]

Writing records once at upload is the right direction: it's what's left of the cost of loading the Crossref dump on F. Four things before it merges:

1. Rebase onto Layout F: a commit's writer holds a run at most #63 (merged). As you note, addPlaced has to count toward layoutConfig.writerRunEntries like add (and go through drain). Otherwise a large commit of packed entries holds every index entry until finish, the bug Layout F: a commit's writer holds a run at most #63 fixed (204k records ran the bulk Worker out of memory on staging).

2. Index keys are changing to (type, id, hash). With keys of (id, hash), a leaf's index range is every type's records whose ids fall between its first and last. With Crossref's DOIs, a small type's leaf spans millions of journal-article entries, all read at once. The fix (branch layout-f-bounded, PR soon) makes recordKey(type, id, hash), RecordWriter.add(type, entry, line), and has leaf reads carry the tree's type. writePacks and addPlaced will need the type too: the upload knows each record's type, and RepoSink has type per tree. Best to rebase onto that rather than land first, or every upload writes keys in the old format.

3. Read locality. Upload packs of 1 MB or more are never merged, so records uploaded out of key order stay scattered until reclaim rewrites them. A leaf's records can then mean hundreds of pack reads, and your full read was ~10× the GETs. Before Crossref relies on this, either the loader sends records in DOI order (and we're sure it does) or there's a key-order repack after commit.

4. Pack count. One pack per upload batch per set, all in the registry, is ~30,000 packs for a 30M-record session. registry.read, mirrorFiles and reclaim's bookkeeping each hold the whole list, so pack count becomes the next thing that grows with the collection. Bigger upload packs, or merging them in key order, would keep it down.