[isTravis]

An audit of layout F after #63 found where memory still grew with the collection. The worst was leaf reads. Index keys were (id, hash), so a leaf's index range held every type's records whose ids fell between its first and last. With Crossref's DOIs, a small type's leaf spans millions of journal-article entries, and the range was read into memory at once.

This changes F's stored format. Only staging test collections are on F; they'll be recreated.

Changes

• Keys are (type, id, hash) (recordKey(type, id, hash)), as record-storage-layout.md's candidate E has them. A leaf's records are one short range of its own type's. Readers pass the tree's type: Leaf.type on Repo.bodyLines/linesAt/entryLine, RepoSource's type, and RepoSink's type per tree (a sink shared by a build is told each tree's type). RecordWriter.add(type, entry, line). F throws if a read has no type.
• Index reads (LayeredIndex.values) take four layers at a time, newest first, and keep only the keys wanted. They stop once every key is found. Before, every layer was read at once, which with the hundreds of runs a large push leaves meant 25 to 50 MB a read.
• PackReader.lines takes each block's lines as it decodes it and lets the block go. A request reads at most 16 blocks, and 8 requests run at once. A leaf scattered over many packs no longer holds every block it touches.
• Merges and reclaim give each input a share of a 16 MB read budget (mergeRead) instead of 1 MB each. A merge of 128 inputs was 128 MB of reads.
• Reclaim plans only at 32 runs or fewer. Each range job reads every run at once, and nothing capped how many there were. Otherwise it throws TooManyRunsError, and the job queues a maintain and retries in 5 minutes.
• reclaimRewrite merges its packs in key order and writes as it goes. It had held every moved entry, plus the block text their keys were sliced from, until the end.
• Sync builds an F leaf's body one gzip member per 256 records, never the leaf raw.
• fsck checks F leaves 256 records at a time.
• Reclaim's mark walks typed roots (recordRoots returns {type, root}).

Tests

• Protocol, 258:
  • a small type's leaf among 20,000 entries of another type is read from at most 3 index pages;
  • 40 runs are read at most 4 directories at a time;
  • reclaim waits for merges past 32 runs;
  • the merge read budget.
• Server, 566, both layouts.

Memory gate

scripts/memory.ts --layout f --records 400000 --small-every 2000 --read-ms 1 --max-mb 60. --small-every makes a dataset type spread over the same DOI space; --max-mb fails the run if any job peaks over the cap. Peaks in MB:

job	parallel commit	serial commit
layout.maintain	54.0	44.7
upload (request)	50.4	48.7
refs.index	48.4	58.0
commit.unit / push.commit	36.4	31.7
push.compact	16.8	41.2
records.ndjson (read whole)	12.0	7.4
records.ndjson.gz (read whole)	10.6	9.1

Both pass. A 1.2M run is going, to check the peaks stay flat. refs.index is today's reference index, not F, and is already filed as growing with the collection.

Not here

Lists that hold one entry per pack (registry.read, mirrorFiles, reclaim's bookkeeping) grow with pack count. That's around 100k packs, about 1 TB, before it matters. A follow-up will page them.

Deploy

No migration. Both Workers. Recreate the staging F test collections afterwards; ones written before this can't be read.