[tefkah]

Stacked on #69 (leveled packs), which is stacked on #71. Merge them in order, pointing each at main once the one below is in.

The index split ("c3x" in the daily-updates simulation: Layout F Daily Updates). In a leveled collection, the index (the layered runs) now serves only level 0. Each pack in a level or in the history carries its own index. Level merges write no index layer. The history is leveled like the current versions. The design note in meta is planning/kf/underlay/leveled-packs-design.md.

Leveled reads stay flat however many versions pile up; index runs no longer hold an entry for every copy of every record. The simulation puts it at 19 GETs per leaf at the median after three years, against #69's 28.

Measured locally (real code, about 1/64 scale)

200,000 records of about 500 B (102 MB), loaded in key order as the bottom level, then 365 days of 2,000 random changes each (1%). It counts a cold read of 1,000 consecutive current records, 20 leaves per checkpoint. Packs were 2 MB, so a leaf is smaller than a pack, as in production. Script: bench-split.mts, in the session folder; #69 ran the same with bench-leveled.mts.

day	#69: GETs a leaf p50 / p90	#69: index	this PR: p50 / p90	this PR: index (+ shards)
1	10 / 11	5	13 / 14	7 + 1
60	18 / 20	13	15 / 19	8 + 2
120	21 / 24	14	15 / 19	8 + 2
180	25 / 27	19	15 / 19	8 + 2
240	31 / 33	26	15 / 17	8 + 2
300	36 / 39	32	15 / 18	8 + 2
365	22 / 39	15	19 / 27	10 + 2

• Pack GETs are the same in both: 4 at the median, 6 at p90.
• Layout F: leveled packs, with replaced versions in history packs #69's index grows with the runs that hold every copy. Here it's the levels' own indexes plus a level 0 index that merges away to nothing: flat after the first days.
• Day 365 caught a commit in level 0 in both runs.
• A cold leaf reads one level shard per level (2 here). The simulation didn't count those: see "Not in this PR".

How it works

• Packs carry their own index. A layer file named after the pack (layers/<pack>i) is written beside the pack, after it (PackInfo.indexed). Merges and the upload writer write one. Commits' packs don't, since they only ever sit in level 0.
• Lookups (#findLeveled):
  • level 0 through the index;
  • then every current level at once, each overlapping pack through its own index;
  • then the history's level 0 and levels, for what's left (old versions' replaced records).
• The index after packs leave level 0:
  • Its entries for those packs are dead weight. stats.indexStale counts them: a pack merged down adds its records, and so does a sorted load adopted as the bottom level. The count only ever overestimates.
  • Once they're half the index and at least levelConfig.staleEntries (50k), every run merges at once. The merge drops entries whose pack is in a level, in the history, retired or gone. An unlisted pack's entries stay, because an upload's packs are adopted only after its commit's runs are in. The merge is the index's size, which is level 0 plus the stale entries. It settles the count.
  • Without this, a sorted load's run (165M entries for Crossref) would never merge: the size rule waits for newer runs to add up to it, and in a leveled collection they shrink.
  • A retired pack stays until a merge of every run that started after it was retired, or until nothing counts as stale. Lookups skip entries naming a retired pack and read the record where it is now.
  • An entry whose pack is gone anyway (through a head older than the sweep) is skipped once its read fails, and the leaf is found again.
• The history is leveled. Manifest.history is the history's level 0; levels h1–h6 are listed in shards like the current ones. A history merge keeps every copy, one of each key, so an old version's lookup reads a pack per level, not one per month. This replaces Layout F: leveled packs, with replaced versions in history packs #69's history packs in the registry (and the monthly compaction it left for later).
• Reclaim:
  • Ranges are cut at level packs' first keys too.
  • A range job counts each level and history pack's live records through the pack's own index.
  • A pack with nothing live is dropped. One under half live is copied in place, within its own level, with its key range marked again: reclaimRewrite(ids, mark). The server's rewrite job runs markRange over the retained versions.
  • Level 0 is reclaimed as before.
• Forks inherit each pack's index file from where the pack lives, and copy the history's level 0. Mirrors copy the index files and the history.
• Level shards are cached in the layout's memory LRU, not an unbounded map.
• Fixed in Layout F: leveled packs, with replaced versions in history packs #69 along the way (pushed there as 2c91484): push_packs.lo_key/hi_key are stored as JSON. SQLite text ends at a key's first NUL, so a sorted load's bottom level got the range "Doc"–"Doc". Layout F: leveled packs, with replaced versions in history packs #69 hid it because the index served every lookup.

Not in this PR

• Per-pack index reads. A level costs a lookup a shard read (cold isolates), the pack index's directory and a page, then the pack's directory and blocks. The simulation assumed one directory per level. Folding a pack's index into its directory would save a GET per level.
• The scale bench at 1M/5M (testing/, scale-testing), checked against the simulator, is still the next measurement.
• Existing tiered collections aren't converted. There are no leveled collections yet: Layout F: leveled packs, with replaced versions in history packs #69 isn't deployed.

Tests

• Protocol: 278 passed. New in test/repo/leveled.test.ts:
  • the index keeps level 0's entries only, the levels are read through their own indexes, the history is leveled;
  • reclaim copies a mostly-dead level pack within its level;
  • an upload not yet adopted keeps its index entries through index merges;
  • a pack that went down is swept only once no run names it;
  • a sorted load's index entries are merged away once it's the bottom level.
  • Each was checked to fail with its fix taken out.
• Server: 576 passed / 12 skipped, both with and without UNDERLAY_TEST_LEVELED=1.
  • The leveled run caught a real bug: a commit's lookups dropped entries of upload packs not yet adopted, and then the NUL keys above.
  • One write budget gained 2 (the upload pack's index).
• lint, fmt and typecheck clean.

Deploy

Migration 0029_push_pack_indexed (push_packs.indexed, nullable) goes with or before the deploy, after #69's 0028.

• Nothing changes until LEVELED_PACKS=1, as in Layout F: leveled packs, with replaced versions in history packs #69.
• Metric: a leveled collection's indexStale returns to 0 after each merge down; its index stays around level 0's size; levelRewrites rises with reclaims.
• Rollback: Layout F: leveled packs, with replaced versions in history packs #69's code can't read a collection made by this code (its levels have no index entries), so rolling back to Layout F: leveled packs, with replaced versions in history packs #69 needs leveled collections deleted first.