[tefkah]

Stacked on #71 (the type cuts, onto main: #68 merged into upload-packs instead). Merge #71 first, then point this at main.

Layout F gets leveled packs, so reading the current version stays about as cheap however many days of changes go in. This is "c3" from the daily-updates simulation: Layout F Daily Updates. The design note, in meta, is planning/kf/underlay/leveled-packs-design.md. It's off by default: LEVELED_PACKS=1 turns it on, and then only for collections that are empty at their first write.

Measured locally (real code, scaled down about 1/64)

The measurement ran 200,000 records of about 500 B (102 MB), loaded in key order, then 120 days of 2,000 random changes each. It counts a cold read of 1,000 consecutive current records, 20 leaves per checkpoint, at the median. Packs were 256 KB and the smallest level target 4 MB. Script: bench-leveled.mts, in the session folder.

day	tiered: pack GETs	leveled: pack GETs	tiered: index GETs	leveled: index GETs
1	4	4	5	5
10	10	4	13	8
20	16	6	14	10
60	26	6	18	10
90	16	6	18	10
120	20	8	18	24

• Packs: leveled pack reads stay at 4–8, while tiered ones grow 2–5× with the days, as the simulation predicted.
• Index: the index is now most of a leveled read, and it varies with the number of index runs (2–10 here). That's the next PR: a current index kept with the levels, and a history index apart.

How it works

• Packs now record their first and last keys (lo/hi). A leveled collection's packs also say their level, or history with the month.
• Levels:
  • Level 0 is what commits and uploads write, listed in the manifest as before.
  • Levels 1–6 each hold packs that never overlap, in key order. Each level is listed in write-once shard objects (manifest/f/levels/<id>.json, up to 512 packs each) that the manifest names with their key ranges.
  • The bottom level of a Crossref-sized collection, tens of thousands of packs, stays out of the manifest every reader reads. A merge rewrites only the shards it touches.
  • Readers never read shards: they find records through the index, as before.
• Sizes, as RocksDB's dynamic levels:
  • The bottom is the largest level, and each level above is 1/10 of the one below.
  • Level 0 merges into the highest level whose target is at least 256 MB, and never straight into the bottom. For a 1 TB collection that is level 3 (about 1 GB).
  • Level 0 merges at 16 packs, or at 64 MB or a tenth of its target level, whichever is more.
• Merges (#mergeLevels, in maintain):
  • Each pass merges from the level most over its target.
  • Level 0's oldest packs merge into the base level. A pack of a lower level merges into the next, choosing the pack whose overlap there is smallest for its size.
  • Both sides stream into the next level's packs, cut at 16 MiB and at types. A new index layer re-points what moved.
  • A merge reads one pack plus the packs it overlaps in the level below: about 180 MB at most. It runs in a maintain job and never needs splitting.
• Replaced versions: where a merge finds a record id with several hashes, the newest copy stays in the levels and the older ones go to the month's history packs. History packs are listed in the registry, which reclaim reads.
  • "Newest" is a guess: a revert brings an old hash back, and it is then read from history. Only locality depends on the guess, since the index says where every copy is.
• A sorted load becomes the bottom level as written. Packs that don't overlap, adopted into an empty collection, are listed as the bottom level directly. The Crossref reload, sent in_key_order into a new collection, goes there in one step, with no rewrite on the way down.
• Reclaim, forks and mirrors know levels:
  • Reclaim drops and rewrites level packs and takes them out of their shards. Its rewritten packs go to level 0.
  • A fork of a leveled collection is leveled, with its own shards listing the parent's packs.
  • A mirror copies the shards and the leveled mark too.
• Which collections are leveled: a marker object, manifest/f/leveled.json, written by the collection's first write when LEVELED_PACKS=1 (layoutConfig.levelNew) finds it empty. A fork gets it from its parent. A collection that already holds packs stays tiered, unchanged.

Not in this PR

• Index split: the next PR.
• Monthly compaction of history packs into sorted runs: until then, history is each merge's sorted pieces. Old-version dumps read more streams than the simulation's 17.
• Existing tiered collections aren't converted; they stay as they are. The design note asks how Layout F: convert a collection from today's layout #67's conversion should land (level 0 or a sorted bottom).
• The scale bench at 1M/5M with 60 dailies (testing/, scale-testing), checked against the simulator, is the next measurement.

Tests

• Protocol: 273 passed. New in test/repo/leveled.test.ts:
  • merges level by level, each level apart, every record readable;
  • replaced versions in history;
  • a sorted load as the bottom level;
  • commits landing during merges;
  • reclaim from levels and history;
  • forks;
  • leveling at first write only for empty collections;
  • tiered collections unchanged.
• Server: 576 passed / 12 skipped, both with and without UNDERLAY_TEST_LEVELED=1. Three tests gained layout-aware expectations: two write budgets (leveled adds the mark and a shard), and the upload test, where a leveled first load is the bottom level rather than the registry.
• lint, fmt and typecheck clean.

Deploy

Migration 0028_push_pack_ranges (push_packs.lo_key, hi_key, nullable) goes with or before the deploy. The keys are stored as JSON strings: SQLite text ends at a key's first NUL (found building the index split; fixed here in 2c91484, with the upload test checking a bottom pack's range after it goes through the database).

• Deploying changes nothing until LEVELED_PACKS is "1" in both Workers' vars. Then new (empty) collections are leveled from their first write.
• Metric: a leveled collection's levelMerges stat rises with commits; level 0 stays under 16 packs; leaf reads' pack GETs stay flat across dailies.
• Rollback: unset LEVELED_PACKS. Leveled collections must then be read by this code: older code doesn't know shards and would miss level packs in reclaim. So a code rollback with leveled collections in place needs them deleted, or this PR kept.