Skip to content

Layout F: packs never span types, and merges take one type's packs - #68

Merged
isTravis merged 1 commit into
upload-packsfrom
type-cuts
Oct 21, 2026
Merged

isTravis merged 1 commit into
upload-packsfrom
type-cuts

Conversation

@tefkah

@tefkah tefkah commented Oct 21, 2026

Copy link
Copy Markdown
Member

Stacked on #64 (base upload-packs); retarget to main once #64 merges.

On layout F a pack now closes where the record type changes, so no pack spans types, and pack merges take one type's packs at a time. That is requirement 9 of the layout ("no pack spans types or sets"), and it is the first of three steps toward leveled maintenance for daily updates: Layout F Daily Updates.

Why

Keys lead with the type, so a type's records are contiguous in key order. A pack that spans a type boundary still mixes the tail of one type with the head of the next, and tier merges then mix types across many packs. In the three-year simulation of Crossref's measured daily updates, a cold leaf read on main reaches 255 GETs at the median (2,556 at p99). With packs cut at types and merges within a type it is 72 (2,079 at p99). It still grows with every day (p90 345 at one year, 835 at three), so this does not replace leveled maintenance. It helps until that lands, and leveled maintenance needs it anyway.

A real-code check of the simulator, with main patched the same way, 150k records for 30 days of dailies: commit units matched exactly, pack counts within a few, and median pack GETs per leaf within one.

What changed

  • PackWriter closes the current pack when a key's type differs from the last one's. PackInfo gains type, the one type a pack holds. It is absent on packs written before, and on keys without a type (some tests).
  • #mergePacks queues non-final packs by (tier, type) instead of by tier, and merges the longest queue. Packs without a type form their own queue and merge among themselves. A test caps a merge at 8 inputs: grouped, eight packs of one type make one pack of 32 records; mixed, they would make two of 16 and move up a tier half-full. The test fails on the old code.
  • Uploads (Layout F: uploads write records straight into packs; runs carry their places #64): push_packs gets a record_type column (migration 0027_push_pack_types), set from the pack-write hook. adoptSessionPacks passes it on, so an upload's small packs merge with their own type's packs once listed.

Cost: a small type with fewer than 8 non-final packs in a tier waits to merge. In the simulation, 242–261 non-final packs stay listed in the manifest against 1–6 before. That's manifest entries, not reads: lookups go through the index.

Tests

  • Protocol: 265 passed, 3 runs. New: packs close at each type and report it; merges take one type's packs.
  • Server: 576 passed / 12 skipped on both layouts. The F upload tests (11) now also check that the type is recorded at upload and every adopted pack keeps it.
  • lint, fmt and typecheck clean.

Deploy

Migration 0027_push_pack_types (push_packs.record_type, nullable) goes with or before the deploy. It goes with #64's 0026, or after it if that is already applied.

  • Main Worker: uploads record each pack's type.
  • Bulk Worker: commit units and merges cut packs at types; merges group by type.

Metric: new packs/ objects each hold one type (type set in the manifest and registry); listed non-final packs stay in the low hundreds per set.

Rollback: redeploy both Workers at their previous versions. Packs with type read the same, and the old merge ignores the field. The column can stay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants