[tefkah]

Compactions (push.compact, push.compactPiece) get their own queue on staging, underlay-staging-compact, which the bulk Worker consumes 32 at a time. Commit units stay on the bulk queue at 8.

Why: on 2026-10-16 the bulk queue went from 100 jobs at once to 8 (#45, #46). The cut was for commit units, which hold whole leaves of 44–216 MB and kept running out of memory. Compaction shares that queue, so it dropped to 8 as well:

Session	Records	Bulk queue, at once	After the upload
a4d41dc5 (Oct 15)	15M	100	compacted and committed in 1.2 h
9f792eef (Oct 17)	30M	8	12.2 h compacting
8870400e (Oct 19)	30M	8	about 10.5 h compacting, after #57 and #58

#57 and #58 cut compaction's CPU about in half: about 53 CPU-minutes an hour on 9f792eef, about 30 on 8870400e. Wall time only went from 12.2 h to 10.5 h. With 7 of 8 slots busy and the isolate's thread half idle, a merge now mostly waits on R2 range GETs, so more merges in flight should go nearly proportionally faster.

A merge holds about 12 MB: the 4 MB prefetch budget plus two 4 MB parts (runs.ts runConfig). A unit holds whole leaves. That's why the two get different limits.

Change:

• jobs.ts:
  • New COMPACT_JOBS set.
  • QueueJobs takes a third queue for those jobs, falling back to the bulk queue.
  • enqueueBatch sends to each distinct queue once. The old if (this.bulk === this.queue) break would send jobs twice once a third queue can fall back to the same object.
• worker.ts: an optional JOBS_COMPACT binding.
• wrangler.bulk.jsonc (staging): the producer binding, plus a consumer for underlay-staging-compact with max_batch_size: 1, max_concurrency: 32, max_retries: 10 and the same dead-letter queue.
• wrangler.jsonc (staging): the producer binding. Uploads queue compactions from the main Worker (session.ts → maybeCompact).
• A routing test: compactions go to the compaction queue, or with the other bulk jobs when there is none, once each.

Unchanged:

• push.commit stays on the bulk queue. When a compaction finishes, it claims the next round, and at the end it runs the commit. When no other push is in its units, a bulk slot frees within seconds, so it doesn't hold compaction back. If two pushes overlap, each round waits for one of the 8 slots. Moving the claim step to the compaction queue would be a follow-up.
• refs.compact* stay on the bulk queue. planCompaction checks for a running compaction and then inserts one without a lock (the known refs.compactPart race). Running more at once would make that worse.
• Prod, whose bulk queue already runs 100 at once.
• Node. Compactions are still in BULK_JOBS, so they keep their place in the order.

Risks:

• Isolate packing. The gain depends on how the runtime spreads invocations over isolates. At 8, every bulk job ran in one isolate, and an isolate holding more than about 8 merges runs out of memory. Merges are idempotent and retried, so a kill costs time, not data.
• Overlapping sessions. Compactions and another session's units can share an isolate if they run at the same time. Loads run one session at a time today.
• Run-index writes. More pieces finishing together means more writes of the same run index. R2 can answer those with a 429, which is retried.
• Dead-lettered pieces. A piece killed 10 times goes to the dead-letter queue. The sweep (recover.ts) queues it again only once none of that session's compactions has moved for its window, and at most 100 per sweep. That's one more reason to halve at the first out-of-memory kills rather than wait.

Deploy: after 8870400e publishes (about 03:00 UTC on Oct 20), not before. From packages/server:

1. wrangler queues create underlay-staging-compact
2. wrangler deploy -c wrangler.bulk.jsonc --env staging (this one consumes the new queue)
3. wrangler deploy --env staging

This deploy also brings #41 to the bulk Worker for the first time. #41 changes only how reads without pointers resolve out-of-line records: they now look up to 16 entries or 4 MB ahead. Units and assembly read with pointers, which pass through unchanged.

Metric:

• Hours of compaction after the next 30M session's upload, against about 10.5.
• exceededMemory on compaction jobs.

Rollback:

• More than one exceededMemory in 15 minutes during compaction: set max_concurrency to 16 (config only).
• Still failing: redeploy both Workers at their versions from just before this deploy. Nothing older than Store compacted runs uncompressed #57's e12f49e9 works, since older versions can't read raw runs.
  • Jobs already on the compaction queue keep draining, because a deploy doesn't detach a consumer. New compactions go to the bulk queue.
  • Then run wrangler queues consumer remove underlay-staging-compact underlay-staging-bulk.

Checked:

• pnpm fmt:check, pnpm lint, and the server typecheck.
• The server suite: 274 tests.
• wrangler deploy --dry-run of both configs: both list JOBS_COMPACT.

[tefkah]

Closing: the rest of the Crossref dump won't load in today's layout (stopped 2026-10-20), and crossref-demo/works will be converted to layout F in place (#61, then the conversion PR), which doesn't compact upload sessions. Upload compaction is unchanged on F, so if a 30M-record load comes back, rebase this branch (compaction-queue, kept) and measure it on that load.

[tefkah]

Reopened: the rest of the Crossref dump, and possibly bigger collections, will be loaded on layout F. Upload compaction is unchanged on F (#61 only made the joined run index write once), so a 30M-record session would compact for ~10.5 h again. Ship after #61's staging deploy, as its own deploy, measured on the next large session.