feat(mem-wal): let a backpressure controller hold back memtable freezes

[hamersaw]

Problem

The memtable seal trigger is blind to whatever the flush lands in. All four arms of memtable_reached_flush_threshold are memtable-local, so an embedder that bounds a downstream tier — a shared page cache over L0, in our case — has no way to stop that tier growing.

Refusing writes does not help: a refused write never runs its seal, and the memtables already frozen flush regardless. The tier grows at (flush rate − compaction rate) × time with nothing to bound it.

Change

BackpressureController gains may_seal(), defaulted to true so lance's own behaviour is unchanged — only an embedder bounding a tier lance cannot see has a reason to say no. When a controller says no, maybe_trigger_memtable_flush returns SealOutcome::Blocked instead of freezing, and the active memtable pins at its cap.

A blocked freeze has to refuse the put that needed it. Growing the memtable instead is not an option: the in-memory indexes are pre-allocated to exactly max_memtable_rows, so an overshoot fails the index apply. So:

• the two pre-insert call sites turn Blocked into Error::backpressure — retryable by construction, since the controller admits freezes again once the tier drains;
• the post-insert rotation ignores it, because those rows already landed; the memtable is left full for the next put's pre-insert check to refuse on.

ShardMemory now carries seal_required, which lets a controller tell a write that still fits in the active memtable from one that can only land after a freeze. The first costs the tier below nothing and there is no reason to slow it; the second is what puts the next generation in it. Gating on that distinction bounds the tier without throttling writes that would not have grown it.

It is answered off the published ArcSwap snapshot rather than under the write lock, so admission can decide before the lock the seal is taken under. To keep one predicate, memtable_reached_flush_threshold now delegates to fill_reached_flush_threshold, which both the writer (under the lock) and the snapshot path call — the check the controller decides on and the one the writer acts on cannot drift.

What is deliberately not gated

force_seal_active. It is how drain and drop-table get bytes out of memory and into storage, and they have to be able to seal whatever the tier below looks like.

Testing

test_a_blocked_seal_pins_the_memtable_and_refuses_the_put drives a real ShardWriter with a controller that blocks freezes and asserts the put is refused with a backpressure error, the generation does not rotate, no rows land, and the seal happens once the block lifts.

cargo test -p lance --lib mem_wal::write::tests — 103 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RY5TX2epNsh5VY6KtbrqyU