Replace the peek with ramped receive dispatch

[LukeButters]

(part 2 see part 1 , some the results include part 0)

Replace the peek with ramped receive dispatch

Problem

The peek's backlog estimate — max(RowVersion) − min(RowVersion) + 1 under READPAST — counts every gap left by other instances' in-flight rows, so under competing consumers it wildly over-reports (e.g. "hundreds" when 3 messages are actually available), and the pump fans out that many concurrent receives. Almost all come up empty, and each empty receive is a full round-trip set: connection acquire, BEGIN TRAN, query, ROLLBACK.

Measured at a production-shaped configuration (12 instances, concurrency-per-instance 256, 100ms peek delay, steady 400 msg/s): 176,319 receive attempts plus 15,141 peeks for 24,000 messages per minute — ~6 wasted round trips per processed message — plus one synchronized receive burst per instance per peek, which spikes latches on the queue index head.

Change

The pump probes with real receives and adapts to what it finds — a receive costs about what a peek costs, but when the answer is "yes" you already have the message instead of a number:

• after an idle/empty period, a single receive probes the queue;
• the dispatch wave doubles while every receive in it finds a message (1 → 2 → 4 → … full width in a few sub-second waves under real backlog);
• a partial wave sets the next wave to the observed availability and continues without backoff (so a busy instance keeps up with its own arrival rate);
• a fully empty wave resets to one probe and waits the configured delay (QueuePeekerOptions.Delay keeps its meaning as the empty backoff);
• waves are capped (64 per instance) so high concurrency limits don't translate into synchronized query bursts.

Mechanically: ReceiveCountdownEvent (which the pump already awaits per batch) now carries whether each receive found a message, so the pump counts wave successes; the peek query no longer runs at all. Processing concurrency semantics are untouched — each receive remains its own connection/transaction/handler, the wave only decides how many independent receives launch between feedback checks, and since the latch is signaled when the receive query resolves (before the handler), waves keep the concurrency limiter saturated under load.

Results

Same benchmark rig as PR #2, on top of the anchored receive (cpu-ms of DB CPU per message at 1 / 6 / 12 instances, concurrency 256, 100ms peek delay, 50ms handler):

	Steady 400 msg/s	Drain 20k backlog @ 12 instances
Stock 9.0.1	3.49 / 5.88 / 6.69	6.28 @ 1,672/s, receive p99 278ms
Anchor only (PR #2)	3.36 / ~5.9 / ~7.8	3.63 @ 2,135/s
Anchor + ramp (this PR)	2.86 / 3.00 / 3.32	2.13 @ 2,124/s, p99 16ms

Wasted attempts at 12 instances drop 152k → 18.5k/min; DB cost per message becomes near-flat with instance count (+16% at 12 instances vs its own single-instance figure, against +92% for stock). In end-user production-shaped testing (Octopus Deploy, 6 small instances) the full series took total DB CPU from 38% to ~20% — below the original single-instance baseline of 22%.

Isolation vs the anchor branch on real endpoints: at modest concurrency (6 instances × 32) this change measures neutral — backlog drain 1.92 vs 1.90 cpu-ms/msg — because 32-wide fan-out was never large enough to hurt. Its win is specifically the high-concurrency configuration that motivated it (hundreds of slots per instance, e.g. 64 × cores): there the peek fan-out is the dominant waste and this change halves steady-state DB cost (table above). Real-endpoint A/B at 16 instances: 4.31 (stock) → 1.87 cpu-ms/msg, +20% throughput. One honest observation from the bursty-arrival test (due-messages moved in batches of 100 at modest concurrency): the ramp trailed the arrival window slightly (52 s vs 46 s to fully drain) as each burst re-ramps — visible only when arrivals are bursty and concurrency is small; at high concurrency the same shape wins outright.

Trade-offs / notes

• After an idle period, receive ramps up over a few sub-second waves instead of jumping straight to a (mis)estimated fan-out. First-message latency is governed by the same empty delay as before.
• The wave cap default of 64 suits larger hosts; on small (2-core) instances we measured 16 as equal-or-better — reasonable to expose as a setting if there's interest.
• The QueuePeeker/peek query becomes unused by the pump (left in place in this PR to keep the diff minimal); anything monitoring peek executions will see them stop.
• Verified: full unit + integration suites green (pump tests rewritten for the new dispatch, incl. a mutation-checked backoff test); 99/101 transport suite with the same 2 pre-existing MSDTC-environment failures as unmodified 9.0.1; real-endpoint A/B at 16 instances: 4.31 → 1.87 cpu-ms/msg, +20% throughput.

🤖 Generated with Claude Code