perf(multicast): share source processing and batched output per worker - #749
Conversation
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_4d4d126f-e890-4cd9-bdf6-74288e56d5e5) |
Documentation previewThe documentation preview has been deployed for this pull request. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_1a5c460e-67df-4e1d-9058-8f01778efc62) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_431cf515-eddf-4359-bfd5-967edfd5ea8d) |
stackia
changed the title
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_9596d334-a750-4263-a69a-184be1bd8336) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_d1aa3b78-287f-434b-a1c5-9983eedc8e0e) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_54102f62-efdc-4f33-aa52-341e490cc827) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_38f5cb81-2e5e-4c25-8410-8430981b7562) |
1 participant
Clients watching the same multicast source in one worker now share one subscription, one ordinary RTP processing path, and immutable output batches. The last subscriber releases the source. Each client retains its own output queue and progress, so partial writes and a slow or departing viewer do not stall the other viewers.
Optimization strategies
1. Share multicast subscriptions and source processing
Each worker matches sources by resolved multicast address, port, SSM source address, effective upstream interface, and FEC port. Channel names,
/rtp/versus/udp/URL spelling, and FCC server parameters do not split an otherwise identical source. Matching viewers share one primary socket and, when required, one FEC socket. Source references govern socket lifetime; event ownership moves to another subscriber when the original subscriber leaves. Workers remain independent processes.Ordinary multicast performs RTP parsing and reordering once per source. Once startup reordering is complete, the expected next packet bypasses reorder-slot insertion/removal when the window is empty and FEC is inactive. Startup, reordered traffic, and FEC retain their full processing paths. This reduces local receive calls, packet processing, and private state; multiple local memberships do not imply that the upstream network carried one complete stream per socket.
2. Batch payloads and share their storage
Payloads are copied once into a 64 KiB batch, preserving complete RTP payloads. A 1,316-byte payload produces batches of 49 packets / 64,484 bytes. The current batch is flushed before the next payload would exceed capacity, avoiding a small tail above 64 KiB. Underfilled batches become eligible for flushing after 100 ms and are flushed by the next worker timer check, also scheduled every 100 ms. Scheduling can add delay; this output batching interval is separate from the receive coalescing interval below.
Clients share the payload allocation, not a mutable queue node. Each
buffer_ref_tview points to an owner and has independent queue links, remaining length, and send offset. The last reference returns the allocation to its worker-owned pool, even if the multicast source has already been destroyed. A sole subscriber uses the original batch descriptor directly, avoiding a separate view allocation.Queue limits charge the full backing allocation until the client releases its reference. A nearly drained 64 KiB batch still counts as 64 KiB for that client's budget. Exhaustion of the bounded batch pool falls back to references to packet-sized buffers; it does not create another multicast subscription.
3. Reduce receive calls, allocation work, and event wakeups
Linux and FreeBSD use
recvmmsgto receive up to 16 datagrams per call. Scratch descriptors and unused packet buffers persist across callbacks. A packet buffer can be reused immediately only when no reorder window, FEC state, or output queue retains another reference. Reception writes directly into pool storage. macOS uses the single-datagramrecvpath, which also reuses its packet buffer.The first readiness notification reads immediately. Later receive work may be coalesced for 1–2 ms according to the actual
SO_RCVBUFvalue and observed packet count: at least 128 KiB permits 1 ms, and at least 256 KiB with a small batch permits 2 ms. These are requested scheduling intervals, not hard latency guarantees. Insufficient or unqueryable buffer capacity selects immediate, level-triggered reads. A busy source also switches to immediate reads; a rate estimate over at least 100 ms can restore coalescing when there is twice the estimated scheduling headroom.EPOLLONESHOT/EV_DISPATCHsuppress redundant wakeups while a delayed read is pending. Only pending sources appear in the receive work list, and detaching the final subscriber removes its pending task. Coalesced reads establish that the socket is empty before sleeping; level-triggered reads can yield after a short batch. A callback processes at most 256 primary datagrams. Trigger-mode changes recreate kqueue filters to replace stickyEV_CLEAR/EV_DISPATCHsemantics, while ordinary rearming reuses the existing registration. Linux applies mode changes withEPOLL_CTL_MOD.4. Schedule writes in the worker before asking the kernel for readiness
Ready output enters a worker-local write queue. Kernel writable notifications are requested only after a write cannot continue, rather than toggling interest for every output batch. Each connection sends at most 256 KiB per turn, and each write-queue pass processes at most 128 tasks; unfinished tasks remain queued. This reduces event-registration work and keeps receive handling, timers, and other clients scheduled.
5. Reuse kernel pages with immutable batch snapshots
On Linux and FreeBSD builds exposing memory-file sealing, nearly full batches with multiple batch subscribers can be written once to a fresh
memfd_createobject and sent to multiple sockets withsendfile. FreeBSD first sizes the object withftruncate. The file is sealed against writing, growing, shrinking, and further seal changes before publication.Every published batch has its own immutable file. Application buffers may be recycled and the application may close its last descriptor while TCP still holds file pages; immutability prevents those queued bytes from being overwritten by a later batch. The platform wrappers account for bytes transferred even when BSD
sendfilealso reports a partial-write error. This removes repeated userspace-to-kernel payload copies for eligible fan-out, but still requires batch assembly and one snapshot write; it is not an end-to-end copy-free pipeline.macOS retains shared memory batches and
sendmsg: it has no compatible sealed-memfd path, and itssendfileaccepts regular files rather than POSIX shared-memory objects. No disk-backed temporary-file path is introduced. See the Apple sendfile documentation and FreeBSD memory-file documentation.6. Allocate memory according to its useful lifetime
Connections allocate RTSP and HTTP proxy state only when those protocols are used. FEC group storage is allocated when recovery groups need to be retained. Ordinary shared multicast avoids per-client reorder arrays; private processing paths allocate them when needed, including clients joining after in-band FEC is detected.
Final connection destruction always releases protocol state through
stream_context_destroy, independently of thestreamingflag. Graceful RTSP TEARDOWN can retain its session asynchronously; completion, timeout, partial initialization, and worker shutdown converge on the same synchronous destructor. Pending RTSP sockets and their poller/fd-map registrations are cancelled before connection memory is freed.Request parsing data and HTTP input occupy independently releasable regions of an anonymous mapping. Input pages are unmapped after parsing/routing, and ordinary stream request pages after response-header generation. HTTP proxy requests retain headers/body while still needed. This returns temporary pages to the OS instead of leaving them in a heap alongside long-lived streams.
Packet pools start with 128 buffers and expand by 128; control pools start with 16 and expand by 16. The batch pool is created on demand, starts with four batches, and expands by four. The worker reclaims entirely idle pool segments while retaining base capacity. Logical client queue budgets remain separate from initial preallocation, preserving buffering allowance while reducing idle memory.
7. Reduce statistics work in the forwarding path
Queue limits, counts, and high-water marks remain current on each operation. Publishing queue snapshots to shared status memory moves to the worker's 100 ms timer, reducing repeated synchronized writes during forwarding. Status displays therefore use the latest published snapshot. Worker ownership checks use a process-local PID cache refreshed after every fork, avoiding repeated
getpid()calls without retaining a parent's identity in a new worker.8. Preserve FCC, FEC, and client isolation
FCC negotiation, unicast delivery, and switching state remain private to each client. Clients share the matching multicast socket during handoff, then join shared batches only after unicast/pending data are handled and sequence state aligns. Existing batches are flushed before promotion to avoid replaying old content to a new subscriber.
Configured FEC or newly detected in-band FEC retains shared sockets but uses private reorder/recovery state. An in-band transition first flushes the shared batch and transfers pending reorder state. Snapshot requests also retain their own processing state. These paths preserve protocol semantics; ordinary multicast benchmark percentages do not describe FCC unicast, FEC recovery, or image extraction costs.
Platform coverage and fallbacks
recvmmsgrecvfallbackrecvmmsgmemfd_create+ seals +sendfile, when availablesendmsgmemfd_create+ftruncate+ seals +sendfile, when availableFallbacks preserve the optimizations that remain applicable:
sendmsg.sendfilefor a client (EINVAL,ENOSYS, orEOPNOTSUPP): mark only that queued view for memory sending. Other views can continue using the snapshot.recvmmsgat runtime (ENOSYSorEOPNOTSUPP): disable batch receive in that worker and userecv, retaining the shared source and shared output.sendfilesupport. Fundamental allocation failures or disconnected clients can still terminate affected work: these fallbacks are not a guarantee of service under unlimited resource pressure.The same implementation is available across the three platforms where the OS supplies the required facilities. The numerical performance results below were measured on ARM64 Linux; equivalent macOS/FreeBSD benchmark percentages have not been measured.
Configuration and compatibility
These optimizations are automatic and add no runtime tuning switches. The existing
--workers/-wcontrols the number of independent sharing domains, and--udp-rcvbuf-size/-Brequests receive capacity; coalescing uses the capacity actually granted by the OS. The existing--buffer-pool-max-size/-balso derives the batch-pool ceiling asmax(4, floor(n * 1536 / 65536)). With the defaultn=16384, that is 384 batches / 24 MiB of payload capacity, allocated on demand. This ceiling is separate from the packet-pool ceiling and excludes descriptors and snapshot storage. Batch size, initial count, growth step, and timing thresholds are internal constants, not newly exposed configuration parameters.Remove
zerocopy-on-send/-Z, theMSG_ZEROCOPYcompletion machinery, related metrics, and OpenWrt/iKuai/container configuration entries. Rename the buffered sending module tosend_queue. The immutablesendfilesnapshots are independent of the removed feature. Web UI source is updated; generatedsrc/embedded_web_data.his excluded from this PR, so rebuild the frontend before building a binary that should include those UI changes.Measurements
The benchmark tools measure the complete service process tree, rotate program order evenly, and produce a separate resource summary. On an Apple M3 Max / Ubuntu 24.04 ARM64 VM with 16 vCPUs, four programs and four workloads were measured four times each, using 5 seconds of warmup and 15 seconds of sampling per trial. rtp2httpd uses one worker; it, msd_lite, and udpxy are pinned to one vCPU. TVGate uses default multicore scheduling with no
GOMAXPROCSoverride or CPU affinity, and only its port/interface are configured. CPU 100% means one vCPU.All three resource means are lower than msd_lite in these four workloads. These are aggregate results for the combined optimizations, not per-optimization attribution or a claim about maximum capacity on every platform. PSS/USS do not include all kernel socket memory or unmapped snapshot pages. The measured rtp2httpd revision is
b4fd92a6; the subsequent kqueue-mode fix retains Linux'sEPOLL_CTL_MODbehavior. The Chinese and English benchmark reports document all four programs equally. Raw records remain local, and performance benchmarks and their helper tests are not added to CI.Comparison with the pre-PR version
A separate before/after run compares
f8c243cbwithbe6b20aaon the same ARM64 Linux VM. Both use one worker pinned to the same vCPU, identical command-line settings, and alternating program order; each workload has four pairs of trials with 5 seconds of warmup and 10 seconds of measurement. For this CPU-efficiency comparison, retain matched pairs where both programs keep all requested clients connected and each sender/client reaches the target bitrate within 2%. All original records remain local.Summing the per-workload mean CPU costs with equal time weights gives a 59.1% reduction, or about 2.4 times the forwarding efficiency per unit of CPU. The 64-client shared-channel workload reduces CPU cost by 78.5% and USS by 93.5%. A rounded public-facing statement is: “Around 60% lower CPU cost across four typical workloads; around 80% lower CPU cost and 90% lower private memory use with 64 viewers on one channel.” These are resource-efficiency measurements on this Linux setup, not a measured increase in maximum client capacity; USS excludes shared and kernel memory. The before/after comparison is kept out of the long-term benchmark documents.
Ownership and code reuse
sendfilesends share one progress-consumption helper. Memory and regular-file entries share head removal, capacity accounting, and reference release. Partial writes reduce remaining payload bytes while retaining the full backing capacity until the entry is removed.Validation
be6b20aa: Linux, macOS, and FreeBSD Release CI pass all 618 tests, including the native ownership and poller-mode checks. Type checks, linters, documentation build, and CodeQL pass._exit(), which would otherwise bypass the usual at-exit check. No sanitizer reports were produced.recvmmsgtests and a full macOS ASan/UBSan suite.Note
High Risk
Touches core multicast forwarding, shared buffer lifetimes, FCC/FEC transitions, and send/backpressure semantics; regressions could cause stream corruption, stuck clients, or memory accounting bugs under concurrency.
Overview
Replaces the optional MSG_ZEROCOPY send path with per-worker shared multicast: one socket subscription, one RTP parse/reorder pass, and ~64 KiB batches fan out to multiple HTTP clients via reference-counted
buffer_ref_tviews (optional sealed memfd + sendfile on Linux/FreeBSD). Queue backpressure now bills backing buffer capacity, not packet count; FCC hands off to shared batches after unicast drains; in-band FEC forces per-client private reorder while still sharing sockets.Removed
--zerocopy-on-send/-Z,zerocopy.c, Docker memlock requirements, and related OpenWrt/iKuai/LuCI options. Sending goes throughsend_queuewith deferred writable polling, capped writes, recvmmsg / receive coalescing, and less frequent status SHM updates. Connections allocate HTTP parse state in anonymous mmap released after routing; buffer pools use smaller initial sizes.Tests & docs:
test_multicast_shared.pyand poller mode regression tests; zerocopy e2e removed. Benchmark harness and EN/ZH reports rewritten (four scenarios,resources.jsonoutput). E2E skill references shared multicast instead of zerocopy.Reviewed by Cursor Bugbot for commit 8c073a5. Configure here.