Skip to content

feat(swarm): host a swarm rendezvous on every node - #262

Draft
grumbach wants to merge 14 commits into
WithAutonomi:mainfrom
grumbach:feat/peer-swarm
Draft

grumbach wants to merge 14 commits into
WithAutonomi:mainfrom
grumbach:feat/peer-swarm

Conversation

@grumbach

@grumbach grumbach commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Every node with a WebRTC Direct endpoint hosts swarm rooms (one without refuses joins as not hosting, so clients go on to the next): it introduces clients reading the same file to each other, carries their WebRTC offers and answers, answers their STUN on its WebRTC Direct socket, and stays the last seed for records. No new binary or port. Design, bounds and rollout: ADR-0026 (new, Proposed). The branch is rebased onto main as linear commits, so the swarm's wire messages follow ADR-0020's get-or-closer request and the check that routes swarm requests by a message's header moves with them. The commits after the feature point the three dependencies at their matching branches, let the notices generator match git dependencies on a branch whose name has a slash, meet Rust 1.99's clippy, make three swarm tests prove what they claim (the pending-offer cap, signal expiry, and that only an offer's recipient can answer it), state the client's send limit and waiting room joins as the client applies them, hold native clients' swarm requests to a budget of their own (64 a second per node, 8 per host, as browsers' already were), host rooms only under the endpoint each refresh of the browser listener published (none from the start of a UPnP-mapped endpoint's withdrawal, ADR-0027, or of a changed endpoint's publication, and none after a refresh that failed), compile a test-only accessor only with the browser listener, lock the dependencies at their branches' heads, give a request frame's swarm charge back on every early exit (a frame taken on the swarm's budget that fails to arrive, open or parse, whose buffers are refused, or whose read is abandoned, spends none of it), and say in ADR-0026 that a client's downloads of one file share their record order.

One change across six repositories. Merge order:

  1. ant-protocol feat(swarm): add swarm rendezvous messages for clients sharing records ant-protocol#50, saorsa-transport feat(webrtc): answer plain STUN Binding requests on the WebRTC Direct socket saorsa-transport#231 and saorsa-core feat(network): send a message on one named channel only saorsa-core#183, independent of each other
  2. ant-node feat(swarm): host a swarm rendezvous on every node #262
  3. ant-client feat(swarm): share records with other readers of a file, in every client ant-client#244
  4. Try https://github.com/WithAutonomi/try/pull/21, once a browser SDK release carries the swarm

This PR depends on: protocol, transport and core, as git branch dependencies on their feat/peer-swarm branches. Once those merge, the dependency commit moves to their main branches (or the released versions at the release train).

Linear issue

Closes V2-1892

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Proposed; a human confirms at review.

Compatibility

  • Wire: handles the swarm messages on the chunk protocol, advertises swarm_rendezvous to browsers and answers plain STUN for hosts that announced in the last 60 s; clients without the swarm see no change.
  • Storage: none (rooms live in memory, bounded)
  • API: none for users; a swarm module the node and its devnet use.

Semver impact

  • breaking
  • feature
  • fix

Test evidence

  • At d39c393f, rebased onto main 34b17c65 (the WebRTC listener summary, pressure-gated retention, the devnet's SIGTERM and run-directory changes, then test nodes on OS-assigned ports with 10 s for DHT requests, a Security Audit that fails only on advisories a pull request introduces, notices generated only when their inputs change, and the release signing key), on this PR's git dependencies at their branches' heads; the last rebase moved this PR's native swarm flood test onto an OS-assigned port, as main's tests now are. As main's CI runs them: lib and bins with test-utils 1307, 10 and 5 passed; cargo test --lib --no-default-features 1241 passed; doc tests 1 passed (2 ignored); webrtc_direct_devnet 6 passed and its ignored five-node test 1 passed; cargo check --lib --no-default-features --locked and RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps clean; cargo fmt --check clean. cargo clippy --all-targets --all-features -- -D warnings (Rust 1.99, macOS) reports only an unnecessary_wraps in main's devnet/run_dir.rs, whose function can fail only on Linux, so CI's Linux lint passes; the library alone (cargo clippy -- -D warnings) also reports main's unused_self in payment/verifier.rs, and without default features main's unused_self lints in replication/audit_metrics.rs too; this PR touches none of these files.
  • Unit tests (src/swarm.rs, src/web_rtc.rs): a request frame admitted on the swarm's budget once the reads' is spent giving that charge back on every early exit (twice the source's swarm budget each of frames with bytes after their end, a channel closing mid-frame, frames that do not open under a real PQ session, headers that do not parse, a chunk read, a read abandoned mid-frame, and frames over the source's byte budget; both swarm buckets full after each, and a swarm request then still admitted on the swarm's budget and keeping its charge; the tests fail with the refund removed), requests naming another node, introductions, answers nobody asked for, long-poll wake-up and shutdown, caps and leaving, expiry, the STUN allowlist, the unanswered-offer cap, browsers never waiting, replayed or stale native requests, histories and members capped per host, ended memberships refused however late, a relayed SDP keeping no public address but its sender's (its IPv4 address, or any in its IPv6 /64; the test relays IPv4), a stored SDP charged the memory it holds, the browser request budgets (swarm requests on a budget of their own, refunded to the reads, taken on the swarm's when the reads' is spent, all-or-nothing across buckets), the native request budget (64 a second per node, 8 per host, an IPv6 /64 sharing one, a flood through the router held to it while another host gets through), only an offer's recipient answering it, the pending-offer cap and signal expiry each on their own, swarm requests told apart by a message's header alone and a get-or-closer request or response never taken for one (the test fails with the swarm's discriminants from before main's get-or-closer pair), and the swarm hosting rooms only under the endpoint a refresh published: none once main's withdrawal removes an endpoint only a gateway mapping reached, none while a changed endpoint's file is being written (a FIFO holds the write, on Unix) or after a refresh that could not write a changed or first endpoint's file, and the old endpoint's again if it comes back before a changed one is published (the moved-mapping and comes-back tests fail if the swarm takes the published endpoint after a failed refresh, they and main's withdrawal test fail if it is never updated, and the FIFO test fails without the clearing before the write).
  • Adversarial: hostile SDPs (spellings a receiver reads leniently, third-party addresses, oversized lists), forged and replayed native requests, a member moving hosts, request floods against the read and swarm budgets.
  • End to end: the client's native swarm tests run this node in-process with real webrtc-rs; the client's browser job runs this node in CI against real Chromium pages.
  • On the rebased heads, in the lab of Many downloaders below (32 nodes of these PRs at 008a76d6, the swarm client at e44ea44 and, as the released client, client main 26199fe; ant-node and ant-client were rebased again since, onto mains whose new commits change tests, CI and the release signing key, and the PRs' own code is unchanged but for one node test moved to an OS-assigned port):
    • 16 readers, four runs of each client in balanced order (two with every reader starting within 3 s, two with one starting every second): the swarm client finished 61 of 64 downloads (16, 15, 15 and 15 of 16) against 27 of 64 (8, 3, 11 and 5; Fisher exact p < 0.0001), its median lower in every pair (218 to 385 s against 541 to 1153 s); the nodes sent 2.6 to 3.4 GiB a run against 6.4 to 12.9 GiB (66% less on average) and used 83 to 172 against 161 to 848 CPU seconds, 306 to 415 MiB coming from peers. The swarm client's 3 failures: 1 reader found no node at start, 2 could not fetch the file's DataMap (after 59 and 78 s); the released client's 37: 34 gave up on a chunk after their retry rounds, 2 found no node at start, 1 could not fetch the DataMap (after 70 s). Against the earlier batch on the merged heads, the swarm client finished 61 rather than 58 of 64 downloads and the released client 27 rather than 40, the nodes sending it 6.4 to 12.9 rather than 4.3 to 8.1 GiB a run; client main changed only its browser download progress, its install script and CI's audit ignores in between. Separate batches do not isolate the cause of the released client's drop; in the first control below, with the same mix of runs, it finished 30 of 64 on node main too.
    • Controls on the same day, the released client alone on 16 readers, builds alternating in balanced order. Against node main f37429a7, which reads saorsa-transport 0.38.2 from crates.io while these nodes read transport main and its unreleased commits since: these nodes finished 24 of 32 downloads against 15 of 32 with one reader starting every second (two runs each), and 20 of 96 against 38 of 96 with every reader starting within 3 s (six runs each; per pair of runs, 3 fewer of 16 on average, 95% interval 8.6 fewer to 2.6 more). Against node main built on the current core, protocol and transport mains with the third-party versions of these nodes' lockfile, so that only these PRs differ: these nodes finished 45 of 64 against 30 of 64 with every reader starting within 3 s (four runs each; 3.75 more of 16 per pair, interval 1.8 fewer to 9.3 more). The same node binary on the same scenario files finished 20 of 96 in the first two controls and 45 of 64 in the third, so this lab's batches swing by more than these differences, and none of these controls rules out a small loss for released clients on these nodes. Two later batches of these two builds broke off early, when the lab host's disk filled and then its memory ran short; their first runs finished 0 of 16 on these nodes, and the first run of every batch so far did badly whichever build it used (6, 0, 2, 0 and 0 of 16). On the code, a released native client's requests take the same path through these nodes as through node main: these PRs add a look at each chunk message's tag (to send swarm requests elsewhere), a sweep of the swarm's rooms every 15 s and a cleanup when a peer disconnects; decoding now refuses bytes after a message's end, which released clients' postcard::to_stdvec encoding never sends; the transport change is on the browsers' WebRTC Direct socket, not the QUIC one; and saorsa-core only gains a method.
  • Before the rebase onto the current mains, on the merged heads, in the lab of Many downloaders below (32 nodes of these PRs at 039ce7c5, the swarm client at 40258e4 and, as the released client, client main 42b68ff, which reads through ADR-0020's get-or-closer lookup; the later commits change only docs, a test's name and locks):
    • 16 readers, four runs of each client in balanced order (two with every reader starting within 3 s, two with one starting every second): the swarm client finished 58 of 64 downloads (16, 16, 15 and 11 of 16) against 40 of 64 (14, 9, 12 and 5; Fisher exact p = 0.0003), its median lower in every pair (249 to 607 s against 355 to 701 s); the nodes sent 3.0 to 5.0 GiB a run against 4.3 to 8.1 GiB (42% less on average) and used 80 to 149 against 87 to 197 CPU seconds, 215 to 389 MiB coming from peers. Against the runs before the merges, the released client finished more (40 of 64 against 21 of 64) and the swarm client fewer and later (64 of 64 at medians of 219 to 283 s), so the gap narrowed. The swarm client's 6 failures: 3 readers found no node at start, 2 could not fetch the file's DataMap (after 61 and 66 s) and 1 gave up on a chunk after its retry rounds; the released client's 24: 17 gave up on a chunk, 4 found no node at start, 3 could not fetch the DataMap (after 66 to 80 s). DataMap fetch failures also occurred before the merges; both clients now read the map through the same lookup.
    • Half the nodes still on the released v0.23.1 (16 of these PRs' nodes beside 16 of v0.23.1, half the readers on each client, three runs): every run overloaded the nodes, harder than before the merges, the nodes sending 16.4 to 21.0 GiB a run (5.3 to 14.3 GiB before). The swarm client finished 11 of 24 downloads (2, 2 and 7 of 8) against 1 of 24 for the released client in the same runs (Fisher exact p = 0.002); 11 of the swarm client's 13 failures and all 23 of the released client's gave up on a chunk after their retry rounds, the other two found no node at start. Today's network beside the same old nodes, run after those three rather than alternated with them (16 nodes of ant-node main e61446b8, every reader on the released client, three runs), finished 2 of 48 downloads, the nodes sending 22.4 to 25.0 GiB a run (7 of 64 and 8.6 to 16.9 GiB before the merges): the harder collapse comes with the merged mains, ADR-0020's read among them, and the mix with the swarm client finished more (12 of 48 against 2 of 48, p = 0.007) with the nodes sending 22% fewer bytes (18.5 against 23.7 GiB a run on average); its released readers did as well as today's (1 of 24 against 2 of 48). The old nodes logged no panic and no ERROR line, and 31 to 325 decode warnings a run beside these PRs' nodes, the swarm client's room requests.
  • Every other lab result below, the browser lab, Many downloaders, Controls and Rollout, was measured before these PRs merged their mains, so before ADR-0020's native chunk read, on both clients and on the nodes.
  • Local lab with browsers, on these PRs as first published (one VM, 56 nodes: 40 built from these PRs, 6 restarted during runs, 10 on the released v0.23.1; 6 Chromium readers, each on its own address and CPU share; the same page on both sides, nodes only with the published SDK 0.1.5 against nodes plus readers with the swarm client), run back to back: where the nodes were the bottleneck, six staggered readers used 15% and 16% fewer node bytes in two pairs and finished 13% and 20% sooner (median 138 to 121 s, 144 to 115 s), each taking 17 to 61 MiB from the others; where readers' own CPU was the limit, 4% fewer node bytes and 3% sooner. First chunk: 3.6 s sooner at the median over the 12 staggered readers, but +0.3 s (95% interval -2.0 to +3.0 s) pooled with 52 busy-node readers of earlier builds, so a small delay is not ruled out. Every reader had the file's SHA-256. The real ant file download CLI joins the same rooms as the pages (25 s beside four pages, which took 144 MiB from each other). No dev testnet run yet.
  • Many downloaders (local lab: one VM, 32 nodes built from these PRs with their upload shared at 100 Mbit/s and 1.5 CPUs between them, each reader the real ant file download on its own address with a fifth of a vCPU, one 40 MiB file, a fresh network for every run), the swarm client at a5cb5a2 (the client's later commits space a file download's last retry rounds after slow rounds, 19715a9, in the rollout runs below, update Cargo.lock: the native devnet's ant-node and a Windows-only dependency resolution, and merge main) against the released one (client main, no swarm), alternating:
    • 16 readers, four runs of each (two with all starting within 3 s, two with one starting every second): the swarm client finished 64 of 64 downloads, median 219 to 283 s, the nodes sending 2.6 to 3.3 GiB and using 63 to 82 s of CPU, 217 to 366 MiB coming from peers; the released client finished 21 of 64 (1, 9, 3 and 8 of 16), median 431 to 815 s, the nodes sending 5.1 to 9.4 GiB and using 114 to 230 s of CPU. The nodes sent 60% less for the swarm client.
    • 8 readers, nine pairs: the nodes' shared link is still the limit, so a run's time follows the bytes the nodes send, which vary from run to run with either client (87 to 214 MiB per file downloaded, for a 40 MiB file). Medians 47 to 116 s (the median run 64 s) against 49 to 140 s (71 s); the swarm client was faster in five pairs and slower in four; the nodes sent 8.2 GiB against 8.7 GiB in all, 0.2 to 73 MiB a run coming from peers. Each client lost one download: the swarm client's reader found no node at start, the released client's gave up on a chunk after its retry rounds, as it also does on main's nodes. The build before the swarm client's last fix (ae8e997, without the waiting room join on read) had been slower in its first two 8-reader pairs (73 and 78 s against 47 and 55 s, the nodes sending 16 to 25% more) and mixed in the next four.
  • Controls on nodes built from ant-node main, which host no rooms, alternating builds:
    • The released client. 8 readers, three pairs: medians 59 to 62 s on these PRs' nodes against 59 to 98 s on main's, 23 of 24 downloads against 24 of 24 (the failure found no node at start). 16 readers, one starting every second, nine pairs: 89 of 144 downloads on these PRs' nodes against 98 of 144 on main's (5 to 15 of 16 a run), fewer in six pairs and more in three; the last five pairs, run in balanced order, gave 50 of 80 against 52 of 80. Run medians 245 to 724 s (the middle run 455 s) against 342 to 548 s (453 s), the nodes sending 2.9 to 8.5 GiB against 4.1 to 6.6 GiB. Failures were alike on both: 52 of 55 and 44 of 46 gave up on a chunk after their retry rounds, the rest found no node at start or, once, missed the file's DataMap. Also 11 and 8 of 16 on main with only these PRs' dependency branches, and 13 and 15 on main with its dependencies' unreleased main branches. Run back to back before builds were alternated, these PRs' nodes had given 7, 5, 13, 4 and 5 and main's 13 and 16. These runs vary widely and are few: the gap, one download a run of 16, is well inside their spread (exact paired test p = 0.6), but its 95% interval runs from 4.8 fewer to 2.8 more, so a loss of that size is not ruled out. Within the node's own chunk routing, the only work added for a released native client is a check that sets swarm requests aside; the compared builds also run different dependency code (among others the protocol's stricter decoding and the transport's unreleased changes), and no cause of these differences has been found.
    • The released client's own chunk fetch at fixed rates, so a slow node cannot slow the load down (8 clients, four 40 MiB files, the same 32 nodes, link and CPU share, builds alternated in balanced order, a fresh network per run): at 1 and 1.5 fetches a second, the nodes' link up to four fifths full, all 928 fetches succeeded on each build in four pairs, the median 1.01 and 0.71 s on these PRs' nodes against 1.09 and 1.03 s on main's (90th percentile 2.2 and 1.9 s against 3.2 and 2.8 s), the nodes using 0.15 and 0.11 CPU seconds per fetch on average against 0.15 and 0.12. From 1.75 a second a run either stays healthy or the client's repeated asks to further nodes fill the link and fetches give up after about 100 s, on every build. At 1.75 a second, seven runs of each: 69.9% of fetches succeeded on these PRs' nodes against 77.2% on main's (one and two runs healthy; exact permutation test p = 0.4), the median successful fetch 2.7 s against 1.9 s and the 90th percentile 10.9 s against 10.7 s; three runs each gave 73.8% on main with only these PRs' dependency branches and 77.1% on main with its dependencies' unreleased main branches; against the first, these PRs' nodes gave 69.9% pooled (p = 0.9) and 76.0% within the batch where both ran, which does not isolate the node's own changes (their locked dependencies also differ slightly). At 2 a second, eleven runs of each: 37.1% against 39.5% (p = 0.4), the median successful fetch 4.1 s against 3.6 s and the 90th percentile 15.7 s against 11.8 s; over the first 30 s of a run the median was 3.6 s against 2.8 s on average (p = 0.07). In overload the released client did slightly worse on these PRs' nodes than on main's in every pooled comparison, here and with 16 readers above, none of the differences significant. Sent straight to one of the chunk's two closest nodes instead (the released client's own GET encoding and wait, GETs started at a fixed rate whatever earlier ones do, no repeated asks, the lookups done before the timed part, four runs of each in balanced order), its GETs were served alike at 2 and 2.75 a second (the link 60 to 65% and 81 to 88% used): a run's median successful GET took 2.35 s on average on these PRs' nodes against 2.37 s on main's at both rates (95% interval of the difference -0.17 to +0.12 s), its 90th percentile 2.80 and 2.82 s against 2.82 and 2.85 s; the nodes sent 5.4 to 5.5% more than the chunks' own bytes on every run of both and used 0.079 and 0.066 CPU seconds per successful GET against 0.080 and 0.066; 48 against 49 GETs found the chunk missing at that node, and none failed otherwise (exact permutation tests p = 0.2 to 1.0). At 3.5 a second, past the link's capacity, 84.1% of these GETs succeeded on these PRs' nodes against 70.7% on main's (p = 0.09), the rest timing out (59 against 224) or finding the chunk missing (78 against 29), a run's median successful GET 3.74 s against 4.10 s (p = 0.2). The lookup each fetch starts with (the released client's own call for the 7 closest peers to the chunk) was served alike too, timed on its own at a fixed rate on the chunks' addresses, four runs of each in balanced order: at 8 a second (the link 56% used) a run's median lookup took 0.512 s on average on these PRs' nodes against 0.509 s (95% interval of the difference -0.015 to +0.021 s), and at 16 a second (94 to 99% used) 9.22 s against 9.31 s (-0.30 to +0.12 s), the 90th percentile 0.61 s on both and 16.11 against 16.12 s; the nodes sent 0.83 and 0.84 MiB and used 0.021 and 0.026 CPU seconds per lookup on both, and all 11,520 lookups were answered. So each step a node takes for the released client's fetch, the lookup and the GET, measured alike on both builds at every load tested, with no significant difference; at normal load the 95% intervals put any slowdown of a run's median at no more than 0.12 s for a GET and 0.02 s for a lookup, though they do not prove the two equal. As one-sided 95% bounds (from 90% intervals of each measure as a share of main's), no step was more than 5.1% worse on these PRs' nodes at any of the five loads, in a run's median or 90th percentile time or in CPU or bytes per successful operation. These tests run the two steps apart and do not reproduce the client's collapse. To size that collapse's own spread, main's binary was run against a second copy of itself, alternating, at 1.75 a second: 67.8% and 70.5% of fetches succeeded (six runs each, single runs 86 to 160 of 160), and the same binary had given 77.2% in the earlier batches; so a gap like 69.9% against 77.2% above also appears between identical builds, and these PRs' seven runs against all nineteen of main's binary gave 69.9% against 72.1% (p = 0.7). A loss of a few points in this overload is still not strictly ruled out.
    • The swarm client against the released one on main's nodes, 8 readers, three pairs: medians 88, 48 and 49 s against 59, 80 and 65 s, every download correct.
  • Rollout (the same lab, nodes and link, a fresh network per run, 16 readers starting one a second; the released client is client main without the swarm; the swarm client at a5cb5a2, or where marked at 19715a9, the client's commit that spaces a file download's last two retry rounds by its slowest round so far):
    • Every node from these PRs, half the readers on each client (five runs, alternated with three runs of every reader on the released client): 39 of 40 swarm-client and 38 of 40 released-client downloads finished (two of the three failures found no node at start), the swarm readers' median 184 s against 300 s and lower in every run. With every reader on the released client, 34 of 48 finished (the released readers, 38 of 40 against 34 of 48: Fisher exact p = 0.005), and the nodes sent 5.3 GiB a run on average against 3.2 GiB with half the readers on the swarm client (exact permutation test p = 0.02) and used 115 CPU seconds against 69 (p = 0.04). Two more runs at 19715a9: 16 of 16 swarm-client and 14 of 16 released-client downloads finished (both failures could not fetch the file's DataMap), median 191 s against 205 s.
    • Half the nodes still on the released v0.23.1 (16 of these PRs' nodes beside 16 of v0.23.1, half the readers on each client, five runs, alternated with four runs of today's network: 16 of main's nodes beside 16 of v0.23.1, every reader on the released client): every one of these runs collapsed, and at a5cb5a2 the swarm client did worse than the released one. Beside these PRs' nodes 4 of 40 swarm-client and 13 of 40 released-client downloads finished (p = 0.03); past the first reader of each client, 0 of 35 against 10 of 35 (p = 0.001), the ten in two runs, at 944 to 1198 s, each time after every swarm reader there had given up (at 618 to 940 s). Beside main's nodes 7 of 64 finished (every reader beside these PRs' nodes, 17 of 80: p = 0.1; their released readers, 13 of 40: p = 0.01), the nodes sending 9.1 to 16.3 GiB against 8.6 to 16.9 GiB a run. Both clients give up on a record after the same four tries. The swarm readers had most records from each other, so their rounds held only the records none of them had, took little time and put all four tries in the worst of the overload: they gave up first (median 729 s against 1014 s), and some released readers finished after every swarm reader there had stopped. Every failed swarm reader lacked a record no other swarm reader held. Five more runs of this mix at 19715a9, after the others: the swarm client finished 36 of 40 against 23 of 40 for the released client in the same runs (p = 0.002), past the first reader 33 of 35 against 21 of 35 (p = 0.001); two of its four failures were the first swarm reader, with no reader ahead of it, and the four gave up at 707 to 1015 s. The released readers did no worse beside it (23 of 40 against 13 of 40 beside a5cb5a2, a later batch, p = 0.04), the nodes sending 5.3 to 14.3 GiB a run. Load varied from run to run: one of the five never overloaded and all 16 finished (without it, 28 of 32 against 15 of 32). The readers also met later than with every node upgraded (the first record from another reader at a median of 167 s against 116 s, at a5cb5a2): a node on the old release cannot read a room request and leaves it unanswered, and a joining reader passes such a node over after about 60 s, then waits 10 s before trying the next, twice as long after each further node that never answers (up to 5 minutes). So with 19715a9, while half the nodes run the old release and the network is overloaded like this, a reader on the swarm client gave up less often than one on the released client.
    • The old nodes' logs, colour codes stripped, in the last mixed run and the last run of today's network: no panic and no ERROR line in either; 55 warnings that a message could not be decoded beside these PRs' nodes and none beside main's. Those are the swarm client's room requests, as these PRs' nodes send swarm messages only in reply. The logs kept from the earlier runs (4 to 11 of 16 a run, four runs) agree: no panic or ERROR line; 40, 37 and 0 such warnings in three runs beside these PRs' nodes and 0 in one beside main's. The five mixed runs at 19715a9: no panic or ERROR line, 0 to 279 such warnings a run.
  • CI at d39c393f: every job passes, Security Audit included: main's audit now fails only on advisories a pull request introduces, and the one vulnerability in this lockfile, RUSTSEC-2026-0387 (crypto-bigint 0.5.5, through elliptic-curve 0.13), is in main's too, whose fixes wait on alloy and webrtc-rs releases. The macOS test job failed once before running any test, when GitHub's release download of Foundry returned 504, and passed when rerun. linear-link and pr-template, which check this description, pass.

New dependency

none

ADR

https://github.com/grumbach/ant-node/blob/feat/peer-swarm/docs/adr/ADR-0026-clients-share-records-nodes-introduce-them.md (new in the ant-node PR, status Proposed)

Mitigation / rollback

Revert: clients pass over nodes that host no rooms and read from nodes alone, as today.

@MaidSafe-Hermes MaidSafe-Hermes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: Changes requested

Blocker

src/web_rtc.rs:1788–1833 — When the ordinary request-rate buckets are spent, try_admit_frame can charge a frame to the swarm buckets. read_webrtc_request then has several error exits (read_pq_payload_after_first, buffer reservations, PQ open, and parse_request_header) before it reaches the only explicit refund_swarm() call. Those exits retain both the per-source and listener-global swarm token even though the frame was not classified as a valid swarm request. A connected client can trigger this with a malformed encrypted request header or an incomplete/malformed frame. Repeated failures can consume swarm capacity needed by other connections sharing the source IP or listener until tokens refill.

Keep a swarm charge provisional: refund it on every error/cancellation path and disarm the refund only after a valid swarm request is classified. Add a regression test that exhausts the ordinary request budget, triggers a post-admission parse/frame error, and verifies both swarm buckets are restored and a subsequent valid swarm request is admitted.

Coverage: 2/2 review seats (sol-1, gpt-6.1-sol; glm-1, z-ai/glm-5.3) reviewed this targeted accounting/error-path scope at d79b7e998ba479a2b7729bc657a03c26a2c5c610; both confirmed the finding. This is not a claim that the panel covered every file in the PR.

The focused Rust test command did not complete during the review; no test result is claimed.

@grumbach
grumbach marked this pull request as draft November 5, 2026 04:12
grumbach added a commit to grumbach/ant-client that referenced this pull request Nov 5, 2026
Patch ant-protocol, saorsa-core, saorsa-transport and ant-node to their
feat/peer-swarm branches, which this change needs: the swarm messages
(WithAutonomi/ant-protocol#50), the send on one named channel
(WithAutonomi/saorsa-core#183), the plain STUN answers
(WithAutonomi/saorsa-transport#231) and the rendezvous on every node
(WithAutonomi/ant-node#262). Once those merge, these point at their
main branches, or at the released versions at the release train.

Cargo.lock starts from main's lock and differs from it only where those
branches need it: the swarm's WebRTC and STUN crates, saorsa-transport's
main dropping its CLI (and its clap and reqwest dependencies), and
ant-node's main adding winapi-util on Windows. iana-time-zone keeps the
windows-core 0.62.2 main's lock has.
Clients reading the same file share its records over WebRTC, BitTorrent
style. Every node now introduces them and stays the last seed: a record no
client has comes from the nodes as before. There is no new binary and no
new port. Design: ADR-0026.

The rendezvous (`src/swarm.rs`):
- Rooms are keyed by a file's root DataMap address. Readers announce at the
  2 closest nodes to it that host rooms, on the connections they already
  have (QUIC for native clients, WebRTC Direct for browsers).
- The node names each member from its connection, hands each offer to a
  different member, and passes an answer back only for an offer it
  delivered to the answerer, once, before it expires (the protocol's 70 s
  signal lifetime).
- An offer or answer is passed on with only the candidates another reader
  may try: no public address but its sender's own host, as the node reaches
  it on that request, no local address or mDNS name unless the node reaches
  the sender at a local address itself, and at most 16 (ant-protocol's
  `constrain_swarm_sdp`), so a stranger cannot point readers' connectivity
  checks at a third party or at their own home networks. A stored SDP is
  charged the memory it holds.
- A native membership continued over a connection from another address
  counts against that address from then on, if it has room.
- Native announces may wait up to 20 s for a signal. Browser announces never
  wait, so they hold no browser request slot.
- A node hosts rooms only once browsers can reach it: until its browser
  listener has an endpoint and a STUN server to name, it refuses announces
  as `NotHosting`, so native readers pass it over at once and meet browsers
  at the closest nodes both reach.
- Native requests are signed and checked for freshness and replay. A
  request naming another node is refused before it touches that history.
- Membership numbers let the node keep each connection's newest
  membership. Ended memberships are remembered for as long as a request sent
  before they ended can be taken, so a late request cannot end, take over
  or revive one.
- Native replies go only on the connection the request came on, through
  saorsa-core's `send_message_on_channel`.
- Hard limits with no configuration:
  - rooms, members per room and in all, and rooms per connection;
  - members, histories, ended memberships and stored SDP bytes per host
    (an IPv4 address or an IPv6 /64);
  - unanswered offers per member and waiting announces.
- The WebRTC Direct socket answers plain STUN Binding requests for hosts
  that announced in the last 60 s, rate limited in saorsa-transport.

Routing: `NativeSwarmRouter` in `src/node.rs` (shared with the devnet)
handles native swarm requests on their own task budget. `src/web_rtc.rs`
handles browser ones before the storage check, so nodes that store no
chunks host rooms too, and advertises `swarm_rendezvous`. Browser swarm
requests run on a request budget of their own (64 a second, 8 per source):
one within it gives the reads' budget back the token it took on arrival,
one arriving when the reads' budget is spent is taken on the swarm's, and
one past the swarm's budget is refused without closing the channel the
reads share, so a room's announces never spend the request budget reads
need, nor lose the channel to it.
Patch ant-protocol, saorsa-core and saorsa-transport to their
feat/peer-swarm branches, which this change needs: the swarm messages
(WithAutonomi/ant-protocol#50), the send on one named channel
(WithAutonomi/saorsa-core#183) and the plain STUN answers
(WithAutonomi/saorsa-transport#231). Once those merge, these point at
their main branches, or at the released versions at the release train.
cargo tree percent-encodes the slash in a git branch's name and cargo
metadata does not, so the notices generator matched no package on such a
branch (this PR's dependency branches are feat/peer-swarm): it now
compares both decoded.

saorsa-transport's main pulls granit-parser 2.0.0, which ships its
licence texts as 1.3.0 did; its entry names the new version.
Rust 1.99's clippy (assert_is_empty, which CI's stable toolchain runs)
asks for assert_eq when a test checks that a value is empty, so a
failure shows the value.
The pending-offer test never let B take its offers, so B's 32-signal inbox
stopped delivery whatever A's own cap on unanswered offers did. B now
collects every offer without answering, and A's 33rd offer is still refused
while B's inbox is empty.

The expiry test swept past the 70 s signal deadline, which is also past the
60 s a silent member lasts, so the members left and took their offers with
them whether or not signals expired. The members are now kept fresh across
that deadline, the offer and its SDP must still be there just before it and
gone just after it, and silent members expiring is a test of its own.

Each test fails with the limit or expiry it covers removed.
The earlier test answered an offer that did not exist, so it passed with the recipient check removed. Now A's live offer goes to B, a third member of the same room answers it with its real id and is refused, the offer stays pending, and B's answer still reaches A. It fails if any member of the room may answer.
The client now counts each of its own fetches from the nodes, not only reads of a file's data records, when it limits what it sends other readers: opening a file's map, a raw read or an existence check needs its CPU and line first too.
Browsers' swarm requests run on a budget of 64 a second per node and 8 per
source, but native clients' only had a cap on tasks running at once. A
native client can make identities for free, so one host could announce
with wait_ms 0 and fresh memberships as fast as the slots turned over, and
every replacement does its work under the rendezvous lock browsers use too.

Native swarm requests now take a place from a budget of their own with the
same figures, 64 a second in all and 8 from one host (an IPv4 address or
an IPv6 /64), before they are decoded or given a task; past it they are
dropped and counted (native_over_budget), and the client reads from nodes
alone. The node's budget is taken first, so once it is spent a flood costs
one check a request. At most 4096 hosts' budgets are remembered, those that
have refilled forgotten first, which always leaves room for a new host
since the node's own budget keeps all but a few full.

Tests: a host's identities and an IPv6 /64 share one budget, many hosts
together get the node's and no more, the hosts remembered stay bounded and
a forgotten one starts afresh, and a flood through the router from one host
is held to its budget while another host gets through. They fail with the
router's check or the per-host budget removed. ADR-0026 says so.
…or read

The client now tries a waiting room join again when the file is joined again or a read of one of its records starts, not only when a record of it comes in, so a file whose reads the nodes fail still joins once a room has gone unread.
The swarm names the browser endpoint's address as its rooms' STUN server and
hosts rooms only while it has one. Publishing a changed endpoint writes the
endpoint file first, which can fail and keep the old publication: if that was
a gateway mapping that has ended or moved, the swarm went on naming an address
the node no longer has.

Every refresh of the endpoint now ends by setting the swarm's STUN server from
its outcome: the published endpoint's address after a refresh that succeeded,
none after one that failed. A refresh that changes or withdraws the endpoint
also stops the swarm hosting before its first await (the file write, or the
DHT record's withdrawal, which can wait on peers), so the swarm never names
the old address meanwhile. This covers an old endpoint published again before
a changed one could be, and a first endpoint that cannot be published. The
refresh's former body is `publish_browser_endpoint`, unchanged but for that
clearing.

Four tests block the endpoint file: a moved mapping (the old endpoint stays
published, the swarm hosts no rooms, then names the moved mapping once the
file can be written), the old mapping coming back before the moved one is
published (the swarm names it again), a first publication, and, on Unix, a
FIFO that holds the moved mapping's write while the test checks the swarm
hosts no rooms. The first two fail if the swarm takes the published endpoint
even after a failed refresh, they and main's withdrawal test fail if it is
never updated, and the FIFO test fails without the clearing before the write.
ADR-0026 says when a node hosts rooms.
…stener

`SwarmService::ice_servers` is read only by the browser listener's tests in
src/web_rtc.rs, a module that exists only with the `webrtc-direct` feature.
Without that feature the accessor went unused, so the CI job that runs
`cargo test --lib --no-default-features` under `-D warnings` failed to
compile the tests. Gate it on `webrtc-direct` as well as `test`.

Without default features, the library's tests build with `-D warnings` and
1190 pass; with default features 1251 pass. The previous gate fails the same
build with "method `ice_servers` is never used".
The ant-protocol, saorsa-core and saorsa-transport feat/peer-swarm branches
were rebased onto their mains (ant-protocol ffbed67b, saorsa-core e04065db,
saorsa-transport 5e6ea135). Cargo.lock follows, starting from main's own
lock, so it differs from main's only where those branches need it:
saorsa-transport's main dropped its CLI, and with it its clap and reqwest
dependencies. winapi-util keeps the windows-sys 0.61.2 main's lock has.
A request frame that arrives once the reads' budget is spent is admitted on
the swarm's own budget, since only the opened frame says whether it is a
swarm request. The charge was given back only when the opened frame turned
out to be some other request. Every earlier exit kept it: a frame that
failed to arrive (bytes after its declared end, the channel closing
partway), that did not open under the PQ session, whose header did not
parse, or whose byte reservation was refused, and a read the connection
abandoned partway through a frame. A connected browser could so spend its
source's and the listener's swarm tokens with malformed frames, and hold
them from the swarm requests of other connections until they refilled.

The charge is now provisional: a guard taken at admission gives it back
when dropped, on every exit and on cancellation, and only a frame known
for a swarm request keeps it. The request reader takes its messages through
a small trait, implemented for the DataChannel, so tests can feed it frames.

Tests exhaust the reads' budget, then send twice the source's swarm budget
of each kind of failing frame, with a real PQ session for those that open,
and check that both swarm buckets are full after each and that a swarm
request is still admitted on the swarm's budget and keeps its charge there.
They fail with the refund removed.
A browser download asked for a file's records in an order picked at
random for that download. Two downloads of one file in the same client
then asked in different orders, so they no longer overlapped record by
record and shared fewer fetches, and main's test that a joined fetch
charges both downloads its GETs could fail. The client now picks one
order per client (ant-client WithAutonomi#244): readers in different clients still
ask in different orders, which is what the order is for, while one
client's downloads of a file keep sharing each record's fetch. The ADR
says so.
grumbach added a commit to grumbach/ant-client that referenced this pull request Nov 5, 2026
Patch ant-protocol, saorsa-core, saorsa-transport and ant-node to their
feat/peer-swarm branches, which this change needs: the swarm messages
(WithAutonomi/ant-protocol#50), the send on one named channel
(WithAutonomi/saorsa-core#183), the plain STUN answers
(WithAutonomi/saorsa-transport#231) and the rendezvous on every node
(WithAutonomi/ant-node#262). Once those merge, these point at their
main branches, or at the released versions at the release train.

Cargo.lock starts from main's lock and differs from it only where those
branches need it: the swarm's WebRTC and STUN crates, saorsa-transport's
main dropping its CLI (and its clap and reqwest dependencies), and
ant-node's main adding winapi-util on Windows. iana-time-zone keeps the
windows-core 0.62.2 main's lock has.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants