[mickvandijke]

Linear issue

Closes V2-1833

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: none. Hint messages are unchanged, and each peer gets the same hints a lookup per key gives.
• Storage: none.
• API: removes replication::config::PAID_LIST_NEAR_MARGIN_BITS, replication::config::PAID_HINT_FULL_SCAN_ENTRIES, paid_list::near_bits_for_radius and PaidList::keys_near. Adds replication::config::PAID_LIST_NEAR_GROUPS and paid_list::near_bits_for_peer_distances. The removed items came in with feat(replication)!: erase out-of-range records only under disk pressure #271, so the only release they have shipped in is 0.24.0-rc.1. Everything else added is crate-private. The Neighbor-sync hint build log line reports candidates=, stored_keys_read= and paid_keys_read= instead of stored_keys= and paid_keys=, and inbound syncs are now logged too.

Semver impact

• breaking
• feature
• fix

Test evidence

Run locally on this branch:

• cargo fmt --all -- --check: clean.
• cargo clippy --all-features -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used: clean.
• cargo doc --all-features --no-deps: clean.
• cargo check --lib --no-default-features --locked: clean.
• cargo test --lib --features test-utils: 1,265 passed.
• cargo test --test e2e --features test-utils -- --test-threads=1 paid_ prun pointer out_of_range_records_stay audit hint_builds neighbor sync replica: 67 passed, and -- hint late_joiner scenario_26: 8 passed, including verify_storage_gate's two hint tests.
• cargo clippy --all-targets --all-features -- -D warnings shows only the six findings already on the base branch, none in code this PR changes.

The close-group map gives the same groups as sorting every candidate by distance:

• Unit tests compare it, in order, for every key tried, with 0 to 250 candidates and groups of 1 to 20: IDs evenly spread, clustered beside one candidate, or all sharing their top bits. Keys tried include both ends of the key space, each candidate's ID and its neighbours one bit away, and 400 random keys per case. They also check that the ranges cover every key once, in order.
• A unit test checks that a map of the retention width gives a prune pass the same retention and strict groups, in the same order, as sorting every candidate.
• hint_builds_match_a_lookup_per_key (e2e, 25 nodes): for every peer in a node's routing table, the paid hints over 400 paid entries and the replica hints over 150 stored records equal what a lookup per key gives.

Each check was confirmed by breaking the code:

• Swapping which side of a bit is closer fails the map's unit tests and both hint e2e tests.
• Ignoring the per-key candidates when choosing ranges to read fails the worked-example unit test and both hint e2e tests.
• Leaving this node out of the candidates fails hint_builds_match_a_lookup_per_key.
• Hinting every key read, without checking that key's group, fails only_keys_whose_group_holds_a_target_are_hinted. The 25-node e2e test didn't catch this one, so that unit test was added for it.
• Mapping the prune pass at the strict width of 7 instead of the retention width of 9, or hint admission at the record group of 7 instead of the paid group of 20, passed every existing test. Unit tests pinning both widths were added, with this node ranked just inside each width, and each now fails.
• Removing the range limit fails a_skewed_set_of_candidates_stays_within_the_range_limit, which uses zero plus one ID per high bit for 30 IDs: unlimited, that splits into C(n, G) ranges, about 2 million for a group of 7 and 30 million for 20.
• Leaving the map's groups unordered fails the map's unit tests, the prune-group unit test and test_prune_veto_for_committed_out_of_range_key.
• Admission rejecting every key it doesn't already know fails scenario_26_paid_list_majority_repairs_missing_replica_below_storage_quorum, test_late_joiner_replicates_responsible_chunks and hint_label_does_not_change_what_gets_stored.
• From the first commit: removing the "too few peers" check fails two PaidList unit tests, and measuring the near range from the closest counted peer fails a third.

Hint-build cost, release build, Apple M3 Ultra, 300-node local network (the node knew 91 peers), one node building for its nearest neighbour:

20,000 keys, five runs	Lookup per key (before)	Range map (after)	Hints, identical in every run
Paid list, group of 20	265–272 ms (13.2–13.6 µs a key)	0.89–0.94 ms	1,548
Record store, group of 7	179–195 ms (8.9–9.8 µs a key)	0.34–0.37 ms	620

These keys were spread over the whole key space, so the neighbour's ranges held under a tenth of them. A real node's keys cluster around it, so a build reads a larger share of its keys. Including the copy of the routing table and building the map, the new path cost under 0.6 µs per key hinted here, so even a build that had to read every key should stay several times cheaper than before. That last part is an estimate, not a measurement.

Typical node hardware is slower, roughly in proportion to its single-core speed, but the ratio should hold.

Not run on a dev testnet. The hints are the same as before by construction and by the tests above, so what a testnet would add is the cost at production table sizes, which Neighbor-sync hint build ... elapsed_ms= lines will show in the RC's testing.

New dependency

none

ADR

ADR-0028, updated here. docs/REPLICATION_DESIGN.md §6.2 rules 8 and 16 are updated too.

Mitigation / rollback

Revert this PR. Nothing stored or sent on the wire changes, so nodes can move either way at any time.

Summary

A follow-up to #271 for rc-2026.11.1. It changes how sync hints are built, and how a node decides that a paid entry is near it.

• Sync hints from close groups per key range: a hint build used to look up every stored record and every paid entry it read in the routing table, one key at a time. Each lookup clones and sorts the whole table, and a build reads every stored record, including the out-of-range ones feat(replication)!: erase out-of-range records only under disk pressure #271 now keeps. A build now takes one copy of the routing table and works out each key's close group once per key range (src/replication/close_group_map.rs). It reads only the ranges where the peer it builds for can be in the group, from the record store and the paid list, which are both kept sorted by key. Each key gets the group a lookup for it would give, so hints don't change.
• Paid hints for keys anywhere: because each key's group is worked out exactly, a peer gets the paid hint for a key however far the key is from this node. This replaces both feat(replication)!: erase out-of-range records only under disk pressure #271's 100,000-entry limit and the near-range read in this PR's first commit (review thread).
• Inbound syncs use the same batch builder as outbound ones, so they take one routing-table copy instead of two, and are logged.
• Prune passes, hint admission and commitment rotation use the same map in place of a lookup per key. A prune pass and a rotation check every stored record, including the out-of-range ones feat(replication)!: erase out-of-range records only under disk pressure #271 now keeps, and admission checks every hinted key this node doesn't already know. The check just before an erasure still does a fresh lookup, since the routing table may have changed while the proofs were gathered.
• Bounds: a map stops splitting after 1,024 ranges (MAX_SPLIT_RANGES), where routing tables gave 13 to 202; a skewed set of peer IDs could otherwise need millions. Past that, each remaining range compares its competing peers for each key, which gives the same groups at more cost a key. The loops over keys let other tasks run in between (consume_budget), as the per-key lookups' awaits did, and a sync request from a peer outside the routing table returns before any map is built.
• Near range, now for eviction only: it reaches the closest routing-table peer three paid close groups out with no margin (the 60th with the default group of 20), instead of the paid group's farthest peer with two extra bits. That covers about the same number of nodes per key (about 121 against 116, simulated on 4,000 nodes), and reads local peer density from more peers. A node knowing fewer than 60 peers counts every key as near and evicts nothing. With evenly spread IDs, a simulation of 500 to 10,000 nodes found no key outside the new range that this node or one of its 20 closest neighbours is among the closest 20 or 40 for, in about 48,000 sampled keys.
• ADR-0028's lookup cost: the figure of about 100 µs per lookup was extrapolated from test builds, which compile at opt-level 1. It's replaced with a release-build measurement.

🤖 Generated with Claude Code

[MaidSafe-Hermes]

Review — Changes requested

Blocker: unbounded map construction can exhaust CPU and memory. CloseGroupMap::new recursively splits key space in Walk::split without a work or range limit. For 30 candidate IDs consisting of zero plus one ID with each of 29 distinct high bits set, and group_size = 20, the branches recur as F(n,k)=F(n-1,k)+F(n-1,k-1), producing C(30,20) = 30,045,015 leaf ranges. Each leaf is stored as a GroupRange. The map is built from routing-table peers during hint builds, admission, and pruning, so a pathological candidate set can turn those paths into an allocation/CPU denial of service. Add a bounded split/range budget and fall back to a range with the remaining open candidates for per-key evaluation; add a regression test for the skewed candidate pattern.

Panel coverage: 2/2 seats reviewed both evidence scopes (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3). They disagreed on the map's worst-case complexity: sol-1 identified the recurrence above; glm-1 assessed the recursion as O(n). I independently checked the split paths and recurrence; the O(n) assessment does not account for the same candidate subsets recurring with different need values.

Checks: PR CI is green except Security Audit, which reports crypto-bigint 0.5.5 / RUSTSEC-2026-0387. Cargo.lock is byte-identical to the base, so this audit failure is inherited, not introduced here. The targeted local Rust test did not finish: dependency compilation exceeded the command time limit. No dev testnet run is reported.

Reviewed head: 7b689c6e2a548fc7fd5bb9d4352d1aa40229d5a6. This review does not authorise a merge.

[MaidSafe-Hermes]

High — bound this recursive construction. With candidate IDs 0 and one-hot IDs at each of 29 successive high bits, each split creates (a) the large remaining subset at need=k and (b) the same subset at need=k-1. At 30 candidates and group width 20 this gives C(30,20) = 30,045,015 leaves, all materialised as GroupRanges. Since routing peers feed this constructor on network-driven hint/admission/prune paths, a skewed set can cause severe CPU and memory use. Add a global split/range budget; once reached, emit the current prefix as one range retaining open and need so group_in does exact per-key selection. Add a regression test that asserts bounded range count and exact groups for this pattern.

[mickvandijke]

Fixed in ca44084b. Your recurrence was right: unlimited, that pattern splits into exactly C(n, G) ranges. I measured 10.4 million ranges, built in 17 s, at 26 IDs and a group of 13.

• The limit: a map now stops splitting after 1,024 ranges (MAX_SPLIT_RANGES). The routing tables I measured gave 13 to 202.
• Past the limit: each prefix still pending on the walk becomes one range that keeps its open candidates and need, and group_in picks the closest of them for each key. Groups stay exact at a higher cost per key, and a map holds at most 1,024 plus one range per bit of depth.
• Regression test: a_skewed_set_of_candidates_stays_within_the_range_limit uses your pattern of zero plus one-hot IDs at 29 successive high bits, with groups of 1, 2, 7 and 20. It asserts the bound and checks every probed key's group against a full sort. With the limit removed, it fails at the group of 7, which splits into about 2 million ranges.

The same commit makes the loops over keys in hint builds, prune passes, admission and commitment rotation call consume_budget, so a large store doesn't hold a runtime worker. Commitment rotation now uses one map too.