fix(replication): build sync hints from close groups per key range, and widen the paid near range

docs/REPLICATION_DESIGN.md

@@ -115,7 +115,7 @@ Parameter safety constraints (MUST hold):
     - Under disk pressure (ADR-0028), audit scheduling orders eligible candidates farthest from self first (candidates whose proofs failed last, oldest failure first), groups a bounded window of them by target peer and sends multi-key requests, split at `max(floor(sqrt(local_in_range_keys)), 1)` keys per request, sized from the records self is in range for so retained records never make a batch too large for a peer to accept. Without pressure, no prune-confirmation audit is sent. The pass is bounded by actual requests and complete candidate proof sets, not candidate-to-peer edges; no partial candidate audit is carried across passes. A responder's size rejection is not renegotiated in-band: since its acceptance limit already carries a 2x margin over the sender size, an honestly-sized peer rejects only once close-group store spread exceeds 4x, so under the assumed ~2x spread a size rejection is graded as an attributable failure (distinct `size_reject` label), never a possession signal.
     - Immediately before audited deletion, revalidate against the current routing table: the key must still be outside the retention width, must not be held by an answerable retained commitment, and the positive reports must still satisfy the CURRENT strict close group (membership churn invalidates stale reports).
     - Prune-confirmation failures emit trust penalties only after fresh responsibility confirmation; first-time bootstrap claims use the same one-time `BOOTSTRAP_CLAIM_GRACE_PERIOD` tracking before abuse penalties apply.
-    `PaidForList` entries stay at or below `paid_list_evict_start_entries` (about 85% of the store's capacity, ADR-0028), whatever the disk. Above it, a pass evicts entries down to the threshold, locally and with no confirmations: those farthest from self first, never those near self (within four to eight times the distance to self's farthest paid-close-group peer) or those whose record self still stores.
+    `PaidForList` entries stay at or below `paid_list_evict_start_entries` (about 85% of the store's capacity, ADR-0028), whatever the disk. Above it, a pass evicts entries down to the threshold, locally and with no confirmations: those farthest from self first, never those near self (within the distance to self's closest peer three paid close groups out, `PAID_LIST_NEAR_GROUPS`) or those whose record self still stores.
 23. Peers claiming bootstrap status are skipped for sync and audit without penalty for up to `BOOTSTRAP_CLAIM_GRACE_PERIOD` from first observation. After the grace period, each continued bootstrap claim emits `BootstrapClaimAbuse` evidence to `TrustEngine` (via `report_trust_event` with `ApplicationFailure(weight)`). If a peer stops claiming bootstrap and later claims bootstrap again, the repeated claim emits `BootstrapClaimAbuse` immediately; the grace period is not reset.
 24. Audit trust-penalty signals require responsibility confirmation: on audit failure, challenger MUST perform fresh local RT closest-peer lookups for each challenged key and only penalize the peer for keys where it is confirmed responsible.
 
@@ -160,14 +160,15 @@ Rules:
 8. In each session, sender-side hint construction uses peer-targeted sets:
     - `ReplicaHintsForPeer`: keys the sender believes the receiver should hold (receiver is among the `CLOSE_GROUP_SIZE` nearest to `K` in sender's `SelfInclusiveRT`).
     - `PaidHintsForPeer`: keys the sender believes the receiver should track in `PaidForList` (receiver is among the `PAID_LIST_CLOSE_GROUP_SIZE` nearest to `K` in sender's `SelfInclusiveRT`).
+    - A sender MAY evaluate both from one `SelfInclusiveRT` snapshot per build, once per key range rather than once per key, reading only the ranges where the receiver can be in the group (ADR-0028). The sets MUST be the ones a per-key evaluation over that snapshot gives.
 9. Transport-level chunking/fragmentation is implementation detail and out of scope for replication logic.
 10. Receiver treats hint sets as unordered collections and deduplicates repeated keys. If a key appears in both `ReplicaHintsForPeer` and `PaidHintsForPeer` in the same session, receiver MUST keep only the replica-hint entry and drop the paid-hint duplicate (single-pipeline processing).
 11. Receiver diffs replica hints against local store and pending sets, then runs per-key admission rules before quorum logic.
 12. Receiver launches quorum checks exactly once per admitted unknown replica key.
 13. Only admitted unknown replica keys that pass presence quorum or paid-list authorization are queued for fetch.
 14. Receiver processes unknown paid hints via Section 7.2 majority checks in a paid-list pipeline: successful checks may update `PaidForList(self)`. If the receiver is also storage-responsible for `K`, successful verification may queue record fetch using verified `Present` sources from the same round. If the same key is also present in replica hints, rule 10 drops the paid-hint duplicate and fetch behavior is governed by the replica-hint pipeline.
 15. Sync payloads MUST NOT include PoP material; PoP remains fresh-replication-only.
-16. Nodes SHOULD use ongoing neighbor sync rounds to re-announce paid hints for locally paid keys to improve paid-list convergence. Above `PAID_HINT_FULL_SCAN_ENTRIES` entries (ADR-0028), a sender reads only the entries near self for these hints, and can miss a key far from self.
+16. Nodes SHOULD use ongoing neighbor sync rounds to re-announce paid hints for locally paid keys to improve paid-list convergence. A sender includes every local paid key whose `PaidCloseGroup` holds the receiver, however far the key is from self.
 17. After each round, node sets `NeighborSyncCursor(self)` to the position after the last scanned peer in the (possibly shrunk) snapshot. Peers removed during scanning (cooldown or unreachable) do not occupy cursor positions — the cursor reflects the snapshot's state after removals. Priority syncs do not advance the cursor unless the peer was also removed from the snapshot.
 18. When `PriorityNeighborSyncOrder(self)` is empty and `NeighborSyncCursor(self) >= |NeighborSyncOrder(self)|`, the cycle is complete (`NeighborSyncCycleComplete(self)`). Node MUST execute post-cycle responsibility pruning (Section 11), then recompute `CloseNeighbors(self)` from current `LocalRT(self)`, take a fresh snapshot, and reset the cursor to `0` to begin the next cycle.
 

docs/adr/ADR-0028-pressure-gated-retention.md

@@ -177,16 +177,42 @@ store shares that partition.
   - **Above it:** each prune pass evicts down to the threshold, at most 100,000 entries,
     farthest from this node first. Eviction is local, with no confirmations.
   - **What stays even then:** entries near this node, and entries of records it still
-    stores. Near means within four to eight times the distance to its farthest paid-group
-    peer, where it or a neighbour may still have a vote. While it knows too few peers to
-    say, every entry counts as near.
-  - **Sync hints:** read every entry up to 100,000 entries, and only the entries near this
-    node above that, so their cost stops growing with the list. Each entry read costs a
-    routing-table lookup: measured at 4.7 µs with 4 peers and 14.5 µs with 24, which
-    extrapolates to about 100 µs with 200. A node builds hints every round and for every
-    inbound sync. Where few peers are known near a key far from this node, this node or a
-    neighbour can still be in its paid close group, so above that size such a hint can be
-    missed. With evenly spread IDs, a simulation of 100 to 2,000 nodes found none.
+    stores. A key is near when its distance has no more significant bits than the
+    distance to this node's closest routing-table peer three paid close groups out
+    (`PAID_LIST_NEAR_GROUPS`): the 60th with the default group of 20. With evenly spread
+    IDs and at most 20 peers per ring, about 120 nodes count a given key as near, from a
+    simulation of 4,000 nodes. While a node knows fewer peers than that, every entry
+    counts as near.
+  - **Sync hints:** each build takes one copy of the routing table and works out each
+    key's close group once per key range, rather than once per key
+    (`src/replication/close_group_map.rs`). It then reads only the ranges where a peer it
+    builds for can be in the group, from the record store and the paid list, which are
+    both kept sorted by key. Each key gets the group a lookup for it would give, so a peer
+    gets the hint for a key however far the key is from this node. The near range plays no
+    part in hints.
+    - **Why:** before this, every stored record and every paid entry read cost a local
+      routing-table lookup, in every build. In a release build on an Apple M3 Ultra core,
+      with 101 to 107 peers known, one took 10–11 µs for a group of 7 and 15–16 µs for a
+      group of 20, so 150,000 took about 2 s. Typical node hardware is slower, roughly in
+      proportion to its single-core speed. A node builds hints every round and for every
+      inbound sync, a neighbour waits 15 s for an answer, and the records and paid entries
+      this ADR keeps add to every build.
+    - **Measured:** in a release build on the same machine, on a 300-node local network
+      where the node knew 91 peers, building hints for its nearest neighbour over 20,000
+      paid entries took 0.9 ms instead of 265–272 ms, and over 20,000 stored records
+      0.35 ms instead of 179–195 ms, with the same hints. Those keys were spread over the
+      whole key space, so the neighbour's ranges held under a tenth of them. A real node's
+      keys cluster around it, so a build reads a larger share, at well under 1 µs a key.
+  - **Prune passes, hint admission and commitment rotation** use the same map in place of
+    a lookup per key: a prune pass and a rotation check every stored record, including the
+    out-of-range ones this ADR keeps, and admission checks every hinted key this node
+    doesn't already know. The check just before an erasure still does a fresh lookup,
+    since the routing table may have changed while the proofs were gathered.
+  - **Bounds:** a map stops splitting after 1,024 ranges, where routing tables gave 13 to
+    202. A skewed set of peer IDs could otherwise need millions. Past that, each remaining
+    range compares its competing peers for each key, which gives the same groups at more
+    cost a key. These loops let other tasks run between keys, as the per-key lookups'
+    awaits did.
   - **Store size:** the LMDB map grows from 256 MiB to 1 GiB, which holds about 17.6
     million entries, measured at about 61 bytes each with single-key commits. The file
     grows as entries do, on every platform. A full store logs one error, since new paid
@@ -199,6 +225,8 @@ store shares that partition.
 - `src/replication/retention.rs`: the pressure state, thresholds and erase order.
 - `src/replication/out_of_range_clocks.rs`: saving and restoring the clocks.
 - `ChunkStore::disk_space`: the measurement.
+- `src/replication/close_group_map.rs`: close groups per key range, for hint builds,
+  prune passes and hint admission.
 
 ### Why this holds
 
@@ -260,13 +288,16 @@ store shares that partition.
 ### Neutral / Operational
 
 - **The prune log line gains `disk_pressure=` and `retained=`.**
+- **The hint-build log line** reports `candidates=`, `stored_keys_read=` and
+  `paid_keys_read=` instead of `stored_keys=` and `paid_keys=`, and inbound syncs are
+  logged too.
 - **Public Rust API breaks:**
   - `PrunePassContext` gains `disk_pressure`.
   - `ReplicationConfig` gains `paid_list_evict_start_entries`.
   - `PruneResult`'s paid counters become `paid_entries_evicted`, and the prune log line's
     paid part becomes `paid total= evicted=`.
-  - `PaidList` loses its paid timers, its prune cursor and `all_keys`, and gains `keys_near`
-    and `evict_far`. Its record timestamps are `SystemTime`s.
+  - `PaidList` loses its paid timers, its prune cursor and `all_keys`, and gains
+    `evict_far`. Its record timestamps are `SystemTime`s.
   - `PointerReplication::prune_pass` takes the pressure state.
   - `NeighborSyncState::prune_cursor` is removed.
   - PR #266's removals are included.
@@ -288,15 +319,28 @@ store shares that partition.
   - record clocks surviving a reopen of the paid list;
   - a clock surviving "in range" during bootstrap;
   - paid-list eviction: farthest first, up to its limit, never what it keeps, never near;
-  - near ranges, and reading only the keys in one;
-  - a far key read for hints while the list can be read whole, and only near keys above that;
+  - near ranges reaching the counted peer, and every key near until enough peers are
+    known;
+  - a far paid entry kept until the node knows 60 peers;
+  - close-group maps matching, in order, a sort of every candidate by distance, for 0 to
+    250 candidates and groups of 1 to 20, with IDs evenly spread, clustered beside one
+    candidate, or sharing their top bits, and ranges covering every key once;
+  - a skewed set of IDs that would split into millions of ranges staying within the limit,
+    with the same groups;
+  - only keys whose group holds a peer being hinted;
+  - a map of the retention width giving a prune pass the retention and strict groups, in
+    the order a lookup gives them;
+  - reading key ranges from the record store and the paid list;
   - the store's file not taking the whole map up front;
   - removing a paid entry leaving its record's clock;
   - a pointer needing all but one of its group to prove it.
 - **End-to-end tests:**
   - `out_of_range_records_stay_until_the_disk_needs_the_room`: with room, a record the
     whole group holds is kept and its group never asked; under pressure it goes.
-  - `a_neighbour_gets_a_paid_hint_for_a_key_far_from_this_node`: a small list is read whole.
+  - `a_neighbour_gets_a_paid_hint_for_a_key_far_from_this_node`: a peer far from a node
+    still gets the hint for a key it is closest to.
+  - `hint_builds_match_a_lookup_per_key`: on 25 nodes, every peer gets exactly the paid
+    and replica hints a lookup per key gives it.
   - `paid_entries_stay_below_the_threshold_and_near_this_node`: disk pressure evicts no
     paid entry, and an entry near this node stays over the threshold.
   - The existing chunk prune tests run under a simulated nearly full disk.
@@ -308,7 +352,8 @@ store shares that partition.
   - it stops at 15% free;
   - below the threshold, `paid ... evicted=0` holds on every node;
   - with a test build's low threshold, only far entries without a stored record go;
-  - hint-build times stay flat as `paid total=` grows past 100,000;
+  - hint builds stay in milliseconds as `paid total=` and retained records grow, from
+    `Neighbor-sync hint build ... elapsed_ms=`;
   - how fast a node under pressure erases, against how fast it receives in-range data;
   - after restarting a node under pressure, `candidates=` doesn't fall to 0 once bootstrap
     has drained;

src/replication/admission.rs

@@ -13,8 +13,10 @@ use std::sync::Arc;
 
 use saorsa_core::identity::PeerId;
 use saorsa_core::P2PNode;
+use tokio::task::coop::consume_budget;
 
 use crate::ant_protocol::XorName;
+use crate::replication::close_group_map::{close_group_candidates, CloseGroupMap};
 use crate::replication::config::ReplicationConfig;
 use crate::replication::paid_list::PaidList;
 use crate::storage::ChunkStore;
@@ -77,22 +79,26 @@ pub async fn is_in_paid_close_group(
 /// Storage responsibility is a *different* question, asked later against live
 /// routing state at the point of download. Keeping the two apart is what makes
 /// the sender's replica/paid labelling unable to influence what we store.
-async fn is_relevant(
+fn is_relevant(
     self_id: &PeerId,
     key: &XorName,
-    p2p_node: &Arc<P2PNode>,
-    config: &ReplicationConfig,
-    storage: &Arc<ChunkStore>,
-    paid_list: &Arc<PaidList>,
+    paid_groups: &CloseGroupMap,
+    storage: &ChunkStore,
+    paid_list: &PaidList,
     pending_keys: &HashSet<XorName>,
 ) -> bool {
     // Fast paths: we already hold the key, already track it, or already know it
     // is paid for. Each means the relevance question is settled -- no
-    // routing-table lookup needed.
+    // close-group check needed.
     storage.exists(key).unwrap_or(false)
         || pending_keys.contains(key)
         || paid_list.contains(key).unwrap_or(false)
-        || is_in_paid_close_group(self_id, key, p2p_node, config.paid_list_close_group_size).await
+        || paid_groups.includes(key, self_id)
+}
+
+/// Every key's `PaidCloseGroup`, for a whole batch of hints.
+fn paid_group_map(candidates: Vec<PeerId>, config: &ReplicationConfig) -> CloseGroupMap {
+    CloseGroupMap::new(candidates, config.paid_list_close_group_size)
 }
 
 /// Admit neighbor-sync hints per Section 7.1 rules.
@@ -122,23 +128,26 @@ pub async fn admit_hints(
         paid_only_keys: Vec::new(),
         rejected_keys: Vec::new(),
     };
+    if replica_hints.is_empty() && paid_hints.is_empty() {
+        return result;
+    }
+    // One routing-table snapshot for the whole batch, instead of a lookup per key.
+    let paid_groups = paid_group_map(close_group_candidates(p2p_node).await, config);
 
     let mut seen_replica = HashSet::new();
     for &key in replica_hints {
         if !seen_replica.insert(key) {
             continue;
         }
+        consume_budget().await;
         if is_relevant(
             self_id,
             &key,
-            p2p_node,
-            config,
+            &paid_groups,
             storage,
             paid_list,
             pending_keys,
-        )
-        .await
-        {
+        ) {
             result.replica_keys.push(key);
         } else {
             result.rejected_keys.push(key);
@@ -156,17 +165,15 @@ pub async fn admit_hints(
         if seen_replica.contains(&key) {
             continue;
         }
+        consume_budget().await;
         if is_relevant(
             self_id,
             &key,
-            p2p_node,
-            config,
+            &paid_groups,
             storage,
             paid_list,
             pending_keys,
-        )
-        .await
-        {
+        ) {
             result.paid_only_keys.push(key);
         } else {
             result.rejected_keys.push(key);
@@ -199,6 +206,18 @@ mod tests {
         [b; 32]
     }
 
+    #[test]
+    fn a_node_ranked_just_inside_the_paid_close_group_finds_a_key_relevant() {
+        const CANDIDATES: u8 = 30;
+        let config = ReplicationConfig::default();
+        // A key of zeroes, so each candidate's rank is its first byte.
+        let candidates: Vec<PeerId> = (1..=CANDIDATES).map(peer_id_from_byte).collect();
+        let rank = |r: usize| peer_id_from_byte(u8::try_from(r).expect("rank fits"));
+        let map = paid_group_map(candidates, &config);
+        assert!(map.includes(&[0u8; 32], &rank(config.paid_list_close_group_size)));
+        assert!(!map.includes(&[0u8; 32], &rank(config.paid_list_close_group_size + 1)));
+    }
+
     // -----------------------------------------------------------------------
     // AdmissionResult construction helpers for pure-logic tests
     //