From 1e8eb94e7e71c0e6653334b306a687347a4ea91e Mon Sep 17 00:00:00 2001 From: grumbach Date: Wed, 21 Oct 2026 12:09:21 +0900 Subject: [PATCH 01/22] feat(payment)!: enforce the close-group price floor on every node The ADR-0006 floor has run in shadow mode since v0.17.1. A week of production telemetry (2026-10-14..20, 699 node instances on 72 hosts) showed that its 65% default would have refused 2.5% of admissions, and 2.1% of chunks at every receiver that saw them. Neighbours gossiping inflated commitments could refuse a chunk everywhere. One accepting node could still replicate a below-floor write through PaidNotify and paid-list repair. And a peer that refused a write on price would be penalised for it by possession checks, responsible audits and prune audits. This makes enforcement safe and turns it on for every node, with nothing to configure. - Cap the reference at the receiver's own price: min(group median, calculate_price(own committed count)). A node never demands more than the tolerance of what it would itself quote, so lying neighbours cannot make it refuse a payment that clears its own price. - Lower the tolerance from 65% to 30%, a deliberate starting point. No measured admission is refused at 30% or at 40%, but the worst clears 30% by 1.78x and 40% by only 1.34x, on a week of 48 upload bursts. A baseline quote is still refused by 91% of receivers (98% at 40%). - Floor paid-list admission too. PaidNotify only announces a client's write, with the proof the client presented, so this reprices no historical receipt. It stops one accepting node from giving a below-floor write the paid-list votes that repair needs. - Judge no peer on a contested write. A payment is contested when the node cannot price it, or its paid price is below 60% of the uncapped group median (twice the floor). The verifier remembers each contested chunk payment it admits, in a memory of its own, and every record path (PUT, fresh offer, PaidNotify) marks the key's paid-list entry from that decision, never from the unchecked proof a cache hit carries; the first paid-list record of a key decides. A marked key goes out in paid hints only, so no repair proof is recorded, no responsible audit samples it and no peer penalises this node as its sole hinter; the paid list keeps its marked keys in memory too, so building hints scans nothing on disk for them. A prune audit still needs its proofs but scores no absent peer; paid-list pruning keeps the marked entry while the record is held; and a mark that cannot be read counts as set, and is logged. A chunk fan-out schedules no possession check when the write is marked (by the verified decision or by the record that won, read after its own is written) or the proof it forwards is contested, and a pointer fan-out none when its proof is contested. Releases before this one only test paid-list presence, so they read a marked key as paid. The 2x margin covers the largest spread measured between receivers' views (1.91x); it is a margin, not a guarantee. - One floor for every node. PriceFloorConfig and PaymentVerifierConfig::price_floor are removed: the tolerance is the constant PRICE_FLOOR_TOLERANCE_PERCENT and the contest threshold a private one. ANT_PRICE_FLOOR_ENFORCE and ANT_PRICE_FLOOR_TOLERANCE_PERCENT are no longer read, and backing the floor out takes a release. - The telemetry line adds target, group_median_price, own_price, paid_quote_age_secs and contested, and keeps enforce=true constant. - A comment on the group sample says why it needs no self-filter: the routing scan includes this node, but the gossip cache never holds it, since ingest accepts only senders in the routing table, so its own commitment joins the sample exactly once. - Repair tests/poc_price_floor_live.rs, which no longer exercised the close-group floor and failed on main. Pin the chunk and pointer possession schedulers, the mark and hints, the prune audit and all three record paths with e2e tests, each with a control. Add an env-gated A/B workload driver, tests/e2e/price_floor_ab.rs. ADR-0006 gains Amendment 2, recording the measurement, these decisions and the residuals. ADR-0008's status row is updated. BREAKING CHANGE: PriceFloorConfig (with from_env, enforce and tolerance_percent), PaymentVerifierConfig::price_floor, PRICE_FLOOR_ENFORCE_ENV and PRICE_FLOOR_TOLERANCE_ENV are removed, and PRICE_FLOOR_TOLERANCE_PERCENT is added. The floor is enforced at 30% on every node. neighbor_sync::build_replica_hints_for_peer and audit::audit_tick_with_repair_proofs take the paid list, and ReplicationEngine::replicate_fresh is only built with test-utils. --- .../ADR-0006-receiver-side-revenue-floor.md | 486 +++++- ...-storage-economics-and-payment-protocol.md | 6 +- src/devnet.rs | 3 +- src/node.rs | 13 +- src/payment/mod.rs | 5 +- src/payment/verifier.rs | 1509 +++++++++++------ src/pointer/service.rs | 3 +- src/replication/audit.rs | 55 +- src/replication/fresh.rs | 134 +- src/replication/mod.rs | 42 +- src/replication/neighbor_sync.rs | 80 +- src/replication/paid_list.rs | 531 +++++- src/replication/pointer.rs | 15 + src/replication/pruning.rs | 141 +- src/storage/handler.rs | 7 +- tests/e2e/data_types/chunk.rs | 5 +- tests/e2e/harness.rs | 35 +- tests/e2e/mod.rs | 3 + tests/e2e/pointer_replication.rs | 95 ++ tests/e2e/price_floor_ab.rs | 743 ++++++++ tests/e2e/replication.rs | 433 ++++- tests/e2e/testnet.rs | 1 - tests/poc_price_floor_live.rs | 235 +-- 23 files changed, 3796 insertions(+), 784 deletions(-) create mode 100644 tests/e2e/price_floor_ab.rs diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 4dc93035..c7194743 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -1,7 +1,8 @@ # ADR-0006: Close-group pricing for the receiver-side revenue floor - **Status:** Proposed -- **Date:** 2026-07-14 (amended 2026-08-05 after canary measurement) +- **Date:** 2026-07-14 (amended 2026-08-05 after canary measurement; Amendment 2 + 2026-10-20: enforced on every node) - **Decision owners:** Anselme (@grumbach) - **Reviewers:** - **Supersedes:** none (amends the economic scope of ADR-0004; see Decision) @@ -129,6 +130,13 @@ prospective rather than observed. ## Decision +> **Amended.** [Amendment 2](#amendment-2-2026-10-20-enforced-on-every-node) +> changes four things below: the tolerance (65% → 30%), the reference (now +> capped at the receiver's own price), the scope (paid-list admission is +> floored too) and the rollout (enforced on every node, with nothing to +> configure). Where the two disagree, the amendment wins. The text here is kept +> as decided, so the reasons for each step stay readable. + On single-node store admissions (direct client PUT and immediate fresh replication), the receiver additionally requires, alongside the existing `settled >= 3 x median` check: @@ -260,6 +268,482 @@ payment when it settles below its close group's median-priced floor. revisiting this ADR. Changing the default tolerance or extending the floor to any currently exempt path also requires revisiting it. +## Amendment 2 (2026-10-20): enforced on every node + +This amendment turns enforcement on for every node, with nothing to +configure. To make that safe it also adds a cap to the reference, lowers the +tolerance, floors paid-list admission, and keeps a refusal under the floor, +within a stated margin and once nodes run this release, from costing the +refuser trust. It answers the review +trigger in Validation above. The manipulation result now has a fix for the +dangerous direction and an accepted bound for the other. One-quote liveness +is accepted as a residual that could not be measured. + +### What the shadow period and a review of the code found + +Shadow telemetry from ant-prod-01, 2026-10-14T00:33Z to 2026-10-20T00:07Z, on +node 0.22.0 and 0.23.0: 2,117 floor lines from 699 node instances on 72 hosts. +54 of them (2.6%) skipped, all `thin_sample` with 14 known neighbours; across +all lines the median receiver knew 18. Replaying the logged formula +reproduced all 53 of the nodes' own `below_floor=true` lines (47 distinct +admissions) exactly. Taking each chunk and receiver once leaves 1,898 +admissions over 536 chunks (944 first seen as a client PUT, 954 as fresh +replication), arriving in 48 upload bursts. Of those, 1,801 admissions over +521 chunks were matched to the receiver's own committed count, from the +`Storage commitment rotated` line written when `current()` changes. + +- **65% was not safe.** Against the group median it would have refused 47 of + the 1,898 admissions (2.5%), all among the 1,801 matched, and 2.1% of + chunks would have been refused by every receiver that evaluated them. The + window 2026-09-10 to 09-16 showed the same shape: 3.4% of admissions at + 65%. +- **Honest payments sit further below the group median than the simulation + assumed.** The lowest observed ratio of the settled amount to `3 x` group + median was 0.473, and 1% of admissions settled below 0.57x. Receivers of the + same chunk also disagree about the median: by 1.08x at the median, 1.36x at + p95 (1.37x among receivers matched to their own commitment) and 1.91x at + worst. A node's own committed count swings hour to hour as neighbours come + and go (3,132 to 5,921 keys within nine hours on one node), while the fleet + median held between about 3,700 and 4,100 keys all week. +- **Griefing takes the whole chunk, not one replica.** Every receiver of a + chunk prices against nearly the same median, so neighbours that push it up + make every receiver refuse, after the money has settled. That was the first + enforcement blocker above. +- **One accepting node was enough to replicate a below-floor write.** The node + that stores a write announces it with `PaidNotify` to its paid close group + before it sends fresh offers. `PaidNotify` was verified as + `PaidListAdmission`, which the floor exempted, so peers recorded the key as + paid and paid-list repair later copied the bytes from that one node. The + floor refused the fresh offers and then let repair deliver the same data. +- **A refusal became a trust penalty.** Shadow mode never refused anything, + so none of this has fired, but four existing checks judge a peer on data it + does not hold: + - the storing node probes every fresh-offer target 5 to 15 minutes later + and penalises a peer that does not hold the data: at weight 5 for a chunk + (about a 48% cut in trust) and weight 1 for a pointer (about 12%); + - neighbour sync records a repair proof for every key a node hints to a + peer as a replica, and from an hour later the responsible audit + challenges that peer on the key, at weight 5; + - a node that has held an out-of-range record for three days challenges + the current close group before deleting it, and penalises an absent peer + at weight 5; + - a peer whose quorum check fails for a key only one node hinted penalises + that node, at weight 1, up to three times per verification round. + + A peer that refused on price would be penalised by the first three for + following policy, and the node that accepted the write by the last. + +### Decision + +1. **The reference is capped at the receiver's own price:** + `reference = min(group median price, calculate_price(own committed count))`. + A receiver never demands more than the tolerance of what it would itself + quote right now. So `settled >= 3 x tolerance x own price` always clears + it, however many neighbours lie. While gossip is honest, a receiver above + the group median prices exactly as before. The withdrawn own-price rule's + failure, honest rejections on the fullest nodes, therefore does not return. +2. **Tolerance 30%, a deliberate starting point.** 40% also refuses no + measured admission, but the week measured is light and 40% leaves less + margin (see below). Starting lower gives up some detection for that + margin, and the trade is accepted on purpose. A follow-up reads enforced + traffic, and only a later amendment raises the tolerance toward 40% (see + Validation of this amendment). +3. **Paid-list admission is floored too.** `PaidNotify` is only sent when a + node stores a client's write, and it carries the proof the client + presented for it, within the same reuse window as the PUT itself. Repair + and fetch never call the verifier, and an already verified key is answered + from the cache. So this floors the one remaining fresh economic event and + still reprices no historical receipt. (A client PUT that hits the payment + cache forwards whatever proof the client attached, unverified, exactly as + before this change; its receivers verify it.) +4. **No peer is judged on a contested write.** When a node's verifier first + admits the payment for a write (as the node a client PUT it to, on a fresh + offer, or on a `PaidNotify`), it prices the proof against its current + view; a later admission of the same write from the payment cache reuses + that decision. The write is *contested* in two cases: + - the node cannot price it (no own commitment, or too thin a view); + - its paid price is below 60% of the **uncapped** group median, twice the + 30% floor. + + An admitted contested write, one the node cannot price or one paid between + the floor and that threshold, is still stored and its offers still sent, + but the node does not judge its close-group peers on it: + - a fan-out (chunk or pointer) schedules no possession check. This is + decided from the proof as it stands, even one a cached PUT forwards + unverified, since its peers verify that proof and may refuse it; + - for a chunk, the key's paid-list entry carries a one-byte mark, which + survives restarts. Releases before this one only test presence, so they + read it as an ordinary paid key; + - a marked key goes out in paid hints only, never as a replica hint. No + repair proof is recorded for it, so no responsible audit challenges a + peer on it, and no peer can penalise this node as its sole hinter. A + paid hint is never punished, and repair still works wherever the group + accepted the key. The node also keeps its marked keys in memory, so + leaving them out of the hints it builds for a peer reads nothing on + disk. Each paid-list write updates that copy in commit order, a new + mark just before its commit, so the copy never lacks a committed mark. + If the marks cannot be read when the node starts, it still starts, as + before, and hints no key as a replica until a restart can read them; + - the responsible audit never samples a marked key, which also covers one + hinted in the moment before its mark was written; + - a prune audit still needs its proofs before deleting a marked key, but + scores no absent peer for it, and paid-list pruning keeps a marked entry + while the record is held; + - a mark that cannot be read counts as set: when in doubt, judge no one. + While reads fail the node judges no peer on any key they hit, so it + warns at the first failure of a run and, when a read succeeds again, + logs how many failed. + + The mark follows the decision the node took when it verified the payment: + the verifier remembers which admitted chunk payments the floor contested, + in a memory of its own sized for 100,000 of them, so a small payment cache + cannot make it forget one before the key is recorded. A write admitted + later from the payment cache is recorded by that decision. The proof + attached to it was never checked, and anyone can attach one to a + `PaidNotify` or a fresh offer, so it can neither set the mark nor veto it. + The first paid-list record of a key decides its mark, so a later payment + can neither add nor remove it. A fan-out skips its possession check + whenever the write is marked, by this node's verified decision or by the + record of the key that won, read after the write records its own, as + well as when the proof it forwards is contested. Pointers need no mark: + the possession check is the only check that penalises a peer for a + missing pointer. + + The paid quote's price stands in for a third of the settled amount; + settlement is at least that, so an overpaid write is only ever judged more + leniently. Deciding from the proof itself means a write stored after + prices moved is judged like any other. + + The 2x margin covers the largest spread measured between receivers' views + of the same chunk (1.91x; 1.37x at p95 among matched receivers). **It is a + margin, not a guarantee**, because only the refusing peer knows its own + view. Replayed over all 5,536 ordered pairs of measured receivers of the + same chunk, at payment levels from 0.2x to 1x the median in 0.02x steps, + no write one would judge was refused by the other. A 1.5x margin left 4 + such pairs, at payments near 0.3x the median. The cost is coverage: on the + measured week these checks are given up for 1.4% of the 1,801 admissions + matched to their receiver's own commitment, plus the 2.6% of evaluations + the floor could not price. A mark stays while the node holds the key, so a + node keeps unchecked every key it recorded while it could not price + (its first hour before a commitment, or while its gossip view refills + after a restart); the other holders of those keys still check them, + unless the mark was front-run (see the consequences below). + Penalising a peer for following the floor is the worse failure, so that + coverage is given up. A real guarantee needs the + refusing peer to tell the sender. A peer that can say "refused on price" + can also dodge ADR-0003's full-node detection that way, so that is a + separate decision. + + The threshold is its own constant, so a raise of the tolerance needs care. + A node still at 30% judges every write paid above 0.6x of its median, and + a peer already at 40% refuses below 0.4x of its own view. On the measured + pairs that leaves 3 of 5,536 where a refusal would be judged. A raise + therefore takes two releases: the first moves this threshold to 80%, and + the tolerance moves only once nearly every node runs it. Nodes that never + upgrade past the first stay a residual. + + The contest reads the **uncapped** median, which the cap does not + protect. Neighbours that overstate their commitments raise it, more of + their group's writes then count as contested, and the peers holding those + writes go unchecked on them. A minority can lift the median only as far as + the dearest honest members. Overstating also needs fake leaves, which fail + the commitment byte spot-checks (ADR-0004). Neighbours that understate at + the storer narrow the margin instead: on the test fixture, seven of + fifteen pull its median down to the cheapest honest member, which takes + the threshold to about 0.33x of the honest median, so a peer whose own + view and price both sit at least 1.11x above the honest median may refuse + a write the storer judges. That needs the same minority that disarms the + floor (see Griefing below). +5. **Enforced on every node, with nothing to configure.** There is no + per-node switch and no tolerance setting: `ANT_PRICE_FLOOR_ENFORCE` and + `ANT_PRICE_FLOOR_TOLERANCE_PERCENT` are no longer read. A node whose floor + sat above its peers' would refuse writes they store and then fail their + possession checks for it, and one below would just admit underpayment, so + the tolerance is the same everywhere. A node that still sets either + variable gets no warning; the variable is ignored. Backing the floor out + takes a release, so the release train's canary is where it is stopped. A + node whose replication engine fails to start runs without one, as before, + and its floor then has nothing to price from and skips every evaluation + (`skip_reason=no_commitment_cache`). So does a local `ant-devnet`, which + runs no replication engine. + +The telemetry line keeps every earlier field. `enforce` is now always +`true`, which tells these nodes apart from shadow-mode releases while the +fleet upgrades (as does the presence of `target`). `reference_price` is +still the value `required` is computed from. It adds `target` (chunk or +pointer), `group_median_price`, `own_price`, `paid_quote_age_secs` and +`contested`. + +### Why the cap, and why 30% + +The measured admissions replayed through both rules, as admissions refused +out of 1,801: + +| tolerance | median only | capped | +|---|---|---| +| 30% (chosen) | 0 | 0 | +| 40% | 0 | 0 | +| 45% | 0 | 0 | +| 50% | 1 | 0 | +| 55% | 11 | 1 | +| 60% | 26 | 5 | +| 65% (old default) | 47 | 11 | + +- **Margin, not the zero count, picks the tolerance.** Zero refusals out of + 1,801 only bounds the true rate below 0.17% per admission, 0.58% per chunk, + or 6.4% per upload burst at 95% confidence, because admissions in one burst + are not independent. The tolerance therefore rests on how far the worst + observed payment sits above it. With the cap, the lowest ratio is 0.534, so + 30% leaves 1.78x of headroom. 40% would leave 1.34x, 45% 1.19x and 50% only + 1.07x. The week measured is light traffic, 48 upload bursts, so the floor + starts where the margin is widest and moves up as enforced traffic adds to + it. +- **Proof reuse.** A client may present the same proof for up to about a + day. A receiver's capped reference can rise with its own count (one node's + price can double within a day) up to its group median, so a reused proof is + safe only against the uncapped median. There, the worst measured admission + sat at 0.473x, 1.58x above 30% (1.18x above 40%). Fleet-wide the median + reference moved by up to about 1.2x in price from one day to the next, + which 1.58x covers. Neither per-group drift over a day nor refusals of aged + proofs were measured. The new `paid_quote_age_secs` field makes them + measurable. +- **What it catches.** The realistic underpayment pays a near-empty, fresh + or sybil quoter. A baseline quote is about a sixth of the group median. + Share of measured receivers that refuse a 3x payment to a quoter, with the + cap: + + | quoter's count, as a share of the group median | 30% (chosen) | 40% | + |---|---|---| + | 0 (a baseline quote) | 91.4% | 98.0% | + | a quarter | 75.4% | 94.6% | + | 40% | 27.9% | 75.2% | + | half | 0% | 39.1% | + + Every receiver observed for a chunk refuses a baseline quote on 84.5% of + chunks at 30%, and 95.2% at 40%. The median-only rule at 40% refuses the + first three rows at 100%, 99.6% and 95.8%. That detection is what the cap + and the lower tolerance give up. +- **What it does not catch.** A quoter 40% as full as its group is admitted + by 72% of receivers, and one half as full by all of them. Shopping for the + cheapest member of the close group is not caught either: members measured + in August spanned about 1.4x in price, so the cheapest sits no lower than + about 0.7x the median. No tolerance that is safe for honest traffic catches + that, with or without the cap. +- **Griefing.** In the worst case, every neighbour gossips the protocol + maximum. The median-only rule then refuses every honest payment. With the + cap, all 1,801 measured admissions still clear at 30% (1,799 at 40%). On + the test fixture, seven of fifteen overstating neighbours flip no verdict + and eight do. Understating is where 30% costs something. Six neighbours + claiming almost nothing leave the floor standing against a baseline quote, + and seven disarm it, because the median then falls to the cheapest honest + member (3,000 keys), and 30% of that member's price is below a baseline + quote. At 40% it took eight. Disarming admits an underpayment and costs no + one an honest payment, so this minority bound is an accepted residual of + starting at 30%. For overstating, the close group's honest majority stays + the trust boundary the rest of the protocol already rests on. + +### Consequences of this amendment + +- **Liveness is a quorum question.** A client needs 4 stores out of up to 20 + targets (a pointer needs 5 of 7). It moves to the next target, with the same + proof, when one refuses. Under the cap, every receiver whose own price is at + most `settled / (3 x 30%)` accepts unconditionally. That is nearly every + receiver for a payment at the group median. The measurement could not + observe the quorum itself: the median chunk shows only two receivers' client + PUT lines. +- **One-quote liveness is an accepted, unmeasured residual.** An honest client + whose only reachable quoter is a fresh node pays 3x baseline, and nearly + every mature receiver refuses it. This cannot be told apart from the attack. + None of the measured payments was at baseline, and the lowest paid issuer + committed 2,147 keys, but that does not bound how often it happens. A client + retries a refused proof + for up to about a day, then re-quotes and pays again. A client-side change, + such as not settling a lone baseline quote when a committed quote is + available, is the remaining fix and belongs to the client. +- **A write below the floor stays where it was accepted.** A near-empty quoter + accepts its own payment, because its cap is its own price. A node with no + commitment or a thin view skips the floor. Mature peers now refuse both the + fresh offer and the `PaidNotify`. Unless the accepting nodes make up a + majority of the group, neither paid-list repair nor presence repair has the + votes to authorise copying the data. The accepting node finds such a write + contested and marks it (item 4), so it judges none of the peers that + refused it while it holds the key. When the accepting nodes are a majority + of the paid group, the refusers repair the key from those votes like any + other, unmarked, since repair carries no proof to price; a refuser whose own + quorum check fails while its neighbours' passes can then be judged on it. +- **Narrow gaps remain on nodes running this release.** + - On the PUT path the record is stored a moment before its paid-list + entry is written. A sync that scans the store in that moment can hint + the key once; the audit filter keeps that from becoming an audit, but a + refusing peer can charge the storer weight 1 once. A crash in that + moment leaves the stored key unrecorded, so unmarked unless a later + verified `PaidNotify` records it. + - A PUT served from the payment cache with no proof attached stores the + record without writing a paid-list entry at all, as before this change, + so it is not marked either. + - The verifier's memory of contested admissions is bounded: a decision is + lost before its key is recorded only if 100,000 newer contested + admissions reach the node in the moments between its verification and + its record. + - A paid-list insert that fails before its commit leaves the key out of + the paid list, so unmarked. When a marked insert's commit is what + failed, the in-memory copy keeps the mark, since the commit may still + have reached the disk, so this node does not hint the key as a replica + while the mark stays; a restart rebuilds the copy from the database. + - Paid-list pruning checks for the record before and after its network + round, but a record admitted between that check and the delete would + lose its mark. Both store paths require this node inside the same width + the prune requires it outside, so that needs its routing view to change + at that moment. +- **The first paid-list record decides a key's mark, which a quoter can + front-run.** A quoter in the 20 closest to an honest key can pay itself + for the key, for gas only, and announce it before the client's write + lands. At a price between the floor and the contest threshold, or at any + price on nodes that cannot price, the nodes it reaches first mark the key + and leave it unchecked, though they still store and repair it; those among + the client's storers skip their possession check too, while storers it did + not reach first still run theirs. Paying above the + threshold first does the opposite: those nodes record the key unmarked and + can judge peers that later refuse the client's cheaper write, until those + peers repair it from the votes the front-run created. +- **Rollout skew is the main rollout risk.** Until the fleet upgrades, a node + on an older release admits every payment and still probes every + fresh-offer target 5 to 15 minutes later. Each upgraded peer that refused a + below-floor chunk it fanned out is penalised at weight 5. In that node's + view the peer falls from a neutral 0.5 to about 0.26, below the 0.35 swap + threshold, for about 9.5 hours (a pointer costs weight 1: 0.5 to 0.44). The + score is local to that node (for peers in its persisted close group it + survives a restart), and falling below 0.35 only lets a better candidate + replace the peer in that node's routing + table when its bucket is full; nothing is blocked. Honest writes are not + refused, so this needs below-floor writes, which an attacker can buy for + little more than gas by paying its own near-empty node within the 20 + closest to the key: one write stored by `s` older nodes yields up to `s` + penalties on each upgraded member of the close group. Such a key also + stays on the older node unmarked, before and after it upgrades: its replica + hints and responsible and prune audits keep judging the refusers on it, + repeatedly, while it holds the key, and the refusers penalise it (weight 1) + as the sole hinter of a key their quorum rejects. On the v0.17.0 rollout the + managed fleet went from 24 of 73 hosts in the first hour to 65 in six and + 71 within about 17; community nodes follow on their own schedule. Enabling + enforcement one release after item 4 ships would remove this, at the cost + of a release that does not enforce. That was not chosen. +- **Retry load is new.** A proof that was accepted before can now be refused. + Refusals are not cached, so every peer a client tries re-verifies it, the + on-chain read included: up to 20 targets over 4 rounds. The load falls on + below-floor proofs, including the honest residuals above (a lone baseline + quote, an aged proof). +- Floor refusals stay economic. The refusal itself feeds no trust or + misbehaviour scoring, and on nodes running this release item 4 keeps the + checks that judge peers on held data from turning it into a penalty, within + the margin stated there and apart from the gaps above. Merkle batches still + bypass the floor entirely. + +### Validation of this amendment + +- Unit tests pin each piece: + - the cap binds below the median, and no number of overstating neighbours + lifts a receiver above its own price; + - seven of fifteen overstating neighbours leave the verdict unchanged and + eight flip it, and six understating neighbours leave the floor standing + while seven disarm it (pinned as known limits); + - the cheapest member of a realistic group is admitted, and a near-empty + quoter is refused; + - paid-list admission and pointer payments are floored; + - a write below the contest threshold, or one the node cannot price, is + contested, and one at the median is not. The threshold is 60% of the + uncapped median to the atto, it stays on the median when the cap binds, + and it follows the view when the write is recorded; + - the tolerance is 30%: exactly `3 x 30%` of the group median clears it + and one atto less does not; + - a node with no floor wired contests nothing; + - the verifier remembers a contested payment it admitted, in a memory a + small payment cache cannot shrink, a later cache-hit admission reads that + decision, and a refused payment is never remembered; the paid-list mark + is written only with a new entry, survives a restart and a later plain + insert, and goes with the entry; + - a marked key is left out of replica hints and of responsible-audit + sampling, and its paid entry outlives paid pruning while its record is + held; + - the in-memory copy of the marks is filled on opening and follows every + write and removal, and a marked insert and a batch removal change it + only in the order writes take; building + replica hints works while the paid list cannot be read at all, and a + paid list whose marks cannot be read on opening still opens and hints no + replica; + - a mark that cannot be read counts as set in the responsible audit, paid + pruning and a write's fan-out, and a run of such reads logs one + warning. The tests make the read fail by holding the thread's one LMDB + reader slot; + - a record reports the winning record's mark, or its own when that is + set, so a plain decision that loses to a marked record still leaves the + write unjudged. +- Four e2e tests run the real schedulers on a local testnet, each with a + control. They settle nothing on chain, so where a test needs a verified + contested admission it stands one in; unit tests pin that a real + verification records it: + - a chunk PUT through the handler schedules no possession check when the + storer's verified decision was contested (the key is marked and hinted + to a close peer as paid only), when only the proof it forwards is + contested, or when a marked record of the key won before its own plain + one, while the control write is penalised; + - a fresh pointer state the storer cannot price is not used to judge its + peers, while the control is; + - a prune audit of a marked record keeps it but scores none of the absent + peers, while the control scores them; + - a PUT, a fresh offer and a `PaidNotify`, each admitted from the payment + cache with a forged proof the floor would contest, mark the key exactly + when the recording node's own verified decision was contested, and never + from that proof. +- 48 mutations of the floor, the contest decision, the verifier's memory, + the mark, its in-memory copy and every check that reads the mark were run + against these unit and e2e tests on this amendment's code, and 47 fail at + least one of them. The one that survives makes prune-audit scoring count + a mark that cannot be read as unset. Only the e2e prune test reaches that + check, and it cannot make a node's paid-list read fail; the same read + counts as set, under test, in the audit, paid pruning and the fan-out. +- `tests/e2e/price_floor_ab.rs` drives the same honest and near-empty-quoter + uploads through a production-wired local testnet. Nodes are seeded to + production-sized commitments, payments are real (on Anvil), and possession + checks are shortened. It reports who stores each write, whose paid list + holds it, and whether any peer is penalised, so two builds can be compared. + It asserts what the floor promises: + - every honest upload reaches a quorum with no floor refusal; + - no peer that refused a near-empty write, or that the quoter fanned it + out to, loses penalty-sized trust in the quoter's view; + - every near-empty write stays off a quorum, with every refusal the + floor's. A mature storer whose own price is low enough may accept it, as + the cap allows. + + It refuses to report when the floor could not evaluate at all. Its tail is + minutes, too short for the responsible audit (an hour) or a prune audit + (three days). The prune e2e test covers the prune audit; the responsible + audit is covered by keeping marked keys out of replica hints, so no repair + proof exists, and by the sampling filter's unit test. It returns at once + unless `PRICE_FLOOR_AB=1`, so CI only compiles it. +- `tests/poc_price_floor_live.rs` runs the floor against a real Anvil vault + and real signed neighbour commitments. It shows that a 3x-baseline + settlement is refused, and that an at-floor overpayment clears while one + atto less does not. +- After release, watch `ant_node::payment::price_floor` for `below_floor=true` + with `enforce=true`, split by `context` and `target`, along with the + `contested=true` and `skipped=true` rates. Any refusal of a payment whose + paid issuer holds a mature commitment is a reason to revisit this amendment. + So is a refusal rate above 0.1% of evaluations. +- **Raising to 40%.** Each evaluated line carries `settled` and + `reference_price`, so it answers whether 40% would have refused it: + `settled < 3 x 40% x reference_price`. The case for 40% is enforced traffic + where that never happens, counted per upload burst: chunks less than 120 s + apart, and one event if any admission in the burst would be refused. Zero + events in 300 bursts bound the rate below 1% at 95% confidence, against + 6.4% from the 47 bursts measured so far that hold a matched admission. + Bursts from one client are not independent, so that bound is the least the + evidence must show, not proof. Production logs are kept for seven days, so + each week's counts must be saved before they expire. Before any raise, + aged proofs (`paid_quote_age_secs`) and + payments to near-empty quoters are read the same way. The raise then + follows the two steps in item 4. + ## Notes for AI-assisted work AI tools may help draft this ADR, but **must not mark it Accepted without diff --git a/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md b/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md index 509f7cc5..aa7bc08e 100644 --- a/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md +++ b/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md @@ -359,7 +359,7 @@ and a node that was actually paid is nominated for a first audit. | Node re-verification of **non-paid** quotes in the bundle | not done (bad signature on an unpaid quote is accepted) | | Node rejection of off-curve quotes (`QUOTE_ARITHMETIC_RECHECK_ENABLED`) | **enforced**, on every quote in the bundle | | Quote/commitment mismatch reported to trust | off (telemetry only) | -| Receiver-side price floor (ADR-0006) | shadow (`ANT_PRICE_FLOOR_ENFORCE`, 65% tolerance), priced against the **close group's median** committed count, not the receiver's own; the own-price basis was withdrawn after canary measurement | +| Receiver-side price floor (ADR-0006) | **enforced** on every node at 30% of a reference price: the **close group's median** committed-count price, capped at the receiver's own; covers client PUT, fresh replication and paid-list admission; no per-node switch (ADR-0006 Amendment 2) | | Payee eligibility gate (ADR-0005) | **not merged**; observe-only on its branch (`ADR5_ENFORCE`) | | Client pays the 3× multiplier on the merkle path | **enforced** by the published client, immediately and unconditionally (no flag, no date) | | Storer requires it (`MERKLE_PARITY_ENFORCED_FROM_UNIX`) | **enforced** by upgraded nodes, for receipts stamped from 2026-08-04 15:00 UTC onward | @@ -452,7 +452,9 @@ indistinguishable from a single receipt. mechanism to reprice. - **Median-of-K is only as honest as the quote set the client presents.** A modified client can shop the neighbourhood and settle the cheapest valid - quote; the receiver-side floor is the counterweight and is not yet enforcing. + quote. The receiver-side floor, enforced on every node (ADR-0006 Amendment + 2), is the counterweight: it refuses near-empty quotes, but not the + cheapest member of the close group. ### Neutral / Operational diff --git a/src/devnet.rs b/src/devnet.rs index 18e459bb..ddf5170c 100644 --- a/src/devnet.rs +++ b/src/devnet.rs @@ -7,7 +7,7 @@ use crate::ant_protocol::CHUNK_PROTOCOL_ID; use crate::config::{default_root_dir, NODES_SUBDIR, NODE_IDENTITY_FILENAME}; use crate::logging::{debug, info, warn}; use crate::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, QuoteGenerator, + EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator, QuotingMetricsTracker, }; use crate::replication::config::ReplicationConfig; @@ -960,7 +960,6 @@ impl Devnet { cache_capacity: DEVNET_PAYMENT_CACHE_CAPACITY, close_group_size: replication_config.close_group_size, local_rewards_address: rewards_address, - price_floor: PriceFloorConfig::default(), }; let payment_verifier = PaymentVerifier::new(payment_config); let metrics_tracker = QuotingMetricsTracker::new(DEVNET_INITIAL_RECORDS); diff --git a/src/node.rs b/src/node.rs index ac1f3588..63580116 100644 --- a/src/node.rs +++ b/src/node.rs @@ -9,9 +9,7 @@ use crate::event::{create_event_channel, NodeEvent, NodeEventsChannel, NodeEvent use crate::logging::{debug, error, info, warn}; use crate::payment::metrics::QuotingMetricsTracker; use crate::payment::wallet::parse_rewards_address; -use crate::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, QuoteGenerator, -}; +use crate::payment::{EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator}; use crate::replication::config::ReplicationConfig; use crate::replication::fresh::FreshWriteEvent; use crate::replication::ReplicationEngine; @@ -504,16 +502,7 @@ impl NodeBuilder { cache_capacity: config.payment.cache_capacity, close_group_size, local_rewards_address: rewards_address, - // Shadow mode by default; enforcement is a per-node canary opt-in - // via ANT_PRICE_FLOOR_ENFORCE (see PriceFloorConfig). - price_floor: PriceFloorConfig::from_env(), }; - if payment_config.price_floor.enforce { - info!( - "Price floor ENFORCEMENT enabled (tolerance {}%)", - payment_config.price_floor.tolerance_percent - ); - } let payment_verifier = PaymentVerifier::new(payment_config); let metrics_tracker = QuotingMetricsTracker::new(0); let mut quote_generator = QuoteGenerator::new(rewards_address, metrics_tracker); diff --git a/src/payment/mod.rs b/src/payment/mod.rs index c9b2a32d..107f16f6 100644 --- a/src/payment/mod.rs +++ b/src/payment/mod.rs @@ -63,8 +63,7 @@ pub use ant_protocol::payment::verify::{ }; pub use single_node::SingleNodePayment; pub use verifier::{ - EvmVerifierConfig, PaymentStatus, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, - VerificationContext, MAX_PAYMENT_PROOF_SIZE_BYTES, MIN_PAYMENT_PROOF_SIZE_BYTES, - PRICE_FLOOR_ENFORCE_ENV, PRICE_FLOOR_TOLERANCE_ENV, + EvmVerifierConfig, PaymentStatus, PaymentVerifier, PaymentVerifierConfig, VerificationContext, + MAX_PAYMENT_PROOF_SIZE_BYTES, MIN_PAYMENT_PROOF_SIZE_BYTES, PRICE_FLOOR_TOLERANCE_PERCENT, }; pub use wallet::{is_valid_address, parse_rewards_address, WalletConfig}; diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 0d9a126d..a8f57c84 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -12,6 +12,7 @@ use crate::payment::proof::{ deserialize_merkle_proof, deserialize_single_node_proof, detect_proof_type, ProofType, }; use crate::replication::commitment::MAX_COMMITMENT_KEY_COUNT; +use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::config::K_BUCKET_SIZE; use crate::storage::ChunkStore; use ant_protocol::payment::verify::{verify_quote_content, verify_quote_signature}; @@ -30,7 +31,7 @@ use saorsa_core::P2PNode; use std::collections::HashMap; use std::num::NonZeroUsize; use std::sync::Arc; -use std::time::Instant; +use std::time::{Instant, SystemTime}; /// Minimum allowed size for a payment proof in bytes. /// @@ -61,29 +62,33 @@ const PAYMENT_VERIFY_SLOW_LOG_MS: u128 = 500; /// This is the Kademlia K width, intentionally wider than `CLOSE_GROUP_SIZE`. const PAID_QUOTE_ISSUER_CLOSENESS_WIDTH: usize = K_BUCKET_SIZE; -/// Default tolerance for the receiver-side price floor, as a percentage of the -/// close group's MEDIAN commitment-bound price (see -/// [`PaymentVerifier::group_reference_price`]). +/// The receiver-side price floor, as a percentage of the reference price. /// -/// Derived from production, not chosen: on ant-prod-01 over 2026-07-29..08-04, -/// pricing each receiver against its OWN price rejected 0.377% of honest stores -/// while still admitting the cheapest-of-K underpayment at 62.9% of receivers, -/// because node prices span ~4x across the fleet and an honest client pays the -/// group median. Against the group median instead, honest payments land near the -/// reference and the underpayment lands well below it, so a tolerance separates -/// them. +/// The reference is the close group's median commitment-bound price, capped +/// at the receiver's own; a single-node payment must settle at least `3x` +/// this share of it. One network value, the same on every node and never +/// configured: a node whose floor sat above its peers' would refuse writes +/// they store and fail their checks for it. Chosen from production telemetry +/// as a deliberate starting point below what the measurements allow; the +/// measurements, and the rule for raising it, live in ADR-0006 Amendment 2. +pub const PRICE_FLOOR_TOLERANCE_PERCENT: u64 = 30; + +/// A write whose paid price is below this percentage of the UNCAPPED group +/// median is contested: a close-group peer whose view of the group sits higher +/// than this node's may have refused it under its floor, so this node does not +/// judge its peers on it (see [`PaymentVerifier::price_contested`]). /// -/// At the minimum permitted view, 65% rejects 0.140% of honest payments while -/// catching ~90% of underpayment, against the own-price floor's measured 0.377% -/// and 37.1%. Both axes improve, by roughly 2.7x on rejections rather than the -/// order of magnitude an earlier revision claimed. 60% is the conservative -/// alternative: 5x safer on honest traffic, at 66% detection. -/// -/// The full sweep, the sample-size table and the simulation's assumptions live -/// in ADR-0006 — deliberately in one place, because the same tables -/// duplicated here and in the ADR had already drifted apart once. Tighten or -/// loosen only from observed shadow telemetry, never speculatively. -const PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT: u64 = 65; +/// Twice the floor of this node's view, so a peer needs a view twice as high +/// before it can refuse a write this node judges. A margin, not a guarantee: +/// ADR-0006 Amendment 2 carries the measured spread between receivers' views +/// and the replay this was checked against. It is its own constant because a +/// raise of the tolerance moves it first, one release ahead. +const PRICE_FLOOR_CONTESTED_PERCENT: u64 = 60; + +/// Contested chunk admissions the verifier remembers, the most recent first, +/// so each at least until its paid-list record is written (see +/// [`PaymentVerifier::verified_payment_contested`]). +const PRICE_CONTESTED_MEMORY_CAPACITY: usize = 100_000; /// Minimum number of fresh NEIGHBOUR commitments (this node's own is counted /// separately) required before the price floor will evaluate at all. @@ -103,13 +108,12 @@ const PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT: u64 = 65; /// [`PAID_QUOTE_ISSUER_CLOSENESS_WIDTH`] closest, with the full table in /// ADR-0006. /// -/// Availability cost is expected to be small — the gossip TTL is hours against a -/// much shorter sweep, so a settled node should normally know most of its -/// neighbours — but that is an assumption, not a measurement, and the sample is -/// drawn from peers close to the CHUNK while the cache is filled from peers -/// close to THIS NODE, so edge-of-range chunks will skip more often. The -/// `skipped=true` / `skip_reason` telemetry exists to measure that in shadow -/// mode before anyone enforces. +/// Availability cost is small, and measured: on ant-prod-01 over +/// 2026-10-14..20, 54 of 2,117 floor lines (2.6%) skipped, all `thin_sample` +/// with 14 known neighbours; across all lines the median receiver knew 18. +/// The sample is drawn from peers close to the CHUNK while the cache is filled +/// from peers close to THIS NODE, so edge-of-range chunks skip more often. The +/// `skipped=true` / `skip_reason` telemetry keeps measuring it. /// /// A node that could suppress its neighbours' gossip can therefore disable this /// receiver's floor, but cannot make it reject honest traffic. That asymmetry is @@ -118,9 +122,9 @@ const PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS: usize = 15; /// Why a price-floor evaluation did not produce a reference price. /// -/// Shadow mode exists to calibrate -/// [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] from real deployments, and these -/// three causes call for completely different responses: a mid-rotation node is +/// The telemetry calibrates [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] from +/// real deployments, and these three causes call for completely different +/// responses: a mid-rotation node is /// expected and self-correcting, an unwired cache is a startup-order bug, and a /// persistently thin sample means the gate is set above what the network can /// supply. Collapsing them into one counter would make the telemetry unable to @@ -151,6 +155,17 @@ impl SkipReason { } } +/// What the price floor priced one admission against. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct FloorReference { + /// The amount the tolerance applies to: the lower of the two below. + price: Amount, + /// The close group's lower-median commitment-bound price. + group_median: Amount, + /// The price this node would itself quote right now. + own: Amount, +} + /// What the price floor could see of the close group on one admission. /// /// `neighbours` is the quantity the gate actually tests; `total` includes this @@ -188,107 +203,6 @@ impl GroupSample { } } -/// Environment variable enabling price-floor ENFORCEMENT (`1`/`true`). -/// -/// Absent or any other value means shadow mode: the floor is computed and -/// logged, never enforced. Per-node so enforcement can roll out canary-first; -/// unsetting it (and restarting) is the kill switch. -pub const PRICE_FLOOR_ENFORCE_ENV: &str = "ANT_PRICE_FLOOR_ENFORCE"; - -/// Environment variable overriding the price-floor tolerance percentage -/// (`0..=100`). Invalid or absent values use the default. -pub const PRICE_FLOOR_TOLERANCE_ENV: &str = "ANT_PRICE_FLOOR_TOLERANCE_PERCENT"; - -/// Receiver-side price floor for single-node store admissions. -/// -/// ADR-0004 gives every quote a price CEILING (`price == -/// calculate_price(committed_key_count)`), but a ceiling is not a revenue -/// floor: a modified client can fetch quotes from the whole neighbourhood and -/// settle only the cheapest valid one in a 1-quote proof. This policy is the -/// floor half: the settled amount must also clear the close group's MEDIAN -/// commitment-bound price, scaled by a tolerance. -/// -/// The reference is the group's median rather than this receiver's own price. -/// A client pays `3x` the median of the quotes it collected, so comparing that -/// to a single node's price compares a median to one draw from a distribution -/// spanning ~4x — which in production rejected honest stores on the fullest -/// nodes while admitting underpayment at the emptiest. The reference is built -/// only from this node's own commitment and its neighbours' TTL-fresh gossiped -/// ones, never from the payment bundle. -/// -/// Rollout is shadow-first: with `enforce == false` (the default) the floor -/// is evaluated and logged on every single-node store admission but never -/// rejects. Enforcement is enabled per node via [`PRICE_FLOOR_ENFORCE_ENV`] -/// for canary rollout. A floor rejection is an economic admission decision -/// only — it must never feed trust/misbehaviour scoring. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct PriceFloorConfig { - /// When true, a below-floor settled payment is rejected. When false - /// (default), the verifier only logs would-reject telemetry. - pub enforce: bool, - /// Floor as a percentage of the close group's median commitment-bound - /// price (`0..=100`). - pub tolerance_percent: u64, -} - -impl Default for PriceFloorConfig { - fn default() -> Self { - Self { - enforce: false, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - } - } -} - -impl PriceFloorConfig { - /// Build from the environment (see [`PRICE_FLOOR_ENFORCE_ENV`] and - /// [`PRICE_FLOOR_TOLERANCE_ENV`]). Never panics. - /// - /// An unset tolerance uses the default. A tolerance that is *present but - /// invalid* (unparseable or `> 100`) is an operator error: rather than - /// silently enforce at the default, this **fails closed to shadow mode** - /// (`enforce = false`) and logs a prominent error, so a fat-fingered - /// tolerance can never enforce an unintended floor against real payments. - #[must_use] - pub fn from_env() -> Self { - let enforce_requested = std::env::var(PRICE_FLOOR_ENFORCE_ENV) - .is_ok_and(|v| matches!(v.trim(), "1" | "true" | "TRUE" | "True")); - - let tolerance_raw = std::env::var(PRICE_FLOOR_TOLERANCE_ENV).ok(); - let valid_tolerance = tolerance_raw - .as_deref() - .map(str::trim) - .and_then(|s| s.parse::().ok()) - .filter(|percent| *percent <= 100); - // A tolerance that is set but does not parse to `0..=100` is an operator - // error. Fail CLOSED: never enforce a tolerance the operator did not - // actually specify. - let tolerance_present_but_invalid = tolerance_raw.is_some() && valid_tolerance.is_none(); - - if tolerance_present_but_invalid { - if enforce_requested { - crate::logging::error!( - "{PRICE_FLOOR_TOLERANCE_ENV}={tolerance_raw:?} is not an integer in 0..=100; \ - refusing to enforce the price floor with an unspecified tolerance. \ - Price-floor enforcement DISABLED (shadow mode). Fix the value and restart \ - to enable enforcement." - ); - } else { - crate::logging::warn!( - "{PRICE_FLOOR_TOLERANCE_ENV}={tolerance_raw:?} is not an integer in 0..=100; \ - using the default {PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT}% for shadow-mode \ - telemetry." - ); - } - } - - Self { - enforce: enforce_requested && !tolerance_present_but_invalid, - tolerance_percent: valid_tolerance.unwrap_or(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT), - } - } -} - #[derive(Clone, Copy)] struct LegacyMedianCandidate<'a> { encoded_peer_id: &'a evmlib::EncodedPeerId, @@ -444,8 +358,6 @@ pub struct PaymentVerifierConfig { /// Kept in the verifier config for payment policies that bind receipts to /// this node's payout address. pub local_rewards_address: RewardsAddress, - /// Receiver-side price floor policy (shadow mode by default). - pub price_floor: PriceFloorConfig, } /// What a payment authorizes, and where it routes. @@ -523,8 +435,12 @@ impl PaymentTarget { /// touching direct-PUT behaviour. /// - **`PaidListAdmission`** — the node is admitting fresh paid-list metadata. /// It runs the same live payment checks, but writes a weaker cache entry -/// that does not authorize future chunk stores. The price floor never -/// applies here: paid-list records reprice no fresh economic decision. +/// that does not authorize future chunk stores. The price floor applies +/// here too: a `PaidNotify` announces a write a client has just made, with +/// the proof the client presented (on a payment-cache hit, whatever proof it +/// attached, as before), and a paid-list vote is what later lets repair copy +/// the data, so exempting it would let one accepting node replicate a +/// below-floor write to its whole group. /// /// The caller must check local receiver/admission membership before invoking /// the verifier for replication admission: fresh chunk replication requires @@ -740,11 +656,6 @@ pub struct PaymentVerifier { /// entirely rather than pricing against anything, so a node in that state /// never rejects. local_commitment_source: RwLock>>, - /// Live price-floor policy, initialized from - /// [`PaymentVerifierConfig::price_floor`]. Behind a lock so tests (and any - /// future ops surface) can flip enforcement without rebuilding the - /// verifier. - price_floor: RwLock, /// ADR-0008: the instant from which a merkle receipt must settle at the /// single-node 3x multiplier, defaulting to /// [`crate::replication::config::MERKLE_PARITY_ENFORCED_FROM_UNIX`]. @@ -769,6 +680,17 @@ pub struct PaymentVerifier { /// Incremented only on first emission for a pool, so it measures /// settlements sampled rather than chunks admitted. merkle_parity_emissions: std::sync::atomic::AtomicUsize, + /// ADR-0006: chunk addresses whose payment this verifier admitted while the + /// price floor contested it (see [`Self::verified_payment_contested`]). + /// Written during a full verification, before the payment cache learns + /// the chunk, so every later record of it, a cache hit included, is judged + /// by the proof that was verified and never by whatever proof came with + /// it. Only contested admissions enter, and a chunk is recorded within + /// moments of its verification, so the fixed + /// [`PRICE_CONTESTED_MEMORY_CAPACITY`], whatever the payment cache is sized + /// to, loses a decision before it is read only if that many newer + /// contested admissions arrive in between. + verified_price_contested: Mutex>, /// Configuration. config: PaymentVerifierConfig, } @@ -858,6 +780,9 @@ impl PaymentVerifier { "pool cache capacity must be > 0" ); let cache = VerifiedCache::with_capacity(config.cache_capacity); + let verified_price_contested = Mutex::new(LruCache::new( + NonZeroUsize::new(PRICE_CONTESTED_MEMORY_CAPACITY).unwrap_or(NonZeroUsize::MIN), + )); let pool_cache_size = NonZeroUsize::new(DEFAULT_POOL_CACHE_CAPACITY).unwrap_or(NonZeroUsize::MIN); let pool_cache = Mutex::new(LruCache::new(pool_cache_size)); @@ -897,7 +822,6 @@ impl PaymentVerifier { ))), monetized_pin_tx: RwLock::new(None), local_commitment_source: RwLock::new(None), - price_floor: RwLock::new(config.price_floor), merkle_parity_from: RwLock::new( crate::replication::config::MERKLE_PARITY_ENFORCED_FROM_UNIX, ), @@ -906,6 +830,7 @@ impl PaymentVerifier { .unwrap_or(NonZeroUsize::MIN), )), merkle_parity_emissions: std::sync::atomic::AtomicUsize::new(0), + verified_price_contested, config, } } @@ -986,17 +911,82 @@ impl PaymentVerifier { debug!("PaymentVerifier: local commitment source attached for price-floor checks"); } - /// The live price-floor policy this verifier applies. - #[must_use] - pub fn price_floor_config(&self) -> PriceFloorConfig { - *self.price_floor.read() + /// Whether this node must not judge its close-group peers on a write routed + /// at `routing` and paid by `proof`: a peer may have refused it under its + /// own floor, and refusing a below-floor payment is network policy, never + /// misbehaviour (ADR-0006). + /// + /// True when the floor cannot price the write here (no own commitment, or + /// too thin a view), or when its paid price is below + /// [`PRICE_FLOOR_CONTESTED_PERCENT`] of the UNCAPPED group median: the + /// question is what a peer whose own price sits above this node's may have + /// refused. The paid quote's price stands in for a third of the settled + /// amount: settlement is at least that, so an overpaid write can only be + /// judged more leniently, never less. Evaluated from the proof as it + /// stands, verified or not: this is what a fan-out uses, since the peers + /// it offers the proof to verify it themselves and may refuse it. + /// + /// False for a merkle proof, which the floor never refuses, for a proof + /// that does not parse (its peers refuse it on structure, as before), and + /// where no floor is wired at all (no cache or no routing view). + pub(crate) async fn price_contested(&self, routing: &XorName, proof: &[u8]) -> bool { + let Some(paid_price) = Self::single_node_paid_price(proof) else { + return false; + }; + let (reference, sample) = self.group_reference_price(routing).await; + Self::contested(paid_price, reference, sample) } - /// Test-only setter for the price-floor policy, so enforcement and - /// tolerance can be exercised without environment variables. + /// Whether this node admitted a payment for the chunk at `address` that + /// the price floor contested when it verified it. + /// + /// Unlike [`Self::price_contested`], this only ever reflects a proof whose + /// signature and settlement this node checked: a write admitted later from + /// the payment cache reads the decision taken then, so a proof attached to + /// a cache hit, which nobody checked, can neither set nor veto it. + pub(crate) fn verified_payment_contested(&self, address: &XorName) -> bool { + self.verified_price_contested.lock().contains(address) + } + + /// Test-only stand-in for a verified admission of the chunk at `address` + /// that the price floor contested, for tests that cannot settle payments + /// on chain. #[cfg(any(test, feature = "test-utils"))] - pub fn set_price_floor_for_tests(&self, config: PriceFloorConfig) { - *self.price_floor.write() = config; + pub fn note_contested_admission_for_tests(&self, address: &XorName) { + self.verified_price_contested.lock().put(*address, ()); + } + + /// The contest decision for a write paid at `paid_price`, from what the + /// floor could price it against. + fn contested( + paid_price: Amount, + reference: Option, + sample: GroupSample, + ) -> bool { + reference.map_or( + matches!( + sample.skip_reason, + Some(SkipReason::NoOwnCommitment | SkipReason::ThinSample) + ), + |reference| paid_price < Self::contested_price(reference.group_median), + ) + } + + /// The paid (median) quote's price in a single-node proof, or `None` for a + /// merkle proof or one that does not parse. + fn single_node_paid_price(proof: &[u8]) -> Option { + if !matches!(detect_proof_type(proof), Some(ProofType::SingleNode)) { + return None; + } + let parsed = deserialize_single_node_proof(proof).ok()?; + let candidates = Self::legacy_median_candidates(&parsed.proof_of_payment).ok()?; + candidates.first().map(|candidate| candidate.quote.price) + } + + /// The paid (per-quote) price below which a write is contested. + fn contested_price(group_median: Amount) -> Amount { + group_median.saturating_mul(Amount::from(PRICE_FLOOR_CONTESTED_PERCENT)) + / Amount::from(100u64) } /// Test-only setter for the ADR-0008 merkle parity boundary, so both the @@ -1475,9 +1465,9 @@ impl PaymentVerifier { Self::validate_quote_arithmetic(payment)?; let candidates = Self::legacy_median_candidates(payment)?; let mut failures = Vec::with_capacity(candidates.len()); - // The paid (median) price and settled on-chain amount of the winning - // candidate, kept for the receiver-side price-floor policy below. - let mut verified_paid_quote: Option<(Amount, Amount)> = None; + // The winning candidate's quote and settled on-chain amount, kept for + // the receiver-side price floor below. + let mut verified_paid_quote: Option<(&PaymentQuote, Amount)> = None; // ADR-0004 Amendment 2: remember WHICH candidate's on-chain settlement // verified — only that peer's commitment actually earned money, so only // its pin is nominated for a deterministic first audit below. The @@ -1486,11 +1476,10 @@ impl PaymentVerifier { let mut paid_peer: Option<[u8; 32]> = None; for candidate in candidates { - let paid_price = candidate.quote.price; let candidate_peer = *candidate.encoded_peer_id.as_bytes(); match self.verify_legacy_median_candidate(target, candidate).await { Ok(settled_amount) => { - verified_paid_quote = Some((paid_price, settled_amount)); + verified_paid_quote = Some((candidate.quote, settled_amount)); // First settlement-verified median candidate wins the paid // slot and the sole first-audit nomination. If a client // settled several TIED-median candidates, only this one is @@ -1504,7 +1493,7 @@ impl PaymentVerifier { } } - let Some((paid_price, settled_amount)) = verified_paid_quote else { + let Some((paid_quote, settled_amount)) = verified_paid_quote else { let xorname_hex = hex::encode(xorname); let details = if failures.is_empty() { "no median-priced candidates were available".to_string() @@ -1516,17 +1505,24 @@ impl PaymentVerifier { ))); }; - // Receiver-side price floor (single-node store admissions only). Runs - // AFTER the paid quote's signature and settlement verified above, so - // unauthenticated bundles can never poison floor telemetry. Shadow - // mode logs; enforcement rejects — an economic admission decision - // only, never trust/misbehaviour evidence. - // The floor compares against the median commitment price of the close - // group that is RESPONSIBLE for the data, so it takes the routing - // address. For a chunk that is the same value; for a pointer the paid - // content is its state, which names no close group at all. - self.enforce_price_floor(target.routing(), paid_price, settled_amount, context) + // Receiver-side price floor, in every context. Runs AFTER the paid + // quote's signature and settlement verified above, so unauthenticated + // bundles can never poison floor telemetry. A rejection is an economic + // admission decision only, never trust/misbehaviour evidence. + // The floor prices against the close group that is RESPONSIBLE for the + // data, so it reads the target's routing address. For a chunk that is + // the content address; for a pointer the paid content is its state, + // which names no close group at all. + let contested = self + .enforce_price_floor(target, paid_quote, settled_amount, context) .await?; + // Remembered before the payment cache learns the chunk, so any record + // of it from a cache hit reads this decision (see + // `verified_payment_contested`). Only ever set: a later payment does + // not unsay an earlier one. + if let (true, PaymentTarget::Chunk(address)) = (contested, target) { + self.verified_price_contested.lock().put(*address, ()); + } // ADR-0004 observe-only telemetry: log off-curve quotes only AFTER the // paid (median) quote's ML-DSA-65 signature has verified above, so @@ -1679,70 +1675,52 @@ impl PaymentVerifier { .collect() } - /// The close group's MEDIAN commitment-bound price for `xorname`, and the - /// number of peers that median was taken over. - /// - /// This is the price floor's reference, and choosing the median of the group - /// rather than this node's own price is the whole point of the policy. - /// - /// A client pays `3x` the MEDIAN of the quotes it collected. Pricing the - /// floor against a receiver's OWN price compares that median to a single - /// draw from a distribution that spans ~4x across the fleet, so the fullest - /// receivers reject honest payments while the emptiest admit deep - /// underpayment — measured on ant-prod-01, 0.377% of honest stores rejected - /// with the cheapest-of-K underpayment still admitted by 62.9% of receivers. - /// Both sides of that are the same defect: the reference is a sample, and - /// the payment is a median. Comparing median to median removes it. + /// The price floor's reference for `routing`: the close group's MEDIAN + /// commitment-bound price, capped at this node's own, plus the sample the + /// median was taken over. /// - /// The reference is built ONLY from state this node already holds: + /// **Why the median.** A client pays `3x` the MEDIAN of the quotes it + /// collected. Pricing the floor against a receiver's OWN price compares + /// that median to a single draw from a distribution that spans ~4x across + /// the fleet, so the fullest receivers reject honest payments while the + /// emptiest admit deep underpayment — measured on ant-prod-01, 0.377% of + /// honest stores rejected with the cheapest-of-K underpayment still + /// admitted by 62.9% of receivers. Comparing median to median removes that. /// - /// * the K peers closest to `xorname` in its own routing table, and - /// * those peers' own signed commitments from the gossip cache, TTL-gated - /// by [`GOSSIP_ANSWERABILITY_TTL`], plus this node's own live commitment. + /// **Why the cap.** The median is taken over gossiped commitments, and a + /// signature proves who said a number, never that the number is true. + /// Neighbours that OVERSTATE their counts push the median up and would + /// make this node refuse payments that are already settled and cannot be + /// refunded. Capping the reference at this node's own price bounds it by + /// the one number no neighbour controls: a payment of `3x tolerance x` + /// this node's own price always clears here, however many neighbours lie. + /// While the gossip is honest an above-median receiver prices against the + /// median exactly as before. The cost is detection: a receiver below the + /// median admits payments down to the tolerance of its own, lower, price. + /// Neighbours that UNDERSTATE still pull the median down, which costs a + /// missed underpayment, the accepted direction. ADR-0006 Amendment 2 + /// carries both bounds. /// - /// Nothing in the payment bundle feeds it. A client cannot pad, prune or - /// reorder quotes to move this number, which is what makes the floor - /// resistant to the quote-selection games it exists to stop. + /// The reference is built ONLY from state this node already holds: the K + /// peers closest to `routing` in its own routing table, those peers' signed + /// commitments from the gossip cache, TTL-gated by + /// [`GOSSIP_ANSWERABILITY_TTL`], and this node's own live commitment. + /// Nothing in the payment bundle feeds it, so a client cannot pad, prune or + /// reorder quotes to move it. Counts above `MAX_COMMITMENT_KEY_COUNT` are + /// dropped, since gossip ingest authenticates the sender without bounding + /// the count. /// - /// The remaining lever is gossiping a false commitment, and it cuts BOTH - /// ways, which is why the sample gate is set where it is: - /// - /// * **Understating** drags the median down and weakens the floor. The cost - /// is a missed underpayment. - /// * **Overstating** drags it up and makes this node reject settled, honest - /// payments. That is the dangerous direction, because the money is already - /// spent and cannot be refunded, so a griefer would be spending nothing to - /// destroy someone else's payment. - /// - /// The bound is NOT "a majority of the group", and it is worth stating - /// precisely because it is what an enforcement decision rests on. The floor - /// does not need the median to reach the liars' value — only to move past - /// the tolerance headroom. Each liar shifts the order statistic by one rank, - /// and on the measured fleet spread one rank is worth roughly 15% of price - /// while a 65% tolerance buys 1.54x. So **about a quarter of the sample** - /// suffices in either direction: measured on this crate's own fixture, 4 of - /// 15 overstating neighbours reject an honest payment sitting at the true - /// median, and 4 of 15 understating ones (placed above the median, where an - /// attacker would put them) admit a cheapest-of-K underpayment. - /// - /// Counts above `MAX_COMMITMENT_KEY_COUNT` are dropped outright, since - /// gossip ingest authenticates the sender without bounding the count. That - /// caps how far a single liar can reach but does not change the cardinality - /// bound above. A signature proves who said a number, never that the number - /// is true, and address grinding into a close group is a known adjacent - /// problem. This is why the policy ships shadow-only: the ~25% bound is a - /// reason not to enforce yet, not a property to rely on. - /// - /// Returns the reference price only when at least + /// Returns a reference only when at least /// [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] neighbour commitments are - /// fresh AND this node has its own live commitment; the `usize` is the - /// sample size either way, so the caller can log how close it came. A `None` - /// price means "do not evaluate". - /// - /// [`GOSSIP_ANSWERABILITY_TTL`]: crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL - async fn group_reference_price(&self, xorname: &XorName) -> (Option, GroupSample) { + /// fresh AND this node has its own live commitment; the sample comes back + /// either way, so the caller can log how close it came. `None` means "do + /// not evaluate". + async fn group_reference_price( + &self, + routing: &XorName, + ) -> (Option, GroupSample) { // This node's own live commitment. It is in the group whenever it is - // being asked to store the chunk, so it belongs in the sample. Bound to + // being asked to store the data, so it belongs in the sample. Bound to // a local first so the source lock is released before the awaits below. // // It is counted separately from the neighbour tally: the gate below asks @@ -1752,13 +1730,13 @@ impl PaymentVerifier { // Wiring faults are reported before state faults, so a startup-order bug // is never mistaken for "this node is mid-rotation" or for "the gate is // above what the network supplies". Those three have three different - // fixes and shadow mode exists to tell them apart. + // fixes and the telemetry exists to tell them apart. let Some(cache) = self.commitment_cache.read().as_ref().map(Arc::clone) else { return (None, GroupSample::none(SkipReason::NoCommitmentCache)); }; // Cache is checked first so a node without one does not pay for a // routing-table scan it will discard. - let closest = self.local_k_closest_peer_ids(xorname).await; + let closest = self.local_k_closest_peer_ids(routing).await; if closest.is_empty() { return (None, GroupSample::none(SkipReason::NoRoutingView)); } @@ -1772,8 +1750,11 @@ impl PaymentVerifier { let Some(own) = own_source.and_then(|source| source.current_binding_snapshot()) else { return (None, GroupSample::none(SkipReason::NoOwnCommitment)); }; - let now = std::time::Instant::now(); - let ttl = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; + // The routing scan includes this node, which the gossip cache never + // holds (ingest only accepts senders in the routing table, and the + // routing table never holds self), so its own commitment joins the + // sample exactly once, below. + let now = Instant::now(); let mut key_counts: Vec = { let guard = cache.read().await; closest @@ -1783,7 +1764,8 @@ impl PaymentVerifier { // Stale gossip is treated as unknown rather than as a low // price, so a peer going quiet can never drag the median // down, nor count toward the minimum sample. - if now.saturating_duration_since(record.received_at) >= ttl { + if now.saturating_duration_since(record.received_at) >= GOSSIP_ANSWERABILITY_TTL + { return None; } record @@ -1827,72 +1809,100 @@ impl PaymentVerifier { .get((sample.total.saturating_sub(1)) / 2) .copied() .unwrap_or_default(); - let reference = calculate_price(usize::try_from(median_key_count).unwrap_or_default()); + let group_median = calculate_price(usize::try_from(median_key_count).unwrap_or_default()); + // The price this node quotes right now, from the same commitment its + // `QuoteGenerator` binds quotes to. + let own_price = calculate_price(usize::try_from(own.key_count).unwrap_or_default()); + let reference = FloorReference { + price: group_median.min(own_price), + group_median, + own: own_price, + }; (Some(reference), sample) } - /// Receiver-side price floor for single-node store admissions. + /// Receiver-side price floor for single-node payments, in every context. /// /// Requires the settled on-chain amount to also clear - /// `PAID_QUOTE_PAYMENT_MULTIPLIER × tolerance% ×` the close group's median - /// commitment-bound price (see [`Self::group_reference_price`]). Together - /// with the per-quote check above, the effective rule is - /// `settled >= 3 × max(paid_price, tolerated_floor)`. + /// `PAID_QUOTE_PAYMENT_MULTIPLIER × tolerance% ×` the reference price (see + /// [`Self::group_reference_price`]). Together with the per-quote check + /// above, the effective rule is `settled >= 3 × max(paid_price, + /// tolerated_floor)`. + /// + /// It applies to direct PUTs, fresh replication and paid-list admission + /// alike. Any one of them left open would let a single accepting node + /// spread a below-floor write to its whole group: fresh replication copies + /// it directly, and a paid-list vote lets repair copy it later. /// - /// When the group reference cannot be built — too few fresh commitments, no - /// P2P handle, no gossip cache — the floor SKIPS the admission rather than - /// falling back to this node's own price. Falling back is what the previous - /// revision did, and it is precisely the comparison that rejected honest - /// stores on the fullest nodes. A skipped evaluation is logged with its - /// real `neighbours` / `group_sample` counts and a `skip_reason`, so shadow + /// When the reference cannot be built — too few fresh commitments, no own + /// commitment, no P2P handle, no gossip cache — the floor SKIPS the + /// admission rather than guessing. A skipped evaluation is logged with its + /// real `neighbours` / `group_sample` counts and a `skip_reason`, so the /// telemetry can separate an unwired node from one whose gate simply sits /// above what the network supplies. /// + /// Returns, for an admitted payment, whether it is `contested`: the + /// decision [`Self::price_contested`] makes from the same reference. A + /// node that stores a contested write does not judge its peers on it. The + /// line logs it too. + /// /// Always emits one telemetry line per evaluated admission (target - /// `ant_node::payment::price_floor`) so shadow mode measures the exact - /// rejections enforcement would cause. Rejects only when - /// [`PriceFloorConfig::enforce`] is set. + /// `ant_node::payment::price_floor`), and rejects every admission below + /// the floor. The line keeps a constant `enforce=true`, which separates + /// these nodes from releases that only logged (`enforce=false`) while the + /// fleet upgrades. async fn enforce_price_floor( &self, - xorname: &XorName, - paid_price: Amount, + target: &PaymentTarget, + paid_quote: &PaymentQuote, settled_amount: Amount, context: VerificationContext, - ) -> Result<()> { - if !context.is_store_admission() { - return Ok(()); - } - - let floor = *self.price_floor.read(); - let tolerance_percent = floor.tolerance_percent.min(100); - - let (reference, group_sample) = self.group_reference_price(xorname).await; - let Some(reference_price) = reference else { + ) -> Result { + let routing = target.routing(); + let paid_price = paid_quote.price; + // Only read by the telemetry, which compiles to nothing without the + // `logging` feature. + #[cfg_attr(not(feature = "logging"), allow(unused_variables))] + let target_kind = match target { + PaymentTarget::Chunk(_) => "chunk", + PaymentTarget::Pointer { .. } => "pointer", + }; + #[cfg_attr(not(feature = "logging"), allow(unused_variables))] + let paid_quote_age_secs = SystemTime::now() + .duration_since(paid_quote.timestamp) + .map_or(0, |age| age.as_secs()); + + let (reference, group_sample) = self.group_reference_price(routing).await; + let contested = Self::contested(paid_price, reference, group_sample); + let Some(reference) = reference else { // Not enough of the close group is known to price anything. Skip // rather than guess: see `group_reference_price`. The reason and - // both counts are logged because shadow telemetry has to separate a + // both counts are logged because the telemetry has to separate a // mid-rotation node from an unwired cache from a gate set above what // the network can supply — three causes with three different fixes. info!( target: "ant_node::payment::price_floor", "Price floor: context={context:?}, xorname={}, paid_price={paid_price}, \ settled={settled_amount}, neighbours={}, group_sample={}, reference_price=0, \ - tolerance_percent={tolerance_percent}, required=0, below_floor=false, \ - skipped=true, skip_reason={}, enforce={}", - hex::encode(xorname), + tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}, required=0, below_floor=false, \ + skipped=true, skip_reason={}, enforce=true, target={target_kind}, \ + group_median_price=0, own_price=0, paid_quote_age_secs={paid_quote_age_secs}, \ + contested={contested}", + hex::encode(routing), group_sample.neighbours, group_sample.total, group_sample.skip_reason_str(), - floor.enforce, ); - return Ok(()); + return Ok(contested); }; - // Amount is a U256: reference_price is bounded by the saturating pricing - // curve and the multipliers are <= 300, so this arithmetic cannot + // Amount is a U256: prices are bounded by the saturating pricing curve + // and every multiplier here is at most 100, so this arithmetic cannot // overflow; saturating keeps that a proof instead of an assumption. - let tolerated_price = - reference_price.saturating_mul(Amount::from(tolerance_percent)) / Amount::from(100u64); + let tolerated_price = reference + .price + .saturating_mul(Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT)) + / Amount::from(100u64); let required_amount = tolerated_price.saturating_mul(Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER)); let below_floor = settled_amount < required_amount; @@ -1901,25 +1911,29 @@ impl PaymentVerifier { target: "ant_node::payment::price_floor", "Price floor: context={context:?}, xorname={}, paid_price={paid_price}, \ settled={settled_amount}, neighbours={}, group_sample={}, \ - reference_price={reference_price}, tolerance_percent={tolerance_percent}, \ + reference_price={}, tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}, \ required={required_amount}, below_floor={below_floor}, skipped=false, \ - skip_reason=none, enforce={}", - hex::encode(xorname), + skip_reason=none, enforce=true, target={target_kind}, group_median_price={}, \ + own_price={}, paid_quote_age_secs={paid_quote_age_secs}, contested={contested}", + hex::encode(routing), group_sample.neighbours, group_sample.total, - floor.enforce, + reference.price, + reference.group_median, + reference.own, ); - if below_floor && floor.enforce { + if below_floor { return Err(Error::Payment(format!( "Settled payment {settled_amount} is below the close-group price floor \ - {required_amount} ({PAID_QUOTE_PAYMENT_MULTIPLIER}x {tolerance_percent}% of \ - the group median price {reference_price}, taken over {} peers)", - group_sample.total + {required_amount} ({PAID_QUOTE_PAYMENT_MULTIPLIER}x {PRICE_FLOOR_TOLERANCE_PERCENT}% of \ + the reference price {}: the lower of the group median {} over {} peers and \ + this node's own price {})", + reference.price, reference.group_median, group_sample.total, reference.own ))); } - Ok(()) + Ok(contested) } fn validate_paid_quote_content( @@ -3770,7 +3784,7 @@ mod tests { use saorsa_core::MlDsa65; use saorsa_pqc::pqc::types::MlDsaSecretKey; use saorsa_pqc::pqc::MlDsaOperations; - use std::time::SystemTime; + use std::time::{Duration, SystemTime}; /// Create a verifier for unit tests. EVM is always on, but tests can /// pre-populate the cache to bypass on-chain verification. @@ -3835,7 +3849,6 @@ mod tests { cache_capacity: 100, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([1u8; 20]), - price_floor: PriceFloorConfig::default(), }; PaymentVerifier::new(config) } @@ -4507,36 +4520,32 @@ mod tests { } } - /// Receiver commitment count for floor tests, chosen so the local price is - /// comfortably more than double the baseline (the 50% floor then rejects a - /// baseline-priced settlement). Asserted in each test so a pricing-curve - /// change fails loudly instead of silently weakening the tests. - const FLOOR_TEST_LOCAL_KEY_COUNT: u32 = 1_000_000; + /// Receiver commitment count for floor tests: the protocol maximum, far + /// above every group member, so the own-price cap never binds in the tests + /// that are about the group median. The cap has tests of its own, built on + /// [`floor_test_verifier_with_own_count`]. + const FLOOR_TEST_LOCAL_KEY_COUNT: u32 = MAX_COMMITMENT_KEY_COUNT; - fn floor_test_verifier(enforce: bool) -> PaymentVerifier { + fn floor_test_verifier() -> PaymentVerifier { + floor_test_verifier_with_own_count(FLOOR_TEST_LOCAL_KEY_COUNT) + } + + /// A verifier whose own live commitment holds `own_key_count` keys, at the + /// default tolerance. + fn floor_test_verifier_with_own_count(own_key_count: u32) -> PaymentVerifier { let verifier = create_test_verifier(); verifier.attach_local_commitment_source(Arc::new(FixedFloorSource { - key_count: FLOOR_TEST_LOCAL_KEY_COUNT, + key_count: own_key_count, })); - verifier.set_price_floor_for_tests(PriceFloorConfig { - enforce, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - }); - let local_price = price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize); - assert!( - local_price > price_at_records(0) * Amount::from(2u64), - "floor tests need a local price above twice baseline, got {local_price}" - ); verifier } - /// The settled amount the floor requires: 3x the tolerated fraction of the - /// close group's median commitment-bound price. - fn group_floor_required_amount() -> Amount { - let tolerated = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) - / Amount::from(100u64); - tolerated * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER) + /// The tolerated fraction of the close group's median commitment-bound + /// price: a quote priced below this fails the floor. + fn tolerated_group_floor_price() -> Amount { + price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) + / Amount::from(100u64) } /// One cheap (baseline-priced) quote paid `settled`: the cheapest-of-K @@ -4557,23 +4566,34 @@ mod tests { /// /// Exactly [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] entries, so these tests /// sit on the gate rather than comfortably past it: drop one and the floor - /// must skip. Spread widely enough that the cheapest sits below the tolerated - /// fraction of the median (asserted in the rejection test), and kept in the - /// low thousands because each entry builds a real signed merkle commitment. + /// must skip. Shaped like a real close group: members measured in August + /// spanned about 1.4x in price around a fleet median near 4,000 keys, and + /// this one spans 1.9x in keys (2.9x in price), so the cheapest member is + /// shown to clear the floor with room to spare. Kept in the low thousands because each + /// entry builds a real signed merkle commitment. const GROUP_TEST_KEY_COUNTS: [u32; 15] = [ - 500, 800, 1_100, 1_400, 1_700, 2_000, 2_300, 2_600, 2_900, 3_200, 3_500, 3_800, 4_100, - 4_400, 4_700, + 3_000, 3_200, 3_400, 3_600, 3_800, 4_000, 4_200, 4_400, 4_600, 4_800, 5_000, 5_200, 5_400, + 5_600, 5_800, ]; /// The LOWER median of [`GROUP_TEST_KEY_COUNTS`] plus this node's own /// [`FLOOR_TEST_LOCAL_KEY_COUNT`]. /// - /// Sixteen samples, so the lower median is the eighth smallest: `2_600`. The + /// Sixteen samples, so the lower median is the eighth smallest: `4_400`. The /// receiver's own count is an extreme outlier at the top and does not move /// it, which is exactly the robustness a median buys. An even sample is /// deliberate — it is the case where upper and lower median differ, and the /// lower one is what production safety depends on. - const GROUP_TEST_MEDIAN_KEY_COUNT: u32 = 2_600; + const GROUP_TEST_MEDIAN_KEY_COUNT: u32 = 4_400; + + /// A quoter holding under a quarter of the group median: the near-empty + /// node a modified client pays instead of the median. Its price sits well + /// below the floor, while the cheapest real group member sits well above. + const GROUP_TEST_NEAR_EMPTY_KEY_COUNT: usize = 1_000; + + /// A quote that clears the 30% floor but sits below the 0.6x contest + /// threshold: about half the group median price. + const GROUP_TEST_NEAR_FLOOR_KEY_COUNT: usize = 2_800; /// Build a gossip commitment cache holding one fresh commitment per entry of /// `key_counts`, attach it, and return the peer ids so the caller can put @@ -4688,10 +4708,10 @@ mod tests { /// does not move it. #[tokio::test] async fn group_reference_price_is_the_close_group_median() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD0u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); verifier.set_paid_quote_k_closest_for_tests(neighbours); let (reference, sample) = verifier.group_reference_price(&xorname).await; @@ -4712,13 +4732,19 @@ mod tests { "a fully populated group must not report a skip reason" ); assert_eq!( - reference, + reference.group_median, price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize), - "the reference must be the group median price, not this node's own" + "the median must be taken over the whole group" ); - assert!( - reference < price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize), - "this node's own price is far above the median and must not be the reference" + assert_eq!( + reference.own, + price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize), + "this node's own price is the price of its own commitment" + ); + assert_eq!( + reference.price, reference.group_median, + "this node's own price is far above the median, so the cap does not bind \ + and the reference is the group median" ); } @@ -4731,10 +4757,10 @@ mod tests { /// of them on the fullest nodes. #[tokio::test] async fn price_floor_prices_against_the_group_median_not_the_receiver() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD1u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let median_price = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize); let own_price = price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize); @@ -4763,23 +4789,23 @@ mod tests { ); } - /// The attack the floor exists to stop: collect the neighbourhood's quotes, - /// settle only the cheapest one, submit it as a one-quote proof. + /// What the floor does NOT catch, stated as a test so nobody reads more + /// into it: settling the cheapest real member of the close group. Members + /// measured in August spanned about 1.4x in price, so the cheapest sits no + /// lower than about 0.7x the median, far above any tolerance that is safe + /// for honest traffic. #[tokio::test] - async fn price_floor_rejects_cheapest_of_group_payment() { - let verifier = floor_test_verifier(true); + async fn price_floor_admits_the_cheapest_member_of_a_realistic_group() { + let verifier = floor_test_verifier(); let xorname = [0xD2u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let cheapest = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0) as usize; let cheapest_price = price_at_records(cheapest); - let tolerated = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) - / Amount::from(100u64); assert!( - cheapest_price < tolerated, - "the cheapest group price must sit below the tolerated floor for this test to bite" + cheapest_price >= tolerated_group_floor_price(), + "the cheapest real member must clear the floor for this test to mean anything" ); let proof_bytes = group_floor_proof( @@ -4790,10 +4816,43 @@ mod tests { &neighbours, ); + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("shopping inside the close group is not what the floor stops"), + PaymentStatus::PaymentVerified + ); + } + + /// The underpayment the floor exists to stop: collect the neighbourhood's + /// quotes, settle only a near-empty quoter, submit it as a one-quote proof. + #[tokio::test] + async fn price_floor_rejects_a_near_empty_quoter() { + let verifier = floor_test_verifier(); + let xorname = [0xD7u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let near_empty_price = price_at_records(GROUP_TEST_NEAR_EMPTY_KEY_COUNT); + assert!( + near_empty_price < tolerated_group_floor_price(), + "the near-empty quote must sit below the tolerated floor for this test to bite" + ); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_NEAR_EMPTY_KEY_COUNT, + three_times(near_empty_price), + &neighbours, + ); + let err = verifier .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) .await - .expect_err("settling only the cheapest group quote must not clear the floor"); + .expect_err("settling a near-empty quoter must not clear the floor"); assert!( format!("{err}").contains("below the close-group price floor"), @@ -4801,20 +4860,359 @@ mod tests { ); } + /// The cap: a receiver below the group median prices against its own, + /// lower, price, so it never demands more than the tolerance of what it + /// would itself quote. The settlement here fails the median-only rule and + /// clears the capped one, so the cap is what decides it. + #[tokio::test] + async fn price_floor_caps_the_reference_at_this_nodes_own_price() { + let own_count = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0); + let verifier = floor_test_verifier_with_own_count(own_count); + let xorname = [0xD8u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours.clone()); + + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + let own_price = price_at_records(own_count as usize); + assert_eq!(reference.own, own_price); + assert!( + reference.group_median > own_price, + "this receiver must sit below its group's median for the cap to bind" + ); + assert_eq!( + reference.price, own_price, + "the reference must be capped at this node's own price" + ); + + // A quote priced between the tolerance of the capped reference and the + // tolerance of the median. + let quote_records = 1_200; + let quote_price = price_at_records(quote_records); + let tolerance = Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT); + assert!( + quote_price >= own_price * tolerance / Amount::from(100u64) + && quote_price < reference.group_median * tolerance / Amount::from(100u64), + "the quote must fail the median-only floor and clear the capped one" + ); + let proof_bytes = group_floor_proof( + &verifier, + xorname, + quote_records, + three_times(quote_price), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("a receiver must not refuse what clears the tolerance of its own price"), + PaymentStatus::PaymentVerified + ); + } + + /// A write paid close to the floor is contested: it clears the floor here, + /// but not the contest threshold, so a peer with a higher view of the + /// group may have refused it. + #[tokio::test] + async fn a_write_near_the_floor_is_contested() { + let verifier = floor_test_verifier(); + let xorname = [0xD9u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let quote_price = price_at_records(GROUP_TEST_NEAR_FLOOR_KEY_COUNT); + let contest_price = PaymentVerifier::contested_price(price_at_records( + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + )); + assert!( + quote_price >= tolerated_group_floor_price() && quote_price < contest_price, + "the quote must clear the floor but not the contest margin" + ); + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_NEAR_FLOOR_KEY_COUNT, + three_times(quote_price), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("a payment above the floor must be admitted"), + PaymentStatus::PaymentVerified + ); + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write inside the contest margin must not be used to judge peers" + ); + assert!( + verifier.verified_payment_contested(&xorname), + "the verified admission must be remembered as contested" + ); + + // A later admission from the payment cache reads that decision, so a + // proof attached to it, here one the floor would not contest, can + // neither set nor veto it. + let median_proof = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + let cached = verifier + .verify_payment( + &xorname, + Some(&median_proof), + VerificationContext::PaidListAdmission, + ) + .await + .expect("a cached payment is admitted"); + assert_eq!(cached, PaymentStatus::CachedAsVerified); + assert!(verifier.verified_payment_contested(&xorname)); + } + + /// A write paid at the group median is not contested: this node keeps + /// judging its peers on it exactly as before. + #[tokio::test] + async fn a_write_at_the_group_median_is_not_contested() { + let verifier = floor_test_verifier(); + let xorname = [0xE3u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("an honest median payment must clear the floor"), + PaymentStatus::PaymentVerified + ); + assert!( + !verifier.price_contested(&xorname, &proof_bytes).await, + "a payment at the group median must leave possession checks in place" + ); + assert!(!verifier.verified_payment_contested(&xorname)); + // A merkle or unparseable proof is never refused on price. + assert!(!verifier.price_contested(&xorname, &[0u8; 8]).await); + } + + /// The memory of contested admissions has its own capacity, so a payment + /// cache configured small cannot make it forget a decision before the key + /// is recorded. + #[test] + fn contested_admissions_outlive_a_small_payment_cache() { + let verifier = PaymentVerifier::new(PaymentVerifierConfig { + evm: EvmVerifierConfig::default(), + cache_capacity: 1, + close_group_size: CLOSE_GROUP_SIZE, + local_rewards_address: RewardsAddress::new([1u8; 20]), + }); + let keys = [[0xF1u8; 32], [0xF2u8; 32], [0xF3u8; 32]]; + for key in &keys { + verifier.note_contested_admission_for_tests(key); + } + assert!(keys + .iter() + .all(|key| verifier.verified_payment_contested(key))); + } + + /// A node with no floor wired (no gossip cache, or no routing view) refuses + /// nothing on price, so it judges its peers on every write as before. + #[tokio::test] + async fn a_node_with_no_floor_wired_contests_nothing() { + let xorname = [0xEDu8; 32]; + let (peer_id, quote) = make_signed_quote(xorname, price_at_records(0), 1); + let proof_bytes = serialize_proof(vec![(peer_id, quote)]); + + let verifier = create_test_verifier(); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + + verifier.attach_commitment_cache(Arc::new(tokio::sync::RwLock::new(HashMap::new()))); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + } + + /// The decision is taken from the proof against the current view when the + /// write is recorded (or, for a pointer, fanned out), so a write stored + /// after the group's prices rose is judged against the view peers will + /// price it against now. + #[tokio::test] + async fn the_contest_follows_the_current_view() { + let verifier = floor_test_verifier(); + let xorname = [0xE7u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + + // Every neighbour's commitment grows to twice the old median, as after + // heavy churn: the same proof is now close to the floor. + forge_key_counts( + &verifier, + Some(&neighbours), + GROUP_TEST_MEDIAN_KEY_COUNT * 2, + ) + .await; + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write that peers would now price near the floor must not be used to judge them" + ); + } + + /// The contest threshold is 60% of the UNCAPPED group median, to the atto: + /// a write paid at it is judged and one paid an atto below is not. Written + /// out rather than read from the constant, since moving it is an ADR + /// decision. + #[tokio::test] + async fn contest_threshold_is_sixty_percent_of_the_group_median() { + let verifier = floor_test_verifier(); + let xorname = [0xEBu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours); + let threshold = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) + * Amount::from(60u64) + / Amount::from(100u64); + let proof_at = |price: Amount| { + let (peer_id, quote) = make_signed_quote(xorname, price, 1); + serialize_proof(vec![(peer_id, quote)]) + }; + + assert!( + !verifier + .price_contested(&xorname, &proof_at(threshold)) + .await + ); + assert!( + verifier + .price_contested(&xorname, &proof_at(threshold - Amount::from(1u64))) + .await + ); + } + + /// A receiver whose own price caps its floor still contests against the + /// group median: a peer whose own price is higher may refuse what this + /// node's lower floor admitted. + #[tokio::test] + async fn contest_reads_the_uncapped_median_when_the_cap_binds() { + let verifier = floor_test_verifier_with_own_count(3_000); + let xorname = [0xECu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours); + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + assert!( + reference.price < reference.group_median, + "the cap must bind for this test to mean anything" + ); + + // Above the contest threshold of the capped reference, below that of + // the median. + let price = (PaymentVerifier::contested_price(reference.price) + + PaymentVerifier::contested_price(reference.group_median)) + / Amount::from(2u64); + let (peer_id, quote) = make_signed_quote(xorname, price, 1); + let proof_bytes = serialize_proof(vec![(peer_id, quote)]); + + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "the contest must be judged against the median, not the capped reference" + ); + } + + /// A pointer payment is floored against the close group of the pointer's + /// address, and its fan-out is contested the same way as a chunk's. + #[tokio::test] + async fn price_floor_floors_and_contests_a_pointer_state() { + let verifier = floor_test_verifier(); + let address = [0xE4u8; 32]; + let state_id = [0xE5u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let quote_price = price_at_records(GROUP_TEST_NEAR_FLOOR_KEY_COUNT); + let proof_bytes = group_floor_proof( + &verifier, + state_id, + GROUP_TEST_NEAR_FLOOR_KEY_COUNT, + three_times(quote_price), + &neighbours, + ); + + verifier + .verify_pointer_payment_in( + &address, + &state_id, + &proof_bytes, + VerificationContext::ClientPut, + ) + .await + .expect("a pointer payment above the floor must be admitted"); + + assert!( + verifier.price_contested(&address, &proof_bytes).await, + "a pointer write inside the contest margin must not be used to judge peers" + ); + + // And a near-empty quoter is refused for a pointer exactly as for a chunk. + let cheap_state = [0xE6u8; 32]; + let cheap_proof = group_floor_proof( + &verifier, + cheap_state, + GROUP_TEST_NEAR_EMPTY_KEY_COUNT, + three_times(price_at_records(GROUP_TEST_NEAR_EMPTY_KEY_COUNT)), + &neighbours, + ); + let err = verifier + .verify_pointer_payment_in( + &address, + &cheap_state, + &cheap_proof, + VerificationContext::ClientPut, + ) + .await + .expect_err("a near-empty quoter must not pay for a pointer state either"); + assert!(format!("{err}").contains("below the close-group price floor")); + } + /// The anti-manipulation property: the reference price is built from this /// node's own routing view and the neighbours' signed gossip, never from the /// bundle. Padding the bundle with cheap quotes (the median-padding shape) /// must not move the floor. #[tokio::test] async fn price_floor_reference_ignores_client_supplied_quotes() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD3u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); - // A bundle stuffed with baseline-priced quotes: the client's own median - // is now the floor price, which is exactly what it would want. - let cheapest = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0) as usize; + // A bundle stuffed with near-empty quotes: the client's own median is + // now the cheap price, which is exactly what it would want. + let cheapest = GROUP_TEST_NEAR_EMPTY_KEY_COUNT; let mut peer_quotes = Vec::new(); let mut k_closest: Vec<[u8; 32]> = neighbours.clone(); for seed in 1..=CLOSE_GROUP_SIZE { @@ -4852,7 +5250,7 @@ mod tests { /// counts (the obvious choice) would pass either way and prove nothing. #[tokio::test] async fn price_floor_skips_when_group_view_is_too_thin() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD4u8; 32]; let thin: Vec = GROUP_TEST_KEY_COUNTS .iter() @@ -4861,7 +5259,7 @@ mod tests { .take(PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS - 1) .collect(); assert_eq!(thin.len(), PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS - 1); - let neighbours = attach_group_commitments(&verifier, &thin, std::time::Duration::ZERO); + let neighbours = attach_group_commitments(&verifier, &thin, Duration::ZERO); // Deep underpayment that this sample, if it were used, would reject. let settled = three_times(price_at_records(0)); @@ -4869,7 +5267,7 @@ mod tests { assert!( settled < price_at_records(would_be_median as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) / Amount::from(100u64) * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER), "the thin sample must price ABOVE the settlement, so deleting the gate fails this test" @@ -4884,13 +5282,21 @@ mod tests { result.expect("a thin group view must skip the floor, not reject"), PaymentStatus::PaymentVerified ); + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write this node cannot price must not be used to judge its peers" + ); + assert!( + verifier.verified_payment_contested(&xorname), + "a write admitted unpriced is remembered as contested" + ); } /// Exactly ON the gate must evaluate, so the boundary is pinned from both /// sides alongside the one-short test above. #[tokio::test] async fn price_floor_evaluates_at_exactly_the_minimum_sample() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDAu8; 32]; assert_eq!( GROUP_TEST_KEY_COUNTS.len(), @@ -4898,7 +5304,7 @@ mod tests { "the fixture must sit exactly on the gate for this boundary test" ); let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let proof_bytes = group_floor_proof( &verifier, @@ -4915,7 +5321,8 @@ mod tests { assert!(format!("{err}").contains("below the close-group price floor")); } - /// A receiver with no live commitment of its own must skip. + /// A receiver with no live commitment of its own must skip, and leave its + /// peers unjudged on the write. /// /// Such a node prices its OWN quotes at baseline, so holding an incoming /// payment to its neighbours' higher median would reject settlements it @@ -4924,13 +5331,9 @@ mod tests { #[tokio::test] async fn price_floor_skips_when_this_node_has_no_commitment() { let verifier = create_test_verifier(); - verifier.set_price_floor_for_tests(PriceFloorConfig { - enforce: true, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - }); let xorname = [0xDBu8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let proof_bytes = group_floor_proof( &verifier, @@ -4948,6 +5351,9 @@ mod tests { result.expect("a receiver with no commitment of its own must skip the floor"), PaymentStatus::PaymentVerified ); + // It cannot price the write, so it cannot tell whether its peers + // refused it either, and must not judge them on it. + assert!(verifier.price_contested(&xorname, &proof_bytes).await); } /// A neighbour gossiping a count no honest node could hold must be dropped, @@ -4959,18 +5365,19 @@ mod tests { /// floor becomes a way to reject other people's settled payments. #[tokio::test] async fn price_floor_ignores_commitments_above_the_protocol_cap() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDCu8; 32]; // The honest builder refuses to construct an over-cap commitment, so // forge one the way a modified node would: build a valid commitment and // overwrite the signed count. Dropping it leaves the sample one short of // the gate, so the floor skips — that cardinality change is what this // test detects if the filter is removed. One over-cap value alone would - // NOT move a 16-sample median; inflating the reference that way needs - // about a quarter of the sample, which is the cardinality bound on - // `group_reference_price` rather than something this filter fixes. + // NOT move a 16-sample median; inflating the reference that way needs a + // majority of the sample at the default tolerance, which is the + // cardinality bound on `group_reference_price` rather than something + // this filter fixes. let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let over_cap = crate::replication::commitment::MAX_COMMITMENT_KEY_COUNT + 1; { let cache = verifier @@ -5012,22 +5419,29 @@ mod tests { ); } - /// Pins the other KNOWN LIMIT: four of fifteen OVERSTATING neighbours push - /// the reference up far enough to reject an honest payment sitting at the - /// true group median. + /// Pins the KNOWN LIMIT for a receiver whose own price sits far above its + /// group, where the cap does not bind: overstating neighbours that make up + /// a MAJORITY of the sample reject an honest payment at the true median. /// /// This is the griefing direction, and the more serious of the two: the /// attacker spends nothing and the victim's money is already settled and - /// unrefundable. Like its understating sibling this test asserts a weakness - /// deliberately. If a future change improves the bound it will fail and must - /// be re-derived rather than deleted. + /// unrefundable. On this realistic fixture 65% falls to seven liars. At 30% + /// the headroom covers every honest member of the group, so liars lift the + /// median only as far as the dearest honest member until they outnumber + /// the rest: seven still clear (pinned by + /// `group_median_resists_seven_overstated_commitments`), eight take the + /// median. A receiver at or below the payment's price is immune at any + /// count (pinned by + /// `capped_reference_cannot_be_griefed_by_any_number_of_overstated_commitments`). + /// This test asserts the weakness deliberately. If a future change improves + /// the bound it will fail and must be re-derived rather than deleted. #[tokio::test] - async fn group_median_is_griefed_by_four_overstated_commitments() { - let verifier = floor_test_verifier(true); + async fn group_median_alone_is_griefed_by_a_majority_of_overstated_commitments() { + let verifier = floor_test_verifier(); let xorname = [0xE0u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - forge_key_counts(&verifier, neighbours.get(..4), MAX_COMMITMENT_KEY_COUNT).await; + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, neighbours.get(..8), MAX_COMMITMENT_KEY_COUNT).await; // A completely honest payment, priced at the TRUE group median. let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); @@ -5043,34 +5457,38 @@ mod tests { .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) .await .expect_err( - "KNOWN LIMIT: four overstated neighbours reject an honest at-median payment. \ - If this now succeeds, the bound improved — re-derive it and update the docs.", + "KNOWN LIMIT: a majority of overstated neighbours rejects an honest at-median \ + payment at a receiver far above its group. If this now succeeds, the bound \ + improved — re-derive it and update the docs.", ); assert!(format!("{err}").contains("below the close-group price floor")); } - /// Pins the KNOWN LIMIT, so it is a fact in the test suite rather than a - /// claim in a comment: four of fifteen understating neighbours DISARM the - /// floor, admitting the cheapest-of-K underpayment it exists to reject. + /// The griefing direction below a majority: seven of fifteen neighbours + /// gossiping the protocol maximum must not make even a receiver far above + /// its group refuse an honest, already settled payment. /// - /// This test asserts a weakness on purpose. It is the concrete reason - /// enforcement stays off, and if a future change improves the bound this - /// test will fail and must be re-derived rather than quietly deleted. + /// They take the seven LOWEST slots, which is where a liar moves the + /// median most, and lift the lower median from `4_400` to `5_800`, the + /// dearest honest member. The honest payment still clears 30% of that by + /// 2.07x (1.55x at 40%). Forged by overwriting the signed count rather than + /// building real 1,000,000-leaf commitments, which cost about a second and + /// 200 MB each. #[tokio::test] - async fn group_median_is_disarmed_by_four_understated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xDFu8; 32]; - let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); - for slot in counts.iter_mut().rev().take(4) { - *slot = 1; - } - let neighbours = attach_group_commitments(&verifier, &counts, std::time::Duration::ZERO); + async fn group_median_resists_seven_overstated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xDDu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, neighbours.get(..7), MAX_COMMITMENT_KEY_COUNT).await; + // An honest payment at the TRUE group median must still be admitted. + let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); let proof_bytes = group_floor_proof( &verifier, xorname, - 0, - three_times(price_at_records(0)), + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + honest, &neighbours, ); @@ -5079,49 +5497,40 @@ mod tests { .await; assert_eq!( - result.expect( - "KNOWN LIMIT: four understated neighbours disarm the floor. If this now \ - rejects, the bound improved — re-derive it and update the docs." - ), + result.expect("a minority of overstated commitments must not reject honest payments"), PaymentStatus::PaymentVerified ); } - /// The griefing direction: a small minority gossiping OVERSTATED - /// commitments must not push the reference up far enough to make this node - /// reject an honest, already settled payment. - /// - /// This is the dangerous direction of a false commitment. Understating only - /// costs a missed underpayment; overstating destroys someone else's money, - /// and the attacker spends nothing to do it. + /// The cap's guarantee: no number of overstating neighbours can make a + /// receiver refuse a payment of `3x tolerance x` its own price. /// - /// The bound is real but partial, and worth stating precisely rather than - /// claiming immunity. Each liar shifts the order statistic up by one rank, - /// so what protects the payment is the tolerance headroom, not the median - /// itself. In this fixture three liars move the reference `2_600` -> - /// `3_500` and an at-median payment still clears, by only 1.9%; four move it - /// to `3_800` - /// and the same payment is REJECTED (pinned by - /// `group_median_is_griefed_by_four_overstated_commitments`). The griefing - /// bound is therefore about a quarter of the sample, the same as the - /// disarming one — NOT "roughly half", which was the incorrect claim this - /// policy originally rested on. - /// - /// Estimator coverage: this case fails under a 25th percentile, and its - /// disarming sibling fails under a mean and under the upper median, so the - /// pair pins the estimator jointly. Neither does so alone. + /// Every neighbour here claims the protocol maximum, so the group median is + /// the liars' value. A receiver that itself holds the group's true median + /// still admits the honest at-median payment, because its reference is its + /// own price. #[tokio::test] - async fn group_median_resists_three_overstated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xDDu8; 32]; - // Three of fifteen neighbours claim the protocol maximum. Forged by - // overwriting the signed count rather than building real 1,000,000-leaf - // commitments, which cost about a second and 200 MB each. + async fn capped_reference_cannot_be_griefed_by_any_number_of_overstated_commitments() { + let verifier = floor_test_verifier_with_own_count(GROUP_TEST_MEDIAN_KEY_COUNT); + let xorname = [0xE1u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - forge_key_counts(&verifier, neighbours.get(..3), MAX_COMMITMENT_KEY_COUNT).await; + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, Some(&neighbours), MAX_COMMITMENT_KEY_COUNT).await; + + verifier.set_paid_quote_k_closest_for_tests(neighbours.clone()); + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + assert_eq!( + reference.group_median, + price_at_records(MAX_COMMITMENT_KEY_COUNT as usize), + "every neighbour lied, so the median is the liars' value" + ); + assert_eq!( + reference.price, + price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize), + "the reference must be capped at this node's own price" + ); - // An honest payment at the TRUE group median must still be admitted. let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); let proof_bytes = group_floor_proof( &verifier, @@ -5136,7 +5545,85 @@ mod tests { .await; assert_eq!( - result.expect("a minority of overstated commitments must not reject honest payments"), + result.expect("lying neighbours must not lift a receiver above its own price"), + PaymentStatus::PaymentVerified + ); + } + + /// Understated commitments pull the reference DOWN and weaken the floor. + /// That costs a missed underpayment rather than anyone's money. + /// + /// Six of fifteen neighbours claiming almost nothing leave the floor + /// standing against a baseline quote. The placement matters and is easy to + /// get wrong: understating a peer that was already below the median + /// changes nothing, so a test that rewrites the CHEAPEST entries passes + /// while testing nothing. These rewrite the DEAREST ones, which is where an + /// attacker would sit. `1`, not `0`: a commitment over an empty key set + /// cannot be built at all, so one key is the least a peer can commit to. + #[tokio::test] + async fn group_median_resists_six_understated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xD6u8; 32]; + let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); + for slot in counts.iter_mut().rev().take(6) { + *slot = 1; + } + let neighbours = attach_group_commitments(&verifier, &counts, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + 0, + three_times(price_at_records(0)), + &neighbours, + ); + + let err = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await + .expect_err("a minority of understated commitments must not disarm the floor"); + + assert!( + format!("{err}").contains("below the close-group price floor"), + "Error should name the group price floor: {err}" + ); + } + + /// Pins the KNOWN LIMIT, so it is a fact in the test suite rather than a + /// claim in a comment: seven understating neighbours of fifteen DISARM the + /// floor, admitting the baseline payment it exists to reject. The lower + /// median then falls to the cheapest honest member (3,000 keys), and 30% of + /// that price is below a baseline quote. At 40% it took eight. + /// + /// This test asserts a weakness on purpose. If a future change improves the + /// bound it will fail and must be re-derived rather than quietly deleted. + #[tokio::test] + async fn group_median_is_disarmed_by_seven_understated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xDFu8; 32]; + let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); + for slot in counts.iter_mut().rev().take(7) { + *slot = 1; + } + let neighbours = attach_group_commitments(&verifier, &counts, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + 0, + three_times(price_at_records(0)), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect( + "KNOWN LIMIT: seven understated neighbours of fifteen disarm the floor. If \ + this now rejects, the bound improved: re-derive it and update the docs." + ), PaymentStatus::PaymentVerified ); } @@ -5149,7 +5636,7 @@ mod tests { /// floor skips instead of pricing against this node's own (huge) count. #[tokio::test] async fn price_floor_ignores_stale_group_commitments() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD5u8; 32]; let stale_age = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL + std::time::Duration::from_mins(1); @@ -5190,14 +5677,13 @@ mod tests { async fn price_floor_excludes_only_the_stale_entries_from_a_mixed_cache() { use crate::replication::commitment_state::PeerCommitmentRecord; - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDEu8; 32]; let stale_age = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL + std::time::Duration::from_mins(1); // Fresh: the full fixture, exactly on the gate. - let fresh = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + let fresh = attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); // Stale: four cheap counts, keeping the total within the 19 neighbours a // real routing view can supply. @@ -5236,83 +5722,19 @@ mod tests { "stale entries must not count toward the gate" ); assert_eq!( - reference, + reference.map(|reference| reference.group_median), Some(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), "the reference must be the fresh-only median; counting the stale cheap \ entries would drag it below" ); } - /// Understated commitments pull the reference DOWN and weaken the floor. - /// - /// Three adversarially-placed liars leave it standing; four disarm it. The - /// placement matters and is easy to get wrong: understating a peer that was - /// already below the median changes nothing, so a test that rewrites the - /// CHEAPEST entries passes while testing nothing. These rewrite the DEAREST - /// ones, which is where an attacker would sit, and the count is set to the - /// largest that still holds rather than to an aspirational majority. - #[tokio::test] - async fn group_median_resists_three_understated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xD6u8; 32]; - // Three of fifteen neighbours claim to store almost nothing. `1`, not - // `0`: a commitment over an empty key set cannot be built at all, so one - // key is the least a peer can honestly (or dishonestly) commit to. - let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); - for slot in counts.iter_mut().rev().take(3) { - *slot = 1; - } - let neighbours = attach_group_commitments(&verifier, &counts, std::time::Duration::ZERO); - - let cheapest_price = price_at_records(0); - let proof_bytes = group_floor_proof( - &verifier, - xorname, - 0, - three_times(cheapest_price), - &neighbours, - ); - - let err = verifier - .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await - .expect_err("a minority of understated commitments must not disarm the floor"); - - assert!( - format!("{err}").contains("below the close-group price floor"), - "Error should name the group price floor: {err}" - ); - } - - #[tokio::test] - async fn test_price_floor_shadow_mode_accepts_below_floor_settlement() { - let verifier = floor_test_verifier(false); - let xorname = [0xC1u8; 32]; - let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - let settled = three_times(price_at_records(0)); - assert!( - settled < group_floor_required_amount(), - "shadow mode is only meaningful on a settlement that WOULD be rejected" - ); - let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); - - let result = verifier - .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await; - - assert_eq!( - result.expect("shadow mode must never reject on the floor"), - PaymentStatus::PaymentVerified - ); - } - #[tokio::test] async fn test_price_floor_enforced_rejects_cheapest_of_k_settlement() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC2u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); @@ -5327,30 +5749,44 @@ mod tests { ); } + /// An honest client may OVERPAY a cheap quote to clear stricter receivers: + /// the floor compares the settled amount, not the quote price, and + /// exactly-at-floor must pass. The floor is 30% of the reference (ADR-0006 + /// Amendment 2), written out here rather than read from the constant, so + /// moving the tolerance either way, an ADR decision, fails here first. #[tokio::test] async fn test_price_floor_enforced_accepts_exactly_at_floor_overpayment() { - // An honest client may OVERPAY a cheap quote to clear stricter - // receivers: the floor compares the settled amount, not the quote - // price, and exactly-at-floor must pass. - let verifier = floor_test_verifier(true); - let xorname = [0xC3u8; 32]; + assert_eq!(PRICE_FLOOR_TOLERANCE_PERCENT, 30); + let verifier = floor_test_verifier(); let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - let proof_bytes = group_floor_proof( - &verifier, - xorname, - 0, - group_floor_required_amount(), - &neighbours, - ); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + let median_price = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize); + let at_thirty_percent = + three_times(median_price * Amount::from(30u64) / Amount::from(100u64)); - let result = verifier + let xorname = [0xC3u8; 32]; + let proof_bytes = group_floor_proof(&verifier, xorname, 0, at_thirty_percent, &neighbours); + let status = verifier .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await; + .await + .expect("exactly 3x 30% of the group median must clear the floor"); + assert_eq!(status, PaymentStatus::PaymentVerified); - assert_eq!( - result.expect("an exactly-at-floor settlement must pass"), - PaymentStatus::PaymentVerified + let xorname = [0xCAu8; 32]; + let one_short = at_thirty_percent - Amount::from(1u64); + let proof_bytes = group_floor_proof(&verifier, xorname, 0, one_short, &neighbours); + let err = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await + .expect_err("one atto under 3x 30% of the group median must not clear the floor"); + assert!( + err.to_string() + .contains("below the close-group price floor"), + "{err}" + ); + assert!( + !verifier.verified_payment_contested(&xorname), + "a refused payment is never remembered as an admitted contested one" ); } @@ -5359,10 +5795,10 @@ mod tests { // Fresh replication is the same fresh economic event as a direct PUT; // exempting it would let one cheap accepting node fan the proof out // around every other storer's floor. - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC4u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); @@ -5381,26 +5817,31 @@ mod tests { ); } + /// Paid-list admission is floored like a store. A `PaidNotify` only ever + /// announces a write that was just paid for, and a paid-list vote is what + /// lets repair copy the data later, so exempting it would let one + /// accepting node replicate a below-floor write to its whole group. #[tokio::test] - async fn test_price_floor_never_reprices_paid_list_admission() { - let verifier = floor_test_verifier(true); + async fn test_price_floor_enforced_applies_to_paid_list_admission() { + let verifier = floor_test_verifier(); let xorname = [0xC5u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); - let result = verifier + let err = verifier .verify_payment( &xorname, Some(&proof_bytes), VerificationContext::PaidListAdmission, ) - .await; + .await + .expect_err("a below-floor proof must not earn a paid-list vote"); - assert_eq!( - result.expect("paid-list admission reprices no fresh economic decision"), - PaymentStatus::PaymentVerified + assert!( + format!("{err}").contains("below the close-group price floor"), + "Error should name the price floor: {err}" ); } @@ -5414,13 +5855,13 @@ mod tests { /// nothing. #[tokio::test] async fn test_price_floor_without_any_group_view_skips() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC6u8; 32]; let settled = three_times(price_at_records(0)); assert!( settled < price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) / Amount::from(100u64) * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER), "the old own-price rule would have rejected this settlement, so the skip is load-bearing" @@ -5435,42 +5876,9 @@ mod tests { result.expect("no group view must skip the floor, never reject"), PaymentStatus::PaymentVerified ); - } - - /// `from_env` must fail CLOSED: a present-but-invalid tolerance can never - /// leave enforcement on with a tolerance the operator did not specify. - /// (This test owns the `PRICE_FLOOR_*` env vars — no other test reads them.) - #[test] - fn price_floor_from_env_fails_closed_on_invalid_tolerance() { - // Defaults with nothing set. - std::env::remove_var(PRICE_FLOOR_ENFORCE_ENV); - std::env::remove_var(PRICE_FLOOR_TOLERANCE_ENV); - let cfg = PriceFloorConfig::from_env(); - assert!(!cfg.enforce); - assert_eq!(cfg.tolerance_percent, PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT); - - // Enforce on, valid tolerance: honoured. - std::env::set_var(PRICE_FLOOR_ENFORCE_ENV, "1"); - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "80"); - let cfg = PriceFloorConfig::from_env(); - assert!(cfg.enforce); - assert_eq!(cfg.tolerance_percent, 80); - - // Enforce on, out-of-range tolerance: enforcement DISABLED (fail closed). - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "150"); - let cfg = PriceFloorConfig::from_env(); - assert!( - !cfg.enforce, - "an out-of-range tolerance must not silently enforce a default" - ); - - // Enforce on, unparseable tolerance: also disabled. - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "loose"); - let cfg = PriceFloorConfig::from_env(); - assert!(!cfg.enforce); - - std::env::remove_var(PRICE_FLOOR_ENFORCE_ENV); - std::env::remove_var(PRICE_FLOOR_TOLERANCE_ENV); + // No cache means no floor is wired here at all, and so nothing to + // contest: possession checks stay as they were. + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); } #[tokio::test] @@ -6988,7 +7396,6 @@ mod tests { cache_capacity: 100, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([1u8; 20]), - price_floor: PriceFloorConfig::default(), }; let verifier = PaymentVerifier::new(config); diff --git a/src/pointer/service.rs b/src/pointer/service.rs index 7d4c866f..8af79e91 100644 --- a/src/pointer/service.rs +++ b/src/pointer/service.rs @@ -561,7 +561,7 @@ fn cross_kind_refusal(address: XorName, chunk_present: Result) -> Option

, paid_list: &PaidList) -> Vec { + sampled + .into_iter() + .filter(|key| !paid_list.counts_as_price_contested(key)) + .collect() +} + // --------------------------------------------------------------------------- // Main audit tick // --------------------------------------------------------------------------- @@ -107,7 +120,7 @@ pub(crate) fn responsible_audit_response_timeout( /// /// This is the production path used by the replication engine. Direct /// callers that have not adopted repair proofs remain conservative and do not -/// audit peers for unproven keys. +/// audit peers for unproven keys, nor for price-contested ones (ADR-0006). #[allow( clippy::implicit_hasher, clippy::too_many_arguments, @@ -116,6 +129,7 @@ pub(crate) fn responsible_audit_response_timeout( pub async fn audit_tick_with_repair_proofs( p2p_node: &Arc, storage: &Arc, + paid_list: &PaidList, config: &ReplicationConfig, sync_history: &HashMap, repair_proofs: &Arc>, @@ -172,6 +186,7 @@ pub async fn audit_tick_with_repair_proofs( .copied() .collect() }; + let sampled_keys = keys_open_to_audit(sampled_keys, paid_list); // Step 4: Filter to keys where the chosen peer is in the close group and // this node has proof that it already sent the peer a repair hint for the @@ -901,6 +916,44 @@ mod tests { /// incoming audit limit (`2 * sqrt(N)`) never rejects small test challenges. const TEST_STORED_CHUNKS: usize = 1_000_000; + /// ADR-0006: the responsible audit never samples a key marked + /// price-contested, and samples every other key as before. + #[tokio::test] + async fn price_contested_keys_are_not_audited() { + let dir = TempDir::new().expect("temp dir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain: XorName = [0x71; 32]; + let contested: XorName = [0x72; 32]; + let unpaid: XorName = [0x73; 32]; + paid_list.insert(&plain).await.expect("insert"); + paid_list + .insert_price_contested(&contested) + .await + .expect("mark"); + + assert_eq!( + keys_open_to_audit(vec![plain, contested, unpaid], &paid_list), + vec![plain, unpaid] + ); + } + + /// ADR-0006: while the paid list cannot be read, the responsible audit + /// samples no key, since any of them may be marked. + #[tokio::test] + async fn an_unreadable_mark_keeps_a_key_out_of_the_audit() { + let dir = TempDir::new().expect("temp dir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain: XorName = [0x74; 32]; + let unpaid: XorName = [0x75; 32]; + paid_list.insert(&plain).await.expect("insert"); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert_eq!( + keys_open_to_audit(vec![plain, unpaid], &paid_list), + Vec::::new() + ); + } + #[test] fn first_challenged_key_label_truncates_to_16_hex_chars() { let mut key = [0u8; 32]; diff --git a/src/replication/fresh.rs b/src/replication/fresh.rs index af656cb2..6de7174c 100644 --- a/src/replication/fresh.rs +++ b/src/replication/fresh.rs @@ -11,7 +11,7 @@ use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; -use crate::logging::{debug, warn}; +use crate::logging::{debug, info, warn}; use bytes::Bytes; use rand::Rng; use saorsa_core::identity::PeerId; @@ -19,6 +19,7 @@ use saorsa_core::P2PNode; use tokio::sync::{mpsc, OwnedSemaphorePermit, Semaphore}; use crate::ant_protocol::XorName; +use crate::payment::PaymentVerifier; use crate::replication::config::{ ReplicationConfig, FRESH_REPLICATION_DELIVERY_MAX_RETRIES, REPLICATION_PROTOCOL_ID, }; @@ -48,6 +49,9 @@ pub struct FreshWriteEvent { pub(crate) struct FreshOfferEvent { pub(crate) key: XorName, pub(crate) payment_proof: Vec, + /// The fan-out must not judge peers on this write (see + /// [`announce_paid_write`]). + pub(crate) contested: bool, /// Storage read-backs attempted so far; see `MAX_FRESH_READ_ATTEMPTS`. pub(crate) read_attempts: u32, } @@ -84,8 +88,48 @@ impl AsRef<[u8]> for EncodedOffer { } } -/// Rules 6-8: record the paid key locally and announce it to -/// `PaidCloseGroup(K)`. +/// Rule 6: a node that validates `PoP` adds K to `PaidForList(self)`, marked +/// when `mark`. +/// +/// `mark` is the decision this node took when it verified the payment +/// (`PaymentVerifier::verified_payment_contested`), so a write +/// admitted from the payment cache is recorded by that decision, never by the +/// proof attached to it, which nobody checked. A contested write is marked +/// (ADR-0006) because a close-group peer may have refused it under its own +/// floor, and refusing a below-floor payment is network policy, never +/// misbehaviour. While this node holds the key, the mark keeps it from judging +/// its peers on it: the key goes out in paid hints only, so no repair proof is +/// recorded and no responsible audit samples it, and a prune audit of it +/// scores no one. Every other check that reads the mark points here. +/// +/// The first record of a key decides its mark: an existing entry is never +/// changed, so a later payment can neither add nor remove it. Returns whether +/// `mark` or the key's record is marked once this call is done, whichever +/// record won: ours, or one a concurrent `PaidNotify` or fresh offer wrote +/// first. A mark that cannot be read counts as set (see +/// `PaidList::counts_as_price_contested`). +pub(crate) async fn record_paid_key(key: &XorName, mark: bool, paid_list: &PaidList) -> bool { + let recorded = if mark { + paid_list.insert_price_contested(key).await + } else { + paid_list.insert(key).await + }; + if let Err(e) = recorded { + warn!("Failed to add key {} to PaidForList: {e}", hex::encode(key)); + } + mark || paid_list.counts_as_price_contested(key) +} + +/// Rules 6-8: record the paid key locally (see [`record_paid_key`]) and +/// announce it to `PaidCloseGroup(K)`. Returns whether the write's fan-out +/// must leave its peers unjudged (ADR-0006). +/// +/// The record is marked by the decision this node took when it verified the +/// payment. The fan-out is left unjudged when that decision or the key's +/// record is marked once recorded, whichever record won, so a marked write is +/// never judged, and also when the proof being forwarded is contested: a +/// cached PUT forwards it unverified, and its peers verify it and may refuse +/// it. /// /// This is the evidence peers need to repair the key later, so it runs the /// moment a write is accepted and is never gated by the pending-offer permit @@ -93,20 +137,28 @@ impl AsRef<[u8]> for EncodedOffer { pub(crate) async fn announce_paid_write( key: &XorName, proof_of_payment: &[u8], + payment_verifier: &PaymentVerifier, paid_list: &PaidList, p2p_node: &Arc, config: &ReplicationConfig, -) { - // Rule 6: Node that validates PoP adds K to PaidForList(self). - if let Err(e) = paid_list.insert(key).await { - warn!("Failed to add key {} to PaidForList: {e}", hex::encode(key)); - } +) -> bool { + let marked = record_paid_key( + key, + payment_verifier.verified_payment_contested(key), + paid_list, + ) + .await; // Rules 7-8: PaidNotify to every member of PaidCloseGroup(K). send_paid_notify(key, proof_of_payment, p2p_node, config).await; + marked + || payment_verifier + .price_contested(key, proof_of_payment) + .await } /// Rules 2-3: send fresh offers to the close group and schedule the delayed -/// possession check (ADR-0003) for the responsible peers. +/// possession check (ADR-0003) for the responsible peers, unless `contested` +/// (see [`announce_paid_write`]). /// /// `pending_offer` is the caller's permit from the pending-offer semaphore; /// it is held with the encoded offer until the last handle to it is dropped. @@ -117,6 +169,7 @@ pub(crate) async fn send_fresh_offers( key: &XorName, data: Vec, proof_of_payment: Vec, + contested: bool, pending_offer: OwnedSemaphorePermit, ) { let self_id = *ctx.p2p_node.peer_id(); @@ -210,12 +263,21 @@ pub(crate) async fn send_fresh_offers( // Schedule the delayed possession check (ADR-0003) for the responsible // close-group peers. A closed receiver (engine shutting down) is ignored. + // Not for a contested write: a peer may have refused it (ADR-0006). if !target_peers.is_empty() { ctx.dispatched.fetch_add(1, Ordering::Relaxed); - let _ = ctx.possession_check_tx.send(PossessionCheckEvent { - key: *key, - peers: target_peers, - }); + if contested { + info!( + "Fresh replication of {}: no possession check, the payment is contested \ + under the price floor", + hex::encode(key) + ); + } else { + let _ = ctx.possession_check_tx.send(PossessionCheckEvent { + key: *key, + peers: target_peers, + }); + } } } @@ -275,6 +337,52 @@ async fn send_paid_notify( mod tests { use super::*; + /// ADR-0006: a contested payment marks a newly recorded key, an + /// uncontested one records it plainly, and the first record of a key + /// decides: a later record, either way, changes nothing. Each call + /// reports the winning record's mark, or its own when that is set, so a + /// write whose plain decision lost to a marked record is still left + /// unjudged. + #[tokio::test] + async fn the_first_record_of_a_key_decides_its_mark() { + let dir = tempfile::tempdir().expect("tempdir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let contested = [0x91u8; 32]; + let plain = [0x92u8; 32]; + + let reported = vec![ + record_paid_key(&contested, true, &paid_list).await, + record_paid_key(&plain, false, &paid_list).await, + record_paid_key(&contested, false, &paid_list).await, + record_paid_key(&plain, true, &paid_list).await, + ]; + + let marks: Vec<(bool, bool)> = [contested, plain] + .iter() + .map(|key| { + ( + paid_list.contains(key).expect("contains"), + paid_list.is_price_contested(key).expect("read mark"), + ) + }) + .collect(); + assert_eq!(marks, vec![(true, true), (true, false)]); + assert_eq!(reported, vec![true, false, true, true]); + } + + /// ADR-0006: a plain write is left unjudged while the paid list cannot be + /// read, since the record that won may be marked. + #[tokio::test] + async fn a_write_whose_mark_cannot_be_read_is_left_unjudged() { + let dir = tempfile::tempdir().expect("tempdir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain = [0x93u8; 32]; + assert!(!record_paid_key(&plain, false, &paid_list).await); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(record_paid_key(&plain, false, &paid_list).await); + } + /// The permit is released with the last handle to the encoded offer, not /// the first: a slice the transport still holds keeps the offer counted. #[test] diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 62937413..2845c847 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -2487,11 +2487,14 @@ impl ReplicationEngine { /// /// Unlike the PUT path, this waits for a pending-offer permit before /// returning, and it offers the caller's bytes rather than reading them - /// back from storage. + /// back from storage. Test-only: production writes enter through the PUT + /// handler's fresh-write channel. + #[cfg(any(test, feature = "test-utils"))] pub async fn replicate_fresh(&self, key: &XorName, data: &[u8], proof_of_payment: &[u8]) { - fresh::announce_paid_write( + let contested = fresh::announce_paid_write( key, proof_of_payment, + &self.payment_verifier, &self.paid_list, &self.p2p_node, &self.config, @@ -2509,6 +2512,7 @@ impl ReplicationEngine { key, data.to_vec(), proof_of_payment.to_vec(), + contested, pending_offer, ) .await; @@ -2550,6 +2554,7 @@ impl ReplicationEngine { ) { let p2p = Arc::clone(&self.p2p_node); let paid_list = Arc::clone(&self.paid_list); + let payment_verifier = Arc::clone(&self.payment_verifier); let config = Arc::clone(&self.config); let shutdown = self.shutdown.clone(); @@ -2565,9 +2570,10 @@ impl ReplicationEngine { // Stage one never waits for chunk back-pressure: the paid-list // entry and PaidNotify are what let peers repair the key later, // so every queued write gets them at arrival rate. - fresh::announce_paid_write( + let contested = fresh::announce_paid_write( &event.key, &event.payment_proof, + &payment_verifier, &paid_list, &p2p, &config, @@ -2577,6 +2583,7 @@ impl ReplicationEngine { .send(fresh::FreshOfferEvent { key: event.key, payment_proof: event.payment_proof, + contested, read_attempts: 0, }) .is_err() @@ -2689,6 +2696,7 @@ impl ReplicationEngine { &event.key, data, event.payment_proof, + event.contested, pending_offer, ) .await; @@ -3539,6 +3547,7 @@ impl ReplicationEngine { fn start_audit_loop(&mut self) { let p2p = Arc::clone(&self.p2p_node); let storage = Arc::clone(&self.storage); + let paid_list = Arc::clone(&self.paid_list); let config = Arc::clone(&self.config); let shutdown = self.shutdown.clone(); let sync_history = Arc::clone(&self.sync_history); @@ -3573,6 +3582,7 @@ impl ReplicationEngine { audit::audit_tick_with_repair_proofs( &p2p, &storage, + &paid_list, &config, &history, &repair_proofs, @@ -3598,6 +3608,7 @@ impl ReplicationEngine { audit::audit_tick_with_repair_proofs( &p2p, &storage, + &paid_list, &config, &history, &repair_proofs, @@ -6602,10 +6613,15 @@ async fn handle_fresh_offer( } } - // Rule 6: add to PaidForList. - if let Err(e) = ctx.paid_list.insert(key).await { - warn!("Failed to add key to PaidForList: {e}"); - } + // Rule 6: add to PaidForList, marked if the price floor contested the + // payment when this node verified it, before the record is stored and + // can be hinted. + fresh::record_paid_key( + key, + ctx.payment_verifier.verified_payment_contested(key), + &ctx.paid_list, + ) + .await; // Store the record. match ctx.storage.put(key, data).await { @@ -6811,9 +6827,15 @@ async fn handle_paid_notify( } } - if let Err(e) = paid_list.insert(¬ify.key).await { - warn!("Failed to add paid notify key to PaidForList: {e}"); - } + // Marked if the price floor contested the payment when this node verified + // it, so a record later repaired onto this node is not used to judge its + // peers either. + fresh::record_paid_key( + ¬ify.key, + payment_verifier.verified_payment_contested(¬ify.key), + paid_list, + ) + .await; Ok(()) } diff --git a/src/replication/neighbor_sync.rs b/src/replication/neighbor_sync.rs index b1f0d859..d035f32f 100644 --- a/src/replication/neighbor_sync.rs +++ b/src/replication/neighbor_sync.rs @@ -13,6 +13,7 @@ use saorsa_core::identity::PeerId; use saorsa_core::P2PNode; use crate::ant_protocol::XorName; +use crate::error::Result; use crate::replication::config::{ReplicationConfig, REPLICATION_PROTOCOL_ID}; use crate::replication::paid_list::PaidList; use crate::replication::protocol::{ @@ -61,27 +62,50 @@ pub(crate) struct PeerSyncHints { /// This intentionally scans all locally stored records, not only records for /// which this node is still responsible. Out-of-range records retained by /// prune hysteresis therefore continue to repair their new close group before -/// this node is allowed to delete them. +/// this node is allowed to delete them. Price-contested records are left out +/// (ADR-0006). pub async fn build_replica_hints_for_peer( peer: &PeerId, storage: &Arc, + paid_list: &PaidList, p2p_node: &Arc, close_group_size: usize, ) -> Vec { - build_replica_hints_for_peer_with_close_groups(peer, storage, p2p_node, close_group_size) - .await - .into_iter() - .map(|hint| hint.key) - .collect() + build_replica_hints_for_peer_with_close_groups( + peer, + storage, + paid_list, + p2p_node, + close_group_size, + ) + .await + .into_iter() + .map(|hint| hint.key) + .collect() +} + +/// Stored keys this node may hint as replicas: every stored key except those +/// marked price-contested in the paid list (see `fresh::record_paid_key`), +/// which go out as paid hints only. A replica hint records a repair proof the +/// responsible audit later judges the peer on, and makes this node the sole +/// hinter a peer penalises when its quorum check fails; a paid hint does +/// neither. The marks are read from memory, so this scans nothing on disk; if +/// they could not be read when the paid list opened, no key is hinted as a +/// replica. +async fn replica_hint_keys(storage: &ChunkStore, paid_list: &PaidList) -> Result> { + let mut keys = storage.all_keys().await?; + paid_list.retain_unmarked(&mut keys); + Ok(keys) } pub(crate) async fn build_replica_hints_for_peer_with_close_groups( peer: &PeerId, storage: &Arc, + paid_list: &PaidList, p2p_node: &Arc, close_group_size: usize, ) -> Vec { - let all_keys = match storage.all_keys().await { + let all_keys = match replica_hint_keys(storage, paid_list).await { Ok(keys) => keys, Err(e) => { warn!("Failed to read stored keys for hint construction: {e}"); @@ -125,7 +149,7 @@ pub(crate) async fn build_sync_hints_for_peers( return hints_by_peer; } - let all_keys = match storage.all_keys().await { + let all_keys = match replica_hint_keys(storage, paid_list).await { Ok(keys) => keys, Err(e) => { warn!("Failed to read stored keys for batch hint construction: {e}"); @@ -521,6 +545,7 @@ pub(crate) async fn handle_sync_request_with_proofs( let sent_replica_hints = build_replica_hints_for_peer_with_close_groups( sender, storage, + paid_list, p2p_node, config.close_group_size, ) @@ -558,6 +583,7 @@ pub(crate) async fn handle_sync_request_with_proofs( mod tests { use super::*; use crate::replication::types::PeerSyncRecord; + use crate::storage::ChunkStoreConfig; use std::collections::HashMap; /// Build a `PeerId` from a single byte (zero-padded to 32 bytes). @@ -567,6 +593,44 @@ mod tests { PeerId::from_bytes(bytes) } + // -- replica_hint_keys --------------------------------------------------- + + /// ADR-0006: a stored key marked price-contested is never hinted as a + /// replica; every other stored key still is. The filter reads no paid-list + /// database, so it works while this thread cannot read one. + #[tokio::test] + async fn price_contested_keys_are_not_hinted_as_replicas() { + let dir = tempfile::tempdir().expect("tempdir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().join("chunks"), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let contents: [&[u8]; 3] = [b"plain record", b"contested record", b"unpaid record"]; + let [plain, contested, unpaid] = contents.map(ChunkStore::compute_address); + for (key, content) in [plain, contested, unpaid].iter().zip(contents) { + storage.put(key, content).await.expect("put"); + } + paid_list.insert(&plain).await.expect("insert"); + paid_list + .insert_price_contested(&contested) + .await + .expect("mark"); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(paid_list.is_price_contested(&contested).is_err()); + let mut hinted = replica_hint_keys(&storage, &paid_list) + .await + .expect("read keys"); + hinted.sort_unstable(); + let mut expected = vec![plain, unpaid]; + expected.sort_unstable(); + + assert_eq!(hinted, expected); + } + // -- select_sync_batch --------------------------------------------------- #[test] diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index 26d01210..d196cd52 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -11,7 +11,11 @@ //! ``` //! //! One unnamed database stores set membership: key = 32-byte `XorName`, -//! value = empty byte slice. +//! value = empty byte slice, or one marker byte for a key whose +//! payment the price floor contested here (ADR-0006). Releases before that +//! marker only ever test presence, so they read a marked key as paid. The +//! marked keys are also held in memory, so hint construction never scans the +//! database for them. //! //! ## Out-of-range timestamps //! @@ -22,24 +26,55 @@ use crate::ant_protocol::XorName; use crate::error::{Error, Result}; -use crate::logging::{debug, trace, warn}; +use crate::logging::{debug, info, trace, warn}; use heed::types::Bytes; -use heed::{Database, Env, EnvOpenOptions}; -use parking_lot::RwLock; -use std::collections::HashMap; +#[cfg(test)] +use heed::WithTls; +use heed::{Database, Env, EnvOpenOptions, RoTxn}; +use parking_lot::{Mutex, RwLock}; +use std::collections::{HashMap, HashSet}; use std::path::Path; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; use std::time::Instant; use tokio::task::spawn_blocking; use tokio_util::task::TaskTracker; use crate::ant_protocol::XORNAME_LEN; +/// Value stored for a key whose payment the price floor contested here (see +/// `fresh::record_paid_key`). +const PRICE_CONTESTED_VALUE: &[u8] = &[1]; + /// Default LMDB map size for the paid list: 256 MiB. /// -/// The paid list stores only 32-byte keys with empty values, so this is -/// generous even for very large close-group memberships. +/// The paid list stores only 32-byte keys with empty or one-byte values, so +/// this is generous even for very large close-group memberships. const DEFAULT_MAP_SIZE: usize = 256 * 1_024 * 1_024; +/// What the open task returns: the environment, the database and the keys +/// marked price-contested in it, if they could be read. +type OpenedPaidList = (Env, Database, Result>); + +/// Every key whose entry in `db` carries the price-contested mark. +fn read_price_contested_keys(db: &Database, txn: &RoTxn) -> Result> { + let mut keys = HashSet::new(); + for result in db + .iter(txn) + .map_err(|e| Error::Storage(format!("Failed to iterate paid-list: {e}")))? + { + let (key_bytes, value) = + result.map_err(|e| Error::Storage(format!("Failed to read paid-list entry: {e}")))?; + if value.is_empty() { + continue; + } + if let Ok(key) = XorName::try_from(key_bytes) { + keys.insert(key); + } + } + Ok(keys) +} + /// Persistent paid-for-list backed by LMDB. /// /// Tracks which keys this node believes are paid-authorized. @@ -47,8 +82,26 @@ const DEFAULT_MAP_SIZE: usize = 256 * 1_024 * 1_024; pub struct PaidList { /// LMDB environment. env: Env, - /// The unnamed default database (key = `XorName` bytes, value = empty). + /// The unnamed default database (key = `XorName` bytes, value = empty, or + /// the price-contested marker). db: Database, + /// In-memory copy of the keys whose entry carries the price-contested + /// mark, or `None` if they could not be read on open. Changed only by the + /// blocking task that writes the database, under `write_order`, so a + /// cancelled caller cannot leave it stale. Readers clone the `Arc` and + /// filter outside the lock; a write while such a snapshot is alive + /// copies the set first (`Arc::make_mut`). + price_contested: Arc>>>>, + /// Held by every database write from before its write transaction until + /// it has updated `price_contested`, so the copy changes in commit order. + /// A new mark goes in just before its commit and a removed one just after, + /// so the copy never lacks a committed mark. A mark whose commit fails + /// stays, since that commit may still have reached the disk: when in + /// doubt, judge no one. + write_order: Arc>, + /// Mark reads that have failed since the last one that succeeded, so a + /// run of failures is logged once (see `counts_as_price_contested`). + failed_mark_reads: AtomicU64, /// In-memory: when each paid key first went out of `PaidCloseGroup` range. /// Cleared on restart (safe: hysteresis clock restarts from zero). paid_out_of_range: RwLock>, @@ -85,7 +138,7 @@ impl PaidList { // Constructor-only blocking task: it runs before `self` (and its // `blocking_tracker`) exists, so it is deliberately untracked. The // constructor awaits it right here, so it cannot outlive this call. - let (env, db) = spawn_blocking(move || -> Result<(Env, Database)> { + let (env, db, price_contested) = spawn_blocking(move || -> Result { // SAFETY: `EnvOpenOptions::open()` is unsafe because LMDB uses // memory-mapped I/O and relies on OS file-locking to prevent // corruption from concurrent access by multiple processes. We @@ -110,15 +163,36 @@ impl PaidList { .map_err(|e| Error::Storage(format!("Failed to create paid-list database: {e}")))?; wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit db creation: {e}")))?; + // A failed read leaves its transaction unable to commit, so the + // scan gets one of its own, dropped uncommitted. A write + // transaction takes no reader slot on this thread. + let price_contested = env + .write_txn() + .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}"))) + .and_then(|scan| read_price_contested_keys(&db, &scan)); - Ok((env, db)) + Ok((env, db, price_contested)) }) .await .map_err(|e| Error::Storage(format!("Paid-list init task failed: {e}")))??; + // A paid list whose marks cannot be read still opens; the node then + // sends no replica hints until it restarts (see `retain_unmarked`). + let price_contested = price_contested + .inspect_err(|e| { + warn!( + "Cannot read the paid list's price-contested marks; sending no replica \ + hints until restart: {e}" + ); + }) + .ok() + .map(Arc::new); let paid_list = Self { env, db, + price_contested: Arc::new(RwLock::new(price_contested)), + write_order: Arc::new(Mutex::new(())), + failed_mark_reads: AtomicU64::new(0), paid_out_of_range: RwLock::new(HashMap::new()), record_out_of_range: RwLock::new(HashMap::new()), paid_prune_cursor: RwLock::new(0), @@ -139,6 +213,25 @@ impl PaidList { /// /// Returns an error if the LMDB write transaction fails. pub async fn insert(&self, key: &XorName) -> Result { + self.insert_with_value(key, &[]).await + } + + /// Insert `key` as paid at a price the floor contested here (see + /// `fresh::record_paid_key`). Like [`Self::insert`], an existing entry is + /// left exactly as it is, so a key already recorded can never gain the + /// mark later. The mark survives restarts and later inserts, and goes + /// with the entry on [`Self::remove`]. + /// + /// Returns `true` if the key was newly added, `false` if it already existed. + /// + /// # Errors + /// + /// Returns an error if the LMDB write transaction fails. + pub async fn insert_price_contested(&self, key: &XorName) -> Result { + self.insert_with_value(key, PRICE_CONTESTED_VALUE).await + } + + async fn insert_with_value(&self, key: &XorName, value: &'static [u8]) -> Result { // Fast-path: avoid write transaction if key already present. if self.contains(key)? { trace!("Paid-list key {} already present", hex::encode(key)); @@ -148,10 +241,13 @@ impl PaidList { let key_owned = *key; let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let was_new = self .blocking_tracker .spawn_blocking(move || -> Result { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -167,11 +263,17 @@ impl PaidList { return Ok(false); } - db.put(&mut wtxn, &key_owned, &[]) + db.put(&mut wtxn, &key_owned, value) .map_err(|e| Error::Storage(format!("Failed to insert into paid-list: {e}")))?; + if !value.is_empty() { + if let Some(keys) = price_contested.write().as_mut() { + Arc::make_mut(keys).insert(key_owned); + } + } wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list insert: {e}")) })?; + drop(ordered); Ok(true) }) @@ -179,12 +281,87 @@ impl PaidList { .map_err(|e| Error::Storage(format!("Paid-list insert task failed: {e}")))??; if was_new { - debug!("Added key {} to paid-list", hex::encode(key)); + debug!( + "Added key {} to paid-list{}", + hex::encode(key), + if value.is_empty() { + "" + } else { + " as price-contested" + } + ); } Ok(was_new) } + /// Whether `key` is paid at a price the floor contested here (see + /// [`Self::insert_price_contested`]). + /// + /// Reads the database, which is authoritative: the in-memory copy behind + /// [`Self::retain_unmarked`] can hold a mark whose commit is under way or + /// failed, or be missing. A caller that has just seen a racing write's + /// entry sees its mark here as well (see `fresh::record_paid_key`). + /// + /// # Errors + /// + /// Returns an error if the LMDB read transaction fails. + pub fn is_price_contested(&self, key: &XorName) -> Result { + let rtxn = self + .env + .read_txn() + .map_err(|e| Error::Storage(format!("Failed to create read txn: {e}")))?; + let marked = self + .db + .get(&rtxn, key.as_ref()) + .map_err(|e| Error::Storage(format!("Failed to read paid-list entry: {e}")))? + .is_some_and(|value| !value.is_empty()); + Ok(marked) + } + + /// Whether `key` must be treated as price-contested: its entry carries the + /// mark, or the mark cannot be read (when in doubt, judge no one). While + /// reads fail no peer is judged on any key they hit, so the first failure + /// of a run is a warning, and the read that ends the run reports how many + /// failed. + pub fn counts_as_price_contested(&self, key: &XorName) -> bool { + match self.is_price_contested(key) { + Ok(marked) => { + let failed = self.failed_mark_reads.swap(0, Ordering::Relaxed); + if failed > 0 { + info!("Price-contested marks are readable again, after {failed} failed reads"); + } + marked + } + Err(e) => { + if self.failed_mark_reads.fetch_add(1, Ordering::Relaxed) == 0 { + warn!( + "Cannot read the price-contested mark of key {}; counting unreadable \ + marks as set until a read succeeds: {e}", + hex::encode(key) + ); + } else { + debug!( + "Price-contested mark of key {} still unreadable: {e}", + hex::encode(key) + ); + } + true + } + } + } + + /// Drop from `keys` every key marked price-contested, using the in-memory + /// copy rather than the database. If the marks could not be read on open, + /// no key is known to be unmarked, so every key is dropped. + pub fn retain_unmarked(&self, keys: &mut Vec) { + let snapshot = self.price_contested.read().clone(); + match snapshot { + Some(marked) => keys.retain(|key| !marked.contains(key)), + None => keys.clear(), + } + } + /// Remove a key from the paid-for set. /// /// Also clears any in-memory out-of-range timestamps for this key. @@ -198,10 +375,13 @@ impl PaidList { let key_owned = *key; let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let existed = self .blocking_tracker .spawn_blocking(move || -> Result { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -211,6 +391,12 @@ impl PaidList { wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list delete: {e}")) })?; + if let Some(keys) = price_contested.write().as_mut() { + if keys.contains(&key_owned) { + Arc::make_mut(keys).remove(&key_owned); + } + } + drop(ordered); Ok(deleted) }) .await @@ -415,10 +601,13 @@ impl PaidList { let keys_owned: Vec = keys.to_vec(); let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let removed_keys = self .blocking_tracker .spawn_blocking(move || -> Result> { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -435,6 +624,15 @@ impl PaidList { wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit batch remove: {e}")))?; + if let Some(keys) = price_contested.write().as_mut() { + if removed.iter().any(|key| keys.contains(key)) { + let keys = Arc::make_mut(keys); + for key in &removed { + keys.remove(key); + } + } + } + drop(ordered); Ok(removed) }) @@ -480,18 +678,46 @@ impl PaidList { self.blocking_tracker.wait().await; self.blocking_tracker.reopen(); } + + /// Hold a read transaction on the calling thread. LMDB gives each thread + /// one reader slot, so until it is dropped every paid-list read on this + /// thread fails, which lets a test drive the paths a mark that cannot be + /// read takes. + #[cfg(test)] + pub(crate) fn hold_reader_for_tests(&self) -> Result> { + self.env + .read_txn() + .map_err(|e| Error::Storage(format!("Failed to create read txn: {e}"))) + } } #[cfg(test)] #[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] mod tests { use super::*; + #[cfg(target_pointer_width = "64")] + use std::fs::OpenOptions; + #[cfg(target_pointer_width = "64")] + use std::io::{Read, Seek, SeekFrom, Write}; + #[cfg(feature = "logging")] + use tracing::subscriber::with_default; + #[cfg(feature = "logging")] + use tracing_subscriber::fmt::writer::{MakeWriter, MutexGuardWriter}; /// Short probe used to prove `wait_idle` is still blocked on a parked op. const WAIT_IDLE_BLOCKED_PROBE: std::time::Duration = std::time::Duration::from_millis(200); /// Generous ceiling for `wait_idle` to complete once the op is released. const WAIT_IDLE_COMPLETE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); + /// The in-memory copy of the marks, which must have been read on open. + fn marked_in_memory(pl: &PaidList) -> HashSet { + pl.price_contested + .read() + .as_deref() + .cloned() + .expect("marks read on open") + } + async fn create_test_paid_list() -> (PaidList, tempfile::TempDir) { let temp_dir = tempfile::TempDir::new().expect("create temp dir"); let paid_list = PaidList::new(temp_dir.path()) @@ -572,6 +798,264 @@ mod tests { } } + /// ADR-0006: the price-contested mark is written only with a new entry, so + /// an already recorded key can never gain it. It survives a restart and a + /// later plain insert, and goes with the entry on removal. + #[tokio::test] + async fn price_contested_mark_persists_until_the_key_is_removed() { + let temp_dir = tempfile::TempDir::new().expect("create temp dir"); + let marked: XorName = [0x51; 32]; + let recorded_first: XorName = [0x52; 32]; + + { + let pl = PaidList::new(temp_dir.path()) + .await + .expect("create paid list"); + assert!(pl.insert(&recorded_first).await.expect("insert")); + assert!(pl.insert_price_contested(&marked).await.expect("mark")); + assert!(!pl + .insert_price_contested(&recorded_first) + .await + .expect("mark an existing key")); + assert!(!pl.insert(&marked).await.expect("re-insert")); + } + + let pl = PaidList::new(temp_dir.path()) + .await + .expect("reopen paid list"); + assert_eq!(pl.count().expect("count"), 2); + assert!(pl.contains(&marked).expect("contains")); + assert!(pl.is_price_contested(&marked).expect("read mark")); + assert!(!pl.is_price_contested(&recorded_first).expect("read mark")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + + pl.remove(&marked).await.expect("remove"); + pl.insert(&marked).await.expect("insert again"); + assert!(!pl.is_price_contested(&marked).expect("read mark")); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// ADR-0006: the in-memory marks follow every insert, removal and batch + /// removal. + #[tokio::test] + async fn the_in_memory_marks_follow_every_write() { + let (pl, _temp) = create_test_paid_list().await; + let first: XorName = [0x61; 32]; + let second: XorName = [0x62; 32]; + let plain: XorName = [0x63; 32]; + pl.insert_price_contested(&first).await.expect("mark"); + pl.insert_price_contested(&second).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + assert_eq!(marked_in_memory(&pl), HashSet::from([first, second])); + let mut keys = vec![first, plain, second]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, vec![plain]); + pl.remove(&first).await.expect("remove"); + assert_eq!(marked_in_memory(&pl), HashSet::from([second])); + pl.remove_batch(&[second, plain]) + .await + .expect("remove batch"); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// ADR-0006: while a database write holds the write order, a marked + /// insert or a batch removal neither commits nor changes the in-memory + /// copy, so writes and the copy move in one order. + #[tokio::test] + async fn the_in_memory_marks_change_in_commit_order() { + let (pl, _temp) = create_test_paid_list().await; + let key: XorName = [0x6B; 32]; + let (release_tx, holder) = hold_write_order(&pl).await; + + let insert = pl.insert_price_contested(&key); + tokio::pin!(insert); + let blocked = tokio::time::timeout(WAIT_IDLE_BLOCKED_PROBE, &mut insert).await; + assert!(blocked.is_err(), "the insert must wait for the write order"); + assert!(!pl.contains(&key).expect("contains")); + assert!(marked_in_memory(&pl).is_empty()); + + drop(release_tx); + holder.await.expect("holder task"); + assert!(insert.await.expect("insert")); + assert_eq!(marked_in_memory(&pl), HashSet::from([key])); + + let (release_tx, holder) = hold_write_order(&pl).await; + + let batch = [key]; + let removal = pl.remove_batch(&batch); + tokio::pin!(removal); + let blocked = tokio::time::timeout(WAIT_IDLE_BLOCKED_PROBE, &mut removal).await; + assert!( + blocked.is_err(), + "the removal must wait for the write order" + ); + assert!(pl.contains(&key).expect("contains")); + assert_eq!(marked_in_memory(&pl), HashSet::from([key])); + + drop(release_tx); + holder.await.expect("holder task"); + assert_eq!(removal.await.expect("remove batch"), 1); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// Byte offsets in an LMDB meta page on a 64-bit target: the magic number + /// follows the 16-byte page header, the free list's record keeps the page + /// size at 40, and the main database's root page number sits at 128. + #[cfg(target_pointer_width = "64")] + const LMDB_MAGIC_AT: u64 = 16; + #[cfg(target_pointer_width = "64")] + const LMDB_PAGE_SIZE_AT: u64 = 40; + #[cfg(target_pointer_width = "64")] + const LMDB_MAIN_ROOT_AT: u64 = 128; + + /// ADR-0006: a paid list whose marks cannot be read still opens, with no + /// marks in memory, so the node hints no key as a replica. Both meta pages + /// point the database's root past the end of the file, so the scan for + /// the marks fails as it would on a lost page. + #[cfg(target_pointer_width = "64")] + #[tokio::test] + async fn a_paid_list_whose_marks_cannot_be_read_still_opens() { + let temp_dir = tempfile::TempDir::new().expect("create temp dir"); + let marked: XorName = [0x70; 32]; + { + let pl = PaidList::new(temp_dir.path()) + .await + .expect("create paid list"); + pl.insert_price_contested(&marked).await.expect("mark"); + } + + let mut data = OpenOptions::new() + .read(true) + .write(true) + .open(temp_dir.path().join("paid_list.mdb").join("data.mdb")) + .expect("open data file"); + let mut read_u32 = |at: u64| { + let mut word = [0u8; 4]; + data.seek(SeekFrom::Start(at)).expect("seek"); + data.read_exact(&mut word).expect("read"); + u32::from_ne_bytes(word) + }; + let page_size = u64::from(read_u32(LMDB_PAGE_SIZE_AT)); + for meta in [0, page_size] { + assert_eq!( + read_u32(meta + LMDB_MAGIC_AT), + 0xBEEF_C0DE, + "LMDB meta page" + ); + } + for meta in [0, page_size] { + data.seek(SeekFrom::Start(meta + LMDB_MAIN_ROOT_AT)) + .expect("seek"); + data.write_all(&(1u64 << 40).to_ne_bytes()).expect("write"); + } + data.sync_all().expect("sync"); + drop(data); + + let pl = PaidList::new(temp_dir.path()) + .await + .expect("a paid list whose marks cannot be read must open"); + assert!(pl.price_contested.read().is_none()); + assert!(pl.is_price_contested(&marked).is_err()); + let mut keys = vec![marked]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + } + + /// ADR-0006: a paid list whose marks could not be read on open shows no + /// key as unmarked, so the node hints no key as a replica, while writes and + /// point reads of the marks still work. + #[tokio::test] + async fn marks_unread_on_open_hint_no_replicas() { + let (pl, _temp) = create_test_paid_list().await; + let marked: XorName = [0x6C; 32]; + let plain: XorName = [0x6D; 32]; + *pl.price_contested.write() = None; + + let mut keys = vec![plain]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + pl.insert_price_contested(&marked).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + pl.remove_batch(&[plain]).await.expect("remove batch"); + assert!(pl.price_contested.read().is_none()); + assert!(pl.counts_as_price_contested(&marked)); + assert!(!pl.counts_as_price_contested(&plain)); + } + + /// ADR-0006: a key counts as price-contested when it is marked, or when + /// its mark cannot be read; an unmarked or unpaid key that reads fine + /// does not. + #[tokio::test] + async fn an_unreadable_mark_counts_as_price_contested() { + let (pl, _temp) = create_test_paid_list().await; + let marked: XorName = [0x64; 32]; + let plain: XorName = [0x65; 32]; + let unpaid: XorName = [0x66; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + + assert!(pl.counts_as_price_contested(&marked)); + assert!(!pl.counts_as_price_contested(&plain)); + assert!(!pl.counts_as_price_contested(&unpaid)); + + let _reader = pl.hold_reader_for_tests().expect("hold reader"); + assert!(pl.counts_as_price_contested(&plain)); + assert!(pl.counts_as_price_contested(&unpaid)); + } + + /// In-memory sink a test subscriber writes its lines to. + #[cfg(feature = "logging")] + #[derive(Clone, Default)] + struct LogSink(Arc>>); + + #[cfg(feature = "logging")] + impl<'a> MakeWriter<'a> for LogSink { + type Writer = MutexGuardWriter<'a, Vec>; + + fn make_writer(&'a self) -> Self::Writer { + self.0.make_writer() + } + } + + /// ADR-0006: a run of mark reads that fail logs one warning, not one per + /// key, and the read that ends the run reports how many failed. + #[cfg(feature = "logging")] + #[tokio::test] + async fn a_run_of_unreadable_marks_warns_once() { + let (pl, _temp) = create_test_paid_list().await; + let first: XorName = [0x6E; 32]; + let second: XorName = [0x6F; 32]; + pl.insert(&first).await.expect("insert"); + let sink = LogSink::default(); + let subscriber = tracing_subscriber::fmt() + .with_writer(sink.clone()) + .with_ansi(false) + .finish(); + + with_default(subscriber, || { + { + let _reader = pl.hold_reader_for_tests().expect("hold reader"); + for key in [first, second, first] { + assert!(pl.counts_as_price_contested(&key)); + } + } + assert!(!pl.counts_as_price_contested(&first)); + assert!(!pl.counts_as_price_contested(&second)); + }); + + let logs = String::from_utf8(sink.0.lock().expect("sink").clone()).expect("utf-8"); + assert_eq!( + logs.matches("Cannot read the price-contested mark").count(), + 1, + "{logs}" + ); + assert_eq!( + logs.matches("readable again, after 3 failed reads").count(), + 1, + "{logs}" + ); + } + #[tokio::test] async fn test_all_keys() { let (pl, _temp) = create_test_paid_list().await; @@ -757,6 +1241,29 @@ mod tests { (release_tx, holder) } + /// Hold the paid list's write order on a blocking thread, so the next + /// write waits before its write transaction. + /// + /// Returns once the order is held. Dropping the returned sender releases + /// it. + async fn hold_write_order( + pl: &PaidList, + ) -> ( + tokio::sync::oneshot::Sender<()>, + tokio::task::JoinHandle<()>, + ) { + let write_order = Arc::clone(&pl.write_order); + let (release_tx, release_rx) = tokio::sync::oneshot::channel::<()>(); + let (held_tx, held_rx) = tokio::sync::oneshot::channel::<()>(); + let holder = tokio::task::spawn_blocking(move || { + let _ordered = write_order.lock(); + let _ = held_tx.send(()); + let _ = release_rx.blocking_recv(); + }); + held_rx.await.expect("write order held"); + (release_tx, holder) + } + /// Dropping an `insert` awaiter does not cancel its `spawn_blocking` LMDB /// transaction; `wait_idle` must wait for that detached write, and the /// paid list must remain usable afterwards. diff --git a/src/replication/pointer.rs b/src/replication/pointer.rs index 7afa5565..88aca32e 100644 --- a/src/replication/pointer.rs +++ b/src/replication/pointer.rs @@ -1450,6 +1450,13 @@ impl PointerReplication { warn!("A fresh pointer write did not parse; not replicating it"); return; }; + // Decided from the proof being forwarded, before it moves into the + // offer: a state the price floor contests here is not used to judge + // this node's peers (ADR-0006). + let contested = self + .payments + .price_contested(&state.address, &write.payment_proof) + .await; let self_id = *self.p2p.peer_id(); let targets: Vec = self .p2p @@ -1504,6 +1511,14 @@ impl PointerReplication { hex::encode(state.address), targets.len() ); + if contested { + info!( + "Fresh pointer {}: no possession check, the payment is contested under \ + the price floor", + hex::encode(state.address) + ); + return; + } self.schedule_possession_check(state, targets); } diff --git a/src/replication/pruning.rs b/src/replication/pruning.rs index da8d7f9f..06052436 100644 --- a/src/replication/pruning.rs +++ b/src/replication/pruning.rs @@ -342,6 +342,7 @@ struct PruneAuditReportState { #[derive(Clone, Copy)] struct PruneAuditContext<'a> { storage: &'a Arc, + paid_list: &'a PaidList, p2p_node: &'a Arc, config: &'a ReplicationConfig, sync_state: &'a Arc>, @@ -360,6 +361,7 @@ pub async fn run_prune_pass_with_context(ctx: PrunePassContext<'_>) -> PruneResu let (paid_count, paid_stats) = prune_paid_entries( ctx.self_id, ctx.paid_list, + ctx.storage, ctx.p2p_node, ctx.config, now, @@ -508,16 +510,8 @@ async fn prune_stored_records(ctx: &PrunePassContext<'_>) -> (usize, RecordPrune ) .await; - let present_by_key = collect_record_prune_proofs( - &audit_selection.candidates, - stored_keys.len(), - ctx.storage, - ctx.p2p_node, - ctx.config, - ctx.sync_state, - ctx.audit_challenge_coordinator, - ) - .await; + let present_by_key = + collect_record_prune_proofs(&audit_selection.candidates, stored_keys.len(), ctx).await; let (keys_to_delete, revalidated_cleared, audit_below_threshold) = revalidated_record_prune_keys(&audit_selection.candidates, &present_by_key, ctx).await; stats.cleared += revalidated_cleared; @@ -801,6 +795,7 @@ fn select_record_prune_audits( async fn prune_paid_entries( self_id: &PeerId, paid_list: &Arc, + storage: &ChunkStore, p2p_node: &Arc, config: &ReplicationConfig, now: Instant, @@ -849,7 +844,9 @@ async fn prune_paid_entries( let elapsed = now .checked_duration_since(first_seen) .unwrap_or(Duration::ZERO); - if elapsed >= config.prune_hysteresis_duration { + if elapsed >= config.prune_hysteresis_duration + && !keeps_price_contested_mark(key, paid_list, storage) + { match select_paid_prune_candidate( key, &closest, @@ -886,7 +883,7 @@ async fn prune_paid_entries( let confirmed_by_key = collect_paid_prune_confirmations(&expired_candidates, p2p_node, config).await; - let (paid_keys_to_delete, revalidated_cleared) = revalidated_paid_prune_keys( + let (mut paid_keys_to_delete, revalidated_cleared) = revalidated_paid_prune_keys( &expired_candidates, &confirmed_by_key, self_id, @@ -895,12 +892,24 @@ async fn prune_paid_entries( config, ) .await; + // Checked again after the network round, in case the record arrived + // meanwhile. + paid_keys_to_delete.retain(|key| !keeps_price_contested_mark(key, paid_list, storage)); stats.cleared += revalidated_cleared; stats.pruned = delete_paid_entries(&paid_keys_to_delete, paid_list).await; (paid_keys.len(), stats) } +/// Whether the paid entry for `key` must outlive the paid-prune rules: it +/// carries the price-contested mark (see `fresh::record_paid_key`) and this +/// node still holds the record the mark protects. State that cannot be read +/// counts as both (see `PaidList::counts_as_price_contested`). Such an entry +/// is not even put to the paid group, so it costs no verification budget. +fn keeps_price_contested_mark(key: &XorName, paid_list: &PaidList, storage: &ChunkStore) -> bool { + paid_list.counts_as_price_contested(key) && storage.exists(key).unwrap_or(true) +} + fn select_paid_prune_candidate( key: &XorName, closest: &[PeerId], @@ -1202,14 +1211,13 @@ async fn delete_stored_records( /// local, and the retained record continues to participate in normal /// neighbor-sync repair because replica hint construction walks all locally /// stored keys, including out-of-range keys retained by hysteresis. +/// +/// A price-contested key still needs its proofs before it is deleted, but no +/// absent peer is scored for it (see `fresh::record_paid_key`). async fn collect_record_prune_proofs( candidates: &[RecordPruneCandidate], local_stored_key_count: usize, - storage: &Arc, - p2p_node: &Arc, - config: &ReplicationConfig, - sync_state: &Arc>, - audit_challenge_coordinator: &Arc, + ctx: &PrunePassContext<'_>, ) -> HashMap> { if candidates.is_empty() { return HashMap::new(); @@ -1220,11 +1228,12 @@ async fn collect_record_prune_proofs( let challenges = build_peer_audit_challenges(candidates, max_keys_per_challenge); let report_state = PruneAuditReportState::default(); let audit_context = PruneAuditContext { - storage, - p2p_node, - config, - sync_state, - audit_challenge_coordinator, + storage: ctx.storage, + paid_list: ctx.paid_list, + p2p_node: ctx.p2p_node, + config: ctx.config, + sync_state: ctx.sync_state, + audit_challenge_coordinator: ctx.audit_challenge_coordinator, report_state: &report_state, }; let mut requests = stream::iter(challenges) @@ -1618,15 +1627,7 @@ async fn peer_proves_records( "failed" }; if !audit_failure_reported - && report_prune_audit_failure_once( - &peer, - &key, - context.p2p_node, - context.config, - context.report_state, - failure_class, - ) - .await + && report_prune_audit_failure_once(&peer, &key, context, failure_class).await { audit_failure_reported = true; } @@ -1671,16 +1672,7 @@ async fn receive_prune_audit_response( }; for key in challenge_keys { - if report_prune_audit_failure_once( - peer, - key, - context.p2p_node, - context.config, - context.report_state, - failure_class, - ) - .await - { + if report_prune_audit_failure_once(peer, key, context, failure_class).await { break; } } @@ -1964,11 +1956,25 @@ fn audit_digest_proves_key( async fn report_prune_audit_failure_once( peer: &PeerId, key: &XorName, - p2p_node: &Arc, - config: &ReplicationConfig, - report_state: &PruneAuditReportState, + context: PruneAuditContext<'_>, audit_failure_class: &'static str, ) -> bool { + let PruneAuditContext { + paid_list, + p2p_node, + config, + report_state, + .. + } = context; + // No peer is scored for a price-contested key (see + // `fresh::record_paid_key`); a mark that cannot be read counts as set. + if paid_list.counts_as_price_contested(key) { + debug!( + "Prune audit: not scoring {peer} for {}, a price-contested key", + hex::encode(key) + ); + return false; + } let should_report = peer_is_currently_responsible(peer, key, p2p_node, config).await && reserve_prune_audit_failure_report(report_state, peer).await; if !should_report { @@ -2118,6 +2124,53 @@ mod tests { RecordPruneCandidate { key, target_peers } } + /// ADR-0006: a price-contested paid entry is kept while its record is + /// held, and prunes like any other once the record is gone. Unmarked + /// entries are unaffected. + #[tokio::test] + async fn a_price_contested_paid_entry_outlives_paid_pruning_while_its_record_is_held() { + let dir = TempDir::new().expect("temp dir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().join("chunks"), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let held_bytes = b"held price-contested record".to_vec(); + let held = ChunkStore::compute_address(&held_bytes); + storage.put(&held, &held_bytes).await.expect("put"); + let gone: XorName = [0x81; 32]; + let plain: XorName = [0x82; 32]; + paid_list.insert_price_contested(&held).await.expect("mark"); + paid_list.insert_price_contested(&gone).await.expect("mark"); + paid_list.insert(&plain).await.expect("insert"); + + assert!(keeps_price_contested_mark(&held, &paid_list, &storage)); + assert!(!keeps_price_contested_mark(&gone, &paid_list, &storage)); + assert!(!keeps_price_contested_mark(&plain, &paid_list, &storage)); + + // A held record whose mark cannot be read is kept as if marked. + let plain_held_bytes = b"held plain record".to_vec(); + let plain_held = ChunkStore::compute_address(&plain_held_bytes); + storage + .put(&plain_held, &plain_held_bytes) + .await + .expect("put"); + paid_list.insert(&plain_held).await.expect("insert"); + assert!(!keeps_price_contested_mark( + &plain_held, + &paid_list, + &storage + )); + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(keeps_price_contested_mark( + &plain_held, + &paid_list, + &storage + )); + } + /// A prune audit judges holders against this node's own copy. A copy that /// no longer hashes to its key would fail every honest holder and hold the /// prune back forever, so it is not used as a reference: the read takes it diff --git a/src/storage/handler.rs b/src/storage/handler.rs index 62e2db75..6212adf4 100644 --- a/src/storage/handler.rs +++ b/src/storage/handler.rs @@ -864,9 +864,9 @@ impl AntProtocol { self.quote_generator.record_store(); // 7. Notify replication engine for fresh fan-out. - // Only emit when a real proof is present — cached-as-verified - // PUTs have no proof to forward, and the chunk would have - // already replicated on the original write that carried one. + // Only emit when a real proof is present. A cached-as-verified + // PUT normally carries none; one that does forwards it + // unverified, and the fan-out prices it as it stands (ADR-0006). if let (Some(ref tx), Some(proof)) = (&self.fresh_write_tx, request.payment_proof) { // ADR-0004: strip any commitment sidecars before forwarding // to replication — the ADR specifies persisted/replicated @@ -1249,7 +1249,6 @@ mod tests { cache_capacity: 100_000, close_group_size: crate::ant_protocol::CLOSE_GROUP_SIZE, local_rewards_address: rewards_address, - price_floor: crate::payment::PriceFloorConfig::default(), }; let payment_verifier = Arc::new(PaymentVerifier::new(payment_config)); let metrics_tracker = QuotingMetricsTracker::new(100); diff --git a/tests/e2e/data_types/chunk.rs b/tests/e2e/data_types/chunk.rs index 650b8e08..96480f88 100644 --- a/tests/e2e/data_types/chunk.rs +++ b/tests/e2e/data_types/chunk.rs @@ -64,8 +64,8 @@ mod tests { use crate::{TestHarness, TestNetwork}; use ant_node::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, - QuoteGenerator, QuotingMetricsTracker, + EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator, + QuotingMetricsTracker, }; use ant_node::storage::{AntProtocol, ChunkStore, ChunkStoreConfig}; use ant_node::ReplicationConfig; @@ -445,7 +445,6 @@ mod tests { cache_capacity: 100, close_group_size: ReplicationConfig::default().close_group_size, local_rewards_address: rewards_address, - price_floor: PriceFloorConfig::default(), }); let metrics_tracker = QuotingMetricsTracker::new(100); let quote_generator = QuoteGenerator::new(rewards_address, metrics_tracker); diff --git a/tests/e2e/harness.rs b/tests/e2e/harness.rs index 786b18c5..efa0531f 100644 --- a/tests/e2e/harness.rs +++ b/tests/e2e/harness.rs @@ -6,12 +6,45 @@ use super::anvil::TestAnvil; use super::testnet::{TestNetwork, TestNetworkConfig, TestNode, TestnetError}; use ant_node::client::XorName; -use evmlib::common::TxHash; +use ant_node::payment::{serialize_single_node_proof, PaymentProof}; +use evmlib::common::{Amount, TxHash}; +use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment, RewardsAddress}; use saorsa_core::P2PNode; use std::collections::HashMap; use std::sync::{Arc, Mutex}; +use std::time::SystemTime; use tracing::{info, warn}; +/// A possession or audit failure is reported at weight 5, which cuts a peer's +/// trust by `1 - (1 - 0.124)^5`, about 48%; lighter events cut 12% to 33%. A +/// fall to this fraction of a peer's score or below is read as that penalty. +pub const POSSESSION_PENALTY_TRUST_RATIO: f64 = 0.6; + +/// A single-node payment proof that parses, naming `content`, whose one quote +/// is never valid: no signature, no settlement. For tests that need a node to +/// see a proof the price floor could price, without anyone accepting it. +#[allow(clippy::expect_used)] +pub fn parseable_single_node_proof(content: XorName) -> Vec { + let quote = PaymentQuote { + content: xor_name::XorName(content), + timestamp: SystemTime::now(), + price: Amount::from(1u64), + rewards_address: RewardsAddress::new([0x11; 20]), + committed_key_count: 0, + commitment_pin: None, + pub_key: Vec::new(), + signature: Vec::new(), + }; + serialize_single_node_proof(&PaymentProof { + proof_of_payment: ProofOfPayment { + peer_quotes: vec![(EncodedPeerId::new([0x22; 32]), quote)], + }, + tx_hashes: Vec::new(), + commitment_sidecars: Vec::new(), + }) + .expect("serialize a single-node proof") +} + /// How many port ranges a network is brought up on before its bind failure /// is reported. /// diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index ec2a74c7..cdec4e56 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -78,6 +78,9 @@ mod fetch_responsibility_recheck; #[cfg(test)] mod first_audit_ab; +#[cfg(test)] +mod price_floor_ab; + #[cfg(test)] mod fresh_offer_capacity; diff --git a/tests/e2e/pointer_replication.rs b/tests/e2e/pointer_replication.rs index 8548a7c0..586ccf0c 100644 --- a/tests/e2e/pointer_replication.rs +++ b/tests/e2e/pointer_replication.rs @@ -8,6 +8,7 @@ #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use super::harness::parseable_single_node_proof; use super::testnet::{TestNetworkConfig, TestNode}; use super::TestHarness; use ant_node::ant_protocol::chunk::{ @@ -32,8 +33,10 @@ use bytes::Bytes; use saorsa_core::identity::PeerId; use saorsa_pqc::api::sig::{ml_dsa_65, MlDsaPublicKey, MlDsaSecretKey}; use serial_test::serial; +use std::collections::HashMap; use std::sync::Arc; use std::time::Duration; +use tokio::sync::RwLock; /// How long to wait for replication to reach a node. const SETTLE: Duration = Duration::from_secs(30); @@ -475,6 +478,98 @@ async fn the_possession_check_penalises_only_a_member_that_dropped_the_record() harness.teardown().await.expect("teardown"); } +/// ADR-0006: a fresh pointer write the offering node cannot price is not +/// used to judge its peers. The checker's price floor is wired (a gossip cache +/// is attached) but it has no commitment of its own, so it cannot price the +/// single-node proof it forwards, and no possession check may follow. The +/// control write that follows forwards a proof the floor never prices, and its +/// check does penalise the members, so the first result is not vacuous. +#[tokio::test] +#[serial] +async fn a_fresh_pointer_write_the_floor_cannot_price_is_not_used_to_judge_peers() { + // Long enough for the scheduled check (at most 500 ms here) and its probes + // to have run, short enough that unrelated trust events are unlikely. + const CONTESTED_WATCH: Duration = Duration::from_secs(5); + let mut config = TestNetworkConfig::minimal(); + config.replication_config = Some(ReplicationConfig { + possession_check_delay_min: Duration::from_millis(200), + possession_check_delay_max: Duration::from_millis(500), + ..ReplicationConfig::default() + }); + let harness = TestHarness::setup_with_config(config).await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let checker = 3; + // The checker only judges peers that understand pointers. + let everyone: Vec = (0..harness.node_count()).collect(); + exchange_hints(&harness, &everyone, &[checker]).await; + + let checker_node = harness.test_node(checker).expect("node"); + checker_node + .ant_protocol + .as_ref() + .expect("protocol") + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + let checker_p2p = checker_node.p2p_node.as_ref().expect("p2p"); + let members: Vec = others(&harness, &[checker]) + .into_iter() + .map(|i| peer(harness.test_node(i).expect("node"))) + .collect(); + + let (pk, sk) = owner(); + let contested = signed(&pk, &sk, 0, 1); + let control = signed(&pk, &sk, 1, 2); + let mut outcomes = Vec::new(); + for (record, proof, watch) in [ + ( + &contested, + parseable_single_node_proof(contested.state_id()), + CONTESTED_WATCH, + ), + (&control, DUMMY_PROOF.to_vec(), SETTLE), + ] { + let before: Vec = members.iter().map(|p| checker_p2p.peer_trust(p)).collect(); + store(checker_node) + .put_bytes(&record.to_bytes()) + .await + .expect("put"); + // Nothing is paid anywhere, so every member refuses the offer and is + // absent when, and only if, the checker probes it. + replication(checker_node) + .replicate_fresh(PointerFreshWrite { + record: record.to_bytes(), + payment_proof: proof, + }) + .await; + + // A pointer possession failure is a weight-1 trust event, so any fall + // counts. + let deadline = tokio::time::Instant::now() + watch; + let mut penalised = false; + while tokio::time::Instant::now() < deadline { + penalised = members + .iter() + .zip(before.iter()) + .any(|(p, &b)| checker_p2p.peer_trust(p) < b - f64::EPSILON); + if penalised { + break; + } + tokio::time::sleep(POLL).await; + } + outcomes.push(penalised); + } + + assert_eq!( + outcomes, + vec![false, true], + "a write the floor cannot price must not be used to penalise members, while the \ + control write must" + ); + + harness.teardown().await.expect("teardown"); +} + /// A transfer written to one node reaches the whole group, and after that no /// node takes a second final state — not even one whose target sorts first, /// which the previous merge rule let displace the first everywhere (ADR-0018). diff --git a/tests/e2e/price_floor_ab.rs b/tests/e2e/price_floor_ab.rs new file mode 100644 index 00000000..c8522344 --- /dev/null +++ b/tests/e2e/price_floor_ab.rs @@ -0,0 +1,743 @@ +//! ADR-0006 price-floor A/B workload test, env-gated and never run in CI. +//! +//! Builds a production-wired local testnet whose nodes hold production-sized +//! commitments, then drives two workloads against it with real on-chain +//! payments (Anvil): +//! +//! - **honest** uploads: quotes from the close group, the median settled 3x, +//! the chunk PUT to every close-group storer; +//! - **near-empty quoter** uploads: a node that joined after seeding (no +//! commitment, so it quotes the baseline price) is paid 3x its quote, and the +//! chunk is PUT to it and to the mature storers. +//! +//! After a tail long enough for fresh replication, `PaidNotify` and the +//! (shortened) possession checks to play out, it reports who stored each key +//! and whose paid list holds it. Compare the `PF-METRIC` lines, plus the log +//! counts the runner greps for (possession failures naming a near-empty key, +//! contested fan-outs, floor refusals), across builds. It returns at once +//! unless `PRICE_FLOOR_AB=1` is set, so CI only compiles it. +//! +//! ```bash +//! PRICE_FLOOR_AB=1 RUST_LOG=info \ +//! cargo test --test e2e --features test-utils price_floor_ab_workload -- --nocapture +//! ``` +//! +//! Neighbour commitments are delivered straight into each node's gossip cache +//! after seeding, exactly the signed commitments neighbour sync would carry. +//! At production-sized key sets the in-process testnet's sync rounds do not +//! deliver them within any useful time, and the floor skips every admission +//! until they arrive. Everything after that (quotes, payments, PUTs, fresh +//! replication, `PaidNotify`, possession checks) runs on the live paths. +//! +//! It fails, rather than reporting, when its own preconditions do not hold: +//! fewer than 15 known neighbour commitments on the median node (the floor +//! would skip everything) or fewer uploads completed than requested. It also +//! asserts what the floor promises: every honest upload reaches a storage +//! quorum with no floor refusal, no peer that refused a near-empty write or +//! that the quoter fanned it out to loses penalty-sized trust in the quoter's +//! view, and every near-empty write stays off a quorum, every refusal being +//! the floor's (a mature storer whose own price is low enough may accept it, +//! as the cap allows). +//! +//! Knobs (env): `PF_NODES`, `PF_KEYS_PER_NODE`, `PF_HONEST`, `PF_NEAR_EMPTY`, +//! `PF_TAIL_SECS`. + +#![allow( + clippy::unwrap_used, + clippy::expect_used, + clippy::panic, + clippy::unnested_or_patterns, + clippy::cast_precision_loss, + clippy::too_many_lines +)] + +use ant_node::ant_protocol::{ + ChunkMessage, ChunkMessageBody, ChunkPutRequest, ChunkPutResponse, ChunkQuoteRequest, + ChunkQuoteResponse, CLOSE_GROUP_SIZE, +}; +use ant_node::client::{compute_address, send_and_await_chunk_response}; +use ant_node::payment::quote::CommitmentSource; +use ant_node::payment::{serialize_single_node_proof, PaymentProof, PRICE_FLOOR_TOLERANCE_PERCENT}; +use ant_node::replication::commitment_state::PeerCommitmentRecord; +use ant_node::ReplicationConfig; +use bytes::Bytes; +use evmlib::common::Amount; +use evmlib::wallet::Wallet; +use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment}; +use rand::Rng; +use saorsa_core::identity::PeerId; +use saorsa_core::P2PNode; +use serial_test::serial; +use std::collections::{BTreeSet, HashMap}; +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tracing_subscriber::EnvFilter; + +use super::anvil::TestAnvil; +use super::harness::POSSESSION_PENALTY_TRUST_RATIO; +use super::testnet::{TestNetwork, TestNetworkConfig, TestNode}; + +/// The verifier accepts a settlement of at least 3x the median quote price. +const PAYMENT_MULTIPLIER: u64 = 3; + +/// Per-request timeout for quote/PUT round trips. +const REQUEST_TIMEOUT: Duration = Duration::from_secs(20); + +/// Substring of the node's price-floor refusal. +const FLOOR_REFUSAL: &str = "below the close-group price floor"; + +/// Size of each synthetic seeding chunk. +const SEED_CHUNK_BYTES: usize = 64; + +/// Neighbour commitments a node must know before its floor evaluates at all +/// (`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`, private to the verifier). +const FLOOR_MIN_NEIGHBOURS: usize = 15; + +/// Stores a chunk upload needs (`CLOSE_GROUP_MAJORITY`). +const STORE_QUORUM: usize = 4; + +fn env_usize(name: &str, default: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(default) +} + +fn xor_distance(a: &[u8; 32], b: &[u8; 32]) -> [u8; 32] { + let mut out = [0u8; 32]; + for (o, (x, y)) in out.iter_mut().zip(a.iter().zip(b.iter())) { + *o = x ^ y; + } + out +} + +/// Indices of `peers` sorted by XOR distance to `address`, closest first. +fn closest_indices(address: &[u8; 32], peers: &[[u8; 32]]) -> Vec { + let mut order: Vec = (0..peers.len()).collect(); + order.sort_by_key(|&i| peers.get(i).map(|p| xor_distance(address, p))); + order +} + +/// Wire one node exactly like production `node.rs`: commitment source into the +/// quote generator, gossip cache into the verifier, monetized-pin sender into +/// the verifier. The e2e testnet does not wire these by default. +fn wire_like_production(node: &TestNode) { + if let (Some(protocol), Some(engine)) = + (node.ant_protocol.as_ref(), node.replication_engine.as_ref()) + { + let concrete = Arc::clone(engine.commitment_state()); + let source: Arc = concrete; + protocol.attach_commitment_source(source); + protocol + .payment_verifier_arc() + .attach_commitment_cache(Arc::clone(engine.last_commitment_by_peer())); + protocol + .payment_verifier_arc() + .attach_monetized_pin_sender(engine.monetized_pin_sender()); + } +} + +fn peer_bytes(node: &TestNode) -> [u8; 32] { + *node + .p2p_node + .as_ref() + .expect("node p2p") + .peer_id() + .as_bytes() +} + +async fn request_quote( + client: &Arc, + peer: &PeerId, + address: [u8; 32], + data_size: u64, +) -> Result { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::QuoteRequest(ChunkQuoteRequest { + address, + data_size, + data_type: 0, + }), + }; + let message_bytes = message + .encode() + .map_err(|e| format!("encode quote request: {e}"))?; + let quote_bytes = send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::QuoteResponse(ChunkQuoteResponse::Success { quote, .. }) => { + Some(Ok(quote)) + } + ChunkMessageBody::QuoteResponse(ChunkQuoteResponse::Error(e)) => { + Some(Err(format!("quote error: {e:?}"))) + } + _ => None, + }, + |e| format!("send quote request: {e}"), + || "quote request timed out".to_string(), + ) + .await?; + rmp_serde::from_slice("e_bytes).map_err(|e| format!("decode quote: {e}")) +} + +/// One storer's answer to a paid PUT. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PutOutcome { + Stored, + FloorRefused, + OtherError, +} + +async fn paid_put( + client: &Arc, + peer: &PeerId, + address: [u8; 32], + data: &[u8], + proof_bytes: Vec, +) -> (PutOutcome, String) { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::PutRequest(ChunkPutRequest::with_payment( + address, + Bytes::copy_from_slice(data), + proof_bytes, + )), + }; + let Ok(message_bytes) = message.encode() else { + return (PutOutcome::OtherError, "encode put".to_string()); + }; + let result = send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::PutResponse(ChunkPutResponse::Success { .. }) + | ChunkMessageBody::PutResponse(ChunkPutResponse::AlreadyExists { .. }) => Some(Ok(())), + ChunkMessageBody::PutResponse(ChunkPutResponse::PaymentRequired { message }) => { + Some(Err(format!("payment required: {message}"))) + } + ChunkMessageBody::PutResponse(ChunkPutResponse::Error(e)) => { + Some(Err(format!("put error: {e:?}"))) + } + _ => None, + }, + |e| format!("send put: {e}"), + || "put timed out".to_string(), + ) + .await; + match result { + Ok(()) => (PutOutcome::Stored, String::new()), + Err(e) if e.contains(FLOOR_REFUSAL) => (PutOutcome::FloorRefused, e), + Err(e) => (PutOutcome::OtherError, e), + } +} + +/// Settle `amount` for `quote` on chain and build the single-quote proof. +async fn pay(wallet: &Wallet, issuer: &PeerId, quote: &PaymentQuote, amount: Amount) -> Vec { + let (tx_by_quote, _gas) = wallet + .pay_for_quotes(vec![(quote.hash(), quote.rewards_address, amount)]) + .await + .expect("pay_for_quotes on Anvil"); + let proof = PaymentProof { + proof_of_payment: ProofOfPayment { + peer_quotes: vec![(EncodedPeerId::new(*issuer.as_bytes()), quote.clone())], + }, + tx_hashes: tx_by_quote.values().copied().collect(), + commitment_sidecars: vec![], + }; + serialize_single_node_proof(&proof).expect("serialize proof") +} + +/// How many of `nodes` store `key`, and how many hold it in their paid list. +fn holders_and_votes(nodes: &[TestNode], key: &[u8; 32]) -> (usize, usize) { + let mut holders = 0; + let mut votes = 0; + for node in nodes { + if let Some(protocol) = node.ant_protocol.as_ref() { + if protocol.storage().exists(key).unwrap_or(false) { + holders += 1; + } + } + if let Some(engine) = node.replication_engine.as_ref() { + if engine.paid_list().contains(key).unwrap_or(false) { + votes += 1; + } + } + } + (holders, votes) +} + +fn median(mut xs: Vec) -> usize { + xs.sort_unstable(); + xs.get(xs.len() / 2).copied().unwrap_or(0) +} + +async fn sync_rounds(network: &TestNetwork, rounds: usize) { + for _ in 0..rounds { + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + engine.trigger_neighbor_sync(); + } + } + tokio::time::sleep(Duration::from_secs(4)).await; + } +} + +async fn cache_sizes(network: &TestNetwork) -> Vec { + let mut sizes = Vec::new(); + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + sizes.push(engine.last_commitment_by_peer().read().await.len()); + } + } + sizes +} + +/// Put every node's current signed commitment into every other node's gossip +/// cache, as neighbour sync would. +async fn deliver_commitments(network: &TestNetwork) { + let commitments: Vec<(PeerId, _)> = network + .nodes() + .iter() + .filter_map(|node| { + let engine = node.replication_engine.as_ref()?; + let current = engine.commitment_state().current()?; + Some(( + PeerId::from_bytes(peer_bytes(node)), + current.commitment().clone(), + )) + }) + .collect(); + let now = Instant::now(); + for node in network.nodes() { + let own = PeerId::from_bytes(peer_bytes(node)); + if let Some(engine) = node.replication_engine.as_ref() { + let mut cache = engine.last_commitment_by_peer().write().await; + for (peer, commitment) in &commitments { + if *peer != own { + cache.insert( + *peer, + PeerCommitmentRecord::from_verified(commitment.clone(), now), + ); + } + } + } + } +} + +#[tokio::test(flavor = "multi_thread")] +#[serial] +async fn price_floor_ab_workload() { + if std::env::var("PRICE_FLOOR_AB").ok().as_deref() != Some("1") { + eprintln!("price_floor_ab_workload skipped: set PRICE_FLOOR_AB=1 to run"); + return; + } + let _ = tracing_subscriber::fmt() + .with_env_filter( + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")), + ) + .with_writer(std::io::stderr) + .try_init(); + + let nodes = env_usize("PF_NODES", 24); + let keys_per_node = env_usize("PF_KEYS_PER_NODE", 4_500); + let honest_uploads = env_usize("PF_HONEST", 12); + let near_empty_uploads = env_usize("PF_NEAR_EMPTY", 8); + + let tail_secs = env_usize("PF_TAIL_SECS", 90); + eprintln!( + "PF-CONFIG nodes={nodes} keys_per_node={keys_per_node} honest={honest_uploads} \ + near_empty={near_empty_uploads} tail_secs={tail_secs}" + ); + + let anvil = TestAnvil::new().await.expect("start anvil"); + let wallet = anvil.create_funded_wallet().expect("funded wallet"); + let replication_config = ReplicationConfig { + neighbor_sync_interval_min: Duration::from_secs(5), + neighbor_sync_interval_max: Duration::from_secs(10), + possession_check_delay_min: Duration::from_secs(20), + possession_check_delay_max: Duration::from_secs(30), + subtree_round1_responder_cooldown: Duration::from_millis(1), + ..ReplicationConfig::default() + }; + let mut config = TestNetworkConfig { + node_count: nodes, + payment_enforcement: true, + replication_config: Some(replication_config), + ..TestNetworkConfig::default() + }; + config.evm_network = Some(anvil.to_network()); + let mut network = TestNetwork::new(config).await.expect("build network"); + network.start().await.expect("start network"); + network.warmup_dht().await.expect("warmup dht"); + for node in network.nodes() { + wire_like_production(node); + } + + let client_index = nodes.saturating_sub(1); + let client = network + .nodes() + .get(client_index) + .and_then(|n| n.p2p_node.as_ref()) + .map(Arc::clone) + .expect("client node p2p"); + + // ---- Seed: write synthetic chunks straight into the stores of the seven + // XOR-closest nodes (each node's own close group in a full routing table), + // then rotate commitments, so every node commits to a production-sized key + // set without thousands of on-chain payments. + let peers: Vec<[u8; 32]> = network.nodes().iter().map(peer_bytes).collect(); + let seed_chunks = keys_per_node * nodes / CLOSE_GROUP_SIZE; + let mut rng = rand::thread_rng(); + let mut assignments: Vec)>> = vec![Vec::new(); nodes]; + for i in 0..seed_chunks { + let mut content = vec![0u8; SEED_CHUNK_BYTES]; + rng.fill(&mut content[..]); + if let Some(prefix) = content.get_mut(..8) { + prefix.copy_from_slice(&(i as u64).to_le_bytes()); + } + let address = compute_address(&content); + for index in closest_indices(&address, &peers) + .into_iter() + .take(CLOSE_GROUP_SIZE) + { + if let Some(slot) = assignments.get_mut(index) { + slot.push((address, content.clone())); + } + } + } + let mut writers = Vec::new(); + for (node, chunks) in network.nodes().iter().zip(assignments) { + if let Some(protocol) = node.ant_protocol.as_ref() { + let storage = protocol.storage(); + writers.push(tokio::spawn(async move { + for (address, content) in chunks { + storage.put(&address, &content).await.expect("seed put"); + } + })); + } + } + for writer in writers { + writer.await.expect("seed writer"); + } + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + engine.rebuild_commitment_now().await.expect("rebuild"); + } + } + let counts: Vec = network + .nodes() + .iter() + .filter_map(|n| n.replication_engine.as_ref()) + .filter_map(|e| e.commitment_state().current()) + .map(|c| c.commitment().key_count as usize) + .collect(); + eprintln!( + "PF-METRIC phase=seed chunks={seed_chunks} committed_nodes={} key_count_min={} \ + key_count_median={} key_count_max={}", + counts.len(), + counts.iter().min().copied().unwrap_or(0), + median(counts.clone()), + counts.iter().max().copied().unwrap_or(0) + ); + + deliver_commitments(&network).await; + let sizes = cache_sizes(&network).await; + eprintln!( + "PF-METRIC phase=gossip mode=delivered cache_min={} cache_median={} cache_max={}", + sizes.iter().min().copied().unwrap_or(0), + median(sizes.clone()), + sizes.iter().max().copied().unwrap_or(0) + ); + assert!( + median(sizes.clone()) >= FLOOR_MIN_NEIGHBOURS, + "precondition unmet: the median node knows fewer than {FLOOR_MIN_NEIGHBOURS} neighbour \ + commitments, so the floor would skip every admission" + ); + + // ---- Honest uploads. + eprintln!("PF-METRIC phase=build tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}"); + let mut honest_keys = Vec::new(); + let (mut honest_stored, mut honest_floor, mut honest_other) = (0usize, 0usize, 0usize); + for i in 0..honest_uploads { + let mut data = vec![0u8; 4096]; + rng.fill(&mut data[..]); + let address = compute_address(&data); + let storers: Vec = closest_indices(&address, &peers) + .into_iter() + .filter(|&j| j != client_index) + .take(CLOSE_GROUP_SIZE) + .filter_map(|j| peers.get(j).map(|p| PeerId::from_bytes(*p))) + .collect(); + let mut quotes = Vec::new(); + for peer in &storers { + if let Ok(q) = request_quote(&client, peer, address, data.len() as u64).await { + quotes.push((*peer, q)); + } + } + quotes.sort_by_key(|(_, q)| q.price); + let Some((issuer, median_quote)) = quotes.get(quotes.len() / 2).cloned() else { + eprintln!("PF-WARN honest {i}: no quotes"); + continue; + }; + let amount = median_quote.price * Amount::from(PAYMENT_MULTIPLIER); + let proof = pay(&wallet, &issuer, &median_quote, amount).await; + let (mut stored, mut floor, mut other) = (0usize, 0usize, 0usize); + for peer in &storers { + match paid_put(&client, peer, address, &data, proof.clone()).await { + (PutOutcome::Stored, _) => stored += 1, + (PutOutcome::FloorRefused, _) => floor += 1, + (PutOutcome::OtherError, e) => { + other += 1; + eprintln!("PF-WARN honest {i} put to {peer}: {e}"); + } + } + } + eprintln!( + "PF-METRIC phase=honest upload={i} key={} quotes={} paid_price={} stored={stored} \ + floor_refused={floor} other_error={other}", + hex::encode(address), + quotes.len(), + median_quote.price + ); + assert!( + stored >= STORE_QUORUM && floor == 0, + "honest upload {i} must reach a storage quorum with no floor refusal" + ); + honest_stored += stored; + honest_floor += floor; + honest_other += other; + honest_keys.push(address); + } + + // ---- A node joins after seeding: no data, no commitment, baseline quotes. + let before = network.node_count(); + let fresh_index = match network.add_node().await { + Ok(index) => index, + // The node is started and registered before the harness waits on its + // replication bootstrap, which can outlast the harness's fixed timeout + // on a seeded network. The node is still usable: it holds no + // commitment either way, which is all this phase needs. + Err(e) if network.node_count() > before => { + eprintln!("PF-WARN fresh node bootstrap: {e}; continuing with it"); + before + } + Err(e) => panic!("add fresh node: {e}"), + }; + if let Some(node) = network.nodes().get(fresh_index) { + wire_like_production(node); + } + let fresh_peer_bytes = network + .nodes() + .get(fresh_index) + .map(peer_bytes) + .expect("fresh node"); + let fresh_peer = PeerId::from_bytes(fresh_peer_bytes); + sync_rounds(&network, 3).await; + let mut all_peers = peers.clone(); + all_peers.push(fresh_peer_bytes); + let fresh_p2p = network + .nodes() + .get(fresh_index) + .and_then(|n| n.p2p_node.as_ref()) + .map(Arc::clone) + .expect("fresh node p2p"); + let trust_before: HashMap<[u8; 32], f64> = peers + .iter() + .map(|p| (*p, fresh_p2p.peer_trust(&PeerId::from_bytes(*p)))) + .collect(); + let mut refusers: BTreeSet<[u8; 32]> = BTreeSet::new(); + let mut fan_out_targets: BTreeSet<[u8; 32]> = BTreeSet::new(); + let fan_out_group_size = ReplicationConfig::default().close_group_size; + + // ---- Near-empty quoter uploads: pay the fresh node's baseline quote 3x. + let mut near_empty_keys = Vec::new(); + let (mut ne_mature_stored, mut ne_mature_floor, mut ne_mature_other, mut ne_fresh_stored) = + (0usize, 0usize, 0usize, 0usize); + let mut attempts = 0usize; + while near_empty_keys.len() < near_empty_uploads && attempts < near_empty_uploads * 20 { + attempts += 1; + let mut data = vec![0u8; 4096]; + rng.fill(&mut data[..]); + let address = compute_address(&data); + let order = closest_indices(&address, &all_peers); + // Only addresses where the fresh node is among the eight closest, so + // every receiver counts it K-close and the floor is what decides. + let Some(rank) = order.iter().position(|&j| j == fresh_index) else { + continue; + }; + if rank >= 8 { + continue; + } + let quote = match request_quote(&client, &fresh_peer, address, data.len() as u64).await { + Ok(q) => q, + Err(e) => { + eprintln!("PF-WARN near-empty quote: {e}"); + continue; + } + }; + let amount = quote.price * Amount::from(PAYMENT_MULTIPLIER); + let proof = pay(&wallet, &fresh_peer, "e, amount).await; + let mature: Vec = order + .iter() + .copied() + .filter(|&j| j != client_index && j != fresh_index) + .take(CLOSE_GROUP_SIZE) + .filter_map(|j| all_peers.get(j).map(|p| PeerId::from_bytes(*p))) + .collect(); + // The quoter first, as a client paying it would: its store is what + // triggers the fan-out whose possession check is under test. + let (fresh_outcome, fresh_err) = + paid_put(&client, &fresh_peer, address, &data, proof.clone()).await; + // The fan-out goes to the quoter's own view of the close group, which + // can differ from the order above and can include the client node. + fan_out_targets.extend( + fresh_p2p + .dht_manager() + .find_closest_nodes_local_with_self(&address, fan_out_group_size) + .await + .iter() + .filter(|n| n.peer_id != fresh_peer) + .map(|n| *n.peer_id.as_bytes()), + ); + let (mut stored, mut floor, mut other) = (0usize, 0usize, 0usize); + for peer in &mature { + match paid_put(&client, peer, address, &data, proof.clone()).await { + (PutOutcome::Stored, _) => stored += 1, + (PutOutcome::FloorRefused, _) => { + floor += 1; + refusers.insert(*peer.as_bytes()); + } + (PutOutcome::OtherError, e) => { + other += 1; + eprintln!("PF-WARN near-empty put to {peer}: {e}"); + } + } + } + assert_eq!( + fresh_outcome, + PutOutcome::Stored, + "the near-empty quoter must store its own write: {fresh_err}" + ); + // A mature storer whose own price is at most the payment over + // 3 x the tolerance accepts by design (the cap), so a few may store it + // in a low-count region. What the floor promises is that the write + // does not reach a storage quorum, and that every refusal is the + // floor's. The quoter's own store counts toward the quorum too. + assert!( + other == 0 && floor + stored == mature.len() && stored + 1 < STORE_QUORUM, + "the floor must keep a near-empty payment off a storage quorum, refusing it \ + on the floor (stored={stored} floor={floor} other={other})" + ); + eprintln!( + "PF-METRIC phase=near_empty upload={} key={} fresh_rank={rank} paid_price={} \ + mature_stored={stored} mature_floor_refused={floor} mature_other_error={other} \ + fresh_node={fresh_outcome:?}", + near_empty_keys.len(), + hex::encode(address), + quote.price + ); + ne_mature_stored += stored; + ne_mature_floor += floor; + ne_mature_other += other; + ne_fresh_stored += usize::from(fresh_outcome == PutOutcome::Stored); + near_empty_keys.push(address); + } + + assert_eq!( + honest_keys.len(), + honest_uploads, + "every requested honest upload must complete" + ); + assert_eq!( + near_empty_keys.len(), + near_empty_uploads, + "precondition unmet: not every requested near-empty upload was attempted" + ); + + // ---- Tail: fresh offers, PaidNotify and possession checks play out. + eprintln!("PF-METRIC phase=tail sleeping_secs={tail_secs}"); + tokio::time::sleep(Duration::from_secs(tail_secs as u64)).await; + + let mut honest_holders = Vec::new(); + let mut honest_votes = Vec::new(); + for key in &honest_keys { + let (h, v) = holders_and_votes(network.nodes(), key); + eprintln!( + "PF-METRIC phase=after kind=honest key={} holders={h} paid_votes={v}", + hex::encode(key) + ); + honest_holders.push(h); + honest_votes.push(v); + } + let mut ne_holders = Vec::new(); + let mut ne_votes = Vec::new(); + for key in &near_empty_keys { + let (h, v) = holders_and_votes(network.nodes(), key); + assert!( + h < STORE_QUORUM, + "the floor must keep a near-empty write off a storage quorum" + ); + eprintln!( + "PF-METRIC phase=after kind=near_empty key={} holders={h} paid_votes={v}", + hex::encode(key) + ); + ne_holders.push(h); + ne_votes.push(v); + } + + // The node that stored the near-empty writes fanned them out; it must not + // have penalised any peer that refused them, nor any peer it fanned them + // out to. + let mut min_ratio = f64::INFINITY; + let mut penalty_sized = 0usize; + let judged: Vec<&[u8; 32]> = refusers.union(&fan_out_targets).collect(); + for &peer in &judged { + let before = trust_before.get(peer).copied().unwrap_or(0.5); + let after = fresh_p2p.peer_trust(&PeerId::from_bytes(*peer)); + let ratio = if before > 0.0 { after / before } else { 1.0 }; + min_ratio = min_ratio.min(ratio); + if ratio <= POSSESSION_PENALTY_TRUST_RATIO { + penalty_sized += 1; + } + } + eprintln!( + "PF-METRIC phase=trust refusers={} fan_out_targets={} judged={} \ + min_trust_ratio={min_ratio:.3} penalty_sized_drops={penalty_sized}", + refusers.len(), + fan_out_targets.len(), + judged.len() + ); + assert_eq!( + penalty_sized, 0, + "no refuser or fan-out target may be penalised for a write under the price floor" + ); + + eprintln!( + "PF-SUMMARY honest_uploads={} honest_stored={honest_stored} honest_floor_refused={honest_floor} \ + honest_other_error={honest_other} honest_holders_median={} honest_paid_votes_median={} \ + near_empty_uploads={} near_empty_mature_stored={ne_mature_stored} \ + near_empty_mature_floor_refused={ne_mature_floor} near_empty_mature_other_error={ne_mature_other} \ + near_empty_fresh_stored={ne_fresh_stored} near_empty_holders_median={} \ + near_empty_paid_votes_median={}", + honest_keys.len(), + median(honest_holders), + median(honest_votes), + near_empty_keys.len(), + median(ne_holders), + median(ne_votes), + ); + + network.shutdown().await.expect("shutdown network"); +} diff --git a/tests/e2e/replication.rs b/tests/e2e/replication.rs index 6ae3fb5b..9cb1b01f 100644 --- a/tests/e2e/replication.rs +++ b/tests/e2e/replication.rs @@ -5,6 +5,7 @@ #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use super::harness::{parseable_single_node_proof, POSSESSION_PENALTY_TRUST_RATIO}; use super::testnet::{TestNetworkConfig, DEFAULT_CHUNK_OPERATION_TIMEOUT_SECS}; use super::TestHarness; use ant_node::ant_protocol::{ChunkMessage, ChunkMessageBody, ChunkPutRequest, ChunkPutResponse}; @@ -16,10 +17,11 @@ use ant_node::replication::config::{ REPLICATION_PROTOCOL_ID, }; use ant_node::replication::fresh::FreshWriteEvent; +use ant_node::replication::neighbor_sync; use ant_node::replication::protocol::{ compute_audit_digest, AuditChallenge, AuditResponse, FetchRequest, FetchResponse, - FreshReplicationOffer, FreshReplicationResponse, NeighborSyncRequest, ReplicationMessage, - ReplicationMessageBody, VerificationRequest, ABSENT_KEY_DIGEST, + FreshReplicationOffer, FreshReplicationResponse, NeighborSyncRequest, PaidNotify, + ReplicationMessage, ReplicationMessageBody, VerificationRequest, ABSENT_KEY_DIGEST, }; use ant_node::replication::pruning; use ant_node::replication::scheduling::ReplicationQueues; @@ -30,7 +32,7 @@ use bytes::Bytes; use saorsa_core::identity::PeerId; use saorsa_core::{P2PNode, TrustEvent}; use serial_test::serial; -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use std::fs; use std::sync::Arc; use std::time::Duration; @@ -551,6 +553,330 @@ async fn fresh_write_pipeline_never_offers_a_corrupt_chunk() { harness.teardown().await.expect("teardown"); } +/// ADR-0006: a fresh write this node cannot price is not used to judge its +/// peers. Node A's price floor is wired (a gossip cache is attached) but A has +/// no commitment of its own, so it cannot price the single-node proof it +/// forwards. +/// +/// - When A's verification of the payment skipped the floor too (stood in for +/// here, as these tests settle nothing on chain), the scheduled possession +/// check must not run, and the key is marked in A's paid list, so A hints it +/// to its peers as paid only, never as a replica they could later be +/// audited on. +/// - When A's verification was not contested but the proof A forwards is, the +/// check must not run either, and the key is recorded plainly. +/// - When a marked record of the key won before the write recorded its own +/// plain decision, as a concurrent `PaidNotify` or fresh offer can, the +/// winning mark keeps the check from running even with a proof the floor +/// never prices. +/// - The control write that follows forwards a proof the floor never prices, +/// on a key with no mark: its check does penalise the same kind of absent +/// peer and its key is hinted as a replica, so the other results are not +/// vacuous. +#[tokio::test] +#[serial] +async fn possession_check_skips_a_write_the_floor_cannot_price() { + let mut net_config = TestNetworkConfig::small(); + net_config.replication_config = Some(ReplicationConfig { + possession_check_delay_min: Duration::from_millis(200), + possession_check_delay_max: Duration::from_millis(500), + ..ReplicationConfig::default() + }); + let harness = TestHarness::setup_with_config(net_config) + .await + .expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let a_idx = 3; + let a = harness.test_node(a_idx).expect("node a"); + let p2p_a = a.p2p_node.as_ref().expect("p2p a"); + let engine_a = a.replication_engine.as_ref().expect("engine a"); + let protocol_a = a.ant_protocol.as_ref().expect("proto a"); + protocol_a + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + let self_a = *p2p_a.peer_id(); + let close_group_size = ReplicationConfig::default().close_group_size; + + let paid_list_a = engine_a.paid_list(); + let mut outcomes = Vec::new(); + for (content, parseable, admitted_contested, recorded_contested) in [ + ( + &b"adr-0006 contested fan-out payload"[..], + true, + true, + false, + ), + ( + &b"adr-0006 contested proof fan-out payload"[..], + true, + false, + false, + ), + ( + &b"adr-0006 marked record fan-out payload"[..], + false, + false, + true, + ), + ( + &b"adr-0006 control fan-out payload"[..], + false, + false, + false, + ), + ] { + let address = compute_address(content); + let proof = if parseable { + parseable_single_node_proof(address) + } else { + dummy_payment_proof() + }; + if admitted_contested { + protocol_a + .payment_verifier_arc() + .note_contested_admission_for_tests(&address); + } + if recorded_contested { + paid_list_a + .insert_price_contested(&address) + .await + .expect("record the winning mark"); + } + let close_group: Vec = p2p_a + .dht_manager() + .find_closest_nodes_local_with_self(&address, close_group_size) + .await + .iter() + .filter(|n| n.peer_id != self_a) + .map(|n| n.peer_id) + .collect(); + let close_peer = *close_group + .first() + .expect("expected a non-empty close group"); + let trust_before: Vec = close_group.iter().map(|p| p2p_a.peer_trust(p)).collect(); + + // A client PUTs the chunk to A, through A's handler and fresh-write + // pipeline. Only A holds the payment in its cache, so the close group + // refuses A's offers and is absent when, and only if, A probes it. + harness.prepopulate_payment_cache_for_peer(&self_a, &address); + put_cached_chunk(&harness, a_idx, content, proof).await; + + let recorded_by = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + while !paid_list_a.contains(&address).unwrap_or(false) + && tokio::time::Instant::now() < recorded_by + { + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + let marked = paid_list_a + .is_price_contested(&address) + .expect("read the mark"); + let replica_hinted = neighbor_sync::build_replica_hints_for_peer( + &close_peer, + &protocol_a.storage(), + paid_list_a, + p2p_a, + close_group_size, + ) + .await + .contains(&address); + let paid_hinted = neighbor_sync::build_paid_hints_for_peer( + &close_peer, + paid_list_a, + p2p_a, + ReplicationConfig::default().paid_list_close_group_size, + ) + .await + .contains(&address); + + let deadline = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + let mut penalised = false; + while tokio::time::Instant::now() < deadline { + penalised = close_group + .iter() + .zip(trust_before.iter()) + .any(|(peer, &before)| { + p2p_a.peer_trust(peer) <= before * POSSESSION_PENALTY_TRUST_RATIO + }); + if penalised { + break; + } + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + outcomes.push((penalised, marked, replica_hinted, paid_hinted)); + } + + assert_eq!( + outcomes, + vec![ + (false, true, false, true), + (false, false, true, true), + (false, true, false, true), + (true, false, true, true), + ], + "a write the floor cannot price, or whose record is marked, must not be used \ + to judge peers (no penalty; a marked key is hinted as paid only), while the \ + control write must" + ); + + harness.teardown().await.expect("teardown"); +} + +/// How a write reaches the node that records its paid key. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum RecordPath { + /// A client PUT, recorded by the storer's fresh-write pipeline. + Put, + /// A fresh offer on its own, with no `PaidNotify` before it. + FreshOffer, + /// A `PaidNotify` on its own. + PaidNotify, +} + +/// ADR-0006: every path that records a paid key marks it exactly when this +/// node's own verification of the payment was contested, and never from the +/// proof that came with the message, which a cache hit leaves unchecked. +/// +/// Each node's floor is wired but cannot price (it has no commitment of its +/// own), so every proof below would be contested if it were priced. Payments +/// are admitted from the payment cache, and the verified contested admission +/// is stood in for on the keys that have one. Each path records one key with +/// that admission, which must be marked, and one without, which must not be. +#[tokio::test] +#[serial] +async fn every_record_path_marks_a_key_from_its_own_verified_decision() { + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + for i in 0..harness.node_count() { + if let Some(protocol) = harness.test_node(i).and_then(|n| n.ant_protocol.as_ref()) { + protocol + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + } + } + + let (source_idx, receiver_idx) = (3, 1); + let source_p2p = harness + .test_node(source_idx) + .and_then(|n| n.p2p_node.as_ref()) + .expect("source p2p"); + let receiver_peer = *harness + .test_node(receiver_idx) + .and_then(|n| n.p2p_node.as_ref()) + .expect("receiver p2p") + .peer_id(); + + let mut outcomes = Vec::new(); + for (path, admitted_contested) in [ + (RecordPath::Put, true), + (RecordPath::Put, false), + (RecordPath::FreshOffer, true), + (RecordPath::FreshOffer, false), + (RecordPath::PaidNotify, true), + (RecordPath::PaidNotify, false), + ] { + let content = format!("adr-0006 record path {path:?} {admitted_contested}").into_bytes(); + let address = compute_address(&content); + let forged_proof = parseable_single_node_proof(address); + harness.prepopulate_payment_cache_everywhere(&address); + let recorder_idx = if path == RecordPath::Put { + source_idx + } else { + receiver_idx + }; + let recorder = harness.test_node(recorder_idx).expect("recorder"); + if admitted_contested { + recorder + .ant_protocol + .as_ref() + .expect("recorder protocol") + .payment_verifier_arc() + .note_contested_admission_for_tests(&address); + } + + match path { + RecordPath::Put => { + put_cached_chunk(&harness, source_idx, &content, forged_proof).await; + } + RecordPath::FreshOffer => { + let response = send_replication_request( + source_p2p, + &receiver_peer, + ReplicationMessage { + request_id: rand::random(), + body: ReplicationMessageBody::FreshReplicationOffer( + FreshReplicationOffer { + key: address, + data: content.clone(), + proof_of_payment: forged_proof, + }, + ), + }, + PROPAGATION_TIMEOUT, + ) + .await; + assert!( + matches!( + response.body, + ReplicationMessageBody::FreshReplicationResponse( + FreshReplicationResponse::Accepted { .. } + ) + ), + "the receiver must accept a cached fresh offer: {:?}", + response.body + ); + } + RecordPath::PaidNotify => { + let notify = ReplicationMessage { + request_id: rand::random(), + body: ReplicationMessageBody::PaidNotify(PaidNotify { + key: address, + proof_of_payment: forged_proof, + }), + }; + source_p2p + .send_message( + &receiver_peer, + REPLICATION_PROTOCOL_ID, + Bytes::from(notify.encode().expect("encode PaidNotify")), + &[], + ) + .await + .expect("send PaidNotify"); + } + } + + let paid_list = recorder + .replication_engine + .as_ref() + .expect("recorder engine") + .paid_list(); + let deadline = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + while !paid_list.contains(&address).unwrap_or(false) + && tokio::time::Instant::now() < deadline + { + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + assert!( + paid_list.contains(&address).unwrap_or(false), + "{path:?} never recorded the key" + ); + let marked = paid_list + .is_price_contested(&address) + .expect("read the mark"); + outcomes.push((path, admitted_contested, marked)); + } + + for (path, admitted_contested, marked) in outcomes { + assert_eq!( + marked, admitted_contested, + "{path:?}: the mark must follow this node's verified decision, never the attached proof" + ); + } + + harness.teardown().await.expect("teardown"); +} + /// Proof bytes for tests that only need to reach the pre-payment gates. fn dummy_payment_proof() -> Vec { vec![DUMMY_PAYMENT_PROOF_BYTE; DUMMY_PAYMENT_PROOF_LEN] @@ -578,14 +904,26 @@ async fn store_paid_chunk(harness: &TestHarness, source_idx: usize, content: &[u /// handler stores it and emits the fresh-write event itself, carrying the /// dummy proof it was paid with. async fn put_paid_chunk(harness: &TestHarness, source_idx: usize, content: &[u8]) -> XorName { + harness.prepopulate_payment_cache_everywhere(&compute_address(content)); + put_cached_chunk(harness, source_idx, content, dummy_payment_proof()).await +} + +/// PUT a chunk through the source node's handler, attaching `proof`. The +/// caller has put the payment in the source's cache, so the proof itself is +/// never checked. +async fn put_cached_chunk( + harness: &TestHarness, + source_idx: usize, + content: &[u8], + proof: Vec, +) -> XorName { let address = compute_address(content); - harness.prepopulate_payment_cache_everywhere(&address); let request = ChunkMessage { request_id: rand::random(), body: ChunkMessageBody::PutRequest(ChunkPutRequest::with_payment( address, Bytes::copy_from_slice(content), - dummy_payment_proof(), + proof, )), }; let protocol = harness @@ -1516,6 +1854,91 @@ async fn test_prune_pass_requires_remote_confirmation_before_delete() { harness.teardown().await.expect("teardown"); } +/// ADR-0006: a peer missing a key whose payment the price floor contested on +/// the pruner may have refused it under its own floor, so the prune audit +/// keeps the record (its proofs are still missing) but scores no one. The +/// control pass, the same shape with an unmarked key, does penalise the +/// absent peers, so the first result is not vacuous. +#[tokio::test] +#[serial] +async fn prune_audit_scores_no_peer_for_a_price_contested_key() { + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let pruner_idx = 3; + let close_group_size = 2; + let config = prune_test_config(close_group_size); + let sync_state = Arc::new(RwLock::new(NeighborSyncState::new_cycle(vec![]))); + let repair_proofs = Arc::new(RwLock::new(RepairProofs::new())); + let audit_challenge_coordinator = Arc::new(AuditChallengeCoordinator::new()); + + let pruner = harness.test_node(pruner_idx).expect("pruner"); + let pruner_p2p = Arc::clone(pruner.p2p_node.as_ref().expect("pruner p2p")); + let pruner_storage = pruner.ant_protocol.as_ref().expect("protocol").storage(); + let pruner_paid_list = Arc::clone( + pruner + .replication_engine + .as_ref() + .expect("pruner replication engine") + .paid_list(), + ); + let pruner_peer = *pruner_p2p.peer_id(); + + let mut outcomes = Vec::new(); + for (label, contested) in [("price-contested", true), ("unmarked", false)] { + let (content, address, targets) = + find_remote_prune_candidate(&harness, pruner_idx, close_group_size, label).await; + pruner_storage + .put(&address, &content) + .await + .expect("put record on pruner"); + if contested { + pruner_paid_list + .insert_price_contested(&address) + .await + .expect("mark"); + } + let trust_before: Vec = targets.iter().map(|p| pruner_p2p.peer_trust(p)).collect(); + + // No target holds the record: every one is absent when audited. + let result = pruning::run_prune_pass_with_context(pruning::PrunePassContext { + self_id: &pruner_peer, + storage: &pruner_storage, + paid_list: &pruner_paid_list, + p2p_node: &pruner_p2p, + config: &config, + sync_state: &sync_state, + repair_proofs: &repair_proofs, + allow_remote_prune_audits: true, + commitment_state: None, + audit_challenge_coordinator: &audit_challenge_coordinator, + }) + .await; + assert!( + pruner_storage.exists(&address).expect("exists"), + "a record no target proves must stay local" + ); + assert_eq!(result.records_pruned, 0); + + let penalised = targets + .iter() + .zip(trust_before.iter()) + .any(|(peer, &before)| { + pruner_p2p.peer_trust(peer) <= before * POSSESSION_PENALTY_TRUST_RATIO + }); + outcomes.push(penalised); + } + + assert_eq!( + outcomes, + vec![false, true], + "an absent peer must not be scored for a price-contested key, and must be for an \ + unmarked one" + ); + + harness.teardown().await.expect("teardown"); +} + /// Pruner-retention veto (ADR-0002): a chunk the pruner is no longer responsible /// for, but which is still committed under a recently-gossiped commitment, must /// NOT be deleted — the storage-commitment audit's round-2 byte challenge could diff --git a/tests/e2e/testnet.rs b/tests/e2e/testnet.rs index ca33a901..ec0d11a3 100644 --- a/tests/e2e/testnet.rs +++ b/tests/e2e/testnet.rs @@ -1174,7 +1174,6 @@ impl TestNetwork { cache_capacity: TEST_PAYMENT_CACHE_CAPACITY, close_group_size: replication_config.close_group_size, local_rewards_address: rewards_address, - price_floor: ant_node::payment::PriceFloorConfig::default(), }; let payment_verifier = PaymentVerifier::new(payment_config); diff --git a/tests/poc_price_floor_live.rs b/tests/poc_price_floor_live.rs index 3ba4d877..545a4be4 100644 --- a/tests/poc_price_floor_live.rs +++ b/tests/poc_price_floor_live.rs @@ -6,74 +6,84 @@ //! //! - a **real deployed payment vault** on a local Anvil chain, paid with a //! **real on-chain `pay_for_quotes` transaction**, read back through the -//! production `completedPayments` contract call (NOT the test override); and -//! - the **real `ResponderCommitmentState`** as the floor's commitment source, -//! so the local price is derived through the same production -//! `current_binding_snapshot` → `calculate_price(committed_key_count)` path -//! this node's own `QuoteGenerator` prices from. +//! production `completedPayments` contract call (NOT the test override); +//! - the **real `ResponderCommitmentState`** as this node's own commitment +//! source, read through the same production `current_binding_snapshot` → +//! `calculate_price(committed_key_count)` path this node's own +//! `QuoteGenerator` prices from; and +//! - a gossip cache of **real signed neighbour commitments**, the close-group +//! view the floor prices against. //! //! What each test asserts maps directly to a PR claim: -//! - shadow mode never rejects, even when the settlement is below floor; -//! - enforcement rejects the cheapest-of-K one-quote settlement; -//! - enforcement accepts an honest at-floor overpayment; +//! - the floor refuses a real 3x-baseline settlement; +//! - an honest at-floor overpayment is admitted, and one atto less is not; //! - reading the floor does not mutate commitment answerability. //! //! Run: `cargo test --test poc_price_floor_live --features test-utils`. #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use std::collections::HashMap; use std::sync::Arc; -use std::time::SystemTime; +use std::time::{Instant, SystemTime}; +use ant_node::payment::quote::CommitmentSource; use ant_node::payment::{ calculate_price, serialize_single_node_proof, EvmVerifierConfig, PaymentProof, PaymentStatus, - PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, VerificationContext, + PaymentVerifier, PaymentVerifierConfig, VerificationContext, PRICE_FLOOR_TOLERANCE_PERCENT, +}; +use ant_node::replication::commitment_state::{ + BuiltCommitment, PeerCommitmentRecord, ResponderCommitmentState, }; -use ant_node::replication::commitment_state::{BuiltCommitment, ResponderCommitmentState}; use ant_node::CLOSE_GROUP_SIZE; use evmlib::common::Amount; use evmlib::testnet::Testnet; use evmlib::wallet::Wallet; -use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment, RewardsAddress}; +use evmlib::{EncodedPeerId, Network, PaymentQuote, ProofOfPayment, RewardsAddress}; +use saorsa_core::identity::PeerId; use saorsa_pqc::api::sig::{ml_dsa_65, MlDsaPublicKey, MlDsaSecretKey}; +use tokio::sync::RwLock; -/// Committed key count for the receiver in the floor tests. Chosen large -/// enough that `calculate_price(count)` sits well above twice the baseline, so -/// a baseline-priced (cheapest-of-K) settlement is genuinely below a 50% -/// floor. Asserted in-test so a pricing-curve change fails loudly. -const RECEIVER_COMMITTED_KEYS: u32 = 500_000; +/// Committed key count for the receiver. Above every neighbour, so the +/// own-price cap does not bind and the reference is the group median. +const RECEIVER_COMMITTED_KEYS: u32 = 8_000; -/// The receiver's own commitment-derived local price, via the real pricing -/// curve — the exact value the production floor computes. -fn receiver_local_price() -> Amount { - calculate_price(RECEIVER_COMMITTED_KEYS as usize) -} +/// The receiver's close group: fifteen neighbours, exactly the floor's minimum +/// sample, shaped like a production group around a ~4,000-key median. +const NEIGHBOUR_KEY_COUNTS: [u32; 15] = [ + 3_000, 3_200, 3_400, 3_600, 3_800, 4_000, 4_200, 4_400, 4_600, 4_800, 5_000, 5_200, 5_400, + 5_600, 5_800, +]; + +/// Lower median of the neighbours plus the receiver's own (larger) count: the +/// eighth smallest of sixteen. +const GROUP_MEDIAN_KEYS: u32 = 4_400; /// 3x baseline: an honest 3x settlement of the *cheapest* possible quote -/// (committed count 0). This is the cheapest-of-K amount a modified client can -/// settle while still paying an honest 3x of the quote it chose. -fn cheapest_of_k_settlement() -> Amount { +/// (committed count 0). This is the near-empty-quoter amount a modified client +/// can settle while still paying an honest 3x of the quote it chose. +fn baseline_settlement() -> Amount { calculate_price(0) * Amount::from(3u64) } -/// The settled amount an at-floor honest overpayment must reach under a 50% -/// floor: `3 x 50% x local_price`. +/// The settled amount an at-floor honest overpayment must reach: +/// `3 x tolerance x` the group median price. fn at_floor_settlement() -> Amount { - receiver_local_price() * Amount::from(50u64) / Amount::from(100u64) * Amount::from(3u64) + calculate_price(GROUP_MEDIAN_KEYS as usize) * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) + / Amount::from(100u64) + * Amount::from(3u64) } fn keypair() -> (MlDsaPublicKey, MlDsaSecretKey) { ml_dsa_65().generate_keypair().unwrap() } -/// Build the receiver's real commitment source, rotated to a commitment over -/// `RECEIVER_COMMITTED_KEYS` keys so `current_binding_snapshot` reports that -/// count — the same object type production wires into both the quote generator -/// and (via this PR) the verifier's floor. -fn receiver_commitment_source() -> Arc { +/// A real signed commitment over `key_count` keys, and its issuer's peer id +/// (BLAKE3 of the public key, matching production binding). +fn signed_commitment(key_count: u32) -> ([u8; 32], BuiltCommitment) { let (public_key, secret_key) = keypair(); let peer_id_bytes = *blake3::hash(&public_key.to_bytes()).as_bytes(); - let entries: Vec<([u8; 32], [u8; 32])> = (0..RECEIVER_COMMITTED_KEYS) + let entries: Vec<([u8; 32], [u8; 32])> = (0..key_count) .map(|i| { let mut k = [0u8; 32]; k[..4].copy_from_slice(&i.to_be_bytes()); @@ -83,7 +93,16 @@ fn receiver_commitment_source() -> Arc { let built = BuiltCommitment::build(entries, &peer_id_bytes, &secret_key, &public_key.to_bytes()) .unwrap(); - assert_eq!(built.commitment().key_count, RECEIVER_COMMITTED_KEYS); + assert_eq!(built.commitment().key_count, key_count); + (peer_id_bytes, built) +} + +/// Build the receiver's real commitment source, rotated to a commitment over +/// `RECEIVER_COMMITTED_KEYS` keys so `current_binding_snapshot` reports that +/// count — the same object type production wires into both the quote generator +/// and the verifier's floor. +fn receiver_commitment_source() -> Arc { + let (_, built) = signed_commitment(RECEIVER_COMMITTED_KEYS); let state = ResponderCommitmentState::new(); state.rotate(built); Arc::new(state) @@ -125,19 +144,38 @@ fn serialize_proof(peer_quotes: Vec<(EncodedPeerId, PaymentQuote)>) -> Vec { serialize_single_node_proof(&proof).unwrap() } -/// Build a verifier wired to the live Anvil vault and the real commitment -/// source, with the given floor policy. No `P2PNode` is attached, so the -/// issuer K-closest check fail-opens under `test-utils` — that isolates the -/// on-chain settlement read and the floor as the load-bearing checks. -fn live_verifier(network: evmlib::Network, floor: PriceFloorConfig) -> PaymentVerifier { +/// Build a verifier wired to the live Anvil vault, the real commitment source +/// and a close-group view of real signed neighbour commitments. No `P2PNode` +/// is attached, so the K-closest set comes from the test override: the +/// neighbours plus every quote issuer in `issuers`. +fn live_verifier( + network: Network, + source: Arc, + issuers: &[&EncodedPeerId], +) -> PaymentVerifier { let verifier = PaymentVerifier::new(PaymentVerifierConfig { evm: EvmVerifierConfig { network }, cache_capacity: 128, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([0x11; 20]), - price_floor: floor, }); - verifier.attach_local_commitment_source(receiver_commitment_source()); + let dyn_source: Arc = source; + verifier.attach_local_commitment_source(dyn_source); + + let received_at = Instant::now(); + let mut cache = HashMap::new(); + let mut k_closest = Vec::with_capacity(NEIGHBOUR_KEY_COUNTS.len() + issuers.len()); + for count in NEIGHBOUR_KEY_COUNTS { + let (peer, built) = signed_commitment(count); + cache.insert( + PeerId::from_bytes(peer), + PeerCommitmentRecord::from_verified(built.commitment().clone(), received_at), + ); + k_closest.push(peer); + } + k_closest.extend(issuers.iter().map(|issuer| *issuer.as_bytes())); + verifier.attach_commitment_cache(Arc::new(RwLock::new(cache))); + verifier.set_paid_quote_k_closest_for_tests(k_closest); verifier } @@ -158,103 +196,82 @@ async fn pay_and_build_proof( serialize_proof(vec![(peer_id, quote)]) } -/// Shadow mode (the shipping default) must NEVER reject on the floor, even when -/// the real on-chain settlement is genuinely below the receiver's floor. -#[tokio::test(flavor = "multi_thread")] -async fn shadow_mode_accepts_below_floor_real_settlement() { - let testnet = Testnet::new().await.unwrap(); +fn anvil_wallet(testnet: &Testnet) -> (Network, Wallet) { let network = testnet.to_network(); let wallet = Wallet::new_from_private_key( network.clone(), &testnet.default_wallet_private_key().unwrap(), ) .unwrap(); - let verifier = live_verifier(network, PriceFloorConfig::default()); - assert!(!verifier.price_floor_config().enforce, "default is shadow"); - assert!( - cheapest_of_k_settlement() < at_floor_settlement(), - "test fixture must keep the cheap settlement below the floor" - ); - - let xorname = [0xA1u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([2; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, cheapest_of_k_settlement()).await; - - let status = verifier - .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) - .await - .expect("shadow mode must accept a real below-floor settlement"); - assert_eq!(status, PaymentStatus::PaymentVerified); + (network, wallet) } -/// Enforcement rejects the cheapest-of-K one-quote settlement: a real 3x-baseline -/// on-chain payment against a fuller receiver, read from the real vault. +/// The floor refuses a real 3x-baseline settlement, paid to a near-empty +/// quoter, at a receiver that knows its close group. #[tokio::test(flavor = "multi_thread")] -async fn enforced_floor_rejects_cheapest_of_k_real_settlement() { +async fn rejects_a_real_below_floor_settlement() { let testnet = Testnet::new().await.unwrap(); - let network = testnet.to_network(); - let wallet = Wallet::new_from_private_key( - network.clone(), - &testnet.default_wallet_private_key().unwrap(), - ) - .unwrap(); - let verifier = live_verifier( - network, - PriceFloorConfig { - enforce: true, - tolerance_percent: 50, - }, + let (network, wallet) = anvil_wallet(&testnet); + assert!( + baseline_settlement() < at_floor_settlement(), + "test fixture must keep the baseline settlement below the floor" ); - let xorname = [0xA2u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([3; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, cheapest_of_k_settlement()).await; + let xorname = [0xA1u8; 32]; + let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([2; 20])); + let verifier = live_verifier(network, receiver_commitment_source(), &[&peer_id]); + let proof = pay_and_build_proof(&wallet, peer_id, quote, baseline_settlement()).await; let err = verifier .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) .await - .expect_err("a real 3x-baseline settlement must not clear a fuller receiver's floor"); + .expect_err("a real 3x-baseline settlement must not clear the floor"); assert!( - err.to_string().contains("below this node's price floor"), + err.to_string() + .contains("below the close-group price floor"), "unexpected rejection reason: {err}" ); } -/// Enforcement accepts an honest at-floor overpayment: the client overpays the -/// cheap quote to `3 x 50% x local_price`, settled for real on-chain. +/// Enforcement accepts an honest at-floor overpayment and refuses one atto +/// less: the client overpays the cheap quote to exactly `3 x tolerance x` the +/// group median, settled for real on-chain. #[tokio::test(flavor = "multi_thread")] async fn enforced_floor_accepts_at_floor_real_overpayment() { let testnet = Testnet::new().await.unwrap(); - let network = testnet.to_network(); - let wallet = Wallet::new_from_private_key( - network.clone(), - &testnet.default_wallet_private_key().unwrap(), - ) - .unwrap(); + let (network, wallet) = anvil_wallet(&testnet); let source = receiver_commitment_source(); + assert!( + calculate_price(RECEIVER_COMMITTED_KEYS as usize) + > calculate_price(GROUP_MEDIAN_KEYS as usize), + "the receiver must sit above its group so the reference is the group median" + ); // Snapshot the commitment hash BEFORE verification so we can prove the // floor read did not rotate/refresh/age retention (claim: non-mutating). let hash_before = source.current().map(|c| c.hash()); - let verifier = PaymentVerifier::new(PaymentVerifierConfig { - evm: EvmVerifierConfig { - network: network.clone(), - }, - cache_capacity: 128, - close_group_size: CLOSE_GROUP_SIZE, - local_rewards_address: RewardsAddress::new([0x11; 20]), - price_floor: PriceFloorConfig { - enforce: true, - tolerance_percent: 50, - }, - }); - let concrete = Arc::clone(&source); - let dyn_source: Arc = concrete; - verifier.attach_local_commitment_source(dyn_source); - let xorname = [0xA3u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([4; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, at_floor_settlement()).await; + let short_name = [0xA3u8; 32]; + let (short_peer, short_quote) = + signed_quote(short_name, calculate_price(0), RewardsAddress::new([4; 20])); + let xorname = [0xA4u8; 32]; + let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([5; 20])); + let verifier = live_verifier(network, Arc::clone(&source), &[&short_peer, &peer_id]); + + let one_short = at_floor_settlement() - Amount::from(1u64); + let short_proof = pay_and_build_proof(&wallet, short_peer, short_quote, one_short).await; + let err = verifier + .verify_payment( + &short_name, + Some(&short_proof), + VerificationContext::ClientPut, + ) + .await + .expect_err("one atto below the floor must be refused"); + assert!(err + .to_string() + .contains("below the close-group price floor")); + let proof = pay_and_build_proof(&wallet, peer_id, quote, at_floor_settlement()).await; let status = verifier .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) .await From a044676a41a39cbb7fe11cac158d7db5473c5ffa Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 22 Oct 2026 14:58:33 +0900 Subject: [PATCH 02/22] fix(replication): drop in-memory price-contested marks a later write clears The paid list keeps its price-contested keys in memory, so building replica hints reads nothing on disk. A marked insert puts its key in that copy just before its commit and keeps it if the commit fails, since the commit may still have reached the disk. When it had not, the copy held a mark the database lacked, and nothing removed it: a later plain insert of the key wrote an unmarked entry, and a batch removal cleared only the keys the database reported deleted. Replica hints then left out a key that every point read, the audit included, treated as unmarked, until a restart. A plain insert that adds the key now drops any in-memory mark for it once its commit succeeds, still under the write order, and a removal, single or batched, drops the mark of every key it names, held or not. A failed commit still leaves the copy as it was, so the copy stays a superset of the committed marks, and equals them for every key a successful write touched. Tests mark a key in memory only, as such a failed commit leaves it, and check that a plain insert, a batch removal and a single removal of the key each clear it, while a key whose entry carries the mark keeps it. --- .../ADR-0006-receiver-side-revenue-floor.md | 13 ++- src/replication/paid_list.rs | 91 ++++++++++++++++--- 2 files changed, 86 insertions(+), 18 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index c7194743..ab07ac54 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -381,8 +381,9 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over accepted the key. The node also keeps its marked keys in memory, so leaving them out of the hints it builds for a peer reads nothing on disk. Each paid-list write updates that copy in commit order, a new - mark just before its commit, so the copy never lacks a committed mark. - If the marks cannot be read when the node starts, it still starts, as + mark just before its commit, so the copy never lacks a committed mark, + and a write that commits the key unmarked, a plain insert that adds it + or a removal, drops any mark the copy holds for it. If the marks cannot be read when the node starts, it still starts, as before, and hints no key as a replica until a restart can read them; - the responsible audit never samples a marked key, which also covers one hinted in the moment before its mark was written; @@ -589,7 +590,9 @@ out of 1,801: the paid list, so unmarked. When a marked insert's commit is what failed, the in-memory copy keeps the mark, since the commit may still have reached the disk, so this node does not hint the key as a replica - while the mark stays; a restart rebuilds the copy from the database. + while the mark stays. A later plain insert that adds the key, or a + removal of it, drops the mark once that write commits, and a restart + rebuilds the copy from the database. - Paid-list pruning checks for the record before and after its network round, but a record admitted between that check and the delete would lose its mark. Both store paths require this node inside the same width @@ -667,7 +670,9 @@ out of 1,801: held; - the in-memory copy of the marks is filled on opening and follows every write and removal, and a marked insert and a batch removal change it - only in the order writes take; building + only in the order writes take; a mark held in memory alone, as a failed + commit can leave it, goes with a later plain insert of the key and with + its removal, single or batched; building replica hints works while the paid list cannot be read at all, and a paid list whose marks cannot be read on opening still opens and hints no replica; diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index d196cd52..0369921f 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -34,6 +34,7 @@ use heed::{Database, Env, EnvOpenOptions, RoTxn}; use parking_lot::{Mutex, RwLock}; use std::collections::{HashMap, HashSet}; use std::path::Path; +use std::slice; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use std::time::Instant; @@ -75,6 +76,20 @@ fn read_price_contested_keys(db: &Database, txn: &RoTxn) -> Result Ok(keys) } +/// Drop `keys` from the in-memory marks, after a commit that left none of +/// them marked in the database. The set is copied only if one of them is in +/// it. +fn unmark_in_memory(marks: &RwLock>>>, keys: &[XorName]) { + if let Some(marked) = marks.write().as_mut() { + if keys.iter().any(|key| marked.contains(key)) { + let marked = Arc::make_mut(marked); + for key in keys { + marked.remove(key); + } + } + } +} + /// Persistent paid-for-list backed by LMDB. /// /// Tracks which keys this node believes are paid-authorized. @@ -97,7 +112,8 @@ pub struct PaidList { /// A new mark goes in just before its commit and a removed one just after, /// so the copy never lacks a committed mark. A mark whose commit fails /// stays, since that commit may still have reached the disk: when in - /// doubt, judge no one. + /// doubt, judge no one. Any later write that commits the key unmarked, a + /// plain insert that adds it or a removal, drops that mark. write_order: Arc>, /// Mark reads that have failed since the last one that succeeded, so a /// run of failures is logged once (see `counts_as_price_contested`). @@ -273,6 +289,9 @@ impl PaidList { wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list insert: {e}")) })?; + if value.is_empty() { + unmark_in_memory(&price_contested, slice::from_ref(&key_owned)); + } drop(ordered); Ok(true) @@ -391,11 +410,7 @@ impl PaidList { wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list delete: {e}")) })?; - if let Some(keys) = price_contested.write().as_mut() { - if keys.contains(&key_owned) { - Arc::make_mut(keys).remove(&key_owned); - } - } + unmark_in_memory(&price_contested, slice::from_ref(&key_owned)); drop(ordered); Ok(deleted) }) @@ -624,14 +639,9 @@ impl PaidList { wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit batch remove: {e}")))?; - if let Some(keys) = price_contested.write().as_mut() { - if removed.iter().any(|key| keys.contains(key)) { - let keys = Arc::make_mut(keys); - for key in &removed { - keys.remove(key); - } - } - } + // Every key of the batch, not only those the database held: a + // mark whose commit failed can be in memory alone. + unmark_in_memory(&price_contested, &keys_owned); drop(ordered); Ok(removed) @@ -898,6 +908,59 @@ mod tests { assert!(marked_in_memory(&pl).is_empty()); } + /// Mark `key` in memory only, with no entry in the database: what a marked + /// insert whose commit failed before reaching the disk leaves. + fn mark_in_memory_only(pl: &PaidList, key: XorName) { + let mut marks = pl.price_contested.write(); + let marks = marks.as_mut().expect("marks read on open"); + Arc::make_mut(marks).insert(key); + } + + /// ADR-0006: a mark left in memory alone by a failed commit goes with a + /// later plain insert that adds the key, so the copy agrees with the + /// database again and the key is hinted as a replica. A plain insert of a + /// key whose entry carries the mark leaves both as they are. + #[tokio::test] + async fn a_plain_insert_drops_a_mark_held_in_memory_only() { + let (pl, _temp) = create_test_paid_list().await; + let stale: XorName = [0x71; 32]; + let marked: XorName = [0x72; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + mark_in_memory_only(&pl, stale); + let mut keys = vec![stale]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + + assert!(pl.insert(&stale).await.expect("insert")); + assert!(!pl.insert(&marked).await.expect("insert")); + assert!(!pl.is_price_contested(&stale).expect("read mark")); + assert!(pl.is_price_contested(&marked).expect("read mark")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + let mut keys = vec![stale, marked]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, vec![stale]); + } + + /// ADR-0006: a mark left in memory alone by a failed commit goes with a + /// later removal of the key, single or batched, although the database + /// holds no entry to delete. A mark the removal does not name stays. + #[tokio::test] + async fn a_removal_drops_a_mark_held_in_memory_only() { + let (pl, _temp) = create_test_paid_list().await; + let batched: XorName = [0x73; 32]; + let single: XorName = [0x74; 32]; + let marked: XorName = [0x75; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + mark_in_memory_only(&pl, batched); + mark_in_memory_only(&pl, single); + + assert_eq!(pl.remove_batch(&[batched]).await.expect("remove batch"), 0); + assert_eq!(marked_in_memory(&pl), HashSet::from([single, marked])); + assert!(!pl.remove(&single).await.expect("remove")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + assert!(pl.is_price_contested(&marked).expect("read mark")); + } + /// Byte offsets in an LMDB meta page on a 64-bit target: the magic number /// follows the 16-byte page header, the free list's record keeps the page /// size at 40, and the main database's root page number sits at 128. From 7a735be906ae69d1e4fce4d2318dffe8c0769768 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 22 Oct 2026 16:37:28 +0900 Subject: [PATCH 03/22] docs(adr): count the reconciliation mutations in ADR-0006 The mutation run on the price-floor code now covers the in-memory mark reconciliation too: 54 mutations, of which 53 fail at least one unit or e2e test. The one survivor is unchanged: prune-audit scoring reading a mark that cannot be read as unset, which only the e2e prune test reaches, and that test cannot make a paid-list read fail. --- docs/adr/ADR-0006-receiver-side-revenue-floor.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index ab07ac54..165e7e44 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -700,9 +700,9 @@ out of 1,801: cache with a forged proof the floor would contest, mark the key exactly when the recording node's own verified decision was contested, and never from that proof. -- 48 mutations of the floor, the contest decision, the verifier's memory, +- 54 mutations of the floor, the contest decision, the verifier's memory, the mark, its in-memory copy and every check that reads the mark were run - against these unit and e2e tests on this amendment's code, and 47 fail at + against these unit and e2e tests on this amendment's code, and 53 fail at least one of them. The one that survives makes prune-audit scoring count a mark that cannot be read as unset. Only the e2e prune test reaches that check, and it cannot make a node's paid-list read fail; the same read From 2252074964d97d1254d3476a62ab2d040c488480 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 29 Oct 2026 10:35:18 +0900 Subject: [PATCH 04/22] docs(adr): say a fan-out, not the record, reads the current view ADR-0006's validation list said the contest decision follows the view when a write is recorded, and the unit test pinning it said the same. That holds for the proof a fan-out forwards, chunk or pointer, which is priced against the view when the write is fanned out. The paid-list mark a chunk is recorded with comes from the decision its verification made, which the verifier remembers, not from the view at record time. Both texts now say so. No code changes. --- docs/adr/ADR-0006-receiver-side-revenue-floor.md | 4 +++- src/payment/verifier.rs | 8 ++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 165e7e44..1ca5ba49 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -656,7 +656,9 @@ out of 1,801: - a write below the contest threshold, or one the node cannot price, is contested, and one at the median is not. The threshold is 60% of the uncapped median to the atto, it stays on the median when the cap binds, - and it follows the view when the write is recorded; + and a fan-out judges the proof it forwards against the view when it fans + out, while a chunk's paid-list mark keeps the decision its verification + made; - the tolerance is 30%: exactly `3 x 30%` of the group median clears it and one atto less does not; - a node with no floor wired contests nothing; diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index a8f57c84..45e8b003 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -5048,10 +5048,10 @@ mod tests { assert!(!verifier.price_contested(&xorname, &proof_bytes).await); } - /// The decision is taken from the proof against the current view when the - /// write is recorded (or, for a pointer, fanned out), so a write stored - /// after the group's prices rose is judged against the view peers will - /// price it against now. + /// A fan-out, chunk or pointer, judges the proof it forwards against the + /// view when it fans out, so a write fanned out after the group's prices + /// rose is judged against the view peers will price it against now. (A + /// chunk's paid-list mark keeps the decision its verification made.) #[tokio::test] async fn the_contest_follows_the_current_view() { let verifier = floor_test_verifier(); From 16e1fc77a561c92421ca94eb8d6d93292d895216 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 29 Oct 2026 10:37:45 +0900 Subject: [PATCH 05/22] test: name the tokio types the new test helpers use by import The paid-list helper that holds the write order spelled out tokio::sync::oneshot and tokio::task::JoinHandle inline, and the verifier test for a node with no floor wired built its gossip cache as tokio::sync::RwLock inline. The paid-list tests now import oneshot and JoinHandle, the helper waits for the order with an assertion instead of expect, and the verifier test uses CommitmentCache::default(). No behaviour changes. --- src/payment/verifier.rs | 2 +- src/replication/paid_list.rs | 17 +++++++---------- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 45e8b003..12ca85c2 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -5044,7 +5044,7 @@ mod tests { let verifier = create_test_verifier(); assert!(!verifier.price_contested(&xorname, &proof_bytes).await); - verifier.attach_commitment_cache(Arc::new(tokio::sync::RwLock::new(HashMap::new()))); + verifier.attach_commitment_cache(CommitmentCache::default()); assert!(!verifier.price_contested(&xorname, &proof_bytes).await); } diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index 0369921f..a6d9c27b 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -709,6 +709,8 @@ mod tests { use std::fs::OpenOptions; #[cfg(target_pointer_width = "64")] use std::io::{Read, Seek, SeekFrom, Write}; + use tokio::sync::oneshot; + use tokio::task::JoinHandle; #[cfg(feature = "logging")] use tracing::subscriber::with_default; #[cfg(feature = "logging")] @@ -1309,21 +1311,16 @@ mod tests { /// /// Returns once the order is held. Dropping the returned sender releases /// it. - async fn hold_write_order( - pl: &PaidList, - ) -> ( - tokio::sync::oneshot::Sender<()>, - tokio::task::JoinHandle<()>, - ) { + async fn hold_write_order(pl: &PaidList) -> (oneshot::Sender<()>, JoinHandle<()>) { let write_order = Arc::clone(&pl.write_order); - let (release_tx, release_rx) = tokio::sync::oneshot::channel::<()>(); - let (held_tx, held_rx) = tokio::sync::oneshot::channel::<()>(); - let holder = tokio::task::spawn_blocking(move || { + let (release_tx, release_rx) = oneshot::channel::<()>(); + let (held_tx, held_rx) = oneshot::channel::<()>(); + let holder = spawn_blocking(move || { let _ordered = write_order.lock(); let _ = held_tx.send(()); let _ = release_rx.blocking_recv(); }); - held_rx.await.expect("write order held"); + assert!(held_rx.await.is_ok(), "write order held"); (release_tx, holder) } From ce058715ad698a67f5105628e9477f96a854b0f9 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 13:14:27 +0900 Subject: [PATCH 06/22] test(replication): pin that eviction takes a price-contested mark with its entry Paid-list eviction (ADR-0028) removes entries through remove_batch. The new test marks two far entries, evicts one and keeps the other, and checks that the evicted entry's mark leaves the in-memory copy while the kept entry stays marked on disk and in memory, so no mark is held in memory alone after eviction. --- src/replication/paid_list.rs | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index 661ff57a..54639a2b 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -1064,6 +1064,32 @@ mod tests { assert!(marked_in_memory(&pl).is_empty()); } + /// ADR-0006 with ADR-0028: eviction drops the in-memory mark of an entry + /// it evicts, so no mark outlives its entry in memory alone, and a marked + /// entry it keeps stays marked. + #[tokio::test] + async fn eviction_takes_an_evicted_entry_s_mark_and_leaves_a_kept_one() { + let (pl, _temp) = create_test_paid_list().await; + let (kept, evicted_key) = (key_with_first_byte(0x80), key_with_first_byte(0x81)); + pl.insert_price_contested(&kept).await.expect("mark"); + pl.insert_price_contested(&evicted_key) + .await + .expect("mark"); + + let evicted = pl + .evict_far([0u8; 32], XORNAME_BITS - 1, 10, move |key| *key == kept) + .await + .expect("evict"); + + assert_eq!(evicted, 1); + assert!(!pl.contains(&evicted_key).expect("contains")); + assert!(pl.is_price_contested(&kept).expect("read mark")); + assert_eq!(marked_in_memory(&pl), HashSet::from([kept])); + let mut keys = vec![kept, evicted_key]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, vec![evicted_key]); + } + /// ADR-0006: while a database write holds the write order, a marked /// insert or a batch removal neither commits nor changes the in-memory /// copy, so writes and the copy move in one order. From 813b10310ea3f6c9315db02f683df08c01862e4e Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 13:24:59 +0900 Subject: [PATCH 07/22] docs(adr): ADR-0006 follows pressure-gated retention ADR-0028 replaced paid-list pruning by rank with eviction above a threshold, and sends prune audits only under disk pressure. Amendment 2 described the mark against the old pruning, so it now says: - paid-list eviction keeps the entry of every record still held, marked or not, and an evicted entry takes its mark with it; the narrow gap where a record admitted mid-pass loses its mark is restated for eviction; - prune audits, and the e2e test that a marked key's audit scores no one, run under disk pressure; - of the 54 mutations run before the merge, two targeted the paid-pruning rule that is now gone, and two more run on the merged code both fail; - opening the paid list reads every entry once to fill the in-memory marks, which a list at the 15 million entry eviction threshold makes about a third of a second with a warm page cache. --- .../ADR-0006-receiver-side-revenue-floor.md | 56 ++++++++++++------- 1 file changed, 36 insertions(+), 20 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 1ca5ba49..3ac44338 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -10,7 +10,8 @@ - **Related:** ADR-0004 (commitment-bound quote pricing), ant-node #136 (flexible 1..=7 proof bundles), ant-node #176 (first implementation, the approach withdrawn below), ant-node #193 (close-group pricing), - ant-client #150 (one-quote uploads) + ant-client #150 (one-quote uploads), ADR-0028 (pressure-gated retention, + which replaced the paid-list pruning Amendment 2 first relied on) ## Context @@ -387,9 +388,11 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over before, and hints no key as a replica until a restart can read them; - the responsible audit never samples a marked key, which also covers one hinted in the moment before its mark was written; - - a prune audit still needs its proofs before deleting a marked key, but - scores no absent peer for it, and paid-list pruning keeps a marked entry - while the record is held; + - a prune audit, which is sent only under disk pressure (ADR-0028), still + needs its proofs before deleting a marked key, but scores no absent peer + for it. Paid-list eviction (ADR-0028) keeps the entry of every record + the node still holds, marked or not, and an entry it evicts takes its + mark with it; - a mark that cannot be read counts as set: when in doubt, judge no one. While reads fail the node judges no peer on any key they hit, so it warns at the first failure of a run and, when a read succeeds again, @@ -593,11 +596,12 @@ out of 1,801: while the mark stays. A later plain insert that adds the key, or a removal of it, drops the mark once that write commits, and a restart rebuilds the copy from the database. - - Paid-list pruning checks for the record before and after its network - round, but a record admitted between that check and the delete would - lose its mark. Both store paths require this node inside the same width - the prune requires it outside, so that needs its routing view to change - at that moment. + - Paid-list eviction checks for the record when it scans, but a record + admitted between that scan and the delete would lose its entry, and the + mark with it. Eviction takes only entries far outside this node's range, + where with evenly spread IDs neither store path admits the record, so + that needs a key whose peers this node barely knows, or a routing view + that changes at that moment. - **The first paid-list record decides a key's mark, which a quoter can front-run.** A quoter in the 20 closest to an honest key can pay itself for the key, for gas only, and announce it before the client's write @@ -636,6 +640,13 @@ out of 1,801: on-chain read included: up to 20 targets over 4 rounds. The load falls on below-floor proofs, including the honest residuals above (a lone baseline quote, an aged proof). +- **Opening the paid list reads every entry once**, to fill the in-memory + copy of the marks, and the node starts only after that read. Since + pressure-gated retention (ADR-0028) the list can hold 15 million entries + before it evicts any. A local probe read 4 million entries in 85 ms with + the file already in the page cache, about 21 ns an entry, so a list at + that size takes about a third of a second, plus the time to read the file + from disk on a cold start. - Floor refusals stay economic. The refusal itself feeds no trust or misbehaviour scoring, and on nodes running this release item 4 keeps the checks that judge peers on held data from turning it into a penalty, within @@ -668,8 +679,8 @@ out of 1,801: is written only with a new entry, survives a restart and a later plain insert, and goes with the entry; - a marked key is left out of replica hints and of responsible-audit - sampling, and its paid entry outlives paid pruning while its record is - held; + sampling, and paid-list eviction drops the mark of an entry it evicts + and leaves a kept entry marked; - the in-memory copy of the marks is filled on opening and follows every write and removal, and a marked insert and a batch removal change it only in the order writes take; a mark held in memory alone, as a failed @@ -678,8 +689,8 @@ out of 1,801: replica hints works while the paid list cannot be read at all, and a paid list whose marks cannot be read on opening still opens and hints no replica; - - a mark that cannot be read counts as set in the responsible audit, paid - pruning and a write's fan-out, and a run of such reads logs one + - a mark that cannot be read counts as set in the responsible audit and a + write's fan-out, and a run of such reads logs one warning. The tests make the read fail by holding the thread's one LMDB reader slot; - a record reports the winning record's mark, or its own when that is @@ -696,19 +707,24 @@ out of 1,801: one, while the control write is penalised; - a fresh pointer state the storer cannot price is not used to judge its peers, while the control is; - - a prune audit of a marked record keeps it but scores none of the absent - peers, while the control scores them; + - a prune audit of a marked record, run under disk pressure, keeps it but + scores none of the absent peers, while the control scores them; - a PUT, a fresh offer and a `PaidNotify`, each admitted from the payment cache with a forged proof the floor would contest, mark the key exactly when the recording node's own verified decision was contested, and never from that proof. - 54 mutations of the floor, the contest decision, the verifier's memory, the mark, its in-memory copy and every check that reads the mark were run - against these unit and e2e tests on this amendment's code, and 53 fail at - least one of them. The one that survives makes prune-audit scoring count - a mark that cannot be read as unset. Only the e2e prune test reaches that - check, and it cannot make a node's paid-list read fail; the same read - counts as set, under test, in the audit, paid pruning and the fan-out. + against these unit and e2e tests on this amendment's code before ADR-0028 + merged, and 53 failed at least one of them. Two of the 54 targeted the + paid-pruning rule ADR-0028 made unnecessary, which is gone. The one that + survived makes prune-audit scoring count a mark that cannot be read as + unset. Only the e2e prune test reaches that check, and it cannot make a + node's paid-list read fail; the same read counts as set, under test, in + the audit and the fan-out. Two more were run on the code merged with + ADR-0028, and both fail: an eviction that leaves an evicted entry's mark + in memory fails four unit tests, and prune-audit scoring that ignores the + mark fails the e2e prune test, which runs under disk pressure. - `tests/e2e/price_floor_ab.rs` drives the same honest and near-empty-quoter uploads through a production-wired local testnet. Nodes are seeded to production-sized commitments, payments are real (on Anvil), and possession From ada05dd9a085743c1b0018c4e00913a478d877d4 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 13:57:37 +0900 Subject: [PATCH 08/22] fix(replication): hint a stored price-contested key at any distance A key marked price-contested goes out as a paid hint only, never as a replica hint (ADR-0006). Since pressure-gated retention (ADR-0028), a node with more than 100,000 paid entries builds paid hints only from keys near itself, while replica hints have no distance limit. So a marked record it retained far from itself reached its close group in neither hint set, where an unmarked one still goes out as a replica hint. Paid-hint candidates now also take every marked key the node stores and still holds a paid entry for, however far it is. The marks and the store's index are in memory, so each far marked key costs one paid-list read. build_paid_hints_for_peer takes the chunk store for this, PaidList gains marked_keys and paid_list gains is_near. ADR-0006 says so, and lists the new unit test. --- .../ADR-0006-receiver-side-revenue-floor.md | 17 ++- src/replication/neighbor_sync.rs | 139 +++++++++++++++--- src/replication/paid_list.rs | 24 ++- tests/e2e/replication.rs | 8 +- 4 files changed, 158 insertions(+), 30 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 3ac44338..88981992 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -379,9 +379,14 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over repair proof is recorded for it, so no responsible audit challenges a peer on it, and no peer can penalise this node as its sole hinter. A paid hint is never punished, and repair still works wherever the group - accepted the key. The node also keeps its marked keys in memory, so - leaving them out of the hints it builds for a peer reads nothing on - disk. Each paid-list write updates that copy in commit order, a new + accepted the key. Above 100,000 entries a node builds paid hints only + from keys near itself (ADR-0028), while replica hints have no such + limit, so a marked key it stores is a paid-hint candidate at any + distance: a record it retains far from itself still reaches whichever + peers it knows in that key's paid close group. The node also keeps its + marked keys in memory, so leaving them out of the hints it builds for a + peer, or adding them, reads nothing on disk but each far stored marked + key's paid entry. Each paid-list write updates that copy in commit order, a new mark just before its commit, so the copy never lacks a committed mark, and a write that commits the key unmarked, a plain insert that adds it or a removal, drops any mark the copy holds for it. If the marks cannot be read when the node starts, it still starts, as @@ -679,8 +684,10 @@ out of 1,801: is written only with a new entry, survives a restart and a later plain insert, and goes with the entry; - a marked key is left out of replica hints and of responsible-audit - sampling, and paid-list eviction drops the mark of an entry it evicts - and leaves a kept entry marked; + sampling, a marked key this node stores is a paid-hint candidate + however far it is, while one it does not store, an unmarked one or a + near one is left to the near-key read, and paid-list eviction drops the + mark of an entry it evicts and leaves a kept entry marked; - the in-memory copy of the marks is filled on opening and follows every write and removal, and a marked insert and a batch removal change it only in the order writes take; a mark held in memory alone, as a failed diff --git a/src/replication/neighbor_sync.rs b/src/replication/neighbor_sync.rs index 8b579bac..4fb2f002 100644 --- a/src/replication/neighbor_sync.rs +++ b/src/replication/neighbor_sync.rs @@ -18,7 +18,7 @@ use crate::error::Result; use crate::replication::config::{ ReplicationConfig, PAID_HINT_FULL_SCAN_ENTRIES, REPLICATION_PROTOCOL_ID, }; -use crate::replication::paid_list::{near_bits_for_radius, PaidList, XORNAME_BITS}; +use crate::replication::paid_list::{is_near, near_bits_for_radius, PaidList, XORNAME_BITS}; use crate::replication::protocol::{ NeighborSyncRequest, NeighborSyncResponse, ReplicationMessage, ReplicationMessageBody, }; @@ -163,16 +163,40 @@ fn paid_hint_scan_bits(entries: u64, near_bits: usize) -> usize { } } -/// The paid-list keys a hint build considers. +/// The paid-list keys a hint build considers (see [`paid_hint_keys`]). async fn paid_hint_candidates( paid_list: &PaidList, + storage: &ChunkStore, p2p_node: &Arc, group_size: usize, ) -> Result> { let self_xor: XorName = *p2p_node.peer_id().as_bytes(); let near_bits = paid_near_bits(p2p_node, group_size).await; let scan_bits = paid_hint_scan_bits(paid_list.count()?, near_bits); - paid_list.keys_near(&self_xor, scan_bits) + paid_hint_keys(paid_list, storage, &self_xor, scan_bits) +} + +/// The paid-list keys within the `scan_bits` around `center`, and the keys +/// marked price-contested (ADR-0006) beyond them that this node stores and +/// still holds a paid entry for. A marked key goes out as a paid hint only, +/// never as a replica hint, and replica hints have no distance limit: without +/// this, a marked record this node retains far from itself (ADR-0028) would +/// reach its close group in neither hint set. The marks and the store's index +/// are in memory, and marks are few, so the far marked keys cost one paid-list +/// read each. +fn paid_hint_keys( + paid_list: &PaidList, + storage: &ChunkStore, + center: &XorName, + scan_bits: usize, +) -> Result> { + let mut keys = paid_list.keys_near(center, scan_bits)?; + keys.extend(paid_list.marked_keys().into_iter().filter(|key| { + !is_near(center, key, scan_bits) + && storage.exists(key).unwrap_or(false) + && paid_list.contains(key).unwrap_or(false) + })); + Ok(keys) } /// Build outbound hint sets for a batch of peers with one scan over local @@ -222,14 +246,20 @@ pub(crate) async fn build_sync_hints_for_peers( } } - let paid_keys = - match paid_hint_candidates(paid_list, p2p_node, paid_list_close_group_size).await { - Ok(keys) => keys, - Err(e) => { - warn!("Failed to read PaidForList for batch hint construction: {e}"); - Vec::new() - } - }; + let paid_keys = match paid_hint_candidates( + paid_list, + storage, + p2p_node, + paid_list_close_group_size, + ) + .await + { + Ok(keys) => keys, + Err(e) => { + warn!("Failed to read PaidForList for batch hint construction: {e}"); + Vec::new() + } + }; let paid_key_count = paid_keys.len(); for key in paid_keys { @@ -284,20 +314,29 @@ pub(crate) async fn build_sync_hints_for_peers( /// /// Returns keys from our `PaidForList` that we believe the peer should /// track (peer is among `PAID_LIST_CLOSE_GROUP_SIZE` nearest to `K`). +/// `storage` tells which price-contested keys this node stores, since those +/// are hinted at any distance (see `paid_hint_keys`). pub async fn build_paid_hints_for_peer( peer: &PeerId, + storage: &Arc, paid_list: &Arc, p2p_node: &Arc, paid_list_close_group_size: usize, ) -> Vec { - let paid_keys = - match paid_hint_candidates(paid_list, p2p_node, paid_list_close_group_size).await { - Ok(keys) => keys, - Err(e) => { - warn!("Failed to read PaidForList for hint construction: {e}"); - return Vec::new(); - } - }; + let paid_keys = match paid_hint_candidates( + paid_list, + storage, + p2p_node, + paid_list_close_group_size, + ) + .await + { + Ok(keys) => keys, + Err(e) => { + warn!("Failed to read PaidForList for hint construction: {e}"); + return Vec::new(); + } + }; let dht = p2p_node.dht_manager(); let mut hints = Vec::new(); @@ -606,6 +645,7 @@ pub(crate) async fn handle_sync_request_with_proofs( .collect::>(); let paid_hints = build_paid_hints_for_peer( sender, + storage, paid_list, p2p_node, config.paid_list_close_group_size, @@ -681,6 +721,67 @@ mod tests { assert_eq!(hinted, expected); } + /// ADR-0006 with ADR-0028: a price-contested key this node stores is read + /// for paid hints at any distance, as it would be for replica hints, while + /// a far key it does not store or one with no mark is left out of a narrow + /// read, and a full read lists each key once. + #[tokio::test] + async fn a_stored_marked_key_is_a_paid_hint_candidate_at_any_distance() { + const NARROW_SCAN_BITS: usize = 8; + let dir = tempfile::tempdir().expect("tempdir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().join("chunks"), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let contents: [&[u8]; 3] = [ + b"near marked record", + b"far marked record", + b"far plain record", + ]; + let [near_marked, far_marked, far_plain] = contents.map(ChunkStore::compute_address); + for (key, content) in [near_marked, far_marked, far_plain].iter().zip(contents) { + storage.put(key, content).await.expect("put"); + } + let far_marked_unstored = ChunkStore::compute_address(b"far marked, not stored"); + for key in [near_marked, far_marked, far_marked_unstored] { + paid_list.insert_price_contested(&key).await.expect("mark"); + } + paid_list.insert(&far_plain).await.expect("insert"); + let center = near_marked; + for key in [far_marked, far_plain, far_marked_unstored] { + assert!(!is_near(¢er, &key, NARROW_SCAN_BITS), "a far key"); + } + + let read = |scan_bits| { + let mut keys = + paid_hint_keys(&paid_list, &storage, ¢er, scan_bits).expect("read keys"); + keys.sort_unstable(); + keys + }; + let sorted = |mut keys: Vec| { + keys.sort_unstable(); + keys + }; + + assert_eq!( + read(NARROW_SCAN_BITS), + sorted(vec![near_marked, far_marked]) + ); + assert_eq!( + read(XORNAME_BITS), + sorted(vec![ + near_marked, + far_marked, + far_plain, + far_marked_unstored + ]), + "a full read covers every key once" + ); + } + /// A key whose last byte is `b`, the rest zero. fn key_ending_in(b: u8) -> XorName { let mut key = [0u8; 32]; diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index 54639a2b..de89e5d9 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -97,6 +97,14 @@ fn prefix_range(key: &XorName, free_bits: usize) -> (XorName, XorName) { (low, high) } +/// Whether `key` shares all but the low `near_bits` bits with `center`, so +/// [`PaidList::keys_near`] reads it. +#[must_use] +pub fn is_near(center: &XorName, key: &XorName, near_bits: usize) -> bool { + let (low, high) = prefix_range(center, near_bits); + low <= *key && *key <= high +} + /// The key ranges outside the `near_bits` around `center`, farthest first: each /// holds the keys whose highest bit differing from `center` is one given bit. fn far_ranges(center: &XorName, near_bits: usize) -> impl Iterator { @@ -290,8 +298,8 @@ impl PaidList { price_contested, )) }) - .await - .map_err(|e| Error::Storage(format!("Paid-list init task failed: {e}")))??; + .await + .map_err(|e| Error::Storage(format!("Paid-list init task failed: {e}")))??; // A paid list whose marks cannot be read still opens; the node then // sends no replica hints until it restarts (see `retain_unmarked`). let price_contested = price_contested @@ -471,6 +479,14 @@ impl PaidList { } } + /// The keys marked price-contested in the in-memory copy, read without + /// touching the database. It is empty if the marks could not be read when + /// the paid list opened. + pub fn marked_keys(&self) -> Vec { + let snapshot = self.price_contested.read().clone(); + snapshot.map_or_else(Vec::new, |marked| marked.iter().copied().collect()) + } + /// Describe a failed insert. A full store is also reported, once, as an /// error: from then on the node can't record that a new key was paid for. fn insert_error(&self, e: &heed::Error) -> Error { @@ -1072,9 +1088,7 @@ mod tests { let (pl, _temp) = create_test_paid_list().await; let (kept, evicted_key) = (key_with_first_byte(0x80), key_with_first_byte(0x81)); pl.insert_price_contested(&kept).await.expect("mark"); - pl.insert_price_contested(&evicted_key) - .await - .expect("mark"); + pl.insert_price_contested(&evicted_key).await.expect("mark"); let evicted = pl .evict_far([0u8; 32], XORNAME_BITS - 1, 10, move |key| *key == kept) diff --git a/tests/e2e/replication.rs b/tests/e2e/replication.rs index 7e1377e4..ed51ef4c 100644 --- a/tests/e2e/replication.rs +++ b/tests/e2e/replication.rs @@ -692,6 +692,7 @@ async fn possession_check_skips_a_write_the_floor_cannot_price() { .contains(&address); let paid_hinted = neighbor_sync::build_paid_hints_for_peer( &close_peer, + &protocol_a.storage(), paid_list_a, p2p_a, ReplicationConfig::default().paid_list_close_group_size, @@ -1935,7 +1936,10 @@ async fn prune_audit_scores_no_peer_for_a_price_contested_key() { disk_pressure: &disk_pressure, }) .await; - assert!(result.disk_pressure, "the pass must run under disk pressure"); + assert!( + result.disk_pressure, + "the pass must run under disk pressure" + ); assert!( pruner_storage.exists(&address).expect("exists"), "a record no target proves must stay local" @@ -2486,8 +2490,10 @@ async fn a_neighbour_gets_a_paid_hint_for_a_key_far_from_this_node() { ); paid_list.insert(&key).await.expect("insert"); + let storage = node.ant_protocol.as_ref().expect("protocol").storage(); let hints = ant_node::replication::neighbor_sync::build_paid_hints_for_peer( &farthest, + &storage, &paid_list, &p2p, ONLY_THE_CLOSEST, From 7bbe0e12d534981f30bbbb189d1b677986b6ee35 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 14:08:17 +0900 Subject: [PATCH 09/22] fix(replication): hint every stored paid key far away while the marks are unknown When the price-contested marks cannot be read as the paid list opens, the node hints no key as a replica, since any of them may be marked. Before ADR-0028 its paid hints still read every paid entry, so its stored records reached their close groups that way. Above 100,000 entries paid hints now read only near keys, so in that state a record the node retained far from itself went out in neither hint set. While the marks are unknown, every key the node stores that has a paid entry is now a paid-hint candidate at any distance, as a marked key is when the marks are known. That moves the per-key work the replica hints no longer do onto the paid hints. PaidList::marked_keys says when the marks are unknown, and the unit test covers that state. ADR-0006 says so. --- .../ADR-0006-receiver-side-revenue-floor.md | 12 ++-- src/replication/neighbor_sync.rs | 72 ++++++++++++++----- src/replication/paid_list.rs | 14 ++-- 3 files changed, 71 insertions(+), 27 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 88981992..bd92b11d 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -389,8 +389,11 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over key's paid entry. Each paid-list write updates that copy in commit order, a new mark just before its commit, so the copy never lacks a committed mark, and a write that commits the key unmarked, a plain insert that adds it - or a removal, drops any mark the copy holds for it. If the marks cannot be read when the node starts, it still starts, as - before, and hints no key as a replica until a restart can read them; + or a removal, drops any mark the copy holds for it. If the marks cannot + be read when the node starts, it still starts, as before, and hints no + key as a replica until a restart can read them; since any key it stores + may then be marked, each one with a paid entry is a paid-hint candidate + at any distance instead; - the responsible audit never samples a marked key, which also covers one hinted in the moment before its mark was written; - a prune audit, which is sent only under disk pressure (ADR-0028), still @@ -694,8 +697,9 @@ out of 1,801: commit can leave it, goes with a later plain insert of the key and with its removal, single or batched; building replica hints works while the paid list cannot be read at all, and a - paid list whose marks cannot be read on opening still opens and hints no - replica; + paid list whose marks cannot be read on opening still opens, hints no + replica and offers every stored key with a paid entry as a paid hint at + any distance; - a mark that cannot be read counts as set in the responsible audit and a write's fan-out, and a run of such reads logs one warning. The tests make the read fail by holding the thread's one LMDB diff --git a/src/replication/neighbor_sync.rs b/src/replication/neighbor_sync.rs index 4fb2f002..89c675d9 100644 --- a/src/replication/neighbor_sync.rs +++ b/src/replication/neighbor_sync.rs @@ -173,7 +173,7 @@ async fn paid_hint_candidates( let self_xor: XorName = *p2p_node.peer_id().as_bytes(); let near_bits = paid_near_bits(p2p_node, group_size).await; let scan_bits = paid_hint_scan_bits(paid_list.count()?, near_bits); - paid_hint_keys(paid_list, storage, &self_xor, scan_bits) + paid_hint_keys(paid_list, storage, &self_xor, scan_bits).await } /// The paid-list keys within the `scan_bits` around `center`, and the keys @@ -184,17 +184,33 @@ async fn paid_hint_candidates( /// reach its close group in neither hint set. The marks and the store's index /// are in memory, and marks are few, so the far marked keys cost one paid-list /// read each. -fn paid_hint_keys( +/// +/// While the marks are unknown, because they could not be read when the paid +/// list opened, any stored key may be marked and none goes out as a replica +/// hint (see [`replica_hint_keys`]), so every stored key is taken as marked +/// here: the per-key work the replica hints no longer do moves to these. +async fn paid_hint_keys( paid_list: &PaidList, storage: &ChunkStore, center: &XorName, scan_bits: usize, ) -> Result> { let mut keys = paid_list.keys_near(center, scan_bits)?; - keys.extend(paid_list.marked_keys().into_iter().filter(|key| { - !is_near(center, key, scan_bits) - && storage.exists(key).unwrap_or(false) - && paid_list.contains(key).unwrap_or(false) + if scan_bits >= XORNAME_BITS { + return Ok(keys); + } + let stored_marked = match paid_list.marked_keys() { + Some(marked) => marked + .into_iter() + .filter(|key| storage.exists(key).unwrap_or(false)) + .collect(), + None => storage.all_keys().await.unwrap_or_else(|e| { + warn!("Failed to read stored keys for paid hints while the marks are unknown: {e}"); + Vec::new() + }), + }; + keys.extend(stored_marked.into_iter().filter(|key| { + !is_near(center, key, scan_bits) && paid_list.contains(key).unwrap_or(false) })); Ok(keys) } @@ -724,7 +740,8 @@ mod tests { /// ADR-0006 with ADR-0028: a price-contested key this node stores is read /// for paid hints at any distance, as it would be for replica hints, while /// a far key it does not store or one with no mark is left out of a narrow - /// read, and a full read lists each key once. + /// read, and a full read lists each key once. While the marks are unknown, + /// every stored key with a paid entry is read at any distance. #[tokio::test] async fn a_stored_marked_key_is_a_paid_hint_candidate_at_any_distance() { const NARROW_SCAN_BITS: usize = 8; @@ -755,23 +772,19 @@ mod tests { assert!(!is_near(¢er, &key, NARROW_SCAN_BITS), "a far key"); } - let read = |scan_bits| { - let mut keys = - paid_hint_keys(&paid_list, &storage, ¢er, scan_bits).expect("read keys"); - keys.sort_unstable(); - keys - }; let sorted = |mut keys: Vec| { keys.sort_unstable(); keys }; - - assert_eq!( - read(NARROW_SCAN_BITS), - sorted(vec![near_marked, far_marked]) - ); + let narrow = paid_hint_keys(&paid_list, &storage, ¢er, NARROW_SCAN_BITS) + .await + .expect("read keys"); + assert_eq!(sorted(narrow), sorted(vec![near_marked, far_marked])); + let full = paid_hint_keys(&paid_list, &storage, ¢er, XORNAME_BITS) + .await + .expect("read keys"); assert_eq!( - read(XORNAME_BITS), + sorted(full), sorted(vec![ near_marked, far_marked, @@ -780,6 +793,27 @@ mod tests { ]), "a full read covers every key once" ); + + // While the marks are unknown any stored key may be marked, and none + // is hinted as a replica, so every stored key with a paid entry is + // read at any distance; a stored key with no paid entry is not. + let unpaid_stored = ChunkStore::compute_address(b"far unpaid record"); + storage + .put(&unpaid_stored, b"far unpaid record") + .await + .expect("put"); + assert!( + !is_near(¢er, &unpaid_stored, NARROW_SCAN_BITS), + "a far key" + ); + paid_list.forget_marks_for_tests(); + let unknown = paid_hint_keys(&paid_list, &storage, ¢er, NARROW_SCAN_BITS) + .await + .expect("read keys"); + assert_eq!( + sorted(unknown), + sorted(vec![near_marked, far_marked, far_plain]) + ); } /// A key whose last byte is `b`, the rest zero. diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index de89e5d9..2119b5d3 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -480,11 +480,11 @@ impl PaidList { } /// The keys marked price-contested in the in-memory copy, read without - /// touching the database. It is empty if the marks could not be read when - /// the paid list opened. - pub fn marked_keys(&self) -> Vec { + /// touching the database, or `None` if the marks could not be read when + /// the paid list opened, so that any key may be marked. + pub fn marked_keys(&self) -> Option> { let snapshot = self.price_contested.read().clone(); - snapshot.map_or_else(Vec::new, |marked| marked.iter().copied().collect()) + snapshot.map(|marked| marked.iter().copied().collect()) } /// Describe a failed insert. A full store is also reported, once, as an @@ -758,6 +758,12 @@ impl PaidList { /// one reader slot, so until it is dropped every paid-list read on this /// thread fails, which lets a test drive the paths a mark that cannot be /// read takes. + /// Act as if the marks could not be read when the paid list opened. + #[cfg(test)] + pub(crate) fn forget_marks_for_tests(&self) { + *self.price_contested.write() = None; + } + #[cfg(test)] pub(crate) fn hold_reader_for_tests(&self) -> Result> { self.env From 605164076771b3f4f8ffe9a649800fb00566f175 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 14:18:07 +0900 Subject: [PATCH 10/22] fix(replication): keep the paid entry of a record that cannot be read for now Paid-list eviction (ADR-0028) kept an entry only while ChunkStore::exists held for its record, and exists turns false while a record is out of service after a failed read, although its file stays in the index. An eviction pass in that moment removed the entry and, with it, the record's price-contested mark (ADR-0006). Once the record read again it went out as an unmarked replica hint, so peers that had refused it under the floor could be judged on it. Eviction now keeps an entry while its record is in the store's index, readable or not, through the named keeps_paid_entry. It only keeps more entries than before. A unit test makes a record unreadable and checks that its entry stays while an entry with no record does not. ADR-0006 says so. --- .../ADR-0006-receiver-side-revenue-floor.md | 9 +-- src/replication/pruning.rs | 55 ++++++++++++++++++- 2 files changed, 57 insertions(+), 7 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index bd92b11d..afed24a4 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -399,8 +399,8 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over - a prune audit, which is sent only under disk pressure (ADR-0028), still needs its proofs before deleting a marked key, but scores no absent peer for it. Paid-list eviction (ADR-0028) keeps the entry of every record - the node still holds, marked or not, and an entry it evicts takes its - mark with it; + still in the node's store, marked or not and readable or not, and an + entry it evicts takes its mark with it; - a mark that cannot be read counts as set: when in doubt, judge no one. While reads fail the node judges no peer on any key they hit, so it warns at the first failure of a run and, when a read succeeds again, @@ -689,8 +689,9 @@ out of 1,801: - a marked key is left out of replica hints and of responsible-audit sampling, a marked key this node stores is a paid-hint candidate however far it is, while one it does not store, an unmarked one or a - near one is left to the near-key read, and paid-list eviction drops the - mark of an entry it evicts and leaves a kept entry marked; + near one is left to the near-key read, paid-list eviction drops the + mark of an entry it evicts and leaves a kept entry marked, and it keeps + the entry of a record that cannot be read for the moment; - the in-memory copy of the marks is filled on opening and follows every write and removal, and a marked insert and a batch removal change it only in the order writes take; a mark held in memory alone, as a failed diff --git a/src/replication/pruning.rs b/src/replication/pruning.rs index 66efb5e9..b8eaa796 100644 --- a/src/replication/pruning.rs +++ b/src/replication/pruning.rs @@ -680,8 +680,8 @@ fn select_record_prune_audits( /// /// Only entries far outside this node's range go, farthest first: with evenly /// spread IDs this node has no vote on them. An entry whose record is still stored -/// stays, since it authorises that record's repair. So does everything while this -/// node knows too few peers to say what is near. +/// stays, since it authorises that record's repair (see [`keeps_paid_entry`]). So +/// does everything while this node knows too few peers to say what is near. async fn evict_excess_paid_entries(ctx: &PrunePassContext<'_>) -> (usize, usize) { let count = match ctx.paid_list.count() { Ok(count) => usize::try_from(count).unwrap_or(usize::MAX), @@ -702,7 +702,7 @@ async fn evict_excess_paid_entries(ctx: &PrunePassContext<'_>) -> (usize, usize) *ctx.self_id.as_bytes(), near_bits, excess.min(MAX_PAID_EVICTIONS_PER_PASS), - move |key| storage.exists(key).unwrap_or(true), + move |key| keeps_paid_entry(&storage, key), ) .await; match evicted { @@ -714,6 +714,15 @@ async fn evict_excess_paid_entries(ctx: &PrunePassContext<'_>) -> (usize, usize) } } +/// Whether eviction keeps `key`'s paid entry: while its record is in the store's +/// index, whether or not it can be read at the moment. The entry authorises the +/// record's repair and carries its price-contested mark (ADR-0006); a record that +/// failed a read is out of service, not gone, and losing its mark would let it be +/// hinted as a replica once it reads again. +fn keeps_paid_entry(storage: &ChunkStore, key: &XorName) -> bool { + storage.is_indexed(key) +} + fn remote_close_group_peers(close_group: &[PeerId], self_id: &PeerId) -> Vec { close_group .iter() @@ -1611,6 +1620,46 @@ mod tests { RecordPruneCandidate { key, target_peers } } + /// ADR-0006 with ADR-0028: eviction keeps the paid entry of a record that + /// cannot be read for the moment, so a retained price-contested record keeps + /// its mark through a read fault; an entry with no record does not stay. + #[cfg(unix)] + #[tokio::test] + async fn eviction_keeps_the_entry_of_a_record_that_cannot_be_read_for_now() { + use std::os::unix::fs::PermissionsExt; + + let dir = TempDir::new().expect("temp dir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().to_path_buf(), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let bytes = b"retained price-contested record".to_vec(); + let key = ChunkStore::compute_address(&bytes); + storage.put(&key, &bytes).await.expect("put"); + let path = dir + .path() + .join(CHUNKS_DIR_NAME) + .join(format!("{:02x}", key.last().copied().unwrap_or(0))) + .join(hex::encode(key)); + let mut perms = std::fs::metadata(&path).expect("meta").permissions(); + perms.set_mode(0o000); + std::fs::set_permissions(&path, perms).expect("chmod"); + + assert!(storage.get(&key).await.is_err(), "the read must fail"); + assert!(!storage.exists(&key).expect("exists"), "out of service"); + assert!(keeps_paid_entry(&storage, &key), "but its entry stays"); + assert!( + !keeps_paid_entry(&storage, &[0x5A; 32]), + "an entry with no record does not" + ); + + let mut perms = std::fs::metadata(&path).expect("meta").permissions(); + perms.set_mode(0o600); + std::fs::set_permissions(&path, perms).expect("chmod back"); + } + /// A prune audit judges holders against this node's own copy. A copy that /// no longer hashes to its key would fail every honest holder and hold the /// prune back forever, so it is not used as a reference: the read takes it From 04513a866e7b11b7411dd015305d78cbbc431bd3 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 14:35:06 +0900 Subject: [PATCH 11/22] perf(replication): find far marked keys among the stored keys, not every mark The far price-contested keys a hint build adds to its paid hints were found by copying every mark and checking each against the store. Marks also sit on paid entries whose records this node never stored, and stay until eviction, so with millions of them every hint build copied and checked them all, however few records the node holds. A hint build now reads the stored keys once, as it already did for replica hints, and splits them against the in-memory marks: the unmarked ones become replica-hint candidates, the marked ones far paid-hint candidates. While the marks are unknown every stored key counts as marked. The inbound sync handler builds both hint sets from that one read. PaidList::split_marked replaces marked_keys. --- src/replication/neighbor_sync.rs | 187 +++++++++++++++++-------------- src/replication/paid_list.rs | 14 ++- 2 files changed, 109 insertions(+), 92 deletions(-) diff --git a/src/replication/neighbor_sync.rs b/src/replication/neighbor_sync.rs index 89c675d9..b838af75 100644 --- a/src/replication/neighbor_sync.rs +++ b/src/replication/neighbor_sync.rs @@ -87,18 +87,32 @@ pub async fn build_replica_hints_for_peer( .collect() } -/// Stored keys this node may hint as replicas: every stored key except those -/// marked price-contested in the paid list (see `fresh::record_paid_key`), -/// which go out as paid hints only. A replica hint records a repair proof the -/// responsible audit later judges the peer on, and makes this node the sole -/// hinter a peer penalises when its quorum check fails; a paid hint does -/// neither. The marks are read from memory, so this scans nothing on disk; if -/// they could not be read when the paid list opened, no key is hinted as a -/// replica. -async fn replica_hint_keys(storage: &ChunkStore, paid_list: &PaidList) -> Result> { - let mut keys = storage.all_keys().await?; - paid_list.retain_unmarked(&mut keys); - Ok(keys) +/// Stored keys split for hinting (ADR-0006): `replica` holds those this node +/// may hint as replicas, `marked` those marked price-contested in the paid list +/// (see `fresh::record_paid_key`), which go out as paid hints only. A replica +/// hint records a repair proof the responsible audit later judges the peer on, +/// and makes this node the sole hinter a peer penalises when its quorum check +/// fails; a paid hint does neither. The marks are read from memory, so this +/// scans nothing on disk; if they could not be read when the paid list opened, +/// every stored key counts as marked and none is hinted as a replica. +#[derive(Default)] +struct StoredHintKeys { + replica: Vec, + marked: Vec, +} + +/// This node's stored keys, split for hinting (see [`StoredHintKeys`]). +async fn stored_hint_keys(storage: &ChunkStore, paid_list: &PaidList) -> StoredHintKeys { + match storage.all_keys().await { + Ok(keys) => { + let (replica, marked) = paid_list.split_marked(keys); + StoredHintKeys { replica, marked } + } + Err(e) => { + warn!("Failed to read stored keys for hint construction: {e}"); + StoredHintKeys::default() + } + } } pub(crate) async fn build_replica_hints_for_peer_with_close_groups( @@ -108,17 +122,21 @@ pub(crate) async fn build_replica_hints_for_peer_with_close_groups( p2p_node: &Arc, close_group_size: usize, ) -> Vec { - let all_keys = match replica_hint_keys(storage, paid_list).await { - Ok(keys) => keys, - Err(e) => { - warn!("Failed to read stored keys for hint construction: {e}"); - return Vec::new(); - } - }; + let stored = stored_hint_keys(storage, paid_list).await; + replica_hints_from(peer, stored.replica, p2p_node, close_group_size).await +} +/// The replica hints among `keys` for `peer`: those whose close group, as this +/// node sees it, holds the peer. +async fn replica_hints_from( + peer: &PeerId, + keys: Vec, + p2p_node: &Arc, + close_group_size: usize, +) -> Vec { let dht = p2p_node.dht_manager(); let mut hints = Vec::new(); - for key in all_keys { + for key in keys { let closest = dht .find_closest_nodes_local_with_self(&key, close_group_size) .await; @@ -166,52 +184,37 @@ fn paid_hint_scan_bits(entries: u64, near_bits: usize) -> usize { /// The paid-list keys a hint build considers (see [`paid_hint_keys`]). async fn paid_hint_candidates( paid_list: &PaidList, - storage: &ChunkStore, + stored_marked: &[XorName], p2p_node: &Arc, group_size: usize, ) -> Result> { let self_xor: XorName = *p2p_node.peer_id().as_bytes(); let near_bits = paid_near_bits(p2p_node, group_size).await; let scan_bits = paid_hint_scan_bits(paid_list.count()?, near_bits); - paid_hint_keys(paid_list, storage, &self_xor, scan_bits).await + paid_hint_keys(paid_list, stored_marked, &self_xor, scan_bits) } -/// The paid-list keys within the `scan_bits` around `center`, and the keys -/// marked price-contested (ADR-0006) beyond them that this node stores and -/// still holds a paid entry for. A marked key goes out as a paid hint only, -/// never as a replica hint, and replica hints have no distance limit: without -/// this, a marked record this node retains far from itself (ADR-0028) would -/// reach its close group in neither hint set. The marks and the store's index -/// are in memory, and marks are few, so the far marked keys cost one paid-list -/// read each. -/// -/// While the marks are unknown, because they could not be read when the paid -/// list opened, any stored key may be marked and none goes out as a replica -/// hint (see [`replica_hint_keys`]), so every stored key is taken as marked -/// here: the per-key work the replica hints no longer do moves to these. -async fn paid_hint_keys( +/// The paid-list keys within the `scan_bits` around `center`, and those of +/// `stored_marked`, the stored keys [`StoredHintKeys`] counts as marked, that +/// lie beyond them and still have a paid entry. A marked key goes out as a +/// paid hint only, never as a replica hint, and replica hints have no distance +/// limit: without this, a marked record this node retains far from itself +/// (ADR-0028) would reach its close group in neither hint set. While the marks +/// are unknown every stored key counts as marked, so the per-key work the +/// replica hints no longer do moves here. Each far key costs one paid-list +/// read. +fn paid_hint_keys( paid_list: &PaidList, - storage: &ChunkStore, + stored_marked: &[XorName], center: &XorName, scan_bits: usize, ) -> Result> { let mut keys = paid_list.keys_near(center, scan_bits)?; - if scan_bits >= XORNAME_BITS { - return Ok(keys); + if scan_bits < XORNAME_BITS { + keys.extend(stored_marked.iter().copied().filter(|key| { + !is_near(center, key, scan_bits) && paid_list.contains(key).unwrap_or(false) + })); } - let stored_marked = match paid_list.marked_keys() { - Some(marked) => marked - .into_iter() - .filter(|key| storage.exists(key).unwrap_or(false)) - .collect(), - None => storage.all_keys().await.unwrap_or_else(|e| { - warn!("Failed to read stored keys for paid hints while the marks are unknown: {e}"); - Vec::new() - }), - }; - keys.extend(stored_marked.into_iter().filter(|key| { - !is_near(center, key, scan_bits) && paid_list.contains(key).unwrap_or(false) - })); Ok(keys) } @@ -237,17 +240,11 @@ pub(crate) async fn build_sync_hints_for_peers( return hints_by_peer; } - let all_keys = match replica_hint_keys(storage, paid_list).await { - Ok(keys) => keys, - Err(e) => { - warn!("Failed to read stored keys for batch hint construction: {e}"); - Vec::new() - } - }; - let stored_key_count = all_keys.len(); + let stored = stored_hint_keys(storage, paid_list).await; + let stored_key_count = stored.replica.len() + stored.marked.len(); let dht = p2p_node.dht_manager(); - for key in all_keys { + for &key in &stored.replica { let closest = dht .find_closest_nodes_local_with_self(&key, close_group_size) .await; @@ -264,7 +261,7 @@ pub(crate) async fn build_sync_hints_for_peers( let paid_keys = match paid_hint_candidates( paid_list, - storage, + &stored.marked, p2p_node, paid_list_close_group_size, ) @@ -338,10 +335,29 @@ pub async fn build_paid_hints_for_peer( paid_list: &Arc, p2p_node: &Arc, paid_list_close_group_size: usize, +) -> Vec { + let stored = stored_hint_keys(storage, paid_list).await; + paid_hints_from( + peer, + paid_list, + &stored.marked, + p2p_node, + paid_list_close_group_size, + ) + .await +} + +/// The paid hints for `peer`, given the stored keys that count as marked. +async fn paid_hints_from( + peer: &PeerId, + paid_list: &PaidList, + stored_marked: &[XorName], + p2p_node: &Arc, + paid_list_close_group_size: usize, ) -> Vec { let paid_keys = match paid_hint_candidates( paid_list, - storage, + stored_marked, p2p_node, paid_list_close_group_size, ) @@ -646,23 +662,19 @@ pub(crate) async fn handle_sync_request_with_proofs( ) -> (NeighborSyncResponse, Vec, bool) { let sender_in_rt = p2p_node.dht_manager().is_in_routing_table(sender).await; - // Build outbound hints (always sent, even to non-RT peers). - let sent_replica_hints = build_replica_hints_for_peer_with_close_groups( - sender, - storage, - paid_list, - p2p_node, - config.close_group_size, - ) - .await; + // Build outbound hints (always sent, even to non-RT peers), from one read + // of the stored keys. + let stored = stored_hint_keys(storage, paid_list).await; + let sent_replica_hints = + replica_hints_from(sender, stored.replica, p2p_node, config.close_group_size).await; let replica_hints = sent_replica_hints .iter() .map(|hint| hint.key) .collect::>(); - let paid_hints = build_paid_hints_for_peer( + let paid_hints = paid_hints_from( sender, - storage, paid_list, + &stored.marked, p2p_node, config.paid_list_close_group_size, ) @@ -699,11 +711,12 @@ mod tests { PeerId::from_bytes(bytes) } - // -- replica_hint_keys --------------------------------------------------- + // -- stored_hint_keys ---------------------------------------------------- /// ADR-0006: a stored key marked price-contested is never hinted as a - /// replica; every other stored key still is. The filter reads no paid-list - /// database, so it works while this thread cannot read one. + /// replica, and is set aside as marked; every other stored key still is + /// hinted. The split reads no paid-list database, so it works while this + /// thread cannot read one. #[tokio::test] async fn price_contested_keys_are_not_hinted_as_replicas() { let dir = tempfile::tempdir().expect("tempdir"); @@ -727,14 +740,14 @@ mod tests { let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); assert!(paid_list.is_price_contested(&contested).is_err()); - let mut hinted = replica_hint_keys(&storage, &paid_list) - .await - .expect("read keys"); + let stored = stored_hint_keys(&storage, &paid_list).await; + let mut hinted = stored.replica; hinted.sort_unstable(); let mut expected = vec![plain, unpaid]; expected.sort_unstable(); assert_eq!(hinted, expected); + assert_eq!(stored.marked, vec![contested]); } /// ADR-0006 with ADR-0028: a price-contested key this node stores is read @@ -776,13 +789,12 @@ mod tests { keys.sort_unstable(); keys }; - let narrow = paid_hint_keys(&paid_list, &storage, ¢er, NARROW_SCAN_BITS) - .await + let stored = stored_hint_keys(&storage, &paid_list).await; + let narrow = paid_hint_keys(&paid_list, &stored.marked, ¢er, NARROW_SCAN_BITS) .expect("read keys"); assert_eq!(sorted(narrow), sorted(vec![near_marked, far_marked])); - let full = paid_hint_keys(&paid_list, &storage, ¢er, XORNAME_BITS) - .await - .expect("read keys"); + let full = + paid_hint_keys(&paid_list, &stored.marked, ¢er, XORNAME_BITS).expect("read keys"); assert_eq!( sorted(full), sorted(vec![ @@ -807,8 +819,9 @@ mod tests { "a far key" ); paid_list.forget_marks_for_tests(); - let unknown = paid_hint_keys(&paid_list, &storage, ¢er, NARROW_SCAN_BITS) - .await + let unknown_marks = stored_hint_keys(&storage, &paid_list).await; + assert!(unknown_marks.replica.is_empty(), "no replica hint"); + let unknown = paid_hint_keys(&paid_list, &unknown_marks.marked, ¢er, NARROW_SCAN_BITS) .expect("read keys"); assert_eq!( sorted(unknown), diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index 2119b5d3..ccb38813 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -479,12 +479,16 @@ impl PaidList { } } - /// The keys marked price-contested in the in-memory copy, read without - /// touching the database, or `None` if the marks could not be read when - /// the paid list opened, so that any key may be marked. - pub fn marked_keys(&self) -> Option> { + /// Split `keys` into those known to be unmarked and those marked + /// price-contested, using the in-memory copy rather than the database. If + /// the marks could not be read on open, no key is known to be unmarked, so + /// every key counts as marked. + pub fn split_marked(&self, keys: Vec) -> (Vec, Vec) { let snapshot = self.price_contested.read().clone(); - snapshot.map(|marked| marked.iter().copied().collect()) + match snapshot { + Some(marked) => keys.into_iter().partition(|key| !marked.contains(key)), + None => (Vec::new(), keys), + } } /// Describe a failed insert. A full store is also reported, once, as an From e9728f59fced39bbfcae33654993c0c0fcea1dc0 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 14:35:06 +0900 Subject: [PATCH 12/22] docs(adr): ADR-0006 records that a marked copy is never a repair source A paid hint makes no possession claim, so a node holding a marked record is never asked for its bytes. If a key's current close group holds no copy and every copy outside it is marked, the group cannot repair the key from them. The amendment's consequences now say so, and that fixing it without exposing the holder to sole-hinter penalties needs a wire change. --- docs/adr/ADR-0006-receiver-side-revenue-floor.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index afed24a4..00b59208 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -583,6 +583,17 @@ out of 1,801: of the paid group, the refusers repair the key from those votes like any other, unmarked, since repair carries no proof to price; a refuser whose own quorum check fails while its neighbours' passes can then be judged on it. +- **A marked copy is never a repair source.** A paid hint makes no + possession claim, so no node is asked for a key's bytes because it + paid-hinted the key: peers fetch from the close-group members that answer + present and from nodes that hinted the key as a replica. If no member of a + key's current close group holds it, and every copy outside the group is + marked on its holder, as after churn moves the group away from the nodes + that accepted a contested write, the group cannot repair the key from + those copies. The copies stay, since deleting a record still needs its + group's proofs. Making a marked holder a fetch source without letting + peers penalise it as a sole hinter needs a wire change, so it is not done + here. - **Narrow gaps remain on nodes running this release.** - On the PUT path the record is stored a moment before its paid-list entry is written. A sync that scans the store in that moment can hint From 7b66a6ceb563ba3129c4f220504f9c9b3b5a1ce7 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 14:52:06 +0900 Subject: [PATCH 13/22] fix(replication): send a far marked paid hint only where a replica hint would go A far price-contested key this node stores went out as a paid hint to every member of its paid close group it knows (20 peers), where a replica hint of the same key reaches its close group (7). So turning such keys from replica hints into paid hints could grow a sync message about three times past what main sends, in the worst case past the 10 MiB message limit, and the whole exchange would be dropped. Paid-hint candidates now keep the near keys read from the paid list apart from the far marked keys: the first still go to the paid close group, the second only to the close group, so they add no more to a message than replica hints of the same keys would. build_paid_hints_for_peer takes the close group size for this. ADR-0006 says so. --- .../ADR-0006-receiver-side-revenue-floor.md | 6 +- src/replication/neighbor_sync.rs | 93 +++++++++++++++---- tests/e2e/replication.rs | 2 + 3 files changed, 79 insertions(+), 22 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 00b59208..1dc1e2c8 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -382,8 +382,10 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over accepted the key. Above 100,000 entries a node builds paid hints only from keys near itself (ADR-0028), while replica hints have no such limit, so a marked key it stores is a paid-hint candidate at any - distance: a record it retains far from itself still reaches whichever - peers it knows in that key's paid close group. The node also keeps its + distance. It goes to the members of the key's close group the node + knows, the peers a replica hint would have reached, so a record it + retains far from itself still reaches them, and a sync message carries + no more such keys than replica hints would have. The node also keeps its marked keys in memory, so leaving them out of the hints it builds for a peer, or adding them, reads nothing on disk but each far stored marked key's paid entry. Each paid-list write updates that copy in commit order, a new diff --git a/src/replication/neighbor_sync.rs b/src/replication/neighbor_sync.rs index b838af75..9a18bfbd 100644 --- a/src/replication/neighbor_sync.rs +++ b/src/replication/neighbor_sync.rs @@ -181,13 +181,48 @@ fn paid_hint_scan_bits(entries: u64, near_bits: usize) -> usize { } } +/// The keys a paid-hint build considers: `near`, read from the paid list, each +/// hinted to the members of its paid close group; and `far_marked`, the far +/// stored keys counted as marked, each hinted only to the members of its close +/// group, the peers a replica hint would have reached, so they add no more to +/// a sync message than replica hints of the same keys would (see +/// [`paid_hint_keys`]). +#[derive(Default)] +struct PaidHintKeys { + near: Vec, + far_marked: Vec, +} + +impl PaidHintKeys { + fn len(&self) -> usize { + self.near.len() + self.far_marked.len() + } + + /// Each key with the size of the group whose members it is hinted to. + fn with_group_sizes( + self, + paid_list_close_group_size: usize, + close_group_size: usize, + ) -> impl Iterator { + let near = self + .near + .into_iter() + .map(move |key| (key, paid_list_close_group_size)); + let far_marked = self + .far_marked + .into_iter() + .map(move |key| (key, close_group_size)); + near.chain(far_marked) + } +} + /// The paid-list keys a hint build considers (see [`paid_hint_keys`]). async fn paid_hint_candidates( paid_list: &PaidList, stored_marked: &[XorName], p2p_node: &Arc, group_size: usize, -) -> Result> { +) -> Result { let self_xor: XorName = *p2p_node.peer_id().as_bytes(); let near_bits = paid_near_bits(p2p_node, group_size).await; let scan_bits = paid_hint_scan_bits(paid_list.count()?, near_bits); @@ -208,14 +243,20 @@ fn paid_hint_keys( stored_marked: &[XorName], center: &XorName, scan_bits: usize, -) -> Result> { - let mut keys = paid_list.keys_near(center, scan_bits)?; - if scan_bits < XORNAME_BITS { - keys.extend(stored_marked.iter().copied().filter(|key| { - !is_near(center, key, scan_bits) && paid_list.contains(key).unwrap_or(false) - })); - } - Ok(keys) +) -> Result { + let near = paid_list.keys_near(center, scan_bits)?; + let far_marked = if scan_bits < XORNAME_BITS { + stored_marked + .iter() + .copied() + .filter(|key| { + !is_near(center, key, scan_bits) && paid_list.contains(key).unwrap_or(false) + }) + .collect() + } else { + Vec::new() + }; + Ok(PaidHintKeys { near, far_marked }) } /// Build outbound hint sets for a batch of peers with one scan over local @@ -270,14 +311,16 @@ pub(crate) async fn build_sync_hints_for_peers( Ok(keys) => keys, Err(e) => { warn!("Failed to read PaidForList for batch hint construction: {e}"); - Vec::new() + PaidHintKeys::default() } }; let paid_key_count = paid_keys.len(); - for key in paid_keys { + for (key, group_size) in + paid_keys.with_group_sizes(paid_list_close_group_size, close_group_size) + { let closest = dht - .find_closest_nodes_local_with_self(&key, paid_list_close_group_size) + .find_closest_nodes_local_with_self(&key, group_size) .await; for node in closest { if target_peers.contains(&node.peer_id) { @@ -328,13 +371,15 @@ pub(crate) async fn build_sync_hints_for_peers( /// Returns keys from our `PaidForList` that we believe the peer should /// track (peer is among `PAID_LIST_CLOSE_GROUP_SIZE` nearest to `K`). /// `storage` tells which price-contested keys this node stores, since those -/// are hinted at any distance (see `paid_hint_keys`). +/// are hinted at any distance, to the members of their close group of +/// `close_group_size` (see `paid_hint_keys`). pub async fn build_paid_hints_for_peer( peer: &PeerId, storage: &Arc, paid_list: &Arc, p2p_node: &Arc, paid_list_close_group_size: usize, + close_group_size: usize, ) -> Vec { let stored = stored_hint_keys(storage, paid_list).await; paid_hints_from( @@ -343,6 +388,7 @@ pub async fn build_paid_hints_for_peer( &stored.marked, p2p_node, paid_list_close_group_size, + close_group_size, ) .await } @@ -354,6 +400,7 @@ async fn paid_hints_from( stored_marked: &[XorName], p2p_node: &Arc, paid_list_close_group_size: usize, + close_group_size: usize, ) -> Vec { let paid_keys = match paid_hint_candidates( paid_list, @@ -372,9 +419,11 @@ async fn paid_hints_from( let dht = p2p_node.dht_manager(); let mut hints = Vec::new(); - for key in paid_keys { + for (key, group_size) in + paid_keys.with_group_sizes(paid_list_close_group_size, close_group_size) + { let closest = dht - .find_closest_nodes_local_with_self(&key, paid_list_close_group_size) + .find_closest_nodes_local_with_self(&key, group_size) .await; if closest.iter().any(|n| n.peer_id == *peer) { hints.push(key); @@ -677,6 +726,7 @@ pub(crate) async fn handle_sync_request_with_proofs( &stored.marked, p2p_node, config.paid_list_close_group_size, + config.close_group_size, ) .await; @@ -792,19 +842,21 @@ mod tests { let stored = stored_hint_keys(&storage, &paid_list).await; let narrow = paid_hint_keys(&paid_list, &stored.marked, ¢er, NARROW_SCAN_BITS) .expect("read keys"); - assert_eq!(sorted(narrow), sorted(vec![near_marked, far_marked])); + assert_eq!(narrow.near, vec![near_marked]); + assert_eq!(narrow.far_marked, vec![far_marked]); let full = paid_hint_keys(&paid_list, &stored.marked, ¢er, XORNAME_BITS).expect("read keys"); assert_eq!( - sorted(full), + sorted(full.near), sorted(vec![ near_marked, far_marked, far_plain, far_marked_unstored ]), - "a full read covers every key once" + "a full read covers every key" ); + assert!(full.far_marked.is_empty(), "and adds none twice"); // While the marks are unknown any stored key may be marked, and none // is hinted as a replica, so every stored key with a paid entry is @@ -823,9 +875,10 @@ mod tests { assert!(unknown_marks.replica.is_empty(), "no replica hint"); let unknown = paid_hint_keys(&paid_list, &unknown_marks.marked, ¢er, NARROW_SCAN_BITS) .expect("read keys"); + assert_eq!(unknown.near, vec![near_marked]); assert_eq!( - sorted(unknown), - sorted(vec![near_marked, far_marked, far_plain]) + sorted(unknown.far_marked), + sorted(vec![far_marked, far_plain]) ); } diff --git a/tests/e2e/replication.rs b/tests/e2e/replication.rs index ed51ef4c..c3fbc8cb 100644 --- a/tests/e2e/replication.rs +++ b/tests/e2e/replication.rs @@ -696,6 +696,7 @@ async fn possession_check_skips_a_write_the_floor_cannot_price() { paid_list_a, p2p_a, ReplicationConfig::default().paid_list_close_group_size, + close_group_size, ) .await .contains(&address); @@ -2497,6 +2498,7 @@ async fn a_neighbour_gets_a_paid_hint_for_a_key_far_from_this_node() { &paid_list, &p2p, ONLY_THE_CLOSEST, + ONLY_THE_CLOSEST, ) .await; From 8d12c1729703ffd0f4e543f863cc616fc12aea5e Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 14:59:37 +0900 Subject: [PATCH 14/22] docs(adr): ADR-0006 narrows the repair-source limit to copies outside the queried groups Verification asks a key's close group and its paid close group, and takes every member that answers present as a fetch source, marked or not. Only copies outside both groups, known to repairers through paid hints alone, are never fetched from. The consequence now says so. --- .../ADR-0006-receiver-side-revenue-floor.md | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 1dc1e2c8..50c279b9 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -585,17 +585,18 @@ out of 1,801: of the paid group, the refusers repair the key from those votes like any other, unmarked, since repair carries no proof to price; a refuser whose own quorum check fails while its neighbours' passes can then be judged on it. -- **A marked copy is never a repair source.** A paid hint makes no - possession claim, so no node is asked for a key's bytes because it - paid-hinted the key: peers fetch from the close-group members that answer - present and from nodes that hinted the key as a replica. If no member of a - key's current close group holds it, and every copy outside the group is - marked on its holder, as after churn moves the group away from the nodes - that accepted a contested write, the group cannot repair the key from - those copies. The copies stay, since deleting a record still needs its - group's proofs. Making a marked holder a fetch source without letting - peers penalise it as a sole hinter needs a wire change, so it is not done - here. +- **A marked copy outside a key's queried groups is not a repair source.** + A paid hint makes no possession claim, so no node is asked for a key's + bytes because it paid-hinted the key. A node repairing a key fetches from + the members of its close group and paid close group that answer present + when it verifies the key, and from nodes that hinted the key as a + replica. If no member of those groups holds the key, and every copy + outside them is marked on its holder, as after churn moves the groups + away from the nodes that accepted a contested write, the key cannot be + repaired from those copies. The copies stay, since deleting a record + still needs its group's proofs. Making such a holder a fetch source + without letting peers penalise it as a sole hinter needs a wire change, + so it is not done here. - **Narrow gaps remain on nodes running this release.** - On the PUT path the record is stored a moment before its paid-list entry is written. A sync that scans the store in that moment can hint From 06eea7957f01c973f2c52554651e3eda7a91daab Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 5 Nov 2026 15:37:55 +0900 Subject: [PATCH 15/22] docs(adr): ADR-0006 counts the mutations run on the merged code Five targeted mutations were run on the code merged with ADR-0028, one for each check the merge added or ported, and each fails its test. The validation list now names all five instead of the first two. --- docs/adr/ADR-0006-receiver-side-revenue-floor.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 50c279b9..9a51f321 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -747,10 +747,13 @@ out of 1,801: survived makes prune-audit scoring count a mark that cannot be read as unset. Only the e2e prune test reaches that check, and it cannot make a node's paid-list read fail; the same read counts as set, under test, in - the audit and the fan-out. Two more were run on the code merged with - ADR-0028, and both fail: an eviction that leaves an evicted entry's mark - in memory fails four unit tests, and prune-audit scoring that ignores the - mark fails the e2e prune test, which runs under disk pressure. + the audit and the fan-out. Five more were run on the code merged with + ADR-0028, and each fails: an eviction that leaves an evicted entry's mark + in memory fails four unit tests; leaving far stored marked keys out of + paid hints, treating unknown marks as no marks, and keeping only readable + records' entries through eviction each fail their unit test; and + prune-audit scoring that ignores the mark fails the e2e prune test, which + runs under disk pressure. - `tests/e2e/price_floor_ab.rs` drives the same honest and near-empty-quoter uploads through a production-wired local testnet. Nodes are seeded to production-sized commitments, payments are real (on Anvil), and possession From 3e87bef08f6908a54dbe437ba2ab8ad8bac8a7b8 Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 6 Nov 2026 11:36:02 +0900 Subject: [PATCH 16/22] fix(replication): keep the paid entry of a record admitted while eviction runs Paid-list eviction scanned for entries whose record the store did not hold, then removed them in a separate write transaction. Nothing ordered that removal against a record being stored, so a record that arrived in between lost its paid entry and its price-contested mark while it stayed stored. The node then hinted it as a replica, which exposes the peers that refused the contested write to possession and audit penalties (ADR-0006). Every store path could land in that window: a fresh offer records the entry and then stores the record, a fetch or a client PUT stores a record whose entry an earlier PaidNotify or verification wrote, and a client PUT's check of a chunk whose file has gone drops the stale index entry and puts the offered copy back. The chunk store now counts record admissions. put, repair and holds_verified each register one before taking the key's lane and keep it until they return, and put_after makes a fresh offer's entry and record a single admission. hold_admissions holds back new admissions until its guard is dropped, and holds_or_admits answers whether a key is indexed or on its way there, reading the admissions, then the writes in flight, then the index, so a put whose caller went away still counts while its write runs. Eviction keeps every entry holds_or_admits accepts. Its scan still picks the candidates, but the removal asks again inside its write transaction, holding admissions back until it has committed. A record stored after the scan, or under way when the removal decides, keeps its entry and mark. A record admitted after the removal finds no entry, as after any eviction. remove_batch shares the removal code with an empty guard. Six tests park the eviction and the admission at fixed points, one interleaving each: a record stored after the scan, a fresh offer between its entry and its record, a put not yet writing, a cancelled put still writing, a PUT check putting a missing file back, and an admission that starts during the removal, which waits for the commit without getting past registering. Each fails without this change. --- src/replication/mod.rs | 25 ++- src/replication/paid_list.rs | 145 +++++++++++--- src/replication/pruning.rs | 375 +++++++++++++++++++++++++++++++++-- src/storage/chunk_store.rs | 148 +++++++++++++- src/storage/mod.rs | 2 +- 5 files changed, 645 insertions(+), 50 deletions(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 6fbb81c9..7903f45a 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -6622,16 +6622,21 @@ async fn handle_fresh_offer( // Rule 6: add to PaidForList, marked if the price floor contested the // payment when this node verified it, before the record is stored and - // can be hinted. - fresh::record_paid_key( - key, - ctx.payment_verifier.verified_payment_contested(key), - &ctx.paid_list, - ) - .await; - - // Store the record. - match ctx.storage.put(key, data).await { + // can be hinted. Then store the record. One admission covers both, so + // paid-list eviction cannot take the entry in between (ADR-0006). + let (_, stored) = ctx + .storage + .put_after( + key, + data, + fresh::record_paid_key( + key, + ctx.payment_verifier.verified_payment_contested(key), + &ctx.paid_list, + ), + ) + .await; + match stored { Ok(_) => { send_replication_response( source, diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index ccb38813..80a3cb16 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -32,6 +32,7 @@ use heed::WithTls; use heed::{Database, Env, EnvOpenOptions, MdbError, RoTxn}; use parking_lot::{Mutex, RwLock}; use std::collections::HashSet; +use std::future::Future; use std::ops::Bound; use std::path::Path; use std::slice; @@ -235,6 +236,19 @@ pub struct PaidList { /// is dropped, so [`Self::wait_idle`] waits on the blocking tasks /// themselves before the environment may be reopened. blocking_tracker: TaskTracker, + /// Test-only: parks an eviction between its scan and its removal. + #[cfg(test)] + test_eviction_scan_gate: Arc>, + /// Test-only: how many evictions have finished their scan. + #[cfg(test)] + test_evictions_scanned: Arc, + /// Test-only: parks a removal on the blocking pool after it has deleted its + /// keys and before it commits. + #[cfg(test)] + test_removal_commit_gate: Arc>, + /// Test-only: how many removals have reached that gate. + #[cfg(test)] + test_removals_committing: Arc, } impl PaidList { @@ -321,6 +335,14 @@ impl PaidList { record_out_of_range, full_reported: AtomicBool::new(false), blocking_tracker: TaskTracker::new(), + #[cfg(test)] + test_eviction_scan_gate: Arc::new(tokio::sync::RwLock::new(())), + #[cfg(test)] + test_evictions_scanned: Arc::new(AtomicU64::new(0)), + #[cfg(test)] + test_removal_commit_gate: Arc::new(RwLock::new(())), + #[cfg(test)] + test_removals_committing: Arc::new(AtomicU64::new(0)), }; let count = paid_list.count()?; @@ -613,19 +635,29 @@ impl PaidList { /// Remove up to `limit` entries outside the `near_bits` around `center`, /// farthest first, leaving those `keep` accepts. Returns how many went. /// + /// `keep` is asked twice: by the scan that picks the candidates, and again + /// for each candidate inside the write transaction that removes them, once + /// `settle` has given its guard. The guard is held until that transaction + /// has committed, so a caller whose answer can change meanwhile passes one + /// that holds those changes back, and the second answer still holds when + /// the removal lands (see `ChunkStore::hold_admissions`). + /// /// # Errors /// /// Returns an error if an LMDB transaction fails. - pub async fn evict_far( + pub async fn evict_far( &self, center: XorName, near_bits: usize, limit: usize, - keep: impl Fn(&XorName) -> bool + Send + 'static, + keep: impl Fn(&XorName) -> bool + Send + Sync + 'static, + settle: impl Future, ) -> Result { if limit == 0 { return Ok(0); } + let keep = Arc::new(keep); + let scan_keep = Arc::clone(&keep); let env = self.env.clone(); let db = self.db; let evictable = self @@ -637,7 +669,7 @@ impl PaidList { let mut evictable = Vec::new(); for (low, high) in far_ranges(¢er, near_bits) { let more = visit_range(db, &rtxn, (&low, &high), |key| { - if !keep(&key) { + if !scan_keep(&key) { evictable.push(key); } evictable.len() < limit @@ -650,7 +682,17 @@ impl PaidList { }) .await .map_err(|e| Error::Storage(format!("Paid-list eviction scan failed: {e}")))??; - self.remove_batch(&evictable).await + #[cfg(test)] + { + self.test_evictions_scanned.fetch_add(1, Ordering::AcqRel); + drop(self.test_eviction_scan_gate.read().await); + } + if evictable.is_empty() { + return Ok(0); + } + let held = settle.await; + self.remove_unkept(evictable, move |key| keep(key), held) + .await } /// Record the `RecordOutOfRangeFirstSeen` timestamp for a key. @@ -695,49 +737,72 @@ impl PaidList { /// /// Returns an error if the LMDB write transaction fails. pub async fn remove_batch(&self, keys: &[XorName]) -> Result { + let count = self.remove_unkept(keys.to_vec(), |_| false, ()).await?; + debug!("Batch-removed {count} keys from paid-list"); + Ok(count) + } + + /// Remove each of `keys` that `keep` rejects, in one write transaction, + /// asking `keep` inside it, and hold `held` until the removal has + /// committed and the in-memory marks follow it. Returns how many entries + /// went. + async fn remove_unkept( + &self, + keys: Vec, + keep: impl Fn(&XorName) -> bool + Send + 'static, + held: G, + ) -> Result { if keys.is_empty() { return Ok(0); } - let keys_owned: Vec = keys.to_vec(); let env = self.env.clone(); let db = self.db; let price_contested = Arc::clone(&self.price_contested); let write_order = Arc::clone(&self.write_order); + #[cfg(test)] + let (commit_gate, committing) = ( + Arc::clone(&self.test_removal_commit_gate), + Arc::clone(&self.test_removals_committing), + ); - let removed_keys = self - .blocking_tracker - .spawn_blocking(move || -> Result> { + self.blocking_tracker + .spawn_blocking(move || -> Result { + // Dropped last, once the commit and the in-memory marks are done + // or the transaction is abandoned. + let _held = held; let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; - let mut removed = Vec::new(); - for key in &keys_owned { + let doomed: Vec = keys.into_iter().filter(|key| !keep(key)).collect(); + let mut removed = 0; + for key in &doomed { let deleted = db.delete(&mut wtxn, key.as_ref()).map_err(|e| { Error::Storage(format!("Failed to delete from paid-list: {e}")) })?; if deleted { - removed.push(*key); + removed += 1; } } + #[cfg(test)] + { + committing.fetch_add(1, Ordering::AcqRel); + drop(commit_gate.read()); + } wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit batch remove: {e}")))?; - // Every key of the batch, not only those the database held: a - // mark whose commit failed can be in memory alone. - unmark_in_memory(&price_contested, &keys_owned); + // Every key removed, not only those the database held: a mark + // whose commit failed can be in memory alone. + unmark_in_memory(&price_contested, &doomed); drop(ordered); Ok(removed) }) .await - .map_err(|e| Error::Storage(format!("Paid-list batch remove task failed: {e}")))??; - - let count = removed_keys.len(); - debug!("Batch-removed {count} keys from paid-list"); - Ok(count) + .map_err(|e| Error::Storage(format!("Paid-list batch remove task failed: {e}")))? } /// Wait until every tracked paid-list LMDB blocking task has finished. @@ -774,6 +839,32 @@ impl PaidList { .read_txn() .map_err(|e| Error::Storage(format!("Failed to create read txn: {e}"))) } + + /// Test-only handle to the gate an eviction waits on after its scan: hold + /// the write half to park the next eviction there. + #[cfg(test)] + pub(crate) fn test_eviction_scan_gate(&self) -> Arc> { + Arc::clone(&self.test_eviction_scan_gate) + } + + /// Test-only: how many evictions have finished their scan. + #[cfg(test)] + pub(crate) fn test_evictions_scanned(&self) -> u64 { + self.test_evictions_scanned.load(Ordering::Acquire) + } + + /// Test-only handle to the gate a removal waits on before its commit, on + /// the blocking pool: hold the write half to park the next removal there. + #[cfg(test)] + pub(crate) fn test_removal_commit_gate(&self) -> Arc> { + Arc::clone(&self.test_removal_commit_gate) + } + + /// Test-only: how many removals have reached the gate before their commit. + #[cfg(test)] + pub(crate) fn test_removals_committing(&self) -> u64 { + self.test_removals_committing.load(Ordering::Acquire) + } } #[cfg(test)] @@ -894,7 +985,7 @@ mod tests { } let evicted = pl - .evict_far(center, near_bits, 1, |_| false) + .evict_far(center, near_bits, 1, |_| false, async {}) .await .expect("evict"); assert_eq!(evicted, 1); @@ -905,7 +996,7 @@ mod tests { assert!(pl.contains(&mid).expect("contains")); let evicted = pl - .evict_far(center, near_bits, 10, |_| false) + .evict_far(center, near_bits, 10, |_| false, async {}) .await .expect("evict"); assert_eq!(evicted, 1, "only the far entry was left to evict"); @@ -921,7 +1012,7 @@ mod tests { let near_bits = XORNAME_BITS - 1; let evicted = pl - .evict_far([0u8; 32], near_bits, 10, move |key| *key == kept) + .evict_far([0u8; 32], near_bits, 10, move |key| *key == kept, async {}) .await .expect("evict"); @@ -937,7 +1028,7 @@ mod tests { pl.insert(&key).await.expect("insert"); let evicted = pl - .evict_far([0u8; 32], XORNAME_BITS, 10, |_| false) + .evict_far([0u8; 32], XORNAME_BITS, 10, |_| false, async {}) .await .expect("evict"); @@ -1101,7 +1192,13 @@ mod tests { pl.insert_price_contested(&evicted_key).await.expect("mark"); let evicted = pl - .evict_far([0u8; 32], XORNAME_BITS - 1, 10, move |key| *key == kept) + .evict_far( + [0u8; 32], + XORNAME_BITS - 1, + 10, + move |key| *key == kept, + async {}, + ) .await .expect("evict"); diff --git a/src/replication/pruning.rs b/src/replication/pruning.rs index b8eaa796..0619d8d9 100644 --- a/src/replication/pruning.rs +++ b/src/replication/pruning.rs @@ -78,6 +78,7 @@ use saorsa_core::P2PNode; use tokio::sync::RwLock; use crate::ant_protocol::XorName; +use crate::error::Result; use crate::replication::audit_coordinator::AuditChallengeCoordinator; use crate::replication::audit_metrics::{self, AuditFailureClass, AuditType}; use crate::replication::commitment_state::ResponderCommitmentState; @@ -695,16 +696,14 @@ async fn evict_excess_paid_entries(ctx: &PrunePassContext<'_>) -> (usize, usize) return (count, 0); } let near_bits = paid_near_bits(ctx.p2p_node, ctx.config.paid_list_close_group_size).await; - let storage = Arc::clone(ctx.storage); - let evicted = ctx - .paid_list - .evict_far( - *ctx.self_id.as_bytes(), - near_bits, - excess.min(MAX_PAID_EVICTIONS_PER_PASS), - move |key| keeps_paid_entry(&storage, key), - ) - .await; + let evicted = evict_paid_entries( + ctx.paid_list, + ctx.storage, + *ctx.self_id.as_bytes(), + near_bits, + excess.min(MAX_PAID_EVICTIONS_PER_PASS), + ) + .await; match evicted { Ok(evicted) => (count.saturating_sub(evicted), evicted), Err(e) => { @@ -714,13 +713,42 @@ async fn evict_excess_paid_entries(ctx: &PrunePassContext<'_>) -> (usize, usize) } } +/// Evict up to `limit` paid entries outside the `near_bits` around `center`, +/// farthest first, keeping those [`keeps_paid_entry`] keeps. Returns how many +/// went. +/// +/// The removal asks [`keeps_paid_entry`] again inside its write transaction, +/// with new record admissions held back until it has committed. So a record +/// stored after the scan, or under way when the removal decides, keeps its +/// entry and its price-contested mark (ADR-0006). A record admitted after the +/// removal finds no entry, as after any eviction. +async fn evict_paid_entries( + paid_list: &PaidList, + storage: &Arc, + center: XorName, + near_bits: usize, + limit: usize, +) -> Result { + let keep_storage = Arc::clone(storage); + paid_list + .evict_far( + center, + near_bits, + limit, + move |key| keeps_paid_entry(&keep_storage, key), + storage.hold_admissions(), + ) + .await +} + /// Whether eviction keeps `key`'s paid entry: while its record is in the store's -/// index, whether or not it can be read at the moment. The entry authorises the +/// index, whether or not it can be read at the moment, or on its way there +/// (`ChunkStore::holds_or_admits`). The entry authorises the /// record's repair and carries its price-contested mark (ADR-0006); a record that /// failed a read is out of service, not gone, and losing its mark would let it be /// hinted as a replica once it reads again. fn keeps_paid_entry(storage: &ChunkStore, key: &XorName) -> bool { - storage.is_indexed(key) + storage.holds_or_admits(key) } fn remote_close_group_peers(close_group: &[PeerId], self_id: &PeerId) -> Vec { @@ -1596,9 +1624,15 @@ async fn peer_is_currently_responsible( #[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] mod tests { use super::*; + use crate::replication::fresh::record_paid_key; + use crate::replication::paid_list::XORNAME_BITS; use crate::storage::chunk_store::CHUNKS_DIR_NAME; use crate::storage::ChunkStoreConfig; + use futures::FutureExt; + use std::sync::atomic::{AtomicBool, Ordering}; use tempfile::TempDir; + use tokio::sync::oneshot; + use tokio::task::JoinHandle; fn peer_id_from_byte(b: u8) -> PeerId { let mut bytes = [0u8; 32]; @@ -1660,6 +1694,323 @@ mod tests { std::fs::set_permissions(&path, perms).expect("chmod back"); } + /// A chunk store and a paid list, as one node holds them. + async fn eviction_fixture() -> (Arc, Arc, TempDir) { + let dir = TempDir::new().expect("temp dir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().join("store"), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let paid_list = PaidList::new(&dir.path().join("paid")) + .await + .expect("paid list"); + (Arc::new(storage), Arc::new(paid_list), dir) + } + + /// A record to admit, and its key. + fn test_record() -> (Vec, XorName) { + let content = b"price-contested record admitted during an eviction".to_vec(); + let key = ChunkStore::compute_address(&content); + (content, key) + } + + /// Evict up to ten entries far from a center whose top bit differs from + /// `key`'s, so `key` is a candidate. + fn spawn_eviction( + storage: &Arc, + paid_list: &Arc, + key: XorName, + ) -> JoinHandle> { + let (storage, paid_list) = (Arc::clone(storage), Arc::clone(paid_list)); + let mut center = key; + if let Some(first) = center.first_mut() { + *first ^= 0x80; + } + tokio::spawn(async move { + evict_paid_entries(&paid_list, &storage, center, XORNAME_BITS - 1, 10).await + }) + } + + /// Store `content` under `key` as a fresh offer does, recording the key's + /// marked entry first. The record waits after the entry until `store` fires, + /// and `recorded` fires once the entry is in. + fn spawn_fresh_admission( + storage: &Arc, + paid_list: &Arc, + (content, key): (Vec, XorName), + recorded: oneshot::Sender<()>, + store: oneshot::Receiver<()>, + ) -> JoinHandle<(bool, Result)> { + let (storage, paid_list) = (Arc::clone(storage), Arc::clone(paid_list)); + tokio::spawn(async move { + storage + .put_after(&key, &content, async move { + let marked = record_paid_key(&key, true, &paid_list).await; + let _ = recorded.send(()); + let _ = store.await; + marked + }) + .await + }) + } + + async fn wait_until(reached: impl Fn() -> bool + Send + Sync) { + tokio::time::timeout(Duration::from_secs(10), async { + while !reached() { + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await + .expect("reached in time"); + } + + /// Hold the write half of `gate` on a blocking thread until the returned + /// sender is dropped, so whatever takes its read half parks there. + async fn hold_gate( + gate: Arc>, + ) -> (oneshot::Sender<()>, JoinHandle<()>) { + let (release_tx, release_rx) = oneshot::channel::<()>(); + let (held_tx, held_rx) = oneshot::channel::<()>(); + let holder = tokio::task::spawn_blocking(move || { + let _held = gate.write(); + let _ = held_tx.send(()); + let _ = release_rx.blocking_recv(); + }); + held_rx.await.expect("gate held"); + (release_tx, holder) + } + + /// The record is stored and its paid entry still carries the mark, in the + /// database and in memory, so it goes out in paid hints only. + fn assert_stored_with_its_mark(storage: &ChunkStore, paid_list: &PaidList, key: &XorName) { + assert!(storage.exists(key).expect("exists"), "the record is stored"); + assert!( + paid_list.is_price_contested(key).expect("read mark"), + "its paid entry and its mark stay" + ); + assert_eq!( + paid_list.split_marked(vec![*key]), + (Vec::new(), vec![*key]), + "and the in-memory copy still marks it" + ); + } + + /// ADR-0006 with ADR-0028: an eviction scan finds the key's marked entry + /// with no record, and a fresh offer then records (finding that entry) and + /// stores the record, all before the eviction removes anything. The entry + /// and its mark must stay, or the stored record would be hinted as a + /// replica. + #[tokio::test] + async fn eviction_keeps_the_entry_of_a_record_stored_after_its_scan() { + let (storage, paid_list, _dir) = eviction_fixture().await; + let (content, key) = test_record(); + assert!(record_paid_key(&key, true, &paid_list).await); + + let scan_gate = paid_list.test_eviction_scan_gate(); + let scanned = scan_gate.write().await; + let eviction = spawn_eviction(&storage, &paid_list, key); + wait_until(|| paid_list.test_evictions_scanned() == 1).await; + + let (marked, stored) = storage + .put_after(&key, &content, record_paid_key(&key, true, &paid_list)) + .await; + assert!(marked); + assert!(stored.expect("put")); + + drop(scanned); + let evicted = eviction.await.expect("eviction task").expect("evict"); + assert_eq!(evicted, 0); + assert_eq!( + paid_list.test_removals_committing(), + 1, + "the removal ran with the key as a candidate" + ); + assert_stored_with_its_mark(&storage, &paid_list, &key); + } + + /// The eviction removes while a fresh offer has recorded the key's entry + /// but not yet stored its record. The offer's admission keeps the entry. + #[tokio::test] + async fn eviction_keeps_the_entry_of_a_record_recorded_but_not_yet_stored() { + let (storage, paid_list, _dir) = eviction_fixture().await; + let record = test_record(); + let key = record.1; + // Already recorded when the eviction scans: by a `PaidNotify`, or by + // this offer an instant earlier. + assert!(record_paid_key(&key, true, &paid_list).await); + + let scan_gate = paid_list.test_eviction_scan_gate(); + let scanned = scan_gate.write().await; + let eviction = spawn_eviction(&storage, &paid_list, key); + wait_until(|| paid_list.test_evictions_scanned() == 1).await; + + let (recorded_tx, recorded_rx) = oneshot::channel(); + let (store_tx, store_rx) = oneshot::channel(); + let admission = spawn_fresh_admission(&storage, &paid_list, record, recorded_tx, store_rx); + recorded_rx.await.expect("entry recorded"); + + drop(scanned); + let evicted = eviction.await.expect("eviction task").expect("evict"); + assert_eq!(evicted, 0); + assert_eq!( + paid_list.test_removals_committing(), + 1, + "the removal ran with the key as a candidate" + ); + + store_tx.send(()).expect("store the record"); + let (marked, stored) = admission.await.expect("admission task"); + assert!(marked); + assert!(stored.expect("put")); + assert_stored_with_its_mark(&storage, &paid_list, &key); + } + + /// The eviction removes while a put of the key has started but is not yet + /// writing. The put's admission keeps the entry. + #[tokio::test] + async fn eviction_keeps_the_entry_of_a_record_whose_put_is_not_yet_writing() { + let (storage, paid_list, _dir) = eviction_fixture().await; + let (content, key) = test_record(); + assert!(record_paid_key(&key, true, &paid_list).await); + + let scan_gate = paid_list.test_eviction_scan_gate(); + let scanned = scan_gate.write().await; + let eviction = spawn_eviction(&storage, &paid_list, key); + wait_until(|| paid_list.test_evictions_scanned() == 1).await; + + let put_gate = storage.test_pre_registration_gate(); + let put_parked = put_gate.write().await; + let put = { + let storage = Arc::clone(&storage); + tokio::spawn(async move { storage.put(&key, &content).await }) + }; + wait_until(|| storage.test_reached_pre_registration() == 1).await; + + drop(scanned); + let evicted = eviction.await.expect("eviction task").expect("evict"); + assert_eq!(evicted, 0); + assert_eq!( + paid_list.test_removals_committing(), + 1, + "the removal ran with the key as a candidate" + ); + + drop(put_parked); + assert!(put.await.expect("put task").expect("put")); + assert_stored_with_its_mark(&storage, &paid_list, &key); + } + + /// The eviction removes while a put whose caller has gone away still has + /// its write on the blocking pool. That write lands afterwards, so the + /// entry stays. + #[tokio::test] + async fn eviction_keeps_the_entry_of_a_record_a_cancelled_put_is_still_writing() { + let (storage, paid_list, _dir) = eviction_fixture().await; + let (content, key) = test_record(); + assert!(record_paid_key(&key, true, &paid_list).await); + + let scan_gate = paid_list.test_eviction_scan_gate(); + let scanned = scan_gate.write().await; + let eviction = spawn_eviction(&storage, &paid_list, key); + wait_until(|| paid_list.test_evictions_scanned() == 1).await; + + let (release_write, write_parked) = hold_gate(storage.test_put_gate()).await; + let put = { + let storage = Arc::clone(&storage); + tokio::spawn(async move { storage.put(&key, &content).await }) + }; + wait_until(|| storage.test_file_tasks_in_flight() == 1).await; + put.abort(); + assert!(put.await.expect_err("aborted").is_cancelled()); + + drop(scanned); + let evicted = eviction.await.expect("eviction task").expect("evict"); + assert_eq!(evicted, 0); + assert_eq!( + paid_list.test_removals_committing(), + 1, + "the removal ran with the key as a candidate" + ); + + drop(release_write); + write_parked.await.expect("gate holder"); + storage.wait_idle().await; + assert_stored_with_its_mark(&storage, &paid_list, &key); + } + + /// An admission that starts while an eviction is removing the key's entry + /// waits until the removal has committed, without getting past registering, + /// so it records the entry afresh rather than trusting one about to go. + #[tokio::test] + async fn an_admission_waits_for_an_eviction_to_commit() { + let (storage, paid_list, _dir) = eviction_fixture().await; + let (content, key) = test_record(); + assert!(record_paid_key(&key, true, &paid_list).await); + + let (release_commit, committing) = hold_gate(paid_list.test_removal_commit_gate()).await; + let eviction = spawn_eviction(&storage, &paid_list, key); + wait_until(|| paid_list.test_removals_committing() == 1).await; + + let started = AtomicBool::new(false); + let admission = storage.put_after(&key, &content, async { + started.store(true, Ordering::Release); + record_paid_key(&key, true, &paid_list).await + }); + tokio::pin!(admission); + assert!( + (&mut admission).now_or_never().is_none(), + "the admission waits for the commit" + ); + assert!( + !started.load(Ordering::Acquire), + "without getting past registering" + ); + + drop(release_commit); + committing.await.expect("gate holder"); + let evicted = eviction.await.expect("eviction task").expect("evict"); + assert_eq!(evicted, 1, "the entry with no record goes"); + let (marked, stored) = admission.await; + assert!(started.load(Ordering::Acquire)); + assert!(marked, "the entry is recorded afresh, marked"); + assert!(stored.expect("put")); + assert_stored_with_its_mark(&storage, &paid_list, &key); + } + + /// A client PUT of a chunk the index names but whose file has gone finds + /// the file missing, drops the stale index entry, and puts the offered + /// copy back. An eviction that removes in between keeps the entry: the + /// check counts as an admission from its start. + #[tokio::test] + async fn eviction_keeps_the_entry_of_a_record_a_put_check_is_putting_back() { + let (storage, paid_list, _dir) = eviction_fixture().await; + let (content, key) = test_record(); + assert!(storage.put(&key, &content).await.expect("put")); + assert!(record_paid_key(&key, true, &paid_list).await); + std::fs::remove_file(storage.test_chunk_path(&key)).expect("lose the file"); + + let repair_gate = storage.test_pre_repair_gate(); + let repair_parked = repair_gate.write().await; + let check = { + let storage = Arc::clone(&storage); + tokio::spawn(async move { storage.holds_verified(&key, &content).await }) + }; + wait_until(|| storage.test_reached_pre_repair() == 1).await; + assert!(!storage.is_indexed(&key), "the stale index entry is gone"); + + let evicted = spawn_eviction(&storage, &paid_list, key) + .await + .expect("eviction task") + .expect("evict"); + assert_eq!(evicted, 0); + + drop(repair_parked); + assert!(check.await.expect("check task"), "the offered copy is back"); + assert_stored_with_its_mark(&storage, &paid_list, &key); + } + /// A prune audit judges holders against this node's own copy. A copy that /// no longer hashes to its key would fail every honest holder and hold the /// prune back forever, so it is not used as a reference: the read takes it diff --git a/src/storage/chunk_store.rs b/src/storage/chunk_store.rs index d6029349..022ff895 100644 --- a/src/storage/chunk_store.rs +++ b/src/storage/chunk_store.rs @@ -43,6 +43,7 @@ use fs2::FileExt; use serde::{Deserialize, Serialize}; use std::collections::{BTreeSet, HashMap, HashSet, VecDeque}; use std::fs::{File, OpenOptions}; +use std::future::Future; use std::io::{ErrorKind, Read, Write}; use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicU64, Ordering}; @@ -533,6 +534,33 @@ impl Drop for WriteInFlight { } } +/// An admission of a record under way, counted in [`ChunkStore::admitting`] until +/// dropped. +struct Admission { + admitting: Arc>>, + address: XorName, +} + +impl Drop for Admission { + fn drop(&mut self) { + let mut admitting = self.admitting.lock(); + match admitting.get_mut(&self.address) { + Some(count) if *count > 1 => *count -= 1, + _ => { + admitting.remove(&self.address); + } + } + } +} + +/// Record admissions held back on a [`ChunkStore`] (see +/// [`ChunkStore::hold_admissions`]). They resume when this is dropped. +#[derive(Debug)] +#[must_use = "admissions resume as soon as this is dropped"] +pub struct AdmissionsHeld { + _gate: tokio::sync::OwnedRwLockWriteGuard<()>, +} + /// What is behind a chunk's name on disk. /// /// Four answers, not two, because "could not read it" must never be treated as "wrong": @@ -678,6 +706,15 @@ pub struct ChunkStore { writing: Arc>>, /// Woken when [`Self::writing`] loses its last entry for a key. write_finished: Arc, + /// Records being admitted: for each address, how many admissions of it are under + /// way (see [`Self::holds_or_admits`]). + /// + /// Counted, like [`Self::writing`]: the admission [`Self::put_after`] holds spans + /// the put it makes, which holds one of its own. + admitting: Arc>>, + /// Taken shared while an admission registers, and exclusively by + /// [`Self::hold_admissions`], so no admission starts while that is held. + admission_gate: Arc>, /// Size-aware free-space predicate. capacity: Arc, /// Monotonic counter that makes temp filenames unique within this store. @@ -734,6 +771,14 @@ pub struct ChunkStore { /// is a flake waiting for a loaded machine. #[cfg(test)] test_reached_pre_registration: Arc, + + /// Test-only: parks [`Self::holds_verified`] after its read and before it repairs. + #[cfg(test)] + test_pre_repair_gate: Arc>, + + /// Test-only: how many checks have reached that gate. + #[cfg(test)] + test_reached_pre_repair: Arc, } impl ChunkStore { @@ -832,6 +877,8 @@ impl ChunkStore { known_wrong: Arc::new(parking_lot::RwLock::new(HashSet::new())), writing: Arc::new(parking_lot::Mutex::new(HashMap::new())), write_finished: Arc::new(tokio::sync::Notify::new()), + admitting: Arc::new(parking_lot::Mutex::new(HashMap::new())), + admission_gate: Arc::new(tokio::sync::RwLock::new(())), capacity, temp_seq: AtomicU64::new(0), nonce: rand::random(), @@ -847,6 +894,10 @@ impl ChunkStore { test_pre_registration_gate: Arc::new(tokio::sync::RwLock::new(())), #[cfg(test)] test_reached_pre_registration: Arc::new(std::sync::atomic::AtomicU64::new(0)), + #[cfg(test)] + test_pre_repair_gate: Arc::new(tokio::sync::RwLock::new(())), + #[cfg(test)] + test_reached_pre_repair: Arc::new(std::sync::atomic::AtomicU64::new(0)), }) } @@ -868,6 +919,9 @@ impl ChunkStore { /// Returns [`Error::Storage`] if the content does not hash to `address`, the disk /// is too full, or the write fails. pub async fn put(&self, address: &XorName, content: &[u8]) -> Result { + // Before the key's lane, so a put kept waiting by `hold_admissions` holds no lane + // a delete needs. + let _admission = self.admit(address).await; // The key's whole transition, not just the part that touches the disk. Registering // the write is what a delete waits for, and everything before that registration // happens outside it: validation, the duplicate read, the capacity reservation. A @@ -1042,6 +1096,25 @@ impl ChunkStore { } } + /// Run `first`, then store `content` under `address` as [`Self::put`] does, all as + /// one admission, and return both results. + /// + /// For a caller that writes something the record depends on, such as its paid-list + /// entry, before the record itself: [`Self::holds_or_admits`] counts the key from + /// before `first` starts, so whatever acts on keys with no record cannot act on this + /// one in between. + pub async fn put_after( + &self, + address: &XorName, + content: &[u8], + first: impl Future, + ) -> (T, Result) { + let _admission = self.admit(address).await; + let first = first.await; + let stored = self.put(address, content).await; + (first, stored) + } + /// Decide what a name that was already taken actually means. /// /// Split out of [`Self::put`] because it is a different question. `put` puts bytes on @@ -1107,6 +1180,47 @@ impl ChunkStore { } } + /// Register an admission of `address`, once no [`Self::hold_admissions`] is held. + async fn admit(&self, address: &XorName) -> Admission { + let _open = self.admission_gate.read().await; + *self.admitting.lock().entry(*address).or_insert(0) += 1; + Admission { + admitting: Arc::clone(&self.admitting), + address: *address, + } + } + + /// Hold back every new admission of a record ([`Self::put`], [`Self::put_after`], + /// [`Self::repair`], [`Self::holds_verified`]) until the returned guard is dropped. + /// + /// While it is held, a key [`Self::holds_or_admits`] answers `false` for gets no + /// record. Waits only for admissions registering at this moment, not for those under + /// way, which `holds_or_admits` answers for. + /// + /// For a caller that acts on keys having no record: paid-list eviction removes such + /// keys' entries, and a record admitted between its decision and the removal would + /// lose its entry, and its price-contested mark with it (ADR-0006). + pub async fn hold_admissions(&self) -> AdmissionsHeld { + AdmissionsHeld { + _gate: Arc::clone(&self.admission_gate).write_owned().await, + } + } + + /// Whether `address` has a record in the index or on its way there: an admission or + /// a write of it is under way. Unlike [`Self::exists`], a record that cannot be read + /// for now counts. + /// + /// Read in this order because each step hands over to the next before it ends: an + /// admission ends only after indexing its key itself or registering the write that + /// will, and a write unregisters only after indexing its key. So a key that has just + /// left one is already in the next. + #[must_use] + pub fn holds_or_admits(&self, address: &XorName) -> bool { + self.admitting.lock().contains_key(address) + || self.writing.lock().contains_key(address) + || self.is_indexed(address) + } + /// Take the critical section for one key. /// /// Held across await points, unlike the write lanes, so a whole logical transition for @@ -1152,6 +1266,10 @@ impl ChunkStore { // longer has. Closing that would mean holding a per-key lock across a network // response, which trades a narrow window for a much worse one. Replication finds // the key missing and re-offers it. + // + // An admission, since its read can drop a stale index entry and its repair then + // puts the key back (see `hold_admissions`). Taken before the lane, as `put` does. + let _admission = self.admit(address).await; let _lane = self.key_lock(address).await; if !self.is_indexed(address) { @@ -1161,7 +1279,16 @@ impl ChunkStore { // does not match, and off Unix replacing a chunk truncates it in place, so acting // on an unanswered question would empty a healthy sole copy. The read below // distinguishes them. - match self.get_raw(address).await { + let read = self.get_raw(address).await; + // Test-only: the window between a read that dropped a stale index entry and the + // repair that puts the key back. + #[cfg(test)] + { + self.test_reached_pre_repair + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + drop(self.test_pre_repair_gate.read().await); + } + match read { // Byte-for-byte what the caller has, and the caller checked those bytes against // the address before getting here. Nothing is wrong with this file. Ok(Some(stored)) if stored == content => { @@ -1275,6 +1402,7 @@ impl ChunkStore { /// Returns [`Error::Storage`] if `content` does not hash to `address`, or the write /// fails. pub async fn repair(&self, address: &XorName, content: &[u8]) -> Result<()> { + let _admission = self.admit(address).await; let _lane = self.key_lock(address).await; self.repair_holding_the_lane(address, content).await } @@ -1890,17 +2018,31 @@ impl ChunkStore { /// Test-only handle to the gate that parks a put before it registers itself. #[cfg(test)] - fn test_pre_registration_gate(&self) -> Arc> { + pub(crate) fn test_pre_registration_gate(&self) -> Arc> { Arc::clone(&self.test_pre_registration_gate) } /// Test-only: how many puts have reached the pre-registration gate. #[cfg(test)] - fn test_reached_pre_registration(&self) -> u64 { + pub(crate) fn test_reached_pre_registration(&self) -> u64 { self.test_reached_pre_registration .load(std::sync::atomic::Ordering::Acquire) } + /// Test-only handle to the gate that parks [`Self::holds_verified`] before it + /// repairs. + #[cfg(test)] + pub(crate) fn test_pre_repair_gate(&self) -> Arc> { + Arc::clone(&self.test_pre_repair_gate) + } + + /// Test-only: how many checks have reached the pre-repair gate. + #[cfg(test)] + pub(crate) fn test_reached_pre_repair(&self) -> u64 { + self.test_reached_pre_repair + .load(std::sync::atomic::Ordering::Acquire) + } + /// Test-only handle to the write lanes, and the index of the one `address` is /// written, repaired and quarantined under. /// diff --git a/src/storage/mod.rs b/src/storage/mod.rs index b573acd9..1a010463 100644 --- a/src/storage/mod.rs +++ b/src/storage/mod.rs @@ -61,7 +61,7 @@ pub(crate) mod traffic; pub use crate::ant_protocol::XorName; pub(crate) use chunk_store::{rename_with_retry, Reservation}; -pub use chunk_store::{ChunkStore, ChunkStoreConfig, DiskSpace}; +pub use chunk_store::{AdmissionsHeld, ChunkStore, ChunkStoreConfig, DiskSpace}; // Crate-private, as it was before the two stores became one: `CapacityVerdict` was // `pub(crate)` on the old store and has no caller outside this crate. pub(crate) use chunk_store::CapacityVerdict; From 5fd128106aa3273a40c16fdcee037cad3b05c49d Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 6 Nov 2026 11:36:02 +0900 Subject: [PATCH 17/22] docs(adr): ADR-0006 records that eviction keeps the entries of records being admitted Paid-list eviction now asks again inside its removal, with new record admissions held back until the removal commits, so the narrow gap the ADR listed for a record admitted between the scan and the delete is closed. The decision text says so, and that a record admitted after the removal finds no entry, as after any eviction; the gap is removed from the consequences; and the validation lists the six interleaving tests. --- .../ADR-0006-receiver-side-revenue-floor.md | 25 +++++++++++++------ 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 9a51f321..3ad60b79 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -401,8 +401,15 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over - a prune audit, which is sent only under disk pressure (ADR-0028), still needs its proofs before deleting a marked key, but scores no absent peer for it. Paid-list eviction (ADR-0028) keeps the entry of every record - still in the node's store, marked or not and readable or not, and an - entry it evicts takes its mark with it; + still in the node's store, marked or not and readable or not, or on its + way there, and an entry it evicts takes its mark with it. It asks again + inside the removal, with new record admissions held back until the + removal commits, and a fresh offer's record counts as on its way from + before the offer writes the entry. So a record admitted while an + eviction runs keeps its entry and its mark. One admitted after the + removal finds no entry, as after any eviction: a fresh offer, or a + client PUT that carries a proof, records it afresh, and a fetch stores + the record without one; - a mark that cannot be read counts as set: when in doubt, judge no one. While reads fail the node judges no peer on any key they hit, so it warns at the first failure of a run and, when a read succeeds again, @@ -618,12 +625,6 @@ out of 1,801: while the mark stays. A later plain insert that adds the key, or a removal of it, drops the mark once that write commits, and a restart rebuilds the copy from the database. - - Paid-list eviction checks for the record when it scans, but a record - admitted between that scan and the delete would lose its entry, and the - mark with it. Eviction takes only entries far outside this node's range, - where with evenly spread IDs neither store path admits the record, so - that needs a key whose peers this node barely knows, or a routing view - that changes at that moment. - **The first paid-list record decides a key's mark, which a quoter can front-run.** A quoter in the 20 closest to an honest key can pay itself for the key, for gas only, and announce it before the client's write @@ -706,6 +707,14 @@ out of 1,801: near one is left to the near-key read, paid-list eviction drops the mark of an entry it evicts and leaves a kept entry marked, and it keeps the entry of a record that cannot be read for the moment; + - paid-list eviction keeps the entry and the mark of a record stored + after its scan, of one a fresh offer has recorded but not yet stored, + of one whose put has started but is not yet writing, of one a + cancelled put is still writing, and of one a client PUT's check finds + missing from disk and puts back, and an admission that starts while + the removal runs waits for its commit without getting past + registering. Each test parks the eviction and the admission at fixed + points, so each runs one interleaving; - the in-memory copy of the marks is filled on opening and follows every write and removal, and a marked insert and a batch removal change it only in the order writes take; a mark held in memory alone, as a failed From 75b050149b491031b175fdb6d74e8d3b35af31d0 Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 6 Nov 2026 12:21:25 +0900 Subject: [PATCH 18/22] test(replication): let the put-check eviction test finish the PUT where repair cannot Off Unix a repair rewrites a chunk's file in place and cannot recreate one that has gone missing, so holds_verified answers no for a missing file and the PUT handler stores the offered copy through the ordinary put. The test that parks holds_verified after it dropped the stale index entry assumed the Unix outcome and failed on Windows after its eviction check had passed. It now asserts the check puts the copy back only on Unix, and elsewhere stores it with put as the handler does, before checking the record kept its paid entry and mark. ADR-0006's test list says the check is under way, not that it puts the copy back. --- .../ADR-0006-receiver-side-revenue-floor.md | 8 ++++---- src/replication/pruning.rs | 19 +++++++++++++------ 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 3ad60b79..b22217c7 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -711,10 +711,10 @@ out of 1,801: after its scan, of one a fresh offer has recorded but not yet stored, of one whose put has started but is not yet writing, of one a cancelled put is still writing, and of one a client PUT's check finds - missing from disk and puts back, and an admission that starts while - the removal runs waits for its commit without getting past - registering. Each test parks the eviction and the admission at fixed - points, so each runs one interleaving; + missing from disk while the check is under way, and an admission that + starts while the removal runs waits for its commit without getting + past registering. Each test parks the eviction and the admission at + fixed points, so each runs one interleaving; - the in-memory copy of the marks is filled on opening and follows every write and removal, and a marked insert and a batch removal change it only in the order writes take; a mark held in memory alone, as a failed diff --git a/src/replication/pruning.rs b/src/replication/pruning.rs index 0619d8d9..6ed12779 100644 --- a/src/replication/pruning.rs +++ b/src/replication/pruning.rs @@ -1980,9 +1980,9 @@ mod tests { } /// A client PUT of a chunk the index names but whose file has gone finds - /// the file missing, drops the stale index entry, and puts the offered - /// copy back. An eviction that removes in between keeps the entry: the - /// check counts as an admission from its start. + /// the file missing, drops the stale index entry, and on Unix puts the + /// offered copy back. An eviction that removes in between keeps the + /// entry: the check counts as an admission from its start. #[tokio::test] async fn eviction_keeps_the_entry_of_a_record_a_put_check_is_putting_back() { let (storage, paid_list, _dir) = eviction_fixture().await; @@ -1994,8 +1994,8 @@ mod tests { let repair_gate = storage.test_pre_repair_gate(); let repair_parked = repair_gate.write().await; let check = { - let storage = Arc::clone(&storage); - tokio::spawn(async move { storage.holds_verified(&key, &content).await }) + let (storage, offered) = (Arc::clone(&storage), content.clone()); + tokio::spawn(async move { storage.holds_verified(&key, &offered).await }) }; wait_until(|| storage.test_reached_pre_repair() == 1).await; assert!(!storage.is_indexed(&key), "the stale index entry is gone"); @@ -2007,7 +2007,14 @@ mod tests { assert_eq!(evicted, 0); drop(repair_parked); - assert!(check.await.expect("check task"), "the offered copy is back"); + // Off Unix a repair rewrites the file in place and cannot recreate a missing + // one, so the check answers no and the PUT stores the copy, as the handler does. + let put_back = check.await.expect("check task"); + #[cfg(unix)] + assert!(put_back, "on Unix the check puts the offered copy back"); + if !put_back { + assert!(storage.put(&key, &content).await.expect("put")); + } assert_stored_with_its_mark(&storage, &paid_list, &key); } From 73e15bbd156661fa46a0c0c55a89f1534d3d440c Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 6 Nov 2026 12:32:05 +0900 Subject: [PATCH 19/22] docs(adr): ADR-0006 counts the mutations run on the eviction and admission coordination Nine targeted mutations of the coordination were run against its six interleaving tests, and each fails at least one: a removal that does not ask again, put_after, put or holds_verified without an admission, an admission check that ignores writes in flight, an admission that does not wait for a held eviction, a guard dropped before the removal, the keep rule back to the index alone, and all of the coordination removed at once, which fails all six. --- docs/adr/ADR-0006-receiver-side-revenue-floor.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index b22217c7..92014a85 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -762,7 +762,14 @@ out of 1,801: paid hints, treating unknown marks as no marks, and keeping only readable records' entries through eviction each fail their unit test; and prune-audit scoring that ignores the mark fails the e2e prune test, which - runs under disk pressure. + runs under disk pressure. Nine more were run on the coordination of + eviction with record admission, and each fails at least one of its six + interleaving tests: a removal that does not ask again; a `put_after`, a + `put` or a `holds_verified` that registers no admission; an admission + check that ignores writes in flight; an admission that does not wait for + a held eviction; a guard dropped before the removal; the keep rule back + to the index alone; and all of the coordination removed at once, which + fails all six. - `tests/e2e/price_floor_ab.rs` drives the same honest and near-empty-quoter uploads through a production-wired local testnet. Nodes are seeded to production-sized commitments, payments are real (on Anvil), and possession From 3901df30b6caf1b96dd661bda5f5ca3722538fd1 Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 6 Nov 2026 15:23:02 +0900 Subject: [PATCH 20/22] fix(storage): keep a client PUT's paid entry from its check to its store The client PUT handler checked whether it already held the chunk, then verified the payment, which can take seconds, and only then stored the chunk. The check and the store were separate admissions, so while the payment was verified paid-list eviction could find the key with no admission, no write in flight and no record (the check may also have dropped a stale index entry for a file that had gone), and remove the key's paid entry and its price-contested mark. The PUT then stored the record unmarked until the fresh-write drainer recorded an entry again, and the node could hint it as a replica in between (ADR-0006). The handler now holds one admission from before its check to the end of its store, so eviction keeps the entry throughout. Nothing is stored before the payment is verified. ChunkStore::admit becomes public for this, returning a RecordAdmission guard. A test parks the eviction after its scan with the key as a candidate, then parks the PUT after its check found the chunk not held, lets the eviction's removal run and commit, and resumes the PUT. Without the handler's admission the removal takes the entry; with it the entry and its mark stay and the stored record is hinted as paid only. --- src/replication/pruning.rs | 2 +- src/storage/chunk_store.rs | 28 +++++--- src/storage/handler.rs | 128 +++++++++++++++++++++++++++++++++++++ src/storage/mod.rs | 2 +- 4 files changed, 149 insertions(+), 11 deletions(-) diff --git a/src/replication/pruning.rs b/src/replication/pruning.rs index 6ed12779..5ac9a3de 100644 --- a/src/replication/pruning.rs +++ b/src/replication/pruning.rs @@ -722,7 +722,7 @@ async fn evict_excess_paid_entries(ctx: &PrunePassContext<'_>) -> (usize, usize) /// stored after the scan, or under way when the removal decides, keeps its /// entry and its price-contested mark (ADR-0006). A record admitted after the /// removal finds no entry, as after any eviction. -async fn evict_paid_entries( +pub(crate) async fn evict_paid_entries( paid_list: &PaidList, storage: &Arc, center: XorName, diff --git a/src/storage/chunk_store.rs b/src/storage/chunk_store.rs index 022ff895..95d1f370 100644 --- a/src/storage/chunk_store.rs +++ b/src/storage/chunk_store.rs @@ -534,14 +534,16 @@ impl Drop for WriteInFlight { } } -/// An admission of a record under way, counted in [`ChunkStore::admitting`] until -/// dropped. -struct Admission { +/// An admission of a record under way, counted until dropped (see +/// [`ChunkStore::admit`]). +#[derive(Debug)] +#[must_use = "the admission ends as soon as this is dropped"] +pub struct RecordAdmission { admitting: Arc>>, address: XorName, } -impl Drop for Admission { +impl Drop for RecordAdmission { fn drop(&mut self) { let mut admitting = self.admitting.lock(); match admitting.get_mut(&self.address) { @@ -1180,18 +1182,26 @@ impl ChunkStore { } } - /// Register an admission of `address`, once no [`Self::hold_admissions`] is held. - async fn admit(&self, address: &XorName) -> Admission { + /// Register an admission of `address`, once no [`Self::hold_admissions`] is held, + /// and keep it until the returned guard is dropped. + /// + /// [`Self::put`], [`Self::put_after`], [`Self::repair`] and [`Self::holds_verified`] + /// each hold one of their own. A caller that admits a record in several steps holds + /// one across all of them, so [`Self::holds_or_admits`] counts the key throughout: + /// the client PUT handler holds one from its check that the chunk is held, through + /// payment verification, to its store. + pub async fn admit(&self, address: &XorName) -> RecordAdmission { let _open = self.admission_gate.read().await; *self.admitting.lock().entry(*address).or_insert(0) += 1; - Admission { + RecordAdmission { admitting: Arc::clone(&self.admitting), address: *address, } } - /// Hold back every new admission of a record ([`Self::put`], [`Self::put_after`], - /// [`Self::repair`], [`Self::holds_verified`]) until the returned guard is dropped. + /// Hold back every new admission of a record ([`Self::admit`], and so [`Self::put`], + /// [`Self::put_after`], [`Self::repair`] and [`Self::holds_verified`]) until the + /// returned guard is dropped. /// /// While it is held, a key [`Self::holds_or_admits`] answers `false` for gets no /// record. Waits only for admissions registering at this moment, not for those under diff --git a/src/storage/handler.rs b/src/storage/handler.rs index 3be0e14c..f3a83f64 100644 --- a/src/storage/handler.rs +++ b/src/storage/handler.rs @@ -411,6 +411,13 @@ pub struct AntProtocol { /// message to a node that does not keep pointers gets a clean refusal /// rather than a silent drop. pointers: Option, + /// Test-only: parks a PUT after its check found the chunk not held, before + /// its payment is verified. + #[cfg(test)] + test_after_check_gate: Arc>, + /// Test-only: how many PUTs have reached that gate. + #[cfg(test)] + test_reached_after_check: Arc, } impl AntProtocol { @@ -449,6 +456,10 @@ impl AntProtocol { fresh_write_tx: None, p2p_node: RwLock::new(None), pointers: None, + #[cfg(test)] + test_after_check_gate: Arc::new(tokio::sync::RwLock::new(())), + #[cfg(test)] + test_reached_after_check: Arc::new(AtomicU64::new(0)), } } @@ -835,6 +846,11 @@ impl AntProtocol { } } + // One admission from the check below to the store in step 6, so paid-list + // eviction cannot take this key's paid entry, and its price-contested mark, + // while the payment is verified (ADR-0006). Nothing is stored before then. + let _admission = self.storage.admit(&address).await; + // 3. Check if already exists (idempotent success) // // Verified against the offered bytes, not answered from the name. A name can @@ -851,6 +867,8 @@ impl AntProtocol { debug!("Chunk {addr_hex} already exists"); return ChunkPutResponse::AlreadyExists { address }; } + #[cfg(test)] + self.test_park_after_check().await; // 4. Cheap disk-space pre-check — runs BEFORE the expensive payment // verification path (ML-DSA pool checks, a Kademlia closeness @@ -1348,6 +1366,28 @@ impl AntProtocol { pub async fn put_local(&self, address: &[u8; 32], content: &[u8]) -> Result { self.storage.put(address, content).await } + + /// Test-only handle to the gate that parks a PUT after its check found the + /// chunk not held. + #[cfg(test)] + fn test_after_check_gate(&self) -> Arc> { + Arc::clone(&self.test_after_check_gate) + } + + /// Test-only: how many PUTs have reached that gate. + #[cfg(test)] + fn test_reached_after_check(&self) -> u64 { + self.test_reached_after_check.load(Ordering::Acquire) + } + + /// Test-only: the window between a check that found the chunk not held and + /// the store that follows a verified payment. Parks while a test holds the + /// gate's write half. + #[cfg(test)] + async fn test_park_after_check(&self) { + self.test_reached_after_check.fetch_add(1, Ordering::AcqRel); + drop(self.test_after_check_gate.read().await); + } } #[cfg(test)] @@ -1357,6 +1397,9 @@ mod tests { use crate::payment::metrics::QuotingMetricsTracker; use crate::payment::{EvmVerifierConfig, PaymentVerifierConfig}; use crate::pointer::{DATA_TYPE_POINTER, POINTER_WIRE_LEN}; + use crate::replication::fresh::record_paid_key; + use crate::replication::paid_list::{PaidList, XORNAME_BITS}; + use crate::replication::pruning::evict_paid_entries; use crate::storage::ChunkStoreConfig; use evmlib::RewardsAddress; use saorsa_core::identity::NodeIdentity; @@ -1417,6 +1460,91 @@ mod tests { (protocol, temp_dir) } + /// ADR-0006 with ADR-0028: a client PUT of a chunk this node does not store, + /// whose marked paid entry an earlier `PaidNotify` wrote, finds the chunk + /// not held and goes on to verify payment. An eviction that removes then + /// must keep the entry and its mark, or the record the PUT stores would be + /// hinted as a replica until its own entry is written. + #[tokio::test] + async fn a_put_keeps_the_paid_entry_from_its_check_to_its_store() { + let (protocol, temp) = create_test_protocol().await; + let protocol = Arc::new(protocol); + let paid_list = Arc::new( + PaidList::new(&temp.path().join("paid")) + .await + .expect("paid list"), + ); + let content = b"price-contested chunk put while an eviction runs".to_vec(); + let address = compute_address(&content); + assert!(record_paid_key(&address, true, &paid_list).await); + protocol.payment_verifier().cache_insert(address); + + // The eviction scans first and finds the entry with no record. + let scan_gate = paid_list.test_eviction_scan_gate(); + let scanned = scan_gate.write().await; + let eviction = { + let (paid_list, storage) = (Arc::clone(&paid_list), Arc::clone(&protocol.storage)); + let mut center = address; + if let Some(first) = center.first_mut() { + *first ^= 0x80; + } + tokio::spawn(async move { + evict_paid_entries(&paid_list, &storage, center, XORNAME_BITS - 1, 10).await + }) + }; + wait_until(|| paid_list.test_evictions_scanned() == 1).await; + + // Then the PUT finds the chunk not held and stops before its payment. + let check_gate = protocol.test_after_check_gate(); + let checked = check_gate.write().await; + let put = { + let protocol = Arc::clone(&protocol); + let request = ChunkPutRequest::new(address, Bytes::from(content)); + tokio::spawn(async move { protocol.handle_put(request).await }) + }; + wait_until(|| protocol.test_reached_after_check() == 1).await; + + // The eviction's removal runs and commits with the key as a candidate. + drop(scanned); + let evicted = eviction.await.expect("eviction task").expect("evict"); + assert_eq!( + paid_list.test_removals_committing(), + 1, + "the removal ran with the key as a candidate" + ); + assert_eq!( + evicted, 0, + "the PUT's entry stays while the PUT is under way" + ); + + drop(checked); + let response = put.await.expect("put task"); + assert!( + matches!(response, ChunkPutResponse::Success { .. }), + "stored: {response:?}" + ); + assert!(protocol.storage.exists(&address).expect("exists")); + assert!( + paid_list.is_price_contested(&address).expect("read mark"), + "the stored record keeps its entry and its mark" + ); + assert_eq!( + paid_list.split_marked(vec![address]), + (Vec::new(), vec![address]), + "so it goes out in paid hints only" + ); + } + + async fn wait_until(reached: impl Fn() -> bool + Send + Sync) { + tokio::time::timeout(Duration::from_secs(10), async { + while !reached() { + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await + .expect("reached in time"); + } + #[tokio::test] async fn test_put_and_get_chunk() { let (protocol, _temp) = create_test_protocol().await; diff --git a/src/storage/mod.rs b/src/storage/mod.rs index 1a010463..971a774e 100644 --- a/src/storage/mod.rs +++ b/src/storage/mod.rs @@ -61,7 +61,7 @@ pub(crate) mod traffic; pub use crate::ant_protocol::XorName; pub(crate) use chunk_store::{rename_with_retry, Reservation}; -pub use chunk_store::{AdmissionsHeld, ChunkStore, ChunkStoreConfig, DiskSpace}; +pub use chunk_store::{AdmissionsHeld, ChunkStore, ChunkStoreConfig, DiskSpace, RecordAdmission}; // Crate-private, as it was before the two stores became one: `CapacityVerdict` was // `pub(crate)` on the old store and has no caller outside this crate. pub(crate) use chunk_store::CapacityVerdict; From c797212e03970dd22f0ca5a7e3d6dbb6d27853f2 Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 6 Nov 2026 15:23:02 +0900 Subject: [PATCH 21/22] docs(adr): ADR-0006 counts the client PUT as one admission from its check to its store The decision text says a client PUT's record counts as on its way from its check that the chunk is held, through payment verification, to its store. The validation lists the seventh interleaving test, and the mutation count is now ten: the client PUT handler without its admission fails that test, and all of the coordination removed at once fails all seven. --- .../ADR-0006-receiver-side-revenue-floor.md | 38 ++++++++++--------- 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 92014a85..18748661 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -404,12 +404,13 @@ replication), arriving in 48 upload bursts. Of those, 1,801 admissions over still in the node's store, marked or not and readable or not, or on its way there, and an entry it evicts takes its mark with it. It asks again inside the removal, with new record admissions held back until the - removal commits, and a fresh offer's record counts as on its way from - before the offer writes the entry. So a record admitted while an - eviction runs keeps its entry and its mark. One admitted after the - removal finds no entry, as after any eviction: a fresh offer, or a - client PUT that carries a proof, records it afresh, and a fetch stores - the record without one; + removal commits. A fresh offer's record counts as on its way from + before the offer writes the entry, and a client PUT's from its check + that the chunk is held, through payment verification, to its store. + So a record admitted while an eviction runs keeps its entry and its + mark. One admitted after the removal finds no entry, as after any + eviction: a fresh offer, or a client PUT that carries a proof, records + it afresh, and a fetch stores the record without one; - a mark that cannot be read counts as set: when in doubt, judge no one. While reads fail the node judges no peer on any key they hit, so it warns at the first failure of a run and, when a read succeeds again, @@ -710,11 +711,12 @@ out of 1,801: - paid-list eviction keeps the entry and the mark of a record stored after its scan, of one a fresh offer has recorded but not yet stored, of one whose put has started but is not yet writing, of one a - cancelled put is still writing, and of one a client PUT's check finds - missing from disk while the check is under way, and an admission that - starts while the removal runs waits for its commit without getting - past registering. Each test parks the eviction and the admission at - fixed points, so each runs one interleaving; + cancelled put is still writing, of one a client PUT's check finds + missing from disk while the check is under way, and of one a client + PUT is verifying payment for after finding it not held, and an + admission that starts while the removal runs waits for its commit + without getting past registering. Each test parks the eviction and the + admission at fixed points, so each runs one interleaving; - the in-memory copy of the marks is filled on opening and follows every write and removal, and a marked insert and a batch removal change it only in the order writes take; a mark held in memory alone, as a failed @@ -762,14 +764,14 @@ out of 1,801: paid hints, treating unknown marks as no marks, and keeping only readable records' entries through eviction each fail their unit test; and prune-audit scoring that ignores the mark fails the e2e prune test, which - runs under disk pressure. Nine more were run on the coordination of - eviction with record admission, and each fails at least one of its six + runs under disk pressure. Ten more were run on the coordination of + eviction with record admission, and each fails at least one of its seven interleaving tests: a removal that does not ask again; a `put_after`, a - `put` or a `holds_verified` that registers no admission; an admission - check that ignores writes in flight; an admission that does not wait for - a held eviction; a guard dropped before the removal; the keep rule back - to the index alone; and all of the coordination removed at once, which - fails all six. + `put`, a `holds_verified` or the client PUT handler that registers no + admission; an admission check that ignores writes in flight; an + admission that does not wait for a held eviction; a guard dropped before + the removal; the keep rule back to the index alone; and all of the + coordination removed at once, which fails all seven. - `tests/e2e/price_floor_ab.rs` drives the same honest and near-empty-quoter uploads through a production-wired local testnet. Nodes are seeded to production-sized commitments, payments are real (on Anvil), and possession From c19b075f5e20a6241e89ce9f11937383ac7c76fd Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 6 Nov 2026 16:11:04 +0900 Subject: [PATCH 22/22] docs(storage): say what still waits for payment in the PUT handler's admission comment The comment on the PUT handler's admission said nothing is stored before the payment is verified. The check at step 3 can repair an already held chunk's damaged bytes from the offered copy before that, as it always has. The comment now says that the store at step 6 still waits for the verified payment. --- src/storage/handler.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/storage/handler.rs b/src/storage/handler.rs index f3a83f64..44d047e1 100644 --- a/src/storage/handler.rs +++ b/src/storage/handler.rs @@ -848,7 +848,8 @@ impl AntProtocol { // One admission from the check below to the store in step 6, so paid-list // eviction cannot take this key's paid entry, and its price-contested mark, - // while the payment is verified (ADR-0006). Nothing is stored before then. + // while the payment is verified (ADR-0006). That store still waits for the + // verified payment. let _admission = self.storage.admit(&address).await; // 3. Check if already exists (idempotent success)