[Nic-dorman]

Summary

test_merkle_payment_across_batch_boundary fails about 3 times in 100 CI runs with "Got 14/15 of 16 merkle candidates". The cause is the client, not the test.

A merkle tree's candidate pools (up to 16) are queried at the same time, and each asks up to 32 peers. So one tree sends about 500 candidate requests at once and gets about 500 answers back in one burst. On a loaded host part of that burst is lost:

• saorsa-core queues inbound messages on 8 dispatch shards chosen by the sender's IP, 32 messages each, and drops a message when its shard is full (V2-1900). Every node of the in-process testnet is on 127.0.0.1, so they all share one shard.
• Under CPU load, part of the burst is also lost below the dispatcher (V2-1901, not yet located).

Each lost answer costs its request the versioned-probe timeout plus a legacy retry. Pools then come up short of 16 candidates, and the payment fails or retries.

Change: one semaphore, shared by every pool of a payment, holds candidate requests to 32 in flight. The legacy retry is included. That's 35 lines in ant-core/src/data/client/merkle.rs, plus a changelog entry.

Why 32

Before choosing, I measured four variants. The full report is in Linear on V2-1893.

Variant	Lost answers per run, local testnet at 2 cores	Mainnet: median time added per tree
main (no cap)	356, 1094	1.5 s
16 slots, held through probe and retry	1, 0	7.5 s
16 slots, released on answer or after 2 s	12, 0	1.9 s
32 slots, held through probe and retry (this PR)	0, 0	2.1 s

• Mainnet figures are 10 trees per variant, with all pools full. Time added is tree time minus its slowest DHT lookup.
• Sharding saorsa-core by socket address instead (no client change) still lost 33 and 443 answers per run, so the upstream fix alone doesn't clear this.
• Alternative if needed: the 2-s release variant can't queue behind slow peers at all. It's the one to switch to if a payment ever meets more than 32 slow peers at once. On mainnet, out of about 5,050 requests per variant, 0–4 peers were silent and 4–12 were slow.

Linear issue

Closes V2-1893

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Proposed T1. Only the client's request pacing changes. Requests, answers and payments are the same messages as before.

Compatibility

• Wire: none (same requests, at most 32 in flight per payment)
• Storage: none
• API: none

Semver impact

• breaking
• feature
• fix

Test evidence

• Measurements, 2026-11-05 (dev2 plus mainnet, numbers in the table above): the runs used a build that set the cap and slot mode with environment variables, and its logic matches this commit.

  • Local test_merkle_payment_across_batch_boundary, 35 nodes. The 32-slot variant passed all 4 of its runs: 3 at 2 cores (2 with no silent nodes, 1 with 4 silent nodes) and 1 at 4 cores with 4 silent nodes (10 s quote timeout).
  • Mainnet: 10 trees with prepare_merkle_batch_external (no wallet, nothing paid), 16/16 pools on every tree.
  • At 1.5 cores every variant fails, main included. Those failures are dial errors and short lookups from a starved testnet, with no dispatcher drops, so 1.5 cores is not a target.

• This commit (26746e3), on dev2:

  • cargo +stable fmt --check passes.
  • cargo +stable clippy --all-targets --all-features -D warnings is clean.
  • ant-core merkle unit tests: 75 passed.
  • e2e_merkle at 4 cores: 8 of 8 passed (1917 s).
  • e2e_merkle at 2 cores: 7 of 7 passed before my 45-min timeout cut the run off. The eighth, test_merkle_payment_across_batch_boundary (the flaky one), then passed on its own at 2 cores (426 s, no short attempts).

• CI on 26746e3: everything is green, including E2E and Merkle E2E on ubuntu and macOS, except Security Audit, which fails with RUSTSEC-2026-0387 as on main (V2-1743; this PR touches no Cargo files).

• Main merged in (e818e8d, 2026-11-06), bringing in feat(core): type a wallet balance shortfall as Error::InsufficientFunds (#152) #224, ci(install-test): make the npm jobs test npm's ant, not Apache Ant's #228 and fix(core): don't mistake a stale daemon PID file for a running daemon (V2-1655) #235. The only conflict was CHANGELOG.md, and both Fixed entries are kept. On dev2 the merged tree passes fmt, strict clippy and the 75 merkle unit tests. The fix itself is unchanged since 26746e3.

New dependency

none (tokio::sync::Semaphore, already a dependency)

ADR

n/a

Mitigation / rollback

Revert this commit, or raise MAX_IN_FLIGHT_MERKLE_CANDIDATE_REQUESTS. Either way, payments go back to today's uncapped behaviour.

🤖 Generated with Claude Code

[MaidSafe-Hermes]

APPROVE. Both seats agree (two model families): 2/2 seats reviewed (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3). The shared semaphore limits all candidate request futures across the pools in one build_candidate_pools invocation; each owned permit spans the versioned attempt and any legacy retry and releases on completion/cancellation. No blocking correctness issue found. The scope is per invocation, not a client-wide aggregate cap, which matches the stated per-payment intent. The performance measurements are author-reported; CI passes the listed build/test checks. Security Audit remains a separate inherited RUSTSEC-2026-0387 merge gate (V2-1743), not a defect introduced by this PR.

Coverage: e818e8d.