Skip to content

Centralize retry/backoff configuration on backon; inject retries where components give up #618

Description

@emlautarom1

Summary

backon is already the workspace retry engine and app/src/retry.rs (the Charon app/retry port) is built on it — but the workspace has drifted into at least 5 distinct backoff parameter sets and 4 drive mechanisms:

  • retry.rs defaults (250ms/12s/1.6) — zero call sites (feat(app): wrap duty callbacks with the retry executor #534 tracks wiring it into the duty callbacks).
  • core::expbackoff fast() (100ms/5s) and default() (1s/120s) — used by scheduler/sse/bootnode, sometimes driven manually via .next() + expect.
  • p2p/src/relay/dial.rs#L104-L126 — a hand-written duplicate of expbackoff::default() (its doc even cites the same Charon config), reimplemented because p2p needs a pollable Duration rather than an async wrapper.
  • quic_upgrade.rs — a third scheme in units of minutes (1→512, doubling).
  • eth1wrap — Alloy's RetryBackoffLayer::new(10, 1000, 100), a fourth policy in a foreign library's units.
  • Fixed-delay loops in dkg/sync (250ms), cli test/peers (for attempt in 0..5, 5s intervals). Same family: dkg busy-polls node signatures on a 100ms ticker, re-locking and cloning the accumulated slot vector every tick (nodesigs.rs#L144-L164), and exchanger.rs#L561 polls with a bare 100ms sleep — both want a Notify/watch signal instead of a poll.

Meanwhile, components that should retry just give up and wait for the next tick: scheduler resolve_duties errors are logged "(retrying next slot)" — a beacon-node blip loses a slot's duty resolution; bcast/recast retries next epoch; the wire-layer store/broadcast sinks log and swallow errors (wire.rs#L739-L746 and siblings) — precisely the points Charon wraps in async-retry.

Proposed change

  • One retry module (the existing app::retry + core::expbackoff, merged or clearly layered) exposing the named policies (fast, default, plus a pollable-Duration helper for poll-based behaviours so relay/dial.rs can delete its copy).
  • Replace the manual .next()/hand-rolled loops with backon's Retryable or the shared pollable helper; express eth1wrap's policy in the same config vocabulary.
  • Inject retries at the give-up sites above (scheduler duty resolution, recast, wire sinks), with per-duty deadlines from the existing DeadlineCalculator plumbing. feat(app): wrap duty callbacks with the retry executor #534 covers the five Charon duty-callback wrap points; this issue covers the config unification and the remaining sites.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestrustPull requests that update rust code

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions