Skip to content

dkg: ceremony aborts on single transient p2p send failure #4685

Description

@KaloyanTanev

Problem

During a `charon alpha edit replace-operator` ceremony, one operator's node completed the Pedersen reshare successfully but then crashed during the lock-hash partial signature exchange:

ERRO cmd Application failed to start: run replace operator protocol: write message: i/o deadline reached {"protocol": "/charon/parsigex/2.0.0"}
    p2p/sender.go:361 .Send
    core/parsigex/parsigex.go:151 .Broadcast
    ...

A single stalled stream write to one peer (likely a dying relay circuit) hit the send deadline and aborted the entire ceremony for everyone.

Root causes

  1. No retries: all DKG transports (`parsigex` exchange, `bcast`, pedersen board, frost round 1) abort on the first failed send, even though ceremony receivers deduplicate messages and a retry would be safe.
  2. No peer in the error: `p2p.Send`/`SendReceive` errors only name the protocol, so the unreachable peer could not be identified from logs.

Related observations from the incident (not addressed by the fix PR):

  • On all released versions (≤ v1.9.5), the other operators hung indefinitely on "Pedersen reshare completed" waiting for the crashed node's signatures — dkg: fail fast on peer restart or death during ceremonies #4616 (unreleased) adds the fail-fast timeout.
  • `ParSigEx` passes its send/receive options only to the receive-side handler, so the DKG's `p2p.WithSendTimeout(--timeout)` is silently dropped and sends run with the hardcoded 7s default deadline — which matches the incident timeline exactly.

Impact

Any transient p2p blip during a DKG or cluster mutation ceremony (dkg, add/remove/replace operator, reshare) fails the whole ceremony, requiring all operators to coordinate a rerun — and on current releases, leaves the remaining operators hung without an error.

No artifacts were written anywhere (writes happen only after all exchanges complete), so the incident was safe to rerun.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingprotocolProtocol Team tickets

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions