You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
During a `charon alpha edit replace-operator` ceremony, one operator's node completed the Pedersen reshare successfully but then crashed during the lock-hash partial signature exchange:
A single stalled stream write to one peer (likely a dying relay circuit) hit the send deadline and aborted the entire ceremony for everyone.
Root causes
No retries: all DKG transports (`parsigex` exchange, `bcast`, pedersen board, frost round 1) abort on the first failed send, even though ceremony receivers deduplicate messages and a retry would be safe.
No peer in the error: `p2p.Send`/`SendReceive` errors only name the protocol, so the unreachable peer could not be identified from logs.
Related observations from the incident (not addressed by the fix PR):
On all released versions (≤ v1.9.5), the other operators hung indefinitely on "Pedersen reshare completed" waiting for the crashed node's signatures — dkg: fail fast on peer restart or death during ceremonies #4616 (unreleased) adds the fail-fast timeout.
`ParSigEx` passes its send/receive options only to the receive-side handler, so the DKG's `p2p.WithSendTimeout(--timeout)` is silently dropped and sends run with the hardcoded 7s default deadline — which matches the incident timeline exactly.
Impact
Any transient p2p blip during a DKG or cluster mutation ceremony (dkg, add/remove/replace operator, reshare) fails the whole ceremony, requiring all operators to coordinate a rerun — and on current releases, leaves the remaining operators hung without an error.
No artifacts were written anywhere (writes happen only after all exchanges complete), so the incident was safe to rerun.
Problem
During a `charon alpha edit replace-operator` ceremony, one operator's node completed the Pedersen reshare successfully but then crashed during the lock-hash partial signature exchange:
A single stalled stream write to one peer (likely a dying relay circuit) hit the send deadline and aborted the entire ceremony for everyone.
Root causes
Related observations from the incident (not addressed by the fix PR):
Impact
Any transient p2p blip during a DKG or cluster mutation ceremony (dkg, add/remove/replace operator, reshare) fails the whole ceremony, requiring all operators to coordinate a rerun — and on current releases, leaves the remaining operators hung without an error.
No artifacts were written anywhere (writes happen only after all exchanges complete), so the incident was safe to rerun.