Problem
During a replace-operator ceremony (also applies to any pedersen reshare/DKG), a node that starts the protocol a few seconds later than its peers fails the ceremony:
07:31:21.901 ERRO pedersen Dropping deal bundle, context done {"from": "16Uiu..."}
07:31:23.144 ERRO pedersen Dropping deal bundle, context done {"from": "16Uiu..."}
07:31:23.340 ERRO pedersen Dropping deal bundle, context done {"from": "16Uiu..."}
07:31:23.932 INFO cmd Starting pedersen reshare...
07:31:33.944 WARN pedersen Sending DKG complaints, deals from these dealers failed verification {"dealer_indices": "[0 1 5]"}
07:31:53.937 ERRO pedersen [dkg-log Public polynomial missing - evicting dealer 5]
07:31:53.937 ERRO pedersen [dkg-log Public polynomial missing - evicting dealer 1]
07:31:53.937 ERRO pedersen [dkg-log Public polynomial missing - evicting dealer 0]
07:31:53.940 ERRO cmd Application failed to start: ... process-justifications: only 3/5 valid deals - dkg abort
Root cause
Peers ahead in the ceremony broadcast their deal bundles before this node's kyber protocol starts reading Board.IncomingDeal(). The p2p handler pushes into an unbuffered channel (dkg/pedersen/board.go), blocks until the request context expires (5s defaultRcvTimeout), and drops the bundle. The drop is permanent:
Sender.SendAsync is fire-and-forget, and the handler acks success even after dropping, so the sender never retransmits.- On
main, the dedup map records the bundle signature before delivery, so even a retransmit would be refused as a duplicate.
The missing deals then cascade deterministically: complaints against the affected dealers → their justifications cannot be verified without their public polynomials (which were in the dropped deal bundles) → dealers evicted → fewer valid deals than oldThreshold → dkg abort.
Affected versions
Verified present on main and v1.10.3 (the unbuffered-channel drop with success ack is identical; v1.10.3 predates the dedup map but the sender never retransmits anyway).
Fix
Buffer the deal/response/justification channels to cluster size so bundles arriving before the protocol starts reading are queued instead of dropped, and un-record dropped bundles from the dedup map so a redelivery is accepted.
Workaround
Retry the ceremony with all operators starting as close to simultaneously as possible, and ensure no node is resource-starved (the failing node was ~5-8s behind its peers).
Problem
During a replace-operator ceremony (also applies to any pedersen reshare/DKG), a node that starts the protocol a few seconds later than its peers fails the ceremony:
Root cause
Peers ahead in the ceremony broadcast their deal bundles before this node's kyber protocol starts reading
Board.IncomingDeal(). The p2p handler pushes into an unbuffered channel (dkg/pedersen/board.go), blocks until the request context expires (5sdefaultRcvTimeout), and drops the bundle. The drop is permanent:Sender.SendAsyncis fire-and-forget, and the handler acks success even after dropping, so the sender never retransmits.main, the dedup map records the bundle signature before delivery, so even a retransmit would be refused as a duplicate.The missing deals then cascade deterministically: complaints against the affected dealers → their justifications cannot be verified without their public polynomials (which were in the dropped deal bundles) → dealers evicted → fewer valid deals than
oldThreshold→dkg abort.Affected versions
Verified present on
mainandv1.10.3(the unbuffered-channel drop with success ack is identical; v1.10.3 predates the dedup map but the sender never retransmits anyway).Fix
Buffer the deal/response/justification channels to cluster size so bundles arriving before the protocol starts reading are queued instead of dropped, and un-record dropped bundles from the dedup map so a redelivery is accepted.
Workaround
Retry the ceremony with all operators starting as close to simultaneously as possible, and ensure no node is resource-starved (the failing node was ~5-8s behind its peers).