daemon: path watchdog resets on rekey-giveup — faster T2 desync recovery#415
Merged
Merged
Conversation
… T2 recovery) The 2026-07-20 director probe (T2) documented the worst reliability failure: after a session-key desync, peers exchange 'encrypted packet but no key' + 'rekey retransmit gave up after maxRekeyAttempts' in a loop, and specialist replies arrive undecryptable and are dropped — nothing surfaces to the caller (a silent 'no reply'). Plain rekey-retransmit can't recover it: it resends the key exchange to the same stale cached endpoint. The per-peer path watchdog CAN (resetPeerPath re-resolves a fresh endpoint + re-punches), but only fired after the ~85s inbound-silence probe budget. Wire the rekey-gave-up state (keyexchange.PeerInRekeyGaveUp) as a fast reset trigger in pathWatchPeer: a gave-up peer is an unambiguous dead session, so reset it on the current ~10s tick instead of waiting out the probe budget. Still honours the post-reset cooldown so a genuinely offline peer can't cause a reset storm. Cuts T2 recovery from ~85s to one tick. Test: TestPathWatchRekeyGaveUpResetsImmediately (resets on gave-up before the probe budget; cooldown suppresses a second reset). pkg/daemon + keyexchange green under -race. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TeoSlayer
added a commit
that referenced
this pull request
Jul 22, 2026
…fied live) (#416) #415 tried to recover the T2 desync by polling the rekey-gave-up state from the path watchdog. A LIVE test (2026-07-22) proved it never fired: a desynced peer has no usable key, so it is not Ready, so ReadyPeerIDs() — the set the watchdog scans — excludes it entirely. 0 firings across dozens of real give-ups. (The unit test passed only because it injected a Ready peer + gave-up state, a combination that can't occur.) Correct fix: event-driven. Add an onGaveUp hook fired from the keyexchange retransmit loop at the exact give-up site, wired through the tunnel to a daemon handler (onRekeyGaveUp) that runs resetPeerPath — the only thing that recovers a desync, because it re-resolves a FRESH endpoint and re-punches (plain rekey-retransmit just resends to the same stale cached endpoint forever). Per-peer 30s cooldown prevents a persistently-offline peer from storming resolves; reset runs async (registry I/O) off the rekey loop goroutine. Reverts #415's ineffective path-watchdog poll. Verified LIVE on a real NAT'd laptop: give-up -> full path reset (fresh resolve + punch + PILA) in ~1.2s, reproduced across peers 17461 and 16392, vs the prior indefinite silent-drop. This is the 'no reply on whatever' fix. Tests: TestOnRekeyGaveUpResetsWithCooldown (hook resets + cooldown suppresses a second give-up). pkg/daemon + keyexchange green under -race. Co-authored-by: Teodor Calin <teodor@vulturelabs.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 2026-07-20 director probe's worst finding (T2): after a session-key desync, replies arrive undecryptable and are silently dropped ('no reply'), and plain rekey-retransmit can't recover (resends to a stale endpoint). The path watchdog's resetPeerPath can (fresh resolve + punch) but only fired after ~85s of inbound silence. This wires the unambiguous rekey-gave-up signal as a fast reset trigger — recovery drops from ~85s to one ~10s tick, with the reset cooldown preventing storms. Directly targets the intermittent no-reply that makes MOM/agents feel like 'Pilot stopped working'. 🤖 Generated with Claude Code