tailcat: add --auto-region to find a server that moved DERP region - #65
Open
gmkbenjamin wants to merge 4 commits into
Open
tailcat: add --auto-region to find a server that moved DERP region#65gmkbenjamin wants to merge 4 commits into
gmkbenjamin wants to merge 4 commits into
Conversation
gmkbenjamin
force-pushed
the
auto-region
branch
from
September 2, 2026 10:55
c30c298 to
e8c13b3
Compare
Member
|
This is an area I care about fixing but I'm still thinking about how to do it best (#7) so I probably won't be merging this too quickly until I've had a moment to give it some proper though. |
gmkbenjamin
force-pushed
the
auto-region
branch
from
September 3, 2026 09:53
e8c13b3 to
e0522fd
Compare
A server keyed with "genkey --region=auto" re-picks its region at every startup, so a token published earlier can name one it has since left and the client just times out. Only the region moves; the node and disco public keys in the token still identify the server. Add DiscoverRegion, which keeps those keys and asks candidate regions whether the server is there, and Client.AutoRegion to use it. A meow is a raw DERP packet, so each probe is just a derphttp client. It has to use the caller's node key, since a server started with --allow ignores any other as silently as an absent server would, and the disco key derived from it, since onMeow keeps the one it first sees. Probes run before the client builds its stack, so they never share a relay with magicsock. The named region is tried alone first, then again with the rest if it stays quiet, so a slow answer isn't read as a move. Updates tailscale#7 Signed-off-by: Ben B <gmkbenjamin@hotmail.com>
Set Client.AutoRegion from a root flag, so every subcommand inherits it and newClient is the only place that reads it. ssh and cp exec the system client with a ProxyCommand that re-runs tailcat, so the flag has to reach that child too; sshProxyCommand takes an options struct rather than a sixth positional parameter. The corrected address goes to stderr, since in pipe mode stdout is the connection itself. --verbose names each region as it is tried. Updates tailscale#7 Signed-off-by: Ben B <gmkbenjamin@hotmail.com>
Replace the TODO about clients coping with a changing DERP map. --fixed-region is still the answer for anything long-lived; searching is a rescue. The issue stays open for multi-region servers and lat/long in tokens. Updates tailscale#7 Signed-off-by: Ben B <gmkbenjamin@hotmail.com>
maxProbeRegions capped how many regions a search covered, not how many it probed at once. The DERP map is expected to grow to around thirty regions, and at that size the cap becomes the failure: the search probes the eight lowest region IDs, never looks at the rest, and reports a server that moved to one of them as gone. Bound the concurrency instead. A pool probes maxProbeConcurrency regions at a time and works through the whole map. That needs a per-probe deadline, which probeOne never had, so a probe now yields its slot after probeSlotTimeout -- but only while regions are still queued, leaving a map small enough for one wave to behave as it did before. To keep a large sweep cheap, each probe also sends a sealed disco ping alongside its meow. derpserver answers a disco packet addressed to a node key it holds no connection for with PeerGone/NotHere, and ignores non-disco packets, which a meow is; that turns an empty region into a round trip rather than a timeout. Nothing replies to the ping, since an unknown peer's disco is dropped, so only a meowed counts as the server answering. A relay answers for the instant it is asked, so NotHere ends a probe only after probeNotHereGrace, and the search takes two looks over one budget: a fast sweep, then a slower one that ignores the relay. The usual reason a search is running at all is that the server has just restarted, so writing one off mid-connect is the mistake to avoid. The sweep runs outward from the region the token names, by the lat/long the DERP map already publishes, and where it found the server is kept in a RegionCache so a moved server costs one search and not one per connection. The cached region is probed alongside the one the token names, never instead of it. For the same reason probeSettle is no longer a flat 250ms: an answer from elsewhere is held until every named region has finished probing, so a token whose own region still works is never rewritten. Updates tailscale#7. Signed-off-by: Ben B <gmkbenjamin@hotmail.com>
gmkbenjamin
force-pushed
the
auto-region
branch
from
September 3, 2026 10:49
e0522fd to
d44adcc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A stop-gap, not the fix. The real fix is server-side: a saved key
should record a concrete region and bind to it at every startup, so a
published token stays true. That needs the server to change, and does
nothing for a token already published or a server already running.
This is entirely client-side, and works against what is deployed today.
Backward compatible with deployed servers
Not just "servers need no change" — this branch changes no server code
at all, so a running server cannot tell the difference.
tailcat.gois insideClient.Server,locoBackend,onMeowandadvertiseEndpointsare untouched.wire.goanddisco.goare untouched: the token layout and the meowpackets are byte-identical, and a corrected token is the same short
RegionID form the server already emits.
one — same meow, same node key.
NotHereout of the relay,is handled by magicsock, not tailcat (
OnDERPRecvsees only non-discopackets). An unknown peer's disco is dropped, so it can't get past
--allow.TestDiscoverRegionAllowliststill comes back empty against a probersending that ping, and
TestClientAutoRegioncompletes a real tunnelsession against a server that received every new packet. Since no server
code changed, those tests are the compatibility test.
The one exception is pre-existing and explicit: a token minted before
servers carried a separate disco key can't be searched, and says so.
What it does
A server keyed with
genkey --region=autore-picks its region at everystartup, so a published token can name one it has since left; the client
then just times out. The keys in the token still identify the server, so
--auto-regionkeeps them, asks candidate regions whether the server isthere, connects, and prints the corrected token on stderr.
The search covers every region in the map, however large it grows; only
concurrency is bounded, at eight at a time. It runs outward from the
region the token names, since a server that re-picks by latency usually
lands near where it was, and asks each relay outright whether it has the
server, which is a round trip rather than a timeout. Where it found the
server is cached in
$XDG_CACHE_HOME/tailcat/regions.json, so a movedserver costs one search and not one per
sshorcp.What it does not fix
A published token is still wrong until someone stores the corrected one;
only clients carrying this change benefit; a server on a relay outside
the map stays unreachable. The region remains a mutable fact inside an
immutable address — the actual defect, still open.
One consequence worth naming: the relay now tells a token-holder which
region has the server, where an unauthorized holder previously saw the
same silence either way. Not a new capability — any DERP client with the
node key could always ask — but tailcat now asks by default under the
flag.
Testing
region_test.gocovers a moved server, an unmoved one, an embeddedtoken, an allowlisted server, a multi-homed server, the not-found error,
the region cache, full coverage of a thirty-region map, ordering, and
the search budget.
cmd/tailcat/autoregion_e2e_test.godrives the realbinary with a negative control and asserts the notice never reaches
stdout. Clean under
-race.Rebased past "harden validation of arguments passed to ssh/scp child
processes":
sshProxyCommandkeeps its hardened args-slice form andquoting, with the flags a ProxyCommand carries into the child bundled in
a
proxyOptsstruct so--auto-regioncan join them.Updates #7 — the client-search part; multi-region servers and lat/long
in tokens are untouched.