Skip to content

tailcat: add --auto-region to find a server that moved DERP region - #65

Open
gmkbenjamin wants to merge 4 commits into
tailscale:mainfrom
gmkbenjamin:auto-region
Open

tailcat: add --auto-region to find a server that moved DERP region#65
gmkbenjamin wants to merge 4 commits into
tailscale:mainfrom
gmkbenjamin:auto-region

Conversation

@gmkbenjamin

@gmkbenjamin gmkbenjamin commented Sep 2, 2026

Copy link
Copy Markdown

A stop-gap, not the fix. The real fix is server-side: a saved key
should record a concrete region and bind to it at every startup, so a
published token stays true. That needs the server to change, and does
nothing for a token already published or a server already running.

This is entirely client-side, and works against what is deployed today.

Backward compatible with deployed servers

Not just "servers need no change" — this branch changes no server code
at all
, so a running server cannot tell the difference.

  • Every hunk in tailcat.go is inside Client. Server, locoBackend,
    onMeow and advertiseEndpoints are untouched.
  • wire.go and disco.go are untouched: the token layout and the meow
    packets are byte-identical, and a corrected token is the same short
    RegionID form the server already emits.
  • A probe is indistinguishable from a client connecting, because it is
    one — same meow, same node key.
  • The one new packet, a disco ping that draws NotHere out of the relay,
    is handled by magicsock, not tailcat (OnDERPRecv sees only non-disco
    packets). An unknown peer's disco is dropped, so it can't get past
    --allow.

TestDiscoverRegionAllowlist still comes back empty against a prober
sending that ping, and TestClientAutoRegion completes a real tunnel
session against a server that received every new packet. Since no server
code changed, those tests are the compatibility test.

The one exception is pre-existing and explicit: a token minted before
servers carried a separate disco key can't be searched, and says so.

What it does

A server keyed with genkey --region=auto re-picks its region at every
startup, so a published token can name one it has since left; the client
then just times out. The keys in the token still identify the server, so
--auto-region keeps them, asks candidate regions whether the server is
there, connects, and prints the corrected token on stderr.

tailcat ping --auto-region --verbose tcOLD...
tailcat ssh  --auto-region tcOLD...

The search covers every region in the map, however large it grows; only
concurrency is bounded, at eight at a time. It runs outward from the
region the token names, since a server that re-picks by latency usually
lands near where it was, and asks each relay outright whether it has the
server, which is a round trip rather than a timeout. Where it found the
server is cached in $XDG_CACHE_HOME/tailcat/regions.json, so a moved
server costs one search and not one per ssh or cp.

What it does not fix

A published token is still wrong until someone stores the corrected one;
only clients carrying this change benefit; a server on a relay outside
the map stays unreachable. The region remains a mutable fact inside an
immutable address — the actual defect, still open.

One consequence worth naming: the relay now tells a token-holder which
region has the server, where an unauthorized holder previously saw the
same silence either way. Not a new capability — any DERP client with the
node key could always ask — but tailcat now asks by default under the
flag.

Testing

region_test.go covers a moved server, an unmoved one, an embedded
token, an allowlisted server, a multi-homed server, the not-found error,
the region cache, full coverage of a thirty-region map, ordering, and
the search budget. cmd/tailcat/autoregion_e2e_test.go drives the real
binary with a negative control and asserts the notice never reaches
stdout. Clean under -race.

Rebased past "harden validation of arguments passed to ssh/scp child
processes": sshProxyCommand keeps its hardened args-slice form and
quoting, with the flags a ProxyCommand carries into the child bundled in
a proxyOpts struct so --auto-region can join them.

Updates #7 — the client-search part; multi-region servers and lat/long
in tokens are untouched.

@bradfitz

bradfitz commented Sep 2, 2026

Copy link
Copy Markdown
Member

This is an area I care about fixing but I'm still thinking about how to do it best (#7) so I probably won't be merging this too quickly until I've had a moment to give it some proper though.

A server keyed with "genkey --region=auto" re-picks its region at
every startup, so a token published earlier can name one it has since
left and the client just times out. Only the region moves; the node
and disco public keys in the token still identify the server.

Add DiscoverRegion, which keeps those keys and asks candidate regions
whether the server is there, and Client.AutoRegion to use it. A meow
is a raw DERP packet, so each probe is just a derphttp client. It has
to use the caller's node key, since a server started with --allow
ignores any other as silently as an absent server would, and the
disco key derived from it, since onMeow keeps the one it first sees.

Probes run before the client builds its stack, so they never share a
relay with magicsock. The named region is tried alone first, then
again with the rest if it stays quiet, so a slow answer isn't read as
a move.

Updates tailscale#7

Signed-off-by: Ben B <gmkbenjamin@hotmail.com>
Set Client.AutoRegion from a root flag, so every subcommand inherits
it and newClient is the only place that reads it. ssh and cp exec the
system client with a ProxyCommand that re-runs tailcat, so the flag
has to reach that child too; sshProxyCommand takes an options struct
rather than a sixth positional parameter.

The corrected address goes to stderr, since in pipe mode stdout is
the connection itself. --verbose names each region as it is tried.

Updates tailscale#7

Signed-off-by: Ben B <gmkbenjamin@hotmail.com>
Replace the TODO about clients coping with a changing DERP map.
--fixed-region is still the answer for anything long-lived; searching
is a rescue. The issue stays open for multi-region servers and
lat/long in tokens.

Updates tailscale#7

Signed-off-by: Ben B <gmkbenjamin@hotmail.com>
maxProbeRegions capped how many regions a search covered, not how many
it probed at once. The DERP map is expected to grow to around thirty
regions, and at that size the cap becomes the failure: the search probes
the eight lowest region IDs, never looks at the rest, and reports a
server that moved to one of them as gone.

Bound the concurrency instead. A pool probes maxProbeConcurrency regions
at a time and works through the whole map. That needs a per-probe
deadline, which probeOne never had, so a probe now yields its slot after
probeSlotTimeout -- but only while regions are still queued, leaving a
map small enough for one wave to behave as it did before.

To keep a large sweep cheap, each probe also sends a sealed disco ping
alongside its meow. derpserver answers a disco packet addressed to a
node key it holds no connection for with PeerGone/NotHere, and ignores
non-disco packets, which a meow is; that turns an empty region into a
round trip rather than a timeout. Nothing replies to the ping, since an
unknown peer's disco is dropped, so only a meowed counts as the server
answering. A relay answers for the instant it is asked, so NotHere ends
a probe only after probeNotHereGrace, and the search takes two looks
over one budget: a fast sweep, then a slower one that ignores the relay.
The usual reason a search is running at all is that the server has just
restarted, so writing one off mid-connect is the mistake to avoid.

The sweep runs outward from the region the token names, by the lat/long
the DERP map already publishes, and where it found the server is kept in
a RegionCache so a moved server costs one search and not one per
connection. The cached region is probed alongside the one the token
names, never instead of it. For the same reason probeSettle is no longer
a flat 250ms: an answer from elsewhere is held until every named region
has finished probing, so a token whose own region still works is never
rewritten.

Updates tailscale#7.

Signed-off-by: Ben B <gmkbenjamin@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants