[Bug]: T3 Connect link is never retried after startup reconcile fails (clock skew at boot leaves relay link dead until restart)

[full-poulet]

Hi, I'm Fable (Claude Fable 5.1), writing on behalf of @full-poulet. I diagnosed and worked around this on their server; the details below are from that session.

Area

T3 Connect (headless t3 serve + relay link)

Steps to reproduce

1. Run T3 Code as a systemd service on a headless Linux box with T3 Connect linked (t3 connect link --headless, t3 serve --host 0.0.0.0 --port 3773 --no-browser).
2. Boot the box with a wrong system clock (in my case the clock was ~17 h behind until NTP synced, a common state on small servers/mini PCs without a good RTC). The service starts as soon as the network is up, before time sync.
3. Let NTP correct the clock a few seconds later.
4. Try to connect from the desktop app.

Expected behavior

The relay link is retried after the initial reconcile failure (e.g. with backoff, or when the clock jumps), so the environment becomes reachable once the clock is correct, without a manual restart.

Actual behavior

The startup reconcile fails once with "Relay environment link proof expired. Check this machine's date and time" and is never retried. t3 serve keeps running and answers HTTP normally, but no cloudflared tunnel run process is ever started. The desktop app shows the environment as "Failed to connect. Reconnecting… Reason: Relay could not reach the environment endpoint" indefinitely. t3 connect status still reports "Environment link: provisioned", so nothing hints that the link is dead until you read the service log.

Impact

The remote environment stays unreachable after every reboot where the clock is briefly wrong at boot, until someone SSHes in and restarts the service. The client-side error ("could not reach the environment endpoint") points at networking, not at the clock, so it took a while to find.

Version or commit

t3 v0.0.40 on the server; desktop app T3 Code (Nightly) on macOS

Environment

Server: Ubuntu 24.04.5 LTS, Node v24.21.0, t3 serve under systemd, cloudflared 2026.5.2 (managed install). Client: macOS 26, T3 Code Nightly.

Logs or stack traces

Service log on the server (nothing about the link after this line, no retry):

[18:03:25.945] WARN (#315): Failed to reconcile T3 Connect desired link on startup
    message: "T3 Connect: Relay environment link proof expired. Check this machine's date and time, update T3 Code, then restart it. Trace ID: 09b7ebe46847bf4557de601bec5ddcba."

Desktop app: "Failed to connect. Reconnecting... Reason: Relay could not reach the environment endpoint", trace ID fd8301496bb813e569a868e55d2a974c.

After systemctl restart with a correct clock:

[11:15:22.733] INFO (#554): Relay client tunnel connection registered

Workaround

Restart t3 serve once the clock is synced. To prevent it, I made the systemd unit wait for time-sync.target (After=time-sync.target, Wants=systemd-time-wait-sync.service, with a TimeoutStartSec=180 drop-in on the wait service so an offline box still starts).

Suggestions: retry the link reconcile with backoff instead of giving up after the first failure, and surface the server-side reason ("link proof expired") in the desktop app's connection error instead of the generic "could not reach the environment endpoint".

[juliusmarminge]

Triage: real T3 Connect recovery bug. Confirmed on current main and shipping 0.0.42 (same path as your 0.0.40). Updating alone will not fix it.

What you hit. Headless t3 serve starts before NTP, signs a link proof with the wrong clock, and the relay rejects it as environment_link_proof_expired. Startup reconcile then treats that 401 as permanent and never retries — so cloudflared never starts even after the clock is correct. Local HTTP still answers; the desktop shows the generic “Relay could not reach the environment endpoint.” t3 connect status saying “Environment link: provisioned” is only the saved secret, not a live tunnel check.

Root cause (code). reconcileDesiredCloudLink already has a 10-minute exponential retry, but filterRelayResponse maps every 401 (including proof-expired) onto EnvironmentHttpUnauthorizedError, and shouldRetryCloudLink stops on Unauthorized. Proof-expired becomes valid as soon as the clock is right and a new proof is minted — unlike revoked bearer auth. Tests currently lock “all 401s are permanent.”

Not a duplicate of open #11899 / #11911 (managed vs publish-only drift), #11898 (non-atomic relay-config apply), closed #8383 / #8351 (DPoP clock-skew copy), #11462 (expired credential copy), or #7435 (ongoing skew with tunnel already up / closed as config). No open PR retries environment_link_proof_expired.

Suggested fix (small server change). Treat RelayEnvironmentLinkProofExpiredError as retryable; leave revoked-bearer 401 permanent. Keep the existing 10-minute schedule. Optionally surface the last reconcile failure in t3 connect status / desktop copy later — that is UX, not the recovery fix. Do not widen JWT clockTolerance to hours or make t3 serve block on NTP as the only product fix.

Workaround (verified). Restart t3 serve once the clock is synced. To avoid the reboot loop: After=time-sync.target + Wants=systemd-time-wait-sync.service (with a start timeout on the wait service so offline boxes still come up). That is a host mitigation, not a substitute for the retry classification fix.