Unrecognised GitHub Enterprise host re-probes fj, tea and glab once per project on every sweep

[JorrinKievit]

What happened

Threads open slowly and the VCS/PR state in the sidebar takes 10–18s to settle. Not tied to a
particular update — it has always behaved this way on this machine. The desktop app intermittently
shows "Reconnecting" whenever anything else on the box uses CPU.

Every repository I work on is hosted on a GitHub Enterprise Cloud tenant with a data-residency
hostname of the form <tenant>.ghe.com. gh is authenticated against that host and works fine
from the terminal.

Diagnosis

A GHE host is classified unknown, can never be refined, and the failed refinement is retried on
every read without ever being cached. Verified against main @ ccf220b.

1. <tenant>.ghe.com is never detected as GitHub.

packages/shared/src/sourceControl.ts:183 matches GitHub as host === "github.com" or a github
DNS label. <tenant>.ghe.com has neither, so detectSourceControlProviderFromRemoteUrl returns
kind: "unknown".

2. Nothing can refine it. Of the five discovery specs, only GitLab
(GitLabSourceControlProvider.ts:107) and Forgejo (managed-cli) register a refineUnknownRemote.
The GitHub spec registers none. So there is no code path by which a GHE host can ever resolve to
github, no matter how many times it is probed.

3. The refinement runs uncached on every read. SourceControlProviderRegistry.ts:261 only
consults providerContextCache when context is undefined. Both hot callers pass context
explicitly, so both bypass it entirely:

• PullRequestService.ts:615 (refineUnknownProjectKinds) — no caching whatsoever
• server.ts:307 (identity resolver refine) — 1 min TTL

Negative results are discarded rather than cached: ForgejoSourceControlProvider.ts:150
(orElseSucceed(() => [])), SourceControlProviderDiscovery.ts:328 (orElseSucceed(() => null)),
and the registry cache uses Duration.zero on failure (SourceControlProviderRegistry.ts:258).

4. firstSuccessOf multiplies it by the number of projects on the host.
refineUnknownProjectKinds groups candidates by baseUrl and runs Effect.firstSuccessOf over
them. Every project on the same GHE host lands in one group, and because none can ever succeed, the
entire list is exhausted on every call. 11 of my 14 projects share one host.

Each candidate spawns fj, tea and glab. None are installed — these are pure failed spawns.

Measured: one PullRequestService.canonicalRef = 12 refinements → 24 listLogins → 20.8s.
PullRequestSyncReactor runs syncGroup at concurrency: 8, every 60s.

5. WSL2 makes each failed spawn ~100x more expensive. This is why it is so much worse here than
it would be on native Linux. The server's PATH contains 36 /mnt/c/... entries, so every ENOENT
lookup is a scan across the 9p filesystem. Measured on this machine, 20 spawns of a missing binary:
3.05s (152ms each) versus 0.043s (2.1ms each) with a stub resolving first on PATH.

The result is a sweep that cannot keep up with its own 60s schedule, and a server with no CPU
headroom — so any unrelated load (a test run) pushes the health endpoint past the desktop app's 10s
timeout and triggers "Reconnecting".

Steps to reproduce

1. Have gh authenticated against a GitHub Enterprise host whose hostname contains no github DNS
   label — e.g. <tenant>.ghe.com, or any GHES vanity domain.
2. Add several projects (I have 11) whose origin remote points at that host.
3. Do not install fj, tea or glab.
4. Open the app and watch server.trace.ndjson.

Every PullRequestSyncReactor sweep re-probes all three CLIs once per project, forever. The effect
scales with the number of projects sharing the host, and is far more visible on WSL2 than on native
Linux or macOS because of the PATH cost above.

Version

0.0.41-nightly.20260916.1795 (t3 triage context reports installed version 0.0.42)

Environment

WSL2 (Ubuntu, kernel 6.6.87.2-microsoft-standard-WSL2) on Windows, x64, 16 cores.
Node v26.8.2. Desktop app against a local server. gh 2.x authenticated to both github.com and
the enterprise host. fj, tea, glab, az not installed.

Evidence

225s window, 4 rotated trace files, 21,847 spans

  n     p50        max        span
  1629  335.0ms    2674.6ms   ForgejoCli.listLogins        (874 failures, ~50%)
  816   1175.8ms   4341.6ms   refineUnknownRemoteProvider
  815   658.4ms    2718.3ms   ForgejoSourceControlProvider.refineUnknownRemote
  1762  476.2ms    3641.2ms   processRunner.runProcessCore

spawn failures in window:  glab 881,  tea 873

slowest spans:
  119405ms  PullRequestSyncReactor.sweep         (on a 60s Schedule.spaced)
   31316ms  ws.rpc.vcs.refreshStatus
   30127ms  ws.rpc.subscribeVcsStatus
   23001ms  PullRequestSyncReactor.syncGroup     (x8 back-to-back, concurrency: 8)
   23001ms  PullRequestService.canonicalRef

one canonicalRef subtree (20.8s):
   12  refineUnknownRemoteProvider
   12  ForgejoSourceControlProvider.refineUnknownRemote
   12  ForgejoCli.readKeys
   24  ForgejoCli.listLogins
   24  processRunner.runProcessCore

glab spawn parentage (796 of 881):
   VcsProcess.runUnbounded < VcsProcess.run < refineUnknownRemoteProvider < PullRequestService.canonicalRef

Related issues

#5087 (GHE support, closed; its PR #6052 closed unmerged) and #5089 (still open) would both remove
the root cause. This issue is about the cost of the failure mode itself: even with GHE detection
shipped, any genuinely unrecognised host still re-probes three CLIs per project per sweep forever,
because the negative result is never cached and firstSuccessOf exhausts the whole candidate group.
#11220 is adjacent — idle sweep cadence and caches that expire before the next sweep — but its root
cause is scheduling, not unrefinable provider detection. #5932 is the change that made hostname
matching deliberately strict.

Fix applied or workaround

No fix for the detection itself; the playbook rules out patching the source. As a local mitigation,
shadowed the three never-installed CLIs so the lookups stop scanning the Windows PATH:

for c in tea fj glab; do
  printf '#!/bin/sh\nexit 1\n' > ~/.local/bin/$c
  chmod +x ~/.local/bin/$c
done

VcsProcess.ts:167 fails on non-zero exit when allowNonZeroExit is unset, so discovery still
reports both providers as missing and refinement still returns unknown — identical behaviour,
cheaper. Measured before/after:

ForgejoCli.listLogins p50   335ms  ->  187ms
canonicalRef p50           20.8s   ->  10.9s
syncGroup max              23.0s   ->  10.9s

Roughly 2x end-to-end — less than the isolated PATH benchmark suggests, because the spans also
include spawn overhead and 8-way contention. It is a mitigation, not a fix: the probe storm is
still there, just cheaper per probe.

Filed by

claude (opus-5) via t3 triage

[juliusmarminge]

Triage

Confirmed on current main (ccf220be2) and shipping 0.0.42. This is a real server perf bug, not a wash of #5087 / #5089 / #11220. Those would stop this host from entering the unknown bucket. This report is the cost of the bucket itself: an unrefinable host is re-probed on every read, and firstSuccessOf multiplies that by every project that shares it.

What the code does

<tenant>.ghe.com is classified unknown. isGitHubHost is host === "github.com" or a complete github DNS label (packages/shared/src/sourceControl.ts). That is #5932 / #6175. acme.ghe.com has neither, so detectSourceControlProviderFromRemoteUrl returns { kind: "unknown", baseUrl: "https://<tenant>.ghe.com" }. There is no ghe.com exception.

Nothing can refine it. Of the five discovery specs, only GitLab and Forgejo (managed fj/tea) register refineUnknownRemote. GitHub’s spec is version + gh auth status only — no remote hook. parseGitHubAuthStatus already walks every host in that JSON, including enterprise ones, and then findAuthenticatedGitHubAccount keeps a single account for the one github discovery row. The authenticated GHE host is sitting in the parse and is never asked “does this remote belong to you?”

Each refinement therefore always runs fj, tea, and glab against a host none of them will claim. GitHub’s gh is on PATH and already logged in to that host, and is not in this loop.

The result is discarded, then asked again:

• SourceControlProviderRegistry.resolveHandle only hits providerContextCache (5s, Duration.zero on failure) when context is omitted. Both hot callers pass context and skip it: PullRequestService.refineUnknownProjectKinds, and server.ts’s identity refine (which also forces kind: "unknown").
• Forgejo’s managed refiner orElseSucceeds missing CLIs to []. CLI specs orElseSucceed to null. A host-level “nobody claimed this” is indistinguishable from a one-off miss, and is not stored.
• RepositoryIdentityResolver does cache the identity for 1 minute. That is a different cache. refineUnknownProjectKinds never consults it.

firstSuccessOf then multiplies it. Candidates are grouped by baseUrl. Every unknown project on the GHE host lands in one group. When no candidate can succeed, the whole list runs. That path is load-bearing for broken checkouts — it does not want eleven healthy checkouts of an unclaimable host.

That is the 12-span canonicalRef. requireProject first refines listWorkspaceProjects({ projectId }) (one candidate). Unknown has no PR API, so it falls through to listWorkspaceProjects({ host }) and firstSuccessOfs the other projects on that host. 1 + N = N+1 refineUnknownRemoteProvider → 2(N+1) ForgejoCli.listLogins. Matches the trace.

PullRequestSyncReactor ticks Schedule.spaced("1 minute") at concurrency: 8. Open + unsettled and null snapshots always are due. Eight of those canonicalRefs at ~20s is enough to blow the 60s gap. The 119s sweep span is that backlog, not a separate scheduler bug.

WSL2 is the amplifier, not the defect. A missing fj/tea/glab walks every PATH entry; /mnt/c/... on 9p is ~100× a native ENOENT. Same storm on macOS/Linux, cheaper per spawn. Desktop “Reconnecting” under extra load is the event-loop losing a health probe, not a separate reconnect bug.

Related, not duplicates

Work	What it did	Why this still happens
Closed #5087 / closed unmerged #6052	GHE as a first-class connector	Never landed. Detection still ends at github.com / a github label.
Open #5089	.ghe.com + GitHub refineUnknownRemote via gh hosts; new github-enterprise kind	Would take this tenant out of unknown. XXL, unvouched, not the negative-cache / firstSuccessOf hole. Do not wait on it, and do not attribute Closes #12060.
Open #11220	Idle PR-discovery / settlement cadence; 60s caches that expire before the next spaced tick	Adjacent load. Different loops, different caches. A longer identity TTL does not stop refineUnknownProjectKinds.
Merged #6175 (closed #5932)	Complete DNS-label match so notgithub.example.com is not GitHub	Why acme.ghe.com is unknown. Keep the strict match; add an explicit GHE rule or a gh-host refine, do not revert substring matching.

No open PR caches a negative refinement or stops firstSuccessOf after a host-level miss.

Suggested fix

Keep this a small server bugfix on unknown-remote refinement. Do not land #5089 to close this, and do not rebuild provider detection.

1. Cache refinement by baseUrl (and host), including “still unknown”. One TTL for the whole group, minutes not 5s, on both success and “no spec claimed this.” resolveHandle({ context }) must read that map. Identity-resolver refine should too. Failures that are cwd-specific (broken checkout, spawn crash) stay uncached so the broken-then-healthy checkout test still walks to /healthy.
2. Stop exhausting the group on a host-level miss. After the first candidate returns kind: "unknown" because every spec returned null, the rest of that baseUrl group will too. firstSuccessOf only needs the next cwd when this checkout failed to produce an answer.
3. Do not do the host-wide refine after the project-scoped one already failed. requireProject’s listWorkspaceProjects({ host }) fallback is for “route this PR to another checkout on a supported host.” For unknown it is a second full probe storm that cannot create an API.
4. Optional, separate, small: classify *.ghe.com as GitHub (or add a GitHub refineUnknownRemote that claims a remote whose host is in gh auth status). That removes the GHE trigger. It does not replace (1)–(3) for GHES vanity domains and everything else that stays unknown.
5. Tests: N unknown projects on one *.ghe.com (or git.example.internal) with fj/tea/glab missing → one refine per host per window, not N; the “broken then healthy GitLab cwd” test still asks both; canonicalRef on an unknown project does not call listWorkspaceProjects({ host }) after the project-scoped refine already returned unknown.
6. Trace close: same multi-project GHE machine, one canonicalRef is one refineUnknownRemoteProvider (or one per host), PullRequestSyncReactor.sweep stays inside the 60s gap, sidebar VCS settles in hundreds of ms.

Do not treat PATH stubs, skipping the idle sweep, or “install glab” as the fix.

Workaround

The reporter’s ~/.local/bin/{tea,fj,glab} exit 1 stubs are the right local mitigation on WSL: discovery still reports missing, refinement still returns unknown, PATH no longer walks /mnt/c. Measured ~2× on that box; the storm is still there. No in-app way to mark a host as GitHub.

Classification: bug · accepted · medium (server source-control; uncached unknown-host refine × projects on the host; WSL PATH makes it user-visible; desktop reconnect is the sideshow)
Labels: add bug, accepted, via-triage
Discord tags: performance, linux