Skip to content

[Plugin] Fix intermittent re-auth from cross-process refresh-token rotation - #44

Open
pragati-agrawal-glean wants to merge 13 commits into
mainfrom
pragati/fix-plugin-token-rotation-reauth
Open

[Plugin] Fix intermittent re-auth from cross-process refresh-token rotation#44
pragati-agrawal-glean wants to merge 13 commits into
mainfrom
pragati/fix-plugin-token-rotation-reauth

Conversation

@pragati-agrawal-glean

@pragati-agrawal-glean pragati-agrawal-glean commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Problem

The plugin can intermittently ask the user to authenticate again — find_skills or run_tool returns [SETUP_REQUIRED] — even though another live plugin process has already refreshed the shared OAuth grant.

Root cause: cross-process refresh-token rotation

Each MCP session runs its own plugin process, but those processes share one credential file. The OAuth server rotates refresh tokens on every refresh and invalidates the presented refresh token. A sibling process can therefore retain a revoked refresh token in memory:

process A: refreshes R0 → receives A1/R1 → persists A1/R1
process B: still holds R0 → refreshes → invalid_grant or invalid_request

Without coordination, the losing process can clear the shared store and surface [SETUP_REQUIRED], potentially taking down the process that already has the valid rotated grant.

Fix

  1. Treat the credential file as the source of truth. tokens() reloads credentials from disk on every access through syncTokensFromDisk(). Correctness no longer depends on file mtime, and there is no mtime-based synchronization or environment override.
  2. Wait before clearing tokens. invalidateCredentials("tokens") waits for a sibling's changed access token for a fixed two-second grace period, polling every 100 ms. If the sibling grant appears, it is adopted; only an unchanged store is cleared.
  3. Retry connect-level refresh failures. createRemoteClient() observes whether a newer access token appeared on disk and retries the connection once. It also handles raw structured OAuth errors with errorCode invalid_request or invalid_grant, rather than matching human-readable error messages.
  4. Keep retries bounded and conservative. A retry requires a changed token and is limited to one attempt, so unrelated OAuth errors and repeated failures are not hidden or retried indefinitely.
  5. Preserve atomic, private credential writes. Credential updates continue to use temp-file-plus-rename writes, with chmodSync(tmpPath, FILE_MODE) applied before the rename so temporary files are 0600 as well.

The registered OAuth client is not replaced during a sibling retry; the new MCP connection reuses the same provider and the newly persisted grant. DCR client reuse remains covered separately by #45.

Verification

Against the non-production SST endpoint:

https://salessavvy-test-be.glean.com/mcp/gateway/proxy

with client_api.oauth.accessToken.defaultExpiryMinutes=2:

  • Two separate Claude Code processes were launched with --strict-mcp-config.
  • Both processes used the candidate plugin MCP server and one shared test credential file.
  • A newly issued access token reported expires_in=120 and the credential file remained mode 0600.
  • After access-token expiry, both real MCP calls succeeded without [SETUP_REQUIRED].
  • Local fingerprints confirmed that both the access token and rotated refresh token changed, while the registered client ID stayed stable.
  • No raw credentials are included here.

Files

  • src/auth-provider.ts: direct disk synchronization, fixed grace-window sibling adoption.
  • src/remote-client.ts: structured OAuth-error handling and bounded sibling retry.
  • src/token-store.ts: atomic credential persistence and temp-file permission hardening.
  • Corresponding auth-provider, remote-client, and token-store tests.

— sent via Glean Desktop

Comment thread src/auth-provider.ts Outdated
Comment thread src/auth-provider.ts Outdated
pragati-agrawal-glean added a commit to gleanwork/agent-plugins that referenced this pull request Jul 30, 2026
…en rotation

Port of gleanwork/glean-plugins-vnext#44 (squashed; full history and E2E
evidence there).

Each host session runs its own plugin process sharing one credentials
file. The Glean OAuth server rotates refresh tokens on every refresh with
no grace period, so when one session refreshes, every other session's
in-memory copy is revoked; their next refresh gets invalid_grant, the SDK
wipes the SHARED store, and the user sees [SETUP_REQUIRED] — plus every
other live session dies with them.

Fixes (E2E-verified on an experimental pod against real prod /oauth —
bug reproduced on demand with the old build, silent recovery in both
race shapes with this change):

- tokens()/syncTokensFromDisk: mtime-guarded re-read of the shared store
  so a sibling's rotated grant is picked up before the SDK refreshes.
- invalidateCredentials('tokens'): adopt a newer on-disk token instead of
  wiping — with a grace-window poll (GLEAN_ROTATION_GRACE_MS, 2s) because
  the loser's invalid_grant usually lands milliseconds before the
  winner's write.
- Connect-level sibling-refresh retry: concurrent refreshes of the same
  grant make fosite fail the loser with invalid_request (not
  invalid_grant — observed live), which the SDK rethrows raw; recognize
  refresh-shaped failures, wait out the grace window, retry once.
- saveCredentials: temp-file + rename so concurrent writers can't leave
  a torn store that parses as wiped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pragati-agrawal-glean

Copy link
Copy Markdown
Contributor Author

Before changes:
Screenshot 2026-08-03 at 1 10 55 AM

After changes:
Screenshot 2026-08-03 at 1 12 43 AM

pragati-agrawal-glean and others added 6 commits August 13, 2026 11:28
…tation

MCP servers are spawned per session, and the OAuth provider reads
credentials from disk once at startup, then serves tokens from that
in-memory snapshot. With Ory single-use refresh-token rotation, when one
process refreshes it persists a new refresh token and invalidates the old
one that every other live process still holds in memory. The next process
to hit a 401 refreshes with its now-dead token -> invalid_grant -> full
re-auth -> [SETUP_REQUIRED]. This is the intermittent "why is it asking me
to auth again", not the (intentional, unchanged) 7-day access-token TTL.

Two fixes, both independent of the PLUGIN_DATA_DIR store split:

1. tokens() re-reads the credentials file when its mtime advances, so a
   process picks up a sibling's freshly-rotated grant before the SDK's
   auth flow reads tokens and attempts a refresh. mtime-guarded so the
   steady state is a single stat(). Conservative on removal: a missing
   file or a tokens-less rewrite does not evict the in-memory token.

2. createRemoteClient retries connect once when, after an auth failure, a
   newer access token has appeared on disk (sibling refresh) -- turning
   the rotation race into a silent reconnect instead of a re-auth. Bounded
   to a single retry.

Tests: provider adopts a sibling's newer token, keeps its token when the
file vanishes, ignores a tokens-less rewrite; credentialsMtimeMs probe;
connect retry fires only when the on-disk token changed.

Follow-ups (not in this PR): collapse the PLUGIN_DATA_DIR / ~/.glean store
split to one canonical path so surfaces share one DCR client; reuse the
DCR client across re-auths to stop orphaned-token pile-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… invalid_grant

The SDK's auth() calls invalidateCredentials("tokens") when a refresh returns
invalid_grant, which wrote {tokens: undefined} to the SHARED credentials file.
But the plugin's OAuth server (legacy /oauth, fosite) rotates refresh tokens
with NO grace period (RevokeRefreshTokenMaybeGracePeriod -> immediate delete),
so invalid_grant is exactly what a sibling's rotation looks like: the sibling
already minted a fresh grant and persisted it, and we only failed because we
refreshed with the now-revoked old token.

Blindly clearing then (a) forced a needless re-auth and (b) clobbered the fresh
token every other session on that store depends on -- one stale session poisoned
the well for all of them. syncFromDisk alone couldn't cover this: the wipe runs
inside connect() (SDK auth() catch), before the connect-retry could re-read.

Now invalidateCredentials("tokens") first checks whether disk holds a token
newer than the one we failed with (mtime-guarded, access_token differs). If so
it adopts that token and keeps it on disk instead of clearing; the SDK's own
post-invalidation retry (authInternal -> refreshAuthorization) then refreshes
with the fresh token and succeeds -- no re-auth, no poisoning. A genuine
invalidation (nothing newer on disk) still clears as before.

Tests: adopt-newer-token-instead-of-wipe (store preserved); clear-when-nothing-
newer. Full suite green, typecheck clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cut the comments added for the cross-process rotation fix down to the
reason/use-case, dropping restatements of what the code already shows.
No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… retry, atomic writes

E2E-verified on an experimental pod against the real prod /oauth (3-min
token-age gate; two plugin sessions sharing one store):

- OLD build: the session losing a refresh race gets 400, wipes the shared
  store, and surfaces [SETUP_REQUIRED] — reproduced on demand.
- With these changes: the loser recovers silently in both race shapes.

Three additions on top of the existing rails:

1. invalidateCredentials('tokens') now polls the store briefly (2s,
   GLEAN_ROTATION_GRACE_MS) before the destructive clear — the loser's
   invalid_grant usually arrives milliseconds BEFORE the winner's write
   lands, and clearing immediately poisons the shared store for every
   session. Skipped when no refresh token was held (no race possible).

2. Connect-level sibling-refresh retry: when two sessions refresh the same
   grant simultaneously, fosite fails the loser with invalid_request (NOT
   invalid_grant — observed live), which the SDK rethrows raw without
   touching invalidateCredentials. createRemoteClient now recognizes
   refresh-shaped failures, waits out the same grace window for the
   sibling's token, and retries once.

3. saveCredentials writes temp-file + rename so concurrent sibling writers
   can never leave a torn store that parses as wiped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pragati-agrawal-glean
pragati-agrawal-glean force-pushed the pragati/fix-plugin-token-rotation-reauth branch from 65c90a9 to 5d53b29 Compare August 13, 2026 05:59
Comment thread src/token-store.ts
Comment thread src/auth-provider.ts Outdated
Comment thread src/auth-provider.ts
Comment thread src/auth-provider.ts Outdated
Comment thread src/auth-provider.ts Outdated
Comment thread src/auth-provider.ts
Comment thread src/auth-provider.ts
Comment thread src/auth-provider.ts Outdated
@pragati-agrawal-glean
pragati-agrawal-glean marked this pull request as draft August 31, 2026 15:42
@pragati-agrawal-glean

Copy link
Copy Markdown
Contributor Author

Simplified the implementation - removed the modified time based logic and added logic to always read the disk stored tokens before using them to ensure a process always has the latest tokens (This assumes new tokens will be written to disk within 2 seconds and the disk write will execute successfully)

@pragati-agrawal-glean
pragati-agrawal-glean marked this pull request as ready for review September 1, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants