[Plugin] Fix intermittent re-auth from cross-process refresh-token rotation - #44
Open
pragati-agrawal-glean wants to merge 13 commits into
Open
[Plugin] Fix intermittent re-auth from cross-process refresh-token rotation#44pragati-agrawal-glean wants to merge 13 commits into
pragati-agrawal-glean wants to merge 13 commits into
Conversation
pragati-agrawal-glean
marked this pull request as ready for review
July 24, 2026 06:46
pragati-agrawal-glean
requested review from
eshwar-sundar-glean,
garvit-scio,
mohit-gupta-glean and
swarup-padhi-glean
as code owners
July 24, 2026 06:46
pragati-agrawal-glean
added a commit
to gleanwork/agent-plugins
that referenced
this pull request
Jul 30, 2026
…en rotation Port of gleanwork/glean-plugins-vnext#44 (squashed; full history and E2E evidence there). Each host session runs its own plugin process sharing one credentials file. The Glean OAuth server rotates refresh tokens on every refresh with no grace period, so when one session refreshes, every other session's in-memory copy is revoked; their next refresh gets invalid_grant, the SDK wipes the SHARED store, and the user sees [SETUP_REQUIRED] — plus every other live session dies with them. Fixes (E2E-verified on an experimental pod against real prod /oauth — bug reproduced on demand with the old build, silent recovery in both race shapes with this change): - tokens()/syncTokensFromDisk: mtime-guarded re-read of the shared store so a sibling's rotated grant is picked up before the SDK refreshes. - invalidateCredentials('tokens'): adopt a newer on-disk token instead of wiping — with a grace-window poll (GLEAN_ROTATION_GRACE_MS, 2s) because the loser's invalid_grant usually lands milliseconds before the winner's write. - Connect-level sibling-refresh retry: concurrent refreshes of the same grant make fosite fail the loser with invalid_request (not invalid_grant — observed live), which the SDK rethrows raw; recognize refresh-shaped failures, wait out the grace window, retry once. - saveCredentials: temp-file + rename so concurrent writers can't leave a torn store that parses as wiped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
…tation MCP servers are spawned per session, and the OAuth provider reads credentials from disk once at startup, then serves tokens from that in-memory snapshot. With Ory single-use refresh-token rotation, when one process refreshes it persists a new refresh token and invalidates the old one that every other live process still holds in memory. The next process to hit a 401 refreshes with its now-dead token -> invalid_grant -> full re-auth -> [SETUP_REQUIRED]. This is the intermittent "why is it asking me to auth again", not the (intentional, unchanged) 7-day access-token TTL. Two fixes, both independent of the PLUGIN_DATA_DIR store split: 1. tokens() re-reads the credentials file when its mtime advances, so a process picks up a sibling's freshly-rotated grant before the SDK's auth flow reads tokens and attempts a refresh. mtime-guarded so the steady state is a single stat(). Conservative on removal: a missing file or a tokens-less rewrite does not evict the in-memory token. 2. createRemoteClient retries connect once when, after an auth failure, a newer access token has appeared on disk (sibling refresh) -- turning the rotation race into a silent reconnect instead of a re-auth. Bounded to a single retry. Tests: provider adopts a sibling's newer token, keeps its token when the file vanishes, ignores a tokens-less rewrite; credentialsMtimeMs probe; connect retry fires only when the on-disk token changed. Follow-ups (not in this PR): collapse the PLUGIN_DATA_DIR / ~/.glean store split to one canonical path so surfaces share one DCR client; reuse the DCR client across re-auths to stop orphaned-token pile-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… invalid_grant
The SDK's auth() calls invalidateCredentials("tokens") when a refresh returns
invalid_grant, which wrote {tokens: undefined} to the SHARED credentials file.
But the plugin's OAuth server (legacy /oauth, fosite) rotates refresh tokens
with NO grace period (RevokeRefreshTokenMaybeGracePeriod -> immediate delete),
so invalid_grant is exactly what a sibling's rotation looks like: the sibling
already minted a fresh grant and persisted it, and we only failed because we
refreshed with the now-revoked old token.
Blindly clearing then (a) forced a needless re-auth and (b) clobbered the fresh
token every other session on that store depends on -- one stale session poisoned
the well for all of them. syncFromDisk alone couldn't cover this: the wipe runs
inside connect() (SDK auth() catch), before the connect-retry could re-read.
Now invalidateCredentials("tokens") first checks whether disk holds a token
newer than the one we failed with (mtime-guarded, access_token differs). If so
it adopts that token and keeps it on disk instead of clearing; the SDK's own
post-invalidation retry (authInternal -> refreshAuthorization) then refreshes
with the fresh token and succeeds -- no re-auth, no poisoning. A genuine
invalidation (nothing newer on disk) still clears as before.
Tests: adopt-newer-token-instead-of-wipe (store preserved); clear-when-nothing-
newer. Full suite green, typecheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cut the comments added for the cross-process rotation fix down to the reason/use-case, dropping restatements of what the code already shows. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… retry, atomic writes
E2E-verified on an experimental pod against the real prod /oauth (3-min
token-age gate; two plugin sessions sharing one store):
- OLD build: the session losing a refresh race gets 400, wipes the shared
store, and surfaces [SETUP_REQUIRED] — reproduced on demand.
- With these changes: the loser recovers silently in both race shapes.
Three additions on top of the existing rails:
1. invalidateCredentials('tokens') now polls the store briefly (2s,
GLEAN_ROTATION_GRACE_MS) before the destructive clear — the loser's
invalid_grant usually arrives milliseconds BEFORE the winner's write
lands, and clearing immediately poisons the shared store for every
session. Skipped when no refresh token was held (no race possible).
2. Connect-level sibling-refresh retry: when two sessions refresh the same
grant simultaneously, fosite fails the loser with invalid_request (NOT
invalid_grant — observed live), which the SDK rethrows raw without
touching invalidateCredentials. createRemoteClient now recognizes
refresh-shaped failures, waits out the same grace window for the
sibling's token, and retries once.
3. saveCredentials writes temp-file + rename so concurrent sibling writers
can never leave a torn store that parses as wiped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pragati-agrawal-glean
force-pushed
the
pragati/fix-plugin-token-rotation-reauth
branch
from
August 13, 2026 05:59
65c90a9 to
5d53b29
Compare
pragati-agrawal-glean
marked this pull request as draft
August 31, 2026 15:42
Contributor
Author
|
Simplified the implementation - removed the modified time based logic and added logic to always read the disk stored tokens before using them to ensure a process always has the latest tokens (This assumes new tokens will be written to disk within 2 seconds and the disk write will execute successfully) |
pragati-agrawal-glean
marked this pull request as ready for review
September 1, 2026 12:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Problem
The plugin can intermittently ask the user to authenticate again —
find_skillsorrun_toolreturns[SETUP_REQUIRED]— even though another live plugin process has already refreshed the shared OAuth grant.Root cause: cross-process refresh-token rotation
Each MCP session runs its own plugin process, but those processes share one credential file. The OAuth server rotates refresh tokens on every refresh and invalidates the presented refresh token. A sibling process can therefore retain a revoked refresh token in memory:
Without coordination, the losing process can clear the shared store and surface
[SETUP_REQUIRED], potentially taking down the process that already has the valid rotated grant.Fix
tokens()reloads credentials from disk on every access throughsyncTokensFromDisk(). Correctness no longer depends on file mtime, and there is no mtime-based synchronization or environment override.invalidateCredentials("tokens")waits for a sibling's changed access token for a fixed two-second grace period, polling every 100 ms. If the sibling grant appears, it is adopted; only an unchanged store is cleared.createRemoteClient()observes whether a newer access token appeared on disk and retries the connection once. It also handles raw structured OAuth errors witherrorCodeinvalid_requestorinvalid_grant, rather than matching human-readable error messages.chmodSync(tmpPath, FILE_MODE)applied before the rename so temporary files are0600as well.The registered OAuth client is not replaced during a sibling retry; the new MCP connection reuses the same provider and the newly persisted grant. DCR client reuse remains covered separately by #45.
Verification
Against the non-production SST endpoint:
with
client_api.oauth.accessToken.defaultExpiryMinutes=2:--strict-mcp-config.expires_in=120and the credential file remained mode0600.[SETUP_REQUIRED].Files
src/auth-provider.ts: direct disk synchronization, fixed grace-window sibling adoption.src/remote-client.ts: structured OAuth-error handling and bounded sibling retry.src/token-store.ts: atomic credential persistence and temp-file permission hardening.— sent via Glean Desktop