Skip to content

core: retry REST OAuth2 token refresh after transient failure - #17768

Open
waterWang wants to merge 1 commit into
apache:mainfrom
waterWang:fix/17756-oauth2-refresh-retry
Open

core: retry REST OAuth2 token refresh after transient failure#17768
waterWang wants to merge 1 commit into
apache:mainfrom
waterWang:fix/17756-oauth2-refresh-retry

Conversation

@waterWang

Copy link
Copy Markdown

Fixes #17756

Problem

One failed background OAuth2 token refresh permanently stops refresh for that AuthSession.

scheduleTokenRefresh chains one-shot scheduled tasks: a successful AuthSession.refresh(client) returns the next expiration, which schedules the following task. When refresh fails (retries exhausted + credential fallback fails), refresh returns null, and the chain simply ends — no manager or request path re-arms it:

  • authenticate keeps attaching the stale token already stored in the session.
  • After the token expires, requests receive 401s for the remaining lifetime of the catalog object even after the token endpoint recovers.

Reproduced by the issue: a token endpoint down for ~8s at refresh time (6 Tasks attempts + 1 credential fallback, ~3.1s of sleep) permanently disabled refresh. StarRocks reports the same production failure (StarRocks/starrocks#76438).

Fix

Distinguish the three outcomes the scheduling chain needs:

  1. Refresh succeeded → schedule from the new expiration (unchanged).
  2. Refresh failed temporarily → schedule a bounded retry (60s) so a transient endpoint outage delays refresh instead of permanently disabling it.
  3. Session closed (stopRefreshing()/close(), i.e. keepRefreshed == false) → do not schedule again.
  • scheduleTokenRefresh: when refresh() returns null but session.config().keepRefreshed() is still true, reschedule with a fixed 60s bounded wait instead of dropping the chain.
  • fromAccessToken initial-refresh path: when the initial refresh of an already-expired token fails while the session stays active, schedule the same bounded retry instead of leaving expiresAtMillis null (which skipped scheduling entirely).

A fixed bounded retry avoids hammering an unavailable token endpoint while still allowing recovery, and it automatically stops for closed sessions (no unconditional null-rescheduling loop).

Tests

Three new tests in TestOAuth2Util:

  • failedRefreshSchedulesBoundedRetryWhileSessionStaysActive — a scheduled refresh that fails (token endpoint down) reschedules with a ~60s delay.
  • closedSessionDoesNotRescheduleAfterFailedRefresh — after close(), a failed refresh does not reschedule.
  • initialRefreshFailureSchedulesBoundedRetryWhenSessionStaysActive — an already-expired token whose initial refresh fails still schedules the bounded retry.

All 12 tests in TestOAuth2Util pass.

One failed background token refresh permanently stopped the refresh chain:
AuthSession.refresh returns null when retries and credential fallback fail,
and scheduleTokenRefresh only rescheduled when a non-null expiration was
returned. After the token expired, requests kept using the stale token and
received 401s for the remaining lifetime of the catalog, even after the
token endpoint recovered.

Fix: when refresh fails but the session is still active (keepRefreshed is
true), schedule a bounded retry (60s) instead of ending the chain. A closed
session (stopRefreshing/close) still stops scheduling. fromAccessToken also
schedules a bounded retry when the initial refresh of an expired token
fails while the session stays active.
@github-actions github-actions Bot added the core label Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REST OAuth2 session never recovers after a failed token refresh

1 participant