Skip to content

fix(github-app): fail over to another app on rate limit, attribute retry-path calls correctly - #5407

Open
atsikham wants to merge 1 commit into
github-aws-runners:mainfrom
atsikham:fix/github-app-reactive-failover
Open

fix(github-app): fail over to another app on rate limit, attribute retry-path calls correctly#5407
atsikham wants to merge 1 commit into
github-aws-runners:mainfrom
atsikham:fix/github-app-reactive-failover

Conversation

@atsikham

@atsikham atsikham commented Sep 10, 2026

Copy link
Copy Markdown

Fixes #5409

Summary

Two related bugs in how the code picks a GitHub App, found while looking into repeated rate-limit stalls when more than one GitHub App is configured:

  1. No failover when a rate limit hits. The code already picks the App with the most budget left before starting a request. But once it picks an App, it keeps using that same App for the whole request — if that App hits a rate limit partway through, the code just waits out the retry-after time, even if another configured App still has budget left. This gets worse when several invocations run at the same time and all pick the same "best" App, because the rate-limit info is only updated after a request finishes or fails.
  2. Retry-path calls always counted against App 0. getOctokit() picks a GitHub App internally, but it only returned the Octokit client — not which App it picked. The retry Lambda called isJobQueued() without telling it which App was used, so metricGitHubAppRateLimit() always counted the call against App 0, no matter which App actually made the call. This breaks the per-App rate-limit numbers that the App-picking logic itself depends on, for every call that goes through the retry path.

Changes

  • The rate-limit handling code (onRateLimit/onSecondaryRateLimit) now checks if another configured App has room left when a rate limit hits. If so, it stops waiting and switches right away instead of sleeping.
  • getOctokitWithFailover() retries the call using a newly picked, different App when that happens, and returns both the client and which App it used.
  • With 3 or more configured Apps, the failover logic remembers every App it has already tried in the current attempt (not just the last one it tried), up to a small limit as a safety net. So if a second App also turns out to be exhausted, it still tries a third instead of giving up.
  • getOctokit() now returns { client, appIndex } instead of just the client. job-retry.ts uses getOctokitWithFailover() for its job-status check, so the rate-limit numbers are always counted against the App that actually made the call.
  • scale-up.ts's own job-status check now works the same way. This was left out at first (only job-retry.ts had it), but scale-up.ts is actually the more important path — it runs for every job, not just for retries. Each group of messages now keeps track of its own current App/client and switches on its own the first time its check hits a rate limit. The switched-to client is then reused for the rest of that group's messages, and for the registration-token/JIT-config calls that come after (so those at least start out using a working App, even though they don't get their own retry — see the note below).

What this does not cover

The registration-token and JIT-config calls inside createStartRunnerConfig (the calls that use up the most rate-limit budget, based on the original incident) don't get their own retry in this PR. They go through ScaleUpComputeProvider, which is a public interface other code can plug into (ghClient: Octokit is passed in as a plain client). Making that support failover would mean either changing that public interface, or making the Octokit client itself silently retry with a different App using octokit.hook.error('request', ...), so the caller never notices. The second option is worth trying later, but it needs its own careful design and testing, so it isn't done here.

Test plan

  • yarn test — the full control-plane test suite passes (382 tests). This includes tests for: picking an App up front and switching after a rate limit, remembering multiple already-tried Apps, a 3-App failover in sequence, the safety limit on retries, counting rate-limit numbers against the right App in the retry path, and scale-up.ts's own failover (single message, reusing the client across a group, no failover when the error isn't a rate limit, no failover when no other App has room, and the safety limit on retries).
  • yarn lint / yarn format
  • tsc --noEmit

@atsikham
atsikham requested a review from a team as a code owner September 10, 2026 22:30
@atsikham
atsikham force-pushed the fix/github-app-reactive-failover branch 2 times, most recently from 1393209 to c833f68 Compare September 12, 2026 11:06
…try-path calls correctly

Two related defects in GitHub App selection:

1. App selection already picks the app with the most budget up front, but
   once an app is selected it stays pinned for the whole request: if it
   hits a rate limit mid-request, the code sleeps out the retry-after
   window even when another configured app still has budget. Concurrent
   invocations that independently pick the same "best" app compound this,
   since the in-memory rate-limit state only updates after a request
   completes or fails.

2. getOctokit() selected a GitHub App internally but only returned the
   Octokit client, not which App was chosen. The retry Lambda called
   isJobQueued() without an app index, so metricGitHubAppRateLimit()
   always attributed the call to app 0 regardless of which app actually
   handled it, corrupting the per-app rate-limit metrics that app
   selection itself relies on for every request going through the retry
   path.

The throttle callbacks now check whether another configured app has
headroom when a rate limit hits; if so, they stop retrying immediately
instead of sleeping, and getOctokitWithFailover() retries the call
against a freshly selected, different app, returning both the client and
the app index it used. With 3+ configured apps, the failover loop tracks
and excludes every app already tried in the current sequence (not just
the most recent one), bounded by a small attempt cap as a safety net, so
a second exhausted app doesn't mean giving up before trying a third.

job-retry.ts uses this for its job-status check. scale-up.ts's own
job-status check gets the same treatment: each message group tracks its
own current app/client and fails over independently the first time its
check actually hits a rate limit, reusing the failed-over client for the
rest of that group's messages and for the registration-token/JIT-config
calls that follow. Those registration-token/JIT-config calls themselves
don't get retried on a rate limit yet - they go through a plugin
interface (ScaleUpComputeProvider) that would need its own design work
to make failover-aware without a breaking change; noted as a follow-up.
@atsikham
atsikham force-pushed the fix/github-app-reactive-failover branch from c833f68 to 4727dfd Compare September 12, 2026 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GitHub App rate limit: no failover to another configured app, and retry-path metrics misattributed

1 participant