fix(github-app): fail over to another app on rate limit, attribute retry-path calls correctly - #5407
Open
atsikham wants to merge 1 commit into
Open
Conversation
atsikham
force-pushed
the
fix/github-app-reactive-failover
branch
2 times, most recently
from
September 12, 2026 11:06
1393209 to
c833f68
Compare
…try-path calls correctly Two related defects in GitHub App selection: 1. App selection already picks the app with the most budget up front, but once an app is selected it stays pinned for the whole request: if it hits a rate limit mid-request, the code sleeps out the retry-after window even when another configured app still has budget. Concurrent invocations that independently pick the same "best" app compound this, since the in-memory rate-limit state only updates after a request completes or fails. 2. getOctokit() selected a GitHub App internally but only returned the Octokit client, not which App was chosen. The retry Lambda called isJobQueued() without an app index, so metricGitHubAppRateLimit() always attributed the call to app 0 regardless of which app actually handled it, corrupting the per-app rate-limit metrics that app selection itself relies on for every request going through the retry path. The throttle callbacks now check whether another configured app has headroom when a rate limit hits; if so, they stop retrying immediately instead of sleeping, and getOctokitWithFailover() retries the call against a freshly selected, different app, returning both the client and the app index it used. With 3+ configured apps, the failover loop tracks and excludes every app already tried in the current sequence (not just the most recent one), bounded by a small attempt cap as a safety net, so a second exhausted app doesn't mean giving up before trying a third. job-retry.ts uses this for its job-status check. scale-up.ts's own job-status check gets the same treatment: each message group tracks its own current app/client and fails over independently the first time its check actually hits a rate limit, reusing the failed-over client for the rest of that group's messages and for the registration-token/JIT-config calls that follow. Those registration-token/JIT-config calls themselves don't get retried on a rate limit yet - they go through a plugin interface (ScaleUpComputeProvider) that would need its own design work to make failover-aware without a breaking change; noted as a follow-up.
atsikham
force-pushed
the
fix/github-app-reactive-failover
branch
from
September 12, 2026 11:18
c833f68 to
4727dfd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5409
Summary
Two related bugs in how the code picks a GitHub App, found while looking into repeated rate-limit stalls when more than one GitHub App is configured:
retry-aftertime, even if another configured App still has budget left. This gets worse when several invocations run at the same time and all pick the same "best" App, because the rate-limit info is only updated after a request finishes or fails.getOctokit()picks a GitHub App internally, but it only returned the Octokit client — not which App it picked. The retry Lambda calledisJobQueued()without telling it which App was used, sometricGitHubAppRateLimit()always counted the call against App 0, no matter which App actually made the call. This breaks the per-App rate-limit numbers that the App-picking logic itself depends on, for every call that goes through the retry path.Changes
onRateLimit/onSecondaryRateLimit) now checks if another configured App has room left when a rate limit hits. If so, it stops waiting and switches right away instead of sleeping.getOctokitWithFailover()retries the call using a newly picked, different App when that happens, and returns both the client and which App it used.getOctokit()now returns{ client, appIndex }instead of just the client.job-retry.tsusesgetOctokitWithFailover()for its job-status check, so the rate-limit numbers are always counted against the App that actually made the call.scale-up.ts's own job-status check now works the same way. This was left out at first (onlyjob-retry.tshad it), butscale-up.tsis actually the more important path — it runs for every job, not just for retries. Each group of messages now keeps track of its own current App/client and switches on its own the first time its check hits a rate limit. The switched-to client is then reused for the rest of that group's messages, and for the registration-token/JIT-config calls that come after (so those at least start out using a working App, even though they don't get their own retry — see the note below).What this does not cover
The registration-token and JIT-config calls inside
createStartRunnerConfig(the calls that use up the most rate-limit budget, based on the original incident) don't get their own retry in this PR. They go throughScaleUpComputeProvider, which is a public interface other code can plug into (ghClient: Octokitis passed in as a plain client). Making that support failover would mean either changing that public interface, or making the Octokit client itself silently retry with a different App usingoctokit.hook.error('request', ...), so the caller never notices. The second option is worth trying later, but it needs its own careful design and testing, so it isn't done here.Test plan
yarn test— the full control-plane test suite passes (382 tests). This includes tests for: picking an App up front and switching after a rate limit, remembering multiple already-tried Apps, a 3-App failover in sequence, the safety limit on retries, counting rate-limit numbers against the right App in the retry path, andscale-up.ts's own failover (single message, reusing the client across a group, no failover when the error isn't a rate limit, no failover when no other App has room, and the safety limit on retries).yarn lint/yarn formattsc --noEmit