fix(gax): map REST fallback transport errors to gRPC status codes - #9346
Open
MarkDuckworth wants to merge 5 commits into
Open
MarkDuckworth wants to merge 5 commits into
MarkDuckworth wants to merge 5 commits into
Conversation
Errors raised by the fallback (REST) transport were forwarded to callers untranslated, so they carried either a Node system error string (e.g. `code: 'ECONNRESET'` for a socket hang up) or a raw HTTP status number instead of a numeric gRPC status code. Retry logic in `normalCalls/retries.ts` matches `err.code` against the numeric retry codes from the service config, so such errors never matched and were treated as permanent: a single transient socket hang up failed the call with zero retries. Client libraries that branch on `err.code` were similarly affected. Two things changed: - `validateStatus` is now passed to `auth.fetch()` so that HTTP error responses resolve and flow through the existing `decodeResponse` path, which produces a `GoogleError` via `GoogleError.parseHttpError()`. 401 and 403 continue to reject so that the auth client can refresh credentials. Previously nothing set `validateStatus`, gaxios rejected on every non-2xx response, and the `!response.ok` branch was unreachable. - Errors that still reject (connection failures, timeouts, aborts) are translated by a new `_toGoogleError()` helper into a `GoogleError` with a numeric `code`, preserving the original error on `cause`. No gaxios-level `retryConfig` is introduced, so retries remain governed by the per-method retry codes in each service config.
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces transport error translation for the fallback service stub, mapping underlying fetch and connection errors to numeric gRPC status codes. It also adds comprehensive unit tests to verify transport parity and error mapping. Feedback on the changes highlights a critical issue where checking fetchError.code === 'AbortError' will fail to match aborted requests in production due to differences in how various fetch implementations represent abort errors, and suggests checking err.name or gaxios-abort instead.
`_toGoogleError()` matched `'AbortError'` and `'TimeoutError'` against
`err.code`, but no fetch implementation reports them that way. Native fetch
rejects with a `DOMException` whose `code` is the numeric DOMException value
(20 for abort, 23 for timeout) and whose `name` carries the string; node-fetch
sets `name` and no `code`. Both therefore fell through to `UNKNOWN`, so a
cancelled call did not surface as `CANCELLED` and a timeout did not surface as
`DEADLINE_EXCEEDED`.
Match on `err.name` instead, consistent with how the rest of this file detects
cancellation, while still accepting a string `code` because gaxios matches
errors in that form.
The existing tests passed only because they constructed the error by hand with
`{code: 'AbortError'}`. They now use the `DOMException` that fetch actually
throws, with the string-`code` form kept as a separate case.
The suite was named for a guarantee it did not provide: it never exercised the gRPC transport, so it compared the fallback transport against a hardcoded table rather than against gRPC. Its closing test pinned that table to rpcCodeFromHttpStatusCode, the same function the transport calls, so it could only fail if one of the two were edited in isolation. Five of its seven cases also duplicated test/unit/status.ts, which already covers the HTTP-to-gRPC map directly. What the suite did uniquely cover is the resolve path: every test in 'transport error translation' uses setMockFallbackError, so none of them put a non-2xx Response through validateStatus and decodeResponse. Two tests are kept for that, folded into the existing suite. One asserts that the code denotes the canonical status the server named; the other that a body without a status field still derives its code from the HTTP status. Both still fail against the pre-fix transport.
The previous commit enumerated system error codes and mapped anything it
did not recognize to UNKNOWN. That table was invented rather than derived,
and it diverged from the transport it is meant to stand in for.
@grpc/grpc-js does not enumerate errnos. It defaults transport failures to
UNAVAILABLE (subchannel-call.js, the write and stream error paths; picker.js
for 'No connection established'; resolver-dns.js for resolution failures)
and inspects errno only to refine an HTTP/2 INTERNAL_ERROR, where it treats
ECONNRESET and ETIMEDOUT as UNAVAILABLE.
Two consequences of the table were wrong:
- ETIMEDOUT became DEADLINE_EXCEEDED, where gRPC reports UNAVAILABLE.
DEADLINE_EXCEEDED is for an elapsed call deadline, not a socket timeout.
Since Firestore's Commit retries only RESOURCE_EXHAUSTED and UNAVAILABLE,
a socket timeout would still not have been retried.
- Any errno absent from the list became UNKNOWN and so non-retryable,
and such a list can only ever be incomplete.
Cancellation and an elapsed deadline stay separate, since gRPC distinguishes
them. Everything else that rejects before a response is produced is now
UNAVAILABLE. Decode errors are handled nearer the decoder and do not reach
this path, so they are not affected.
danieljbruce
approved these changes
Sep 16, 2026
shivanee-p
approved these changes
Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes a bug where errors from the fallback (REST) transport carry an HTTP status number or a Node system error string on
err.code, instead of a numeric gRPC status code.Existing retry logic in
normalCalls/retries.tsexpects gRPC status codes — it matcheserr.codeagainst the numericretryCodesbuilt from each method's service config:So for example, Firestore's
Commitis configured with"retry_codes_name": "resource_exhausted_unavailable", which resolves toRESOURCE_EXHAUSTEDandUNAVAILABLE—[8, 14]. When the connection drops mid-call, the fallback transport reportserr.code === 'ECONNRESET'; when the backend returns 503, it reportserr.code === 503. Neither is in[8, 14], so both are classified as permanent and retried zero times — a single socket hang up fails the call. Client libraries branching onerr.codesee the same divergence.Root cause
Two defects, both introduced when the transport moved to
auth.fetch()ine7c686a(googleapis/gax-nodejs#1766):validateStatus— which defaults to 2xx-only and throws otherwise — and gax never sets it. So the.then()handler only ever runs for 2xx responses, which makes theresponse.okit forwards to the decoder alwaystrue. That leavesdecodeResponse'sif (!ok)branch — the code that builds aGoogleErrorviaparseHttpError()— unreachable. Every HTTP error takes the.catchbelow instead..catchforwards the rawGaxiosErrorstraight to the gRPC-style callback.That PR's scope was fixing silently-failing tests ("minimal change to
gax/src/") and it shipped as a plainfix:with no discussion of error codes; deliberate changes in this area (googleapis/gax-nodejs#1736, googleapis/gax-nodejs#1633) were markedfix!.Why CI stayed green:
setMockFallbackResponse()intest/unit/utils.tsmocksAuthClient#requestto resolve with a failedResponse; real gaxios throws. The suite assertscode === 3for a 400 andcode === 7for a 403 and passes, while production returns400and403.The fix
Pass
validateStatus: status => status !== 401 && status !== 403toauth.fetch(), so HTTP errors resolve and flow through the existingdecodeResponse→parseHttpErrorpath. 401/403 keep rejecting sogoogle-auth-librarycan refresh credentials and retry —oauth2clientbranches on a thrown 401/403.Add
_toGoogleError()for errors that still reject, preserving the original oncause:rpcCodeFromHttpStatusCode(status)AbortErrorCANCELLEDTimeoutErrorDEADLINE_EXCEEDEDUNAVAILABLEThe last row follows
@grpc/grpc-jsrather than a hand-written errno list. grpc-js defaults transport failures toUNAVAILABLE— write and stream errors insubchannel-call.ts,'No connection established'inpicker.ts, resolution failures inresolver-dns.ts— and inspects errno only to refine an HTTP/2INTERNAL_ERROR, where it treats bothECONNRESETandETIMEDOUTasUNAVAILABLE. Enumerating errnos here would classify anything left off the list as non-retryable, and such a list can only ever be incomplete. Note this meansETIMEDOUTisUNAVAILABLE, notDEADLINE_EXCEEDED; the latter is for an elapsed call deadline.Aborts and timeouts are matched on
err.name, noterr.code:AbortSignal.abort()andAbortSignal.timeout()raise aDOMExceptionwhosecodeis the numeric legacy value (20and23). This matches the existingerr.name !== 'AbortError'checks in the same file. A stringcodeis accepted too, since gaxios normalizes aDOMException's name ontocode.An existing
GoogleError, or a non-Error, passes through untouched. Applies to unary and server-streaming. Errors raised while decoding a response are handled nearer the decoder and do not reach this path.No gaxios-level
retryConfigis added, so retries stay governed by the per-method retry codes in each service config.Behaviour change
err.codefrom the fallback transport becomes a numeric gRPC status where it was an HTTP number or system string. This restores the documented contract —GoogleError.codeis typed as the gRPCStatusenum, and the Java and Python REST transports normalize the same way — but it is observable for anyone who adapted to the current behaviour.