Skip to content

fix(gax): map REST fallback transport errors to gRPC status codes - #9346

Open
MarkDuckworth wants to merge 5 commits into
mainfrom
fix-gax-fallback-transport-error-codes
Open

MarkDuckworth wants to merge 5 commits into
mainfrom
fix-gax-fallback-transport-error-codes

Conversation

@MarkDuckworth

@MarkDuckworth MarkDuckworth commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR fixes a bug where errors from the fallback (REST) transport carry an HTTP status number or a Node system error string on err.code, instead of a numeric gRPC status code.

Existing retry logic in normalCalls/retries.ts expects gRPC status codes — it matches err.code against the numeric retryCodes built from each method's service config:

if (retry.retryCodes.indexOf(err!.code!) < 0) {
  // "...not classified as transient"

So for example, Firestore's Commit is configured with "retry_codes_name": "resource_exhausted_unavailable", which resolves to RESOURCE_EXHAUSTED and UNAVAILABLE[8, 14]. When the connection drops mid-call, the fallback transport reports err.code === 'ECONNRESET'; when the backend returns 503, it reports err.code === 503. Neither is in [8, 14], so both are classified as permanent and retried zero times — a single socket hang up fails the call. Client libraries branching on err.code see the same divergence.

Root cause

Two defects, both introduced when the transport moved to auth.fetch() in e7c686a (googleapis/gax-nodejs#1766):

  1. HTTP errors never reach the decoder. gaxios resolves only when a response passes validateStatus — which defaults to 2xx-only and throws otherwise — and gax never sets it. So the .then() handler only ever runs for 2xx responses, which makes the response.ok it forwards to the decoder always true. That leaves decodeResponse's if (!ok) branch — the code that builds a GoogleError via parseHttpError() — unreachable. Every HTTP error takes the .catch below instead.
  2. Connection errors are never classified. The terminal .catch forwards the raw GaxiosError straight to the gRPC-style callback.

That PR's scope was fixing silently-failing tests ("minimal change to gax/src/") and it shipped as a plain fix: with no discussion of error codes; deliberate changes in this area (googleapis/gax-nodejs#1736, googleapis/gax-nodejs#1633) were marked fix!.

Why CI stayed green: setMockFallbackResponse() in test/unit/utils.ts mocks AuthClient#request to resolve with a failed Response; real gaxios throws. The suite asserts code === 3 for a 400 and code === 7 for a 403 and passes, while production returns 400 and 403.

The fix

  • Pass validateStatus: status => status !== 401 && status !== 403 to auth.fetch(), so HTTP errors resolve and flow through the existing decodeResponseparseHttpError path. 401/403 keep rejecting so google-auth-library can refresh credentials and retry — oauth2client branches on a thrown 401/403.

  • Add _toGoogleError() for errors that still reject, preserving the original on cause:

    Source
    carries an HTTP status rpcCodeFromHttpStatusCode(status)
    name AbortError CANCELLED
    name TimeoutError DEADLINE_EXCEEDED
    any other failure before a response UNAVAILABLE

    The last row follows @grpc/grpc-js rather than a hand-written errno list. grpc-js defaults transport failures to UNAVAILABLE — write and stream errors in subchannel-call.ts, 'No connection established' in picker.ts, resolution failures in resolver-dns.ts — and inspects errno only to refine an HTTP/2 INTERNAL_ERROR, where it treats both ECONNRESET and ETIMEDOUT as UNAVAILABLE. Enumerating errnos here would classify anything left off the list as non-retryable, and such a list can only ever be incomplete. Note this means ETIMEDOUT is UNAVAILABLE, not DEADLINE_EXCEEDED; the latter is for an elapsed call deadline.

    Aborts and timeouts are matched on err.name, not err.code: AbortSignal.abort() and AbortSignal.timeout() raise a DOMException whose code is the numeric legacy value (20 and 23). This matches the existing err.name !== 'AbortError' checks in the same file. A string code is accepted too, since gaxios normalizes a DOMException's name onto code.

    An existing GoogleError, or a non-Error, passes through untouched. Applies to unary and server-streaming. Errors raised while decoding a response are handled nearer the decoder and do not reach this path.

No gaxios-level retryConfig is added, so retries stay governed by the per-method retry codes in each service config.

Behaviour change

err.code from the fallback transport becomes a numeric gRPC status where it was an HTTP number or system string. This restores the documented contract — GoogleError.code is typed as the gRPC Status enum, and the Java and Python REST transports normalize the same way — but it is observable for anyone who adapted to the current behaviour.

Errors raised by the fallback (REST) transport were forwarded to callers
untranslated, so they carried either a Node system error string (e.g.
`code: 'ECONNRESET'` for a socket hang up) or a raw HTTP status number
instead of a numeric gRPC status code.

Retry logic in `normalCalls/retries.ts` matches `err.code` against the
numeric retry codes from the service config, so such errors never matched
and were treated as permanent: a single transient socket hang up failed the
call with zero retries. Client libraries that branch on `err.code` were
similarly affected.

Two things changed:

- `validateStatus` is now passed to `auth.fetch()` so that HTTP error
  responses resolve and flow through the existing `decodeResponse` path,
  which produces a `GoogleError` via `GoogleError.parseHttpError()`. 401 and
  403 continue to reject so that the auth client can refresh credentials.
  Previously nothing set `validateStatus`, gaxios rejected on every non-2xx
  response, and the `!response.ok` branch was unreachable.
- Errors that still reject (connection failures, timeouts, aborts) are
  translated by a new `_toGoogleError()` helper into a `GoogleError` with a
  numeric `code`, preserving the original error on `cause`.

No gaxios-level `retryConfig` is introduced, so retries remain governed by
the per-method retry codes in each service config.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces transport error translation for the fallback service stub, mapping underlying fetch and connection errors to numeric gRPC status codes. It also adds comprehensive unit tests to verify transport parity and error mapping. Feedback on the changes highlights a critical issue where checking fetchError.code === 'AbortError' will fail to match aborted requests in production due to differences in how various fetch implementations represent abort errors, and suggests checking err.name or gaxios-abort instead.

Comment thread core/packages/gax/src/fallbackServiceStub.ts Outdated
`_toGoogleError()` matched `'AbortError'` and `'TimeoutError'` against
`err.code`, but no fetch implementation reports them that way. Native fetch
rejects with a `DOMException` whose `code` is the numeric DOMException value
(20 for abort, 23 for timeout) and whose `name` carries the string; node-fetch
sets `name` and no `code`. Both therefore fell through to `UNKNOWN`, so a
cancelled call did not surface as `CANCELLED` and a timeout did not surface as
`DEADLINE_EXCEEDED`.

Match on `err.name` instead, consistent with how the rest of this file detects
cancellation, while still accepting a string `code` because gaxios matches
errors in that form.

The existing tests passed only because they constructed the error by hand with
`{code: 'AbortError'}`. They now use the `DOMException` that fetch actually
throws, with the string-`code` form kept as a separate case.
The suite was named for a guarantee it did not provide: it never exercised
the gRPC transport, so it compared the fallback transport against a
hardcoded table rather than against gRPC. Its closing test pinned that
table to rpcCodeFromHttpStatusCode, the same function the transport calls,
so it could only fail if one of the two were edited in isolation.

Five of its seven cases also duplicated test/unit/status.ts, which already
covers the HTTP-to-gRPC map directly.

What the suite did uniquely cover is the resolve path: every test in
'transport error translation' uses setMockFallbackError, so none of them
put a non-2xx Response through validateStatus and decodeResponse. Two
tests are kept for that, folded into the existing suite. One asserts that
the code denotes the canonical status the server named; the other that a
body without a status field still derives its code from the HTTP status.

Both still fail against the pre-fix transport.
@MarkDuckworth
MarkDuckworth removed the request for review from feywind September 15, 2026 22:19
The previous commit enumerated system error codes and mapped anything it
did not recognize to UNKNOWN. That table was invented rather than derived,
and it diverged from the transport it is meant to stand in for.

@grpc/grpc-js does not enumerate errnos. It defaults transport failures to
UNAVAILABLE (subchannel-call.js, the write and stream error paths; picker.js
for 'No connection established'; resolver-dns.js for resolution failures)
and inspects errno only to refine an HTTP/2 INTERNAL_ERROR, where it treats
ECONNRESET and ETIMEDOUT as UNAVAILABLE.

Two consequences of the table were wrong:

  - ETIMEDOUT became DEADLINE_EXCEEDED, where gRPC reports UNAVAILABLE.
    DEADLINE_EXCEEDED is for an elapsed call deadline, not a socket timeout.
    Since Firestore's Commit retries only RESOURCE_EXHAUSTED and UNAVAILABLE,
    a socket timeout would still not have been retried.

  - Any errno absent from the list became UNKNOWN and so non-retryable,
    and such a list can only ever be incomplete.

Cancellation and an elapsed deadline stay separate, since gRPC distinguishes
them. Everything else that rejects before a response is produced is now
UNAVAILABLE. Decode errors are handled nearer the decoder and do not reach
this path, so they are not affected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants