fix: retry rate limits and transient errors on Google read requests

[boss477]

What changed

• Added automatic retries with exponential backoff and Retry-After header parsing for idempotent GET read requests in GoogleClient.
• Read requests now retry up to MAX_READ_RETRIES = 2 (up to 3 total attempts) on HTTP 429, 500, 502, 503, 504, Google 403 rateLimitExceeded / userRateLimitExceeded, and transient network failures.
• Default retry backoff now starts at 1000ms and adds random jitter so parallel Gmail/Calendar reads do not retry in lockstep.
• Retry-After still wins when Google sends it, capped at 30 seconds.
• Malformed or oversized error JSON is not retried; the HTTP rejection remains visible.
• Successful read responses with malformed/oversized JSON are not retried.
• Preserved strict write safety invariants: mutating requests (POST, PATCH, DELETE) are never retried and immediately throw OutcomeUnknownError on uncertain network or server errors.
• Merged current main and preserved the newer charset, attachment filename, calendar zone, and calendar range fixes.

Verification

• git diff --check
• node --import tsx/esm --test tests/google.test.ts (46 passed)

[kvnloo]

The read/write split here looks good. I think the retryable-read predicate may be a little too narrow, though.

Right now idempotent GETs retry only 429 and 503. Google's guidance for transient server failures also covers the usual retryable server-error cases such as 500, 502, and 504, while writes correctly remain non-retryable here because their outcome may be uncertain.

Would it be cleaner to encode that as one explicit read-only retry predicate rather than special-casing 503?

For example, an explicit set such as 429 | 500 | 502 | 503 | 504 keeps the policy auditable without accidentally retrying definite client failures. Then a regression for one additional transient 5xx (say 502) can pin the same backoff and bounded-attempt behavior.

That keeps the important invariant simple: retry eligibility is determined by operation safety first, then transient response classification.

AI-use note: I used an AI assistant to help inspect the retry boundary and draft this review; I verified the cited behavior against the current PR diff and existing Google write-safety tests before posting.

[boss477]

Thanks for the thoughtful review and suggestion, @kvnloo!

I've pushed commit \42755f5\ addressing this feedback:

1. Explicit Retryable-Read Predicate: Extracted and exported \RETRYABLE_READ_STATUS_CODES\ (\Set([429, 500, 502, 503, 504])) and \isRetryableReadStatus(status: number)\ in [\packages/integrations/src/google.ts](https://github.com/CopilotKit/openmuse/blob/fix/google-rate-limit-retry/packages/integrations/src/google.ts) to auditably capture Google's guidance for transient server failures and rate limits.
2. Operation Safety First: Retries strictly apply only to idempotent GET reads (!write && isRetryableReadStatus(response.status)). All mutations (\POST, \PATCH, \DELETE) continue to fail immediately with \OutcomeUnknownError\ on any 5xx/408/network failure without retrying.
3. Test Coverage: Added test cases in [\ ests/google.test.ts](https://github.com/CopilotKit/openmuse/blob/fix/google-rate-limit-retry/tests/google.test.ts) covering:
   • \isRetryableReadStatus\ predicate validation across transient codes and client rejection codes.
   • Idempotent GET retry recovery on transient \502 Bad Gateway.
   • Bounded attempts and backoff progression up to \MAX_READ_RETRIES\ on transient \504 Gateway Timeout.
   • Verification that write mutations across all transient 5xx codes (\500, \502, \503, \504) remain strictly single-attempt without retrying.

[davidmckayv]

Needs changes before merge:

• Google's guidance for Gmail and Calendar is to wait at least one second before the first retry, with exponential backoff. Set DEFAULT_RETRY_DELAY_MS to 1000 and add random jitter. listMail fetches 30 messages in parallel, so without jitter a 429 burst retries in lockstep.
• Also retry 403 when the error reason is rateLimitExceeded or userRateLimitExceeded, which both docs list as retryable.
• Do not retry when readJson fails. An oversized or malformed body fails the same way again.
• Revert the charset regex change (["'] to ['"]). It is unrelated.
• The description says 429 and 503. The code retries 429, 500, 502, 503 and 504. Update it.

Refs: https://developers.google.com/workspace/gmail/api/guides/handle-errors and https://developers.google.com/workspace/calendar/api/guides/errors

[jerelvelarde]

Value: bounded retries improve Google read reliability. Template/security: read retries remain bounded and writes retain single-attempt OutcomeUnknownError semantics; 33 Google tests passed, no actionable security defect found. Please integrate current main before approval: packages/integrations/src/google.ts and tests/google.test.ts conflict with the merged charset, filename, calendar-zone and calendar-range fixes. Preserve those behaviors and rerun combined regressions. Also update the description's retry-status list to include the implemented 500/502/504 statuses.

[boss477]

Updated in 1f5c3c1:

• merged current main and resolved the conflicts in packages/integrations/src/google.ts and tests/google.test.ts
• changed default read backoff to 1000ms plus jitter
• added retry support for Google 403 rateLimitExceeded / userRateLimitExceeded
• stopped retrying malformed/oversized error JSON and successful malformed read responses
• kept upstream charset, attachment filename, calendar-zone, and calendar-range fixes
• updated the PR description retry-status list

Verification: git diff --check and node --import tsx/esm --test tests/google.test.ts (46 passed).