fix(connectors): stop hydration after rate limits

[waleedlatif1]

Summary

• stop deferred connector hydration after a structured upstream rate limit so later batches do not amplify one provider throttle into per-document failures
• preserve the typed provider error and retry delay for the existing connector backoff scheduler while leaving ordinary document failures isolated
• use GitHub’s documented object media type for file metadata and a byte-bounded raw blob stream for large content, surfacing non-404 failures and oversized blobs explicitly
• add connector and sync-engine regressions for the exact rate-limit, large-file, absence, permission, and byte-cap behaviors

Testing

• bun run --cwd apps/sim test -- connectors/github/github.test.ts lib/knowledge/documents/utils.test.ts lib/knowledge/connectors/sync-engine.test.ts
• bun run --cwd apps/sim type-check
• bun run agent-stream-docs:generate
• bun run skills:sync
• bun run lint
• bun run apps/sim/scripts/check-block-registry.ts origin/staging
• bun run check:audits

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 29, 2026 3:22am

[greptile-apps]

Greptile Summary

The PR stops deferred connector hydration after a structured provider rate limit while preserving the provider error for backoff scheduling.

• Adds structured Google Drive rate-limit classification across the complete provider-reason set.
• Uses GitHub object metadata and byte-bounded raw blob streaming for large files.
• Adds regression coverage for rate limits, permissions, missing files, and oversized content.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/sim/lib/knowledge/connectors/sync-engine.ts	Detects structured throttling after an active hydration batch and propagates the typed error to the existing sync backoff path.
apps/sim/lib/knowledge/documents/utils.ts	Adds cause-chain-aware rate-limit detection based on normalized provider signals and structured HTTP evidence.
apps/sim/connectors/google-drive/google-drive-errors.ts	Classifies all normalized Google Drive reasons while retaining a bounded diagnostic reason list.
apps/sim/connectors/github/github.ts	Fetches object metadata and streams raw blobs through the shared byte limit, explicitly handling missing, forbidden, binary, and oversized files.
apps/sim/lib/knowledge/connectors/sync-engine.test.ts	Verifies that throttling stops later hydration batches and preserves the provider retry delay.
apps/sim/connectors/github/github.test.ts	Covers large-file hydration, absence, authorization failures, bodyless responses, and byte-cap handling.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Connector lists documents] --> B[Hydrate deferred batch]
  B --> C{Structured rate limit?}
  C -->|Yes| D[Stop later hydration batches]
  D --> E[Preserve provider error and retry delay]
  E --> F[Schedule connector backoff]
  C -->|No| G[Isolate ordinary document failures]
  G --> H[Process successful documents]

Loading

_{Reviews (3): Last reviewed commit: "fix(connectors): inspect all drive error..." | Re-trigger Greptile}

[cubic-dev-ai]

2 issues found across 6 files

Confidence score: 3/5

• In apps/sim/lib/knowledge/documents/utils.ts, structured Google Drive rateLimitExceeded and userRateLimitExceeded 403 responses without rate-limit headers are missed, leaving later hydration batches without the intended rate-limit handling; detect these typed provider errors explicitly.
• In apps/sim/connectors/github/github.ts, a bodyless 200 blob without trustworthy Content-Length can be recorded as a false size-limit skip rather than a hydration failure; reject unavailable bodies separately and classify size-limit skips only when the size is bounded.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/knowledge/documents/utils.ts">

<violation number="1" location="apps/sim/lib/knowledge/documents/utils.ts:245">
P2: When Google Drive returns a structured `rateLimitExceeded` or `userRateLimitExceeded` 403, it does not necessarily send rate-limit headers. This branch misses that typed provider error, so later hydration batches still amplify the throttle; normalize provider-specific rate-limit markers into a shared signal before requiring headers here.</violation>
</file>

<file name="apps/sim/connectors/github/github.ts">

<violation number="1" location="apps/sim/connectors/github/github.ts:171">
P2: When GitHub returns a bodyless 200 blob without a trustworthy `Content-Length`, this path records a false size-limit skip instead of a hydration failure. Reject unavailable bodies separately and classify only a bounded-read failure with observed bytes above `MAX_FILE_SIZE` as oversized.

(Based on your team's feedback about distinguishing oversized responses from unavailable bodies.)</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic

[cubic-dev-ai]

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

1 issue found across 8 files

Confidence score: 3/5

• In apps/sim/connectors/google-drive/google-drive-errors.ts, rateLimited can miss a later rate-limit reason when more than 16 preceding reasons are present, potentially misclassifying throttling errors and causing incorrect retry or user-facing handling—compute the throttle signal from the uncapped validation data.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/connectors/google-drive/google-drive-errors.ts">

<violation number="1" location="apps/sim/connectors/google-drive/google-drive-errors.ts:111">
P2: When a structured error contains more than 16 preceding reason entries, `rateLimited` misses a later rate-limit reason because it inspects the capped diagnostic list. Compute the throttle signal from the uncapped validated reasons before applying the diagnostic cap, so deferred hydration still stops on every structured rate limit.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic

[cubic-dev-ai]

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 8 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}