Parallel sync & rate limit

[Yostra]

Summary

v2 dropped parallel backfill support, causing all Stripe list API calls to run sequentially. This restores parallel sync by splitting backfills into time-based segments using the account's created timestamp.

Streams that support created filter are split into 200 time segments and fetched concurrently via mergeAsync
Token-bucket rate limiter (default 25 req/s, configurable via RATE_LIMIT env) prevents hitting Stripe rate limits
Segment state is persisted in stream state for resumability across restarts
Streams without created filter support fall back to sequential pagination (existing behavior)

Test plan

• Run backfill against a Stripe account with substantial data and verify parallel segments are fetched concurrently
• Interrupt and resume a sync to verify segment checkpoint/resume works
• Verify streams without supportsCreatedFilter still sync sequentially
• Test with RATE_LIMIT env var to confirm rate limiting is respected

[Copilot]

Pull request overview

Restores parallelized Stripe list-API backfill in source-stripe by segmenting by created time ranges and adds request-rate limiting (in-memory by default, with an Engine-side Postgres-backed limiter when syncing to Postgres).

Changes:

• Add segmented parallel backfill for supportsCreatedFilter streams, with resumable per-segment checkpoints.
• Introduce a token-bucket rate limiter abstraction + in-memory implementation; plumb it through the Stripe source.
• In Engine API, optionally inject a Postgres-backed distributed limiter (via @stripe/sync-util-postgres) when the destination is Postgres.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pnpm-lock.yaml	Adds workspace dependency wiring for @stripe/sync-util-postgres.
packages/source-stripe/src/src-list-api.ts	Implements segment-based parallel backfill, merge of async generators, and rate-limited pagination.
packages/source-stripe/src/rate-limiter.ts	Adds RateLimiter contract + in-memory token-bucket implementation.
packages/source-stripe/src/index.ts	Extends config/state schema (rate_limit, backfill_concurrency, segments) and injects limiter into backfill.
packages/source-stripe/src/index.test.ts	Adds tests for segment resume/checkpointing, sequential fallback, and rate limiter behavior.
apps/engine/src/api/app.ts	Adds Postgres-backed limiter creation and injects it into the Stripe source during /read and /sync.
apps/engine/package.json	Adds @stripe/sync-util-postgres dependency to support distributed rate limiting.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.