Skip to content

feat(plugins): add ReplicationPlugin for external-to-internal data replication (#72) - #331

Closed
jeffrey-graham wants to merge 3 commits into
outerbase:mainfrom
jeffrey-graham:feat/replication-plugin-72
Closed

jeffrey-graham wants to merge 3 commits into
outerbase:mainfrom
jeffrey-graham:feat/replication-plugin-72

Conversation

@jeffrey-graham

Copy link
Copy Markdown

Summary

Adds ReplicationPlugin (plugins/replication/index.ts) to replicate tables from an external data source (e.g., Supabase / PostgreSQL / MySQL / Turso / D1) into StarbaseDB's internal Durable Object SQLite database so the instance can serve as a low-latency close-to-edge replica.

Key Features

  • Configurable Tables & Polling Intervals: Users can specify which tables to replicate (tables: (string | ReplicationTableConfig)[]), per-table or global intervalSeconds, and batchSize.
  • Incremental Cursor / Watermark Tracking: Supports defining a cursorColumn (such as id or created_at) per table and persists the latest watermark in tmp_replication_state inside DO SQLite for append-only incremental polling (WHERE <cursorColumn> > ? ORDER BY <cursorColumn> ASC LIMIT ?).
  • Automatic Local Table Provisioning & Upsert: Infers SQLite column types and primary keys on first pull (CREATE TABLE IF NOT EXISTS) and upserts rows via INSERT OR REPLACE INTO.
  • HTTP & Query Lifecycle Hooks: Exposes GET /replication/status and admin-protected POST /replication/sync (for all tables or a single table), plus optional non-blocking background sync via ExecutionContext.waitUntil on beforeQuery.
  • Unit Tests: Includes comprehensive Vitest unit test coverage in plugins/replication/index.test.ts.

Fixes #72
/claim #72

@jeffrey-graham

Copy link
Copy Markdown
Author

Closing this PR to avoid adding duplicate noise on #72, since earlier ReplicationPlugin PRs (#75, #95) are already awaiting review and the bounty label on #72 is no longer active.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

1 participant