Skip to content

feat(plugins): add ReplicationPlugin to pull external data into internal SQLite (#72) - #344

Open
KrazyArcher wants to merge 1 commit into
outerbase:mainfrom
KrazyArcher:fix-issue-72
Open

KrazyArcher wants to merge 1 commit into
outerbase:mainfrom
KrazyArcher:fix-issue-72

Conversation

@KrazyArcher

Copy link
Copy Markdown

Fixes #72

Summary

  • Implements ReplicationPlugin (plugins/replication/index.ts) extending StarbasePlugin to pull data from external databases/endpoints (e.g., Supabase Postgres, external REST/DataSource adapters) into the Durable Object's internal SQLite database for low-latency edge querying.
  • Key Features:
    • Automatic SQLite table schema inference & creation (CREATE TABLE IF NOT EXISTS) with primary key mapping.
    • Paginated pull batches with idempotent INSERT OR REPLACE upserts.
    • Incremental cursor/watermark persistence in _starbase_replication_state (cursorColumn support).
    • Exposes POST /replication/sync (on-demand/cron triggerable, supports selective table sync) and GET /replication/status endpoints.
  • Adds unit tests in plugins/replication/index.test.ts covering full replication, cursor advancement, and failure reporting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

1 participant