Skip to content

feat(plugins): add ReplicationPlugin to replicate external data into internal SQLite - #346

Open
khuxaiphi-stack wants to merge 5 commits into
outerbase:mainfrom
khuxaiphi-stack:feat/replication-plugin
Open

khuxaiphi-stack wants to merge 5 commits into
outerbase:mainfrom
khuxaiphi-stack:feat/replication-plugin

Conversation

@khuxaiphi-stack

Copy link
Copy Markdown

/claim #72
Fixes #72

Summary

Implements ReplicationPlugin (plugins/replication) to replicate tables and rows from an external database source (PostgreSQL, MySQL, Turso, Cloudflare D1, or StarbaseDB) into the internal StarbaseDB Durable Object SQLite database.

Key Changes

  • plugins/replication/index.ts:
    • POST /replication/sync: Admin-authenticated endpoint to trigger on-demand replication across configured or explicitly specified tables.
    • GET /replication/status: Admin-authenticated endpoint returning replication status and per-table metadata (table_name, rows_synced, last_synced_at, status) stored in _starbase_replication_meta.
    • beforeQuery Auto-Sync: Optional periodic background replication when autoSyncOnQuery is enabled and intervalSeconds has elapsed.
    • Automatic Table Discovery & Schema Inference: Automatically discovers external tables across PostgreSQL, MySQL, and SQLite dialects when tables is omitted, infers SQLite column types (INTEGER, REAL, BLOB, TEXT), and creates local tables.
    • Batched Transactions & Conflict Strategies: Inserts rows in configurable batchSize chunks via executeTransaction (with fallback to executeQuery) supporting both replace (INSERT OR REPLACE) and ignore (INSERT OR IGNORE) conflict strategies.
    • SQL Identifier Validation: Strictly validates table and column names against /^[a-zA-Z_][a-zA-Z0-9_]*$/ before constructing DDL/DML queries.
  • plugins/replication/index.test.ts: 7 comprehensive Vitest unit tests covering initialization, batched transaction sync, empty table handling, SQL identifier validation, admin route authorization, and beforeQuery auto-sync.
  • plugins/replication/meta.json & plugins/replication/README.md: Plugin metadata and usage documentation.
  • dist/plugins.ts: Exports ReplicationPlugin.

Verification

  • npx vitest run plugins/replication/index.test.ts — 7 / 7 tests passing
  • npx tsc --noEmit — 0 TypeScript errors

Solana Payout Wallet: FyexYHp1TZ6uzFLUvFNfofx14LQaBrYHbQksTo18xRgD

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

1 participant