Skip to content

feat(plugins): add ReplicationPlugin for external-to-internal data sync (#72) - #349

Open
SuperPiIot wants to merge 1 commit into
outerbase:mainfrom
SuperPiIot:feat/replication-plugin-72
Open

SuperPiIot wants to merge 1 commit into
outerbase:mainfrom
SuperPiIot:feat/replication-plugin-72

Conversation

@SuperPiIot

Copy link
Copy Markdown

/claim #72

Fixes #72

Summary

Implements ReplicationPlugin (plugins/replication/) for StarbaseDB to replicate data from external sources into internal StarbaseDB tables inside Durable Objects.

Features & Architecture

  • Plugin Lifecycle (StarbasePlugin + Operation + CronPlugin): Extends StarbasePlugin and integrates cleanly with StarbaseDB's Operation / OperationCollection AST pipeline and CronPlugin scheduler for automatic background synchronization.
  • Multi-Source Replication: Supports HTTP/REST JSON APIs (with bearer/header auth, query params, and nested JSON dataPath extraction) as well as programmatic custom puller functions (registerPuller()).
  • Incremental & Full Sync Modes:
    • Incremental mode: Tracks persisted watermarks/cursors in _starbase_replication_state (including updated timestamp / cursor token and sync status).
    • Full mode: Replicates complete datasets with batch INSERT OR REPLACE / upserts into destination tables (with optional autoCreateTable schema inference).
  • Field Mapping & Deduplication: Supports source-to-destination column mapping and configurable conflict/primary keys.
  • HTTP Endpoints: Exposes POST /replication/sync (on-demand sync trigger) and GET /replication/status (sync watermarks and health status).
  • Exports & Metadata: Adds plugins/replication/meta.json, plugins/replication/README.md, and exports ReplicationPlugin from dist/plugins.ts.

Verification

  • TypeScript (plugins/replication): npx tsc --noEmit 2>&1 | grep -n "plugins/replication" returned 0 errors in plugins/replication/.
  • Replication Plugin Unit Tests: npx vitest run plugins/replication/index.test.ts — 1 passed (1 file), 5 passed (5 tests).
  • Full Test Suite: npx vitest run — 19 passed / 1 failed (20 files), 156 passed / 4 failed (160 tests). The 4 failures are pre-existing upstream failures in untouched src/rls/index.test.ts (should modify SELECT queries with WHERE conditions, should apply RLS policies to tables in JOIN conditions, should apply RLS policies to multiple tables in a JOIN, should apply RLS policies to subqueries inside FROM clause).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

1 participant