diff --git a/wiki/AI-Copilot.md b/wiki/AI-Copilot.md new file mode 100644 index 0000000..933723c --- /dev/null +++ b/wiki/AI-Copilot.md @@ -0,0 +1,94 @@ +# AI Copilot + +SyncFlow includes an LLM-powered AI copilot for pipeline assistance, review, and optimization. + +## Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/ai/chat` | Chat with AI copilot | +| `POST` | `/api/ai/plan` | Create reasoning plan | +| `POST` | `/api/ai/analyze` | Multi-agent analysis | +| `POST` | `/api/ai/document` | Search knowledge base | +| `POST` | `/api/ai/review` | AI pipeline review | +| `POST` | `/api/ai/recommend` | Optimization recommendations | +| `GET` | `/api/ai/history` | Conversation history | + +## Features + +### Chat + +Conversational interface for pipeline questions: + +```json +POST /api/ai/chat +{ + "message": "How do I set up CDC from PostgreSQL to MySQL?", + "sessionId": "optional-session-id" +} +``` + +### Pipeline Review + +AI analyzes a pipeline design and suggests improvements: + +```json +POST /api/ai/review +{ + "pipelineId": "pipeline-uuid" +} +``` + +Review covers: +- Column type compatibility +- Missing transformations +- Performance bottlenecks +- Security concerns +- Best practices + +### Optimization Recommendations + +```json +POST /api/ai/recommend +{ + "pipelineId": "pipeline-uuid", + "metrics": { + "throughput": 1000, + "latency": 50, + "errorRate": 0.01 + } +} +``` + +### Knowledge Base Search + +Semantic search across pipeline documentation: + +```json +POST /api/ai/document +{ + "query": "how to handle schema changes in CDC" +} +``` + +## Configuration + +```yaml +syncflow: + ai: + endpoint: ${SYNCFLOW_AI_ENDPOINT:https://api.openai.com/v1/chat/completions} + model: ${SYNCFLOW_AI_MODEL:gpt-4o} + api-key: ${SYNCFLOW_AI_API_KEY:} + max-tokens: 4096 + temperature: 0.3 +``` + +## Authentication + +AI endpoints require `AI_USE` permission. Check via `AuthorizationService`. + +## Privacy + +- Prompts are not stored permanently +- Conversation history is session-scoped +- No sensitive data (credentials, tokens) included in prompts diff --git a/wiki/API-Reference.md b/wiki/API-Reference.md new file mode 100644 index 0000000..a3e89b5 --- /dev/null +++ b/wiki/API-Reference.md @@ -0,0 +1,174 @@ +# API Reference + +Base URL: `http://localhost:8080` + +All endpoints except auth require `Authorization: Bearer ` header. + +Swagger UI: `http://localhost:8080/swagger-ui.html` +OpenAPI docs: `http://localhost:8080/v3/api-docs` + +--- + +## Authentication + +| Method | Path | Description | Auth | +|--------|------|-------------|------| +| `POST` | `/api/auth/login` | Login (returns JWT) | No | +| `POST` | `/api/auth/change-password` | Change password | Yes | +| `GET` | `/api/auth/me` | Current user info | Yes | + +--- + +## Connections + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/connections` | Create connection | +| `GET` | `/api/connections` | List connections | +| `GET` | `/api/connections/{id}` | Get connection | +| `PUT` | `/api/connections/{id}` | Update connection | +| `DELETE` | `/api/connections/{id}` | Delete connection | +| `POST` | `/api/connections/test` | Test connection | +| `GET` | `/api/connections/{id}/health` | Health check | + +--- + +## Metadata Discovery + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/connections/{id}/metadata` | Discover schemas | +| `GET` | `/api/connections/{id}/schemas/{schema}/tables` | Discover tables | +| `GET` | `/api/connections/{id}/schemas/{schema}/tables/{table}/columns` | Discover columns | +| `GET` | `/api/connections/{id}/schemas/{schema}/tables/{table}/indexes` | Discover indexes | +| `GET` | `/api/connections/{id}/schemas/{schema}/tables/{table}/constraints` | Discover constraints | +| `POST` | `/api/connections/{id}/metadata/refresh` | Refresh cache | + +--- + +## Pipelines + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/pipelines` | Create pipeline | +| `GET` | `/api/pipelines` | List pipelines | +| `GET` | `/api/pipelines/{id}` | Get pipeline | +| `PUT` | `/api/pipelines/{id}` | Update pipeline | +| `DELETE` | `/api/pipelines/{id}` | Delete pipeline | +| `POST` | `/api/pipelines/{id}/validate` | Validate pipeline | +| `POST` | `/api/pipelines/{id}/rollback` | Rollback to version | +| `GET` | `/api/pipelines/{id}/versions` | List versions | +| `GET` | `/api/pipelines/{id}/preview` | Preview output | +| `GET` | `/api/pipelines/{id}/conflicts` | Detect conflicts | + +--- + +## Snapshots + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/pipelines/{id}/snapshot` | Start snapshot | +| `GET` | `/api/snapshots` | List snapshots | +| `GET` | `/api/snapshots/{id}` | Get snapshot | +| `GET` | `/api/snapshots/{id}/progress` | Get progress | +| `GET` | `/api/snapshots/{id}/events` | SSE progress stream | +| `POST` | `/api/snapshots/{id}/cancel` | Cancel snapshot | + +--- + +## CDC Capture + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/pipelines/{id}/capture/start` | Start CDC | +| `POST` | `/api/pipelines/{id}/capture/stop` | Stop CDC | +| `POST` | `/api/pipelines/{id}/capture/pause` | Pause CDC | +| `POST` | `/api/pipelines/{id}/capture/resume` | Resume CDC | +| `GET` | `/api/pipelines/{id}/capture/status` | Capture status | + +--- + +## Agent Fleet + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/agents/register` | Register agent | +| `POST` | `/api/agents/heartbeat` | Agent heartbeat | +| `GET` | `/api/agents` | List agents | +| `GET` | `/api/agents/{id}` | Get agent | +| `POST` | `/api/agents/{id}/drain` | Drain agent | +| `POST` | `/api/agents/{id}/restart` | Restart agent | +| `GET` | `/api/agents/{id}/metrics` | Agent metrics | + +--- + +## Plugins + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/plugins` | List plugins | +| `GET` | `/api/plugins/{id}` | Get plugin | +| `POST` | `/api/plugins/install` | Install plugin | +| `POST` | `/api/plugins/{id}/enable` | Enable plugin | +| `POST` | `/api/plugins/{id}/disable` | Disable plugin | +| `DELETE` | `/api/plugins/{id}` | Uninstall plugin | +| `GET` | `/api/plugins/{id}/capabilities` | Plugin capabilities | + +--- + +## AI Copilot + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/ai/chat` | Chat with copilot | +| `POST` | `/api/ai/plan` | Create reasoning plan | +| `POST` | `/api/ai/analyze` | Multi-agent analysis | +| `POST` | `/api/ai/document` | Search knowledge base | +| `POST` | `/api/ai/review` | Pipeline review | +| `POST` | `/api/ai/recommend` | Optimization recommendations | +| `GET` | `/api/ai/history` | Conversation history | + +--- + +## Admin & Multi-Tenancy + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/admin/organizations` | Create organization | +| `POST` | `/api/admin/workspaces` | Create workspace | +| `POST` | `/api/admin/projects` | Create project | +| `POST` | `/api/admin/apikeys` | Issue API key | +| `DELETE` | `/api/admin/apikeys/{id}` | Revoke API key | +| `GET` | `/api/admin/quotas` | Get tenant quota | +| `GET` | `/api/admin/audit` | List audit records | +| `GET` | `/api/admin/tenants` | Current tenant context | + +--- + +## Dashboard & Diagnostics + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/dashboard/overview` | Dashboard overview | +| `GET` | `/api/dashboard/pipelines` | Pipeline dashboard | +| `GET` | `/api/dashboard/connections` | Connection dashboard | +| `GET` | `/api/dashboard/connectors` | Connector dashboard | +| `GET` | `/api/dashboard/jobs` | Job dashboard | +| `GET` | `/api/dashboard/metrics` | Metrics dashboard | +| `GET` | `/api/dashboard/errors` | Error dashboard | +| `GET` | `/api/diagnostics/system` | System diagnostics | +| `GET` | `/api/diagnostics/connectors` | Connector diagnostics | +| `GET` | `/api/diagnostics/pipelines` | Pipeline diagnostics | +| `GET` | `/api/diagnostics/executions` | Execution diagnostics | + +--- + +## Health & Operations + +| Method | Path | Description | Auth | +|--------|------|-------------|------| +| `GET` | `/api/health` | Health check | No | +| `GET` | `/actuator/health` | Spring health | No | +| `GET` | `/actuator/prometheus` | Prometheus metrics | No | +| `GET` | `/actuator/metrics` | Micrometer metrics | No | +| `POST` | `/actuator/shutdown` | Graceful shutdown | No | diff --git a/wiki/Agent-Data-Plane.md b/wiki/Agent-Data-Plane.md new file mode 100644 index 0000000..357239c --- /dev/null +++ b/wiki/Agent-Data-Plane.md @@ -0,0 +1,92 @@ +# Agent / Data Plane + +The agent module is a standalone Spring Boot application that runs in the data plane, executing snapshot and CDC operations close to the databases. + +## Architecture + +``` +Control Plane (syncflow-api :8080) + | + +-- POST /api/agents/register (agent registers on startup) + +-- POST /api/agents/heartbeat (every 15s, with HW metrics) + +-- GET /api/agents (list all agents) + +-- POST /api/agents/{id}/drain (drain agent) + +-- POST /api/agents/{id}/restart (restart agent) + | +Data Plane (syncflow-agent :9090) + ├── AgentRegistrar (registers with control plane) + └── HeartbeatSender (sends hardware metrics) +``` + +## Running the Agent + +```bash +# Build +./gradlew :syncflow-agent:bootRun + +# Or with Docker +docker build -f docker/Dockerfile.agent -t syncflow-agent . +docker run -e SYNCFLOW_AGENT_CONTROL_PLANE=http://control-plane:8080 syncflow-agent +``` + +## Registration + +On startup, the agent registers with the control plane: + +```json +POST /api/agents/register +{ + "version": "0.1.0", + "capabilities": ["SNAPSHOT", "CDC", "SYNCHRONIZATION", "METADATA"], + "labels": {"type": "standard"}, + "environment": "customer", + "region": "us-east-1", + "hostname": "agent-1.example.com" +} +``` + +## Heartbeat + +Every 15 seconds, the agent sends hardware metrics: + +```json +POST /api/agents/heartbeat +{ + "agentId": "agent-uuid", + "cpuPercent": 45.2, + "memoryUsed": 2147483648, + "memoryTotal": 4294967296, + "runningJobs": 2 +} +``` + +## Agent States + +| State | Description | +|-------|-------------| +| `ONLINE` | Agent is registered and sending heartbeats | +| `DRAINING` | Agent is finishing current jobs, not accepting new ones | +| `OFFLINE` | Agent missed heartbeats (detected by control plane) | + +## Fleet Management + +`FleetManager` in the control plane manages the agent fleet: + +- Tracks online/offline status +- Assigns jobs to agents based on capability and load +- Handles agent drain (move jobs before shutdown) +- Monitors hardware metrics for capacity planning + +## Configuration + +| Variable | Description | Default | +|----------|-------------|---------| +| `SYNCFLOW_AGENT_CONTROL_PLANE` | Control plane URL | `http://localhost:8080` | +| `SYNCFLOW_AGENT_VERSION` | Agent version | `0.1.0` | + +## Use Cases + +1. **Remote Execution** — Run snapshots/CDC on machines close to source databases +2. **Load Distribution** — Spread workload across multiple agents +3. **Network Isolation** — Agents in VPC with database access; control plane in DMZ +4. **Edge Deployment** — Agents at edge locations syncing to central control plane diff --git a/wiki/Architecture-Decisions.md b/wiki/Architecture-Decisions.md new file mode 100644 index 0000000..bbfd1c4 --- /dev/null +++ b/wiki/Architecture-Decisions.md @@ -0,0 +1,171 @@ +# Architecture Decision Records + +All ADRs are in `docs/adr/`. + +## ADR Index + +| ADR | Decision | Status | +|-----|----------|--------| +| [[ADR-001]] | Spring Modulith for module system | Accepted | +| [[ADR-002]] | Hexagonal (Ports & Adapters) Architecture | Accepted | +| [[ADR-003]] | EventPublisher abstraction over direct Kafka | Accepted | +| [[ADR-004]] | Virtual Threads (JDK 25) for all concurrency | Accepted | +| [[ADR-005]] | PostgreSQL for metadata store | Accepted | +| [[ADR-006]] | pgvector for semantic search (deferred) | Deferred | +| [[ADR-007]] | OpenTelemetry for observability | Accepted | +| [[ADR-008]] | REST/HTTP for agent communication | Accepted | +| [[ADR-009]] | Control Plane / Data Plane separation | Accepted | +| [[ADR-010]] | Plugin SDK (syncflow-plugin-api) | Accepted | +| MODULE_SPLIT_DEFERRED | Runtime module split deferred | Deferred | + +--- + +## ADR-001: Spring Modulith + +**Context:** Need compile-time module boundaries without full microservice overhead. + +**Decision:** Use Spring Modulith for module system. + +**Consequences:** +- Compile-time boundary enforcement +- Event-driven inter-module communication +- Single-JVM integration tests +- Future extraction path to microservices + +--- + +## ADR-002: Hexagonal Architecture + +**Context:** Core domain logic should not depend on frameworks or infrastructure. + +**Decision:** Hexagonal (Ports & Adapters) architecture. + +**Consequences:** +- Core has zero framework dependencies +- 33+ core tests run in milliseconds +- Connector isolation via SPI +- Plugin readiness + +--- + +## ADR-003: EventPublisher Abstraction + +**Context:** Need to publish events without coupling to Kafka. + +**Decision:** `EventPublisher` abstraction with in-memory default. + +**Consequences:** +- In-memory default for dev/testing +- Contract stability +- Kafka adapter is ~50 lines when needed + +--- + +## ADR-004: Virtual Threads + +**Context:** Need high concurrency without thread pool sizing. + +**Decision:** JDK 25 virtual threads for all concurrency. + +**Consequences:** +- No thread pool sizing needed +- Blocking I/O is fine +- `StructuredTaskScope` for parallel fan-out +- `ReentrantLock` instead of `synchronized` (prevents carrier-thread pinning) +- `spring.threads.virtual.enabled: true` for Tomcat + +--- + +## ADR-005: PostgreSQL for Metadata + +**Context:** Need ACID compliance, JSONB support, and ecosystem compatibility. + +**Decision:** PostgreSQL as primary metadata store. + +**Consequences:** +- ACID compliance +- JSONB for flexible schemas +- Flyway migrations +- Compatible with Debezium, pgvector + +--- + +## ADR-006: pgvector (Deferred) + +**Context:** AI Copilot needs semantic search for knowledge base. + +**Decision:** Use pgvector for vector similarity search (deferred until doc count > 1000). + +**Consequences:** +- Same database as metadata +- Semantic search for AI Copilot +- Incremental adoption + +--- + +## ADR-007: OpenTelemetry + +**Context:** Need end-to-end tracing across services. + +**Decision:** OpenTelemetry for observability. + +**Consequences:** +- End-to-end tracing +- Vendor neutrality +- MDC integration +- Micrometer bridge + +--- + +## ADR-008: REST/HTTP for Agents + +**Context:** Agents need to communicate with control plane. + +**Decision:** REST/HTTP (not gRPC). + +**Consequences:** +- Existing infrastructure +- Simple request-response contract +- Debuggability with curl +- gRPC deferred for streaming + +--- + +## ADR-009: Control Plane / Data Plane + +**Context:** Need security isolation and scalability. + +**Decision:** Separate control plane (API) and data plane (agent). + +**Consequences:** +- Security: credentials stay in VPC +- Resilience: agents run without control plane +- Scalability: agents scale independently +- Multi-tenancy: agents scoped to tenants + +--- + +## ADR-010: Plugin SDK + +**Context:** Need extensibility without modifying core. + +**Decision:** Standalone `syncflow-plugin-api` module. + +**Consequences:** +- Zero core dependencies +- Isolated ClassLoader per plugin +- Manifest-driven loading +- Versioned compatibility + +--- + +## MODULE_SPLIT_DEFERRED + +**Context:** Architecture analysis recommended extracting `syncflow-runtime` module. + +**Decision:** Defer module split. + +**Rationale:** +- Single consumer today (API module) +- 490+ tests at risk +- JPA types already in natural boundary packages (`com.syncflow.api.{cdc,sync}.entity`) diff --git a/wiki/Architecture.md b/wiki/Architecture.md new file mode 100644 index 0000000..365e243 --- /dev/null +++ b/wiki/Architecture.md @@ -0,0 +1,69 @@ +# Architecture + +## Module Structure + +``` +syncflow-common Shared primitives, tenant context, exceptions +syncflow-core Domain models, SPI interfaces, pipeline/snapshot/CDC logic +syncflow-connectors Concrete connectors (JDBC, Debezium, MongoDB, writers) +syncflow-persistence JPA entities, Spring Data repos, Flyway migrations +syncflow-api REST controllers, orchestration, runtime state +syncflow-security Security configuration, agent token filter +syncflow-monitoring Micrometer metrics, OpenTelemetry integration +syncflow-plugin-api Standalone SPI for third-party connector plugins +syncflow-agent Standalone agent for distributed execution +``` + +## Data Flow + +``` +Pipeline Design (DDL + column mapping) + | + v ++-------------------+ +-------------------+ +| SnapshotExecutor | | Source Connector | +| (batch read + |<--->| (keyset/offset | +| transform + | | pagination) | +| write) | +--------+----------+ ++--------+----------+ | + | v + | CDC Events (WAL/binlog) + v ++-------------------+ +-------------------+ +| CaptureLifecycle |---->| Debezium Engine | +| (start/stop/ | | (streaming) | +| pause/resume) | +--------+----------+ ++--------+----------+ | + | v + v +-------------------+ ++-------------------+ | SyncOrchestrator | +| DestinationRouter |<--| (queue + process | +| (batched writes) | | + route + DLQ) | ++-------------------+ +-------------------+ +``` + +## Design Decisions + +See [[Architecture-Decisions]] for the full ADR index. Key decisions: + +- **Hexagonal Architecture** — Core has zero framework dependencies; connectors implement SPI interfaces +- **Virtual Threads** — JDK 25 virtual threads for all concurrency; `StructuredTaskScope` for parallel fan-out +- **ReentrantLock over synchronized** — Prevents carrier-thread pinning with virtual threads +- **PostgreSQL Advisory Locks** — Distributed locking without Redis dependency +- **EventPublisher Abstraction** — In-memory default, Kafka adapter optional +- **Spring Modulith** — Compile-time module boundary enforcement + +## Technology Stack + +| Layer | Technology | +|-------|-----------| +| Language | Java 25 (preview features enabled) | +| Framework | Spring Boot 3.5.x, Spring Modulith | +| Build | Gradle 9.x with Spotless formatting | +| Database | PostgreSQL 16+ (primary), MySQL 8.4, MongoDB 7.0 | +| CDC Engine | Debezium | +| Serialization | Jackson (JavaTimeModule) | +| Migrations | Flyway (18 versions) | +| Observability | Micrometer + OpenTelemetry + Prometheus + Grafana | +| Testing | JUnit 5, Testcontainers, ArchUnit, JMH | +| Deployment | Docker, Kubernetes (Kustomize), Helm, Terraform, ArgoCD | diff --git a/wiki/CDC-Capture.md b/wiki/CDC-Capture.md new file mode 100644 index 0000000..5df5a93 --- /dev/null +++ b/wiki/CDC-Capture.md @@ -0,0 +1,84 @@ +# CDC Capture + +CDC (Change Data Capture) captures real-time changes from source databases using Debezium, then routes them to destination writers. + +## Architecture + +``` +Source DB (WAL/binlog) + | + v +Debezium Engine (runs on virtual thread) + | + v +CaptureLifecycle (start/stop/pause/resume) + | + +--> In-memory event queue (per pipeline) + | | + | v + | SyncOrchestrator (drain + transform + write) + | | + | v + | DestinationRouter (batched writes) + | + +--> [Optional] Kafka transport (when syncflow.kafka.enabled=true) +``` + +## API Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/pipelines/{id}/capture/start` | Start CDC capture | +| `POST` | `/api/pipelines/{id}/capture/stop` | Stop CDC capture | +| `POST` | `/api/pipelines/{id}/capture/pause` | Pause CDC capture | +| `POST` | `/api/pipelines/{id}/capture/resume` | Resume CDC capture | +| `GET` | `/api/pipelines/{id}/capture/status` | Get capture status + event count | + +## Offset Management + +CDC offsets track the position in the source's WAL/binlog. Stored in `debezium_offsets` (Postgres BYTEA for durable key/value pairs). + +On restart, Debezium resumes from the last committed offset — no data loss, no duplication. + +## Event Processing + +`SyncOrchestrator` processes CDC events: + +1. **Drain** — Pulls up to `batch-size` events from the queue +2. **Group** — Groups by table and operation type +3. **Transform** — Applies column mappings and transformations +4. **Write** — Calls `DestinationRouter.writeBatch()` (single flush + commit per table) +5. **Idempotency** — `markProcessedIfAbsent` prevents duplicate processing + +## Dead Letter Queue (DLQ) + +Events that fail after `max-attempts` retries go to the DLQ: + +- Stored in `dead_letter_events` table (JPA-backed, survives restarts) +- Viewable via dashboard +- Replayable via API +- Each replay increments `replay_count` + +## Backpressure + +- **Bounded queue** — `syncflow.runtime.sync.queue-capacity` (default 10000) +- **Circuit breaker** — `CircuitBreakerEventPublisher` pauses capture when destination is unhealthy +- **DLQ overflow** — Events exceeding retry count are routed to DLQ, not dropped + +## Exactly-Once Semantics + +CDC provides at-least-once delivery. SyncFlow achieves effective exactly-once via: + +1. **Idempotent writes** — `markProcessedIfAbsent` in `processed_events` table +2. **Offset commit** — Only advances offset after successful write +3. **Transactional boundary** — Write + offset commit in same transaction where possible + +## Kafka Transport + +When `syncflow.kafka.enabled=true`, CDC events are published to Kafka topics instead of in-memory queues. This enables: + +- Cross-pod event distribution +- Event replay from Kafka +- Decoupled producers/consumers + +Topics are named `{prefix}.{pipelineId}` with configurable partitions and replication. diff --git a/wiki/Configuration.md b/wiki/Configuration.md new file mode 100644 index 0000000..5cfffc1 --- /dev/null +++ b/wiki/Configuration.md @@ -0,0 +1,108 @@ +# Configuration + +## Environment Variables + +All configuration is externalizable via environment variables. The pattern is `SYNCFLOW_
_`. + +### Required + +| Variable | Description | Default | +|----------|-------------|---------| +| `SYNCFLOW_ENCRYPTION_KEY` | Base64-encoded AES key (16/24/32 bytes) for credential encryption | *(none — fails fast if missing)* | +| `SYNCFLOW_JWT_SECRET` | Base64-encoded HMAC secret (>= 32 bytes) for JWT signing | *(none — fails fast if missing)* | + +### Application + +| Variable | Description | Default | +|----------|-------------|---------| +| `SYNCFLOW_JWT_ISSUER` | JWT issuer claim | `syncflow` | +| `SYNCFLOW_JWT_EXPIRY_MINUTES` | JWT token expiry in minutes | `60` | + +### Runtime Tunables + +| Variable | Description | Default | +|----------|-------------|---------| +| `SYNCFLOW_RUNTIME_SYNC_QUEUE_CAPACITY` | Per-pipeline CDC event queue capacity | `10000` | +| `SYNCFLOW_RUNTIME_SYNC_BATCH_SIZE` | Max events drained + written per batch | `100` | +| `SYNCFLOW_RUNTIME_SYNC_POLL_TIMEOUT` | Poll timeout when queue is empty | `500ms` | +| `SYNCFLOW_RUNTIME_SNAPSHOT_CHECKPOINT_INTERVAL_BATCHES` | Checkpoint every N batches | `5` | +| `SYNCFLOW_RUNTIME_RETRY_MAX_ATTEMPTS` | Max delivery attempts before DLQ | `3` | +| `SYNCFLOW_RUNTIME_RETRY_BASE_DELAY` | Exponential backoff base delay | `1000ms` | + +### Kafka (Optional) + +| Variable | Description | Default | +|----------|-------------|---------| +| `SYNCFLOW_KAFKA_ENABLED` | Enable Kafka transport | `false` | +| `SYNCFLOW_KAFKA_BOOTSTRAP_SERVERS` | Kafka broker list | `localhost:9092` | +| `SYNCFLOW_KAFKA_TOPIC_PREFIX` | Topic name prefix | `syncflow` | + +### Agent + +| Variable | Description | Default | +|----------|-------------|---------| +| `SYNCFLOW_AGENT_CONTROL_PLANE` | Control plane URL | `http://localhost:8080` | +| `SYNCFLOW_AGENT_VERSION` | Agent version string | `0.1.0` | + +### AI Copilot + +| Variable | Description | Default | +|----------|-------------|---------| +| `SYNCFLOW_AI_ENDPOINT` | LLM API endpoint | `https://api.openai.com/v1/chat/completions` | +| `SYNCFLOW_AI_MODEL` | LLM model name | `gpt-4o` | +| `SYNCFLOW_AI_API_KEY` | LLM API key | *(empty)* | +| `SYNCFLOW_AI_MAX_TOKENS` | Max tokens per request | `4096` | +| `SYNCFLOW_AI_TEMPERATURE` | Sampling temperature | `0.3` | + +## Spring Properties + +Key `application.yml` properties: + +```yaml +spring: + threads: + virtual: + enabled: true # Virtual threads for Tomcat + datasource: + url: jdbc:postgresql://localhost:5432/syncflow + hikari: + maximum-pool-size: 10 + jpa: + hibernate: + ddl-auto: validate + +management: + endpoints: + web: + exposure: + include: health,info,metrics,prometheus,shutdown +``` + +## RuntimeProperties + +Bound to `@ConfigurationProperties("syncflow.runtime")` with `@Validated`. Constructor injection of `ActiveCaptureRepository` and `DistributedLockService`. + +```yaml +syncflow: + runtime: + sync: + queue-capacity: 10000 + batch-size: 100 + poll-timeout: 500ms + snapshot: + checkpoint-interval-batches: 5 + progress-publish-interval-batches: 10 + parallelism: 4 + max-chunks: 64 + retry: + max-attempts: 3 + base-delay: 1000ms +``` + +## Profiles + +| Profile | Purpose | +|---------|---------| +| `local` | Local development with PostgreSQL | +| `test` | Integration tests with Testcontainers | +| `production` | Hardened settings for deployment | diff --git a/wiki/Connectors.md b/wiki/Connectors.md new file mode 100644 index 0000000..8112646 --- /dev/null +++ b/wiki/Connectors.md @@ -0,0 +1,85 @@ +# Connectors + +SyncFlow uses a SPI (Service Provider Interface) architecture for connectors. Each connector implements one or more capability interfaces. + +## Connector SPI Hierarchy + +``` +Connector (base) + ├── type(), capabilities(), connect(), disconnect(), validate() + ├── discoverSchemas(), discoverTables(), health(), metadata() + │ + ├── MetadataCapableConnector + │ └── discoverColumns(), discoverConstraints(), discoverIndexes() + │ + ├── SnapshotCapableConnector + │ └── readBatch(), estimateRows(), rangeChunks(), snapshotClone(), streamRows() + │ + └── CdcCapableConnector + └── startCDC(), stopCDC(), pauseCDC(), resumeCDC(), captureStatus(), currentOffset() +``` + +## Supported Connectors + +| Connector | Type | Capabilities | Implementation | +|-----------|------|-------------|----------------| +| PostgreSQL | `POSTGRESQL` | CDC, Snapshot, Metadata | `PostgresCdcConnector`, `PostgresMetadataConnector` | +| MySQL | `MYSQL` | CDC, Snapshot, Metadata | `MySqlCdcConnector`, `MySqlMetadataConnector` | +| MongoDB | `MONGODB` | CDC, Snapshot, Metadata | `MongoDbCdcConnector`, `MongoDbMetadataConnector` | +| Redis | `REDIS` | Metadata | `RedisMetadataConnector` | +| Kafka | `KAFKA` | CDC | `KafkaConnector` | + +## Writers + +| Writer | Purpose | +|--------|---------| +| `JdbcBatchWriter` | JDBC batch inserts/updates/deletes | +| `PooledJdbcBatchWriter` | HikariCP-pooled version (production) | +| `PostgresWriter` | PostgreSQL-specific with UPSERT | +| `MySqlWriter` | MySQL-specific with UPSERT | + +## Connector Capabilities + +```java +public record ConnectorCapabilities( + boolean supportsCdc, + boolean supportsSnapshot, + boolean supportsSchemaDiscovery, + boolean supportsTransactions, + boolean supportsOffsetTracking +) {} +``` + +## Connection Types + +Defined in `ConnectorType` enum: + +``` +POSTGRESQL, MYSQL, MONGODB, KAFKA, SQLSERVER, ORACLE, +ELASTICSEARCH, REDIS, GENERIC_JDBC +``` + +Not all types have implementations yet. The SPI allows adding new connectors without modifying core code. + +## Writing a Custom Connector + +See [[Plugin-System]] for third-party connector development. For internal connectors: + +1. Implement `CdcCapableConnector` (or subset) +2. Add to `syncflow-connectors` module +3. Register via `@Component` (auto-discovered by Spring) +4. Add validator if needed (implements `ConnectorValidator`) + +## Metadata Discovery + +`MetadataCapableConnector` provides schema introspection: + +``` +GET /api/connections/{id}/metadata → schemas +GET /api/connections/{id}/schemas/{schema}/tables → tables +GET /api/connections/{id}/schemas/{schema}/tables/{table}/columns → columns +GET /api/connections/{id}/schemas/{schema}/tables/{table}/indexes → indexes +GET /api/connections/{id}/schemas/{schema}/tables/{table}/constraints → constraints +``` + +Results are cached with configurable TTL (`syncflow.metadata.cache-ttl`, default 5m). diff --git a/wiki/Database-Schema.md b/wiki/Database-Schema.md new file mode 100644 index 0000000..2d5d21e --- /dev/null +++ b/wiki/Database-Schema.md @@ -0,0 +1,160 @@ +# Database Schema + +PostgreSQL 16+ with Flyway migrations (V1–V18). + +## Migration History + +| Version | Tables | Purpose | +|---------|--------|---------| +| V1 | `pipelines`, `pipeline_events` | Core pipeline entities | +| V2 | `connections` | Connection registry with encrypted credentials | +| V3 | `data_governance`, `data_lineage`, `column_tags` | Data governance | +| V4 | `pipeline_designs`, `pipeline_design_versions` | Pipeline designer with versioning | +| V5 | `cdc_offsets` | Debezium offset storage | +| V6 | `kafka_topics`, `kafka_messages` | Kafka sync persistence | +| V7 | `dead_letter_events` | DLQ with nullable event column | +| V8 | — | DLQ: add `replay_count` column | +| V9 | `app_users` | JWT authentication, default admin | +| V10 | — | Users: add `must_change_password` | +| V11 | — | Multi-tenancy: add `tenant_id` to domain tables | +| V12 | `snapshot_jobs`, `snapshot_checkpoints`, `sync_jobs`, `workflow_instances`, `quotas`, `audit_records`, `api_keys`, `agents`, `alert_events` | Runtime state persistence | +| V13 | `debezium_offsets` | Durable CDC offsets (BYTEA) | +| V14 | `active_captures` | Durable CDC capture state | +| V15 | — | Snapshot checkpoints: add `chunk_index` | +| V16 | — | Debezium offsets partitioning | +| V17 | — | Logical replication setup | +| V18 | `event_queue_snapshots` | CDC event queue snapshots | + +## Key Tables + +### connections +```sql +CREATE TABLE connections ( + id VARCHAR(36) PRIMARY KEY, + tenant_id VARCHAR(36) NOT NULL DEFAULT 'system', + name VARCHAR(255) NOT NULL, + connection_type VARCHAR(50) NOT NULL, + host VARCHAR(255), + port INTEGER, + database_name VARCHAR(255), + options JSONB, + encrypted_username TEXT, + encrypted_password TEXT, + status VARCHAR(20) DEFAULT 'ACTIVE', + db_version VARCHAR(100), + driver_name VARCHAR(100), + last_latency_ms BIGINT, + last_checked TIMESTAMP, + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL +); +``` + +### pipeline_designs +```sql +CREATE TABLE pipeline_designs ( + id VARCHAR(36) PRIMARY KEY, + tenant_id VARCHAR(36) NOT NULL DEFAULT 'system', + name VARCHAR(255) NOT NULL, + source_connection_id VARCHAR(36), + destination_connection_id VARCHAR(36), + payload JSONB NOT NULL, + status VARCHAR(20) DEFAULT 'CREATED', + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL +); +``` + +### snapshot_jobs +```sql +CREATE TABLE snapshot_jobs ( + id VARCHAR(36) PRIMARY KEY, + tenant_id VARCHAR(36) NOT NULL DEFAULT 'system', + pipeline_id VARCHAR(36) NOT NULL, + status VARCHAR(20) NOT NULL, + payload JSONB NOT NULL, + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL +); +``` + +### snapshot_checkpoints +```sql +CREATE TABLE snapshot_checkpoints ( + id VARCHAR(36) PRIMARY KEY, + tenant_id VARCHAR(36) NOT NULL, + pipeline_id VARCHAR(36) NOT NULL, + source_table VARCHAR(255) NOT NULL, + chunk_index INTEGER NOT NULL, + last_batch_number INTEGER NOT NULL, + rows_processed BIGINT NOT NULL, + cursor TEXT, + created_at TIMESTAMP NOT NULL +); +``` + +### active_captures +```sql +CREATE TABLE active_captures ( + id VARCHAR(36) PRIMARY KEY, + tenant_id VARCHAR(36) NOT NULL, + pipeline_id VARCHAR(36) NOT NULL, + status VARCHAR(20) NOT NULL, + offset JSONB, + started_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL, + UNIQUE(tenant_id, pipeline_id) +); +``` + +### dead_letter_events +```sql +CREATE TABLE dead_letter_events ( + id VARCHAR(36) PRIMARY KEY, + tenant_id VARCHAR(36) NOT NULL DEFAULT 'system', + pipeline_id VARCHAR(36) NOT NULL, + event JSONB, + error_message TEXT, + error_code VARCHAR(50), + retry_count INTEGER DEFAULT 0, + replay_count INTEGER DEFAULT 0, + created_at TIMESTAMP NOT NULL, + replayed_at TIMESTAMP +); +``` + +### app_users +```sql +CREATE TABLE app_users ( + id VARCHAR(36) PRIMARY KEY, + username VARCHAR(100) UNIQUE NOT NULL, + password_hash VARCHAR(255) NOT NULL, + roles TEXT[], + must_change_password BOOLEAN DEFAULT FALSE, + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL +); +``` + +### debezium_offsets +```sql +CREATE TABLE debezium_offsets ( + id VARCHAR(36) PRIMARY KEY, + tenant_id VARCHAR(36) NOT NULL, + pipeline_id VARCHAR(36) NOT NULL, + key BYTEA NOT NULL, + value BYTEA NOT NULL, + created_at TIMESTAMP NOT NULL, + updated_at TIMESTAMP NOT NULL +); +``` + +## Indexes + +- `connections`: `(tenant_id)`, `(status)` +- `pipeline_designs`: `(tenant_id)`, `(status)` +- `snapshot_jobs`: `(tenant_id, pipeline_id)`, `(status)` +- `snapshot_checkpoints`: `(tenant_id, pipeline_id, source_table, chunk_index)` +- `active_captures`: `(tenant_id, pipeline_id)` UNIQUE +- `dead_letter_events`: `(tenant_id, pipeline_id)`, `(created_at)` +- `debezium_offsets`: `(tenant_id, pipeline_id)` diff --git a/wiki/Deployment.md b/wiki/Deployment.md new file mode 100644 index 0000000..b016c90 --- /dev/null +++ b/wiki/Deployment.md @@ -0,0 +1,100 @@ +# Deployment + +## Docker Compose (Development) + +```bash +# Minimal: PostgreSQL only +docker compose -f docker/docker-compose.yml up -d postgres + +# Full stack with all services +docker compose -f docker/docker-compose.yml up --build +``` + +### Services + +| Service | Port | Always | Profile | +|---------|------|--------|---------| +| PostgreSQL | 5432 | Yes | — | +| App | 8080 | Yes | — | +| MySQL | 3306 | No | `mysql` | +| MongoDB | 27017 | No | `mongodb` | +| Redis | 6379 | No | `redis` | +| Kafka | 9092 | No | `kafka` | +| Prometheus | 9090 | No | `monitoring` | +| Grafana | 3000 | No | `monitoring` | + +## Kubernetes (Kustomize) + +Base manifests in `k8s/base/`: + +```bash +# Apply base configuration +kubectl apply -k k8s/base/ + +# Or with overlays +kubectl apply -k k8s/overlays/production/ +``` + +### Resources + +| Resource | Purpose | +|----------|---------| +| `deployment.yaml` | Application pods | +| `service.yaml` | ClusterIP service | +| `ingress.yaml` | NGINX ingress | +| `configmap.yaml` | Non-sensitive configuration | +| `externalsecret.yaml` | External Secrets for sensitive values | +| `hpa.yaml` | Horizontal Pod Autoscaler | +| `keda-scaledobject.yaml` | KEDA event-driven autoscaling | +| `pdb.yaml` | Pod Disruption Budget | +| `networkpolicy.yaml` | Network policies | +| `prometheusrule.yaml` | Prometheus alerting rules | + +## Helm + +```bash +helm install syncflow helm/syncflow/ \ + --set image.tag=latest \ + --set postgres.host=your-pg-host +``` + +## Terraform + +Infrastructure provisioning in `terraform/`: + +```bash +cd terraform +terraform init +terraform plan +terraform apply +``` + +## ArgoCD + +```bash +kubectl apply -f argocd/application.yaml +``` + +## Multi-Region + +Single-primary, multi-replica architecture: + +- **RTO:** 1-2 minutes (automatic failover) +- **RPO:** < 30 seconds (continuous logical replication) +- **Failover:** `RegionalFailoverManager` with Postgres advisory locks +- **DNS:** Route53 failover routing + +See `docs/deployment/F17_MULTI_REGION_DEPLOYMENT.md` for details. + +## Production Checklist + +- [ ] Set `SYNCFLOW_ENCRYPTION_KEY` (AES-256) +- [ ] Set `SYNCFLOW_JWT_SECRET` (HMAC, >= 32 bytes) +- [ ] Change default admin password +- [ ] Enable TLS termination at ingress +- [ ] Configure `syncflow.region.replication-enabled` for multi-region +- [ ] Set appropriate HikariCP pool sizes +- [ ] Configure Prometheus scraping +- [ ] Set up Grafana dashboards +- [ ] Review NetworkPolicies +- [ ] Configure PodDisruptionBudget diff --git a/wiki/Home.md b/wiki/Home.md new file mode 100644 index 0000000..a62fa44 --- /dev/null +++ b/wiki/Home.md @@ -0,0 +1,41 @@ +# SyncFlow Wiki + +**Enterprise Change Data Capture Platform** + +SyncFlow is a production-grade, connector-based CDC platform for synchronizing data between heterogeneous databases in near real-time. Built on Java 25 with virtual threads and structured concurrency. + +--- + +## Getting Started + +- [[Quick-Start]] — Build, run, and verify in 5 minutes +- [[Architecture]] — Module structure, data flow, and design decisions +- [[Configuration]] — All tunable properties and environment variables + +## Core Concepts + +- [[Pipelines]] — Design, validate, and manage data synchronization pipelines +- [[Connectors]] — Supported source/destination connectors and the SPI +- [[Snapshots]] — Parallel initial data loading with checkpoint resume +- [[CDC-Capture]] — Real-time change data capture via Debezium + +## Operations + +- [[Deployment]] — Docker, Kubernetes, Helm, Terraform, ArgoCD +- [[Multi-Tenancy]] — Tenant isolation, organizations, workspaces +- [[Security]] — Authentication, authorization, encryption, API keys +- [[Observability]] — Metrics, tracing, logging, dashboards + +## Advanced + +- [[Plugin-System]] — Build and install custom connectors +- [[Agent-Data-Plane]] — Distributed execution with the agent module +- [[AI-Copilot]] — LLM-powered pipeline assistance +- [[Runbooks]] — Troubleshooting guides for common failure scenarios + +## Reference + +- [[API-Reference]] — Complete REST endpoint catalog +- [[Database-Schema]] — Flyway migrations and table reference +- [[Architecture-Decisions]] — ADR index +- [[Roadmap]] — Feature status and upcoming work diff --git a/wiki/Multi-Tenancy.md b/wiki/Multi-Tenancy.md new file mode 100644 index 0000000..6b626dc --- /dev/null +++ b/wiki/Multi-Tenancy.md @@ -0,0 +1,67 @@ +# Multi-Tenancy + +SyncFlow supports multi-tenancy via tenant context threading and row-level isolation. + +## Tenant Context + +```java +public record TenantContext( + TenantId tenantId, + String organizationId, + String workspaceId, + String projectId, + String userId, + List roles, + Instant establishedAt +) {} +``` + +## How It Works + +1. **JWT Authentication** — Login returns a JWT with tenant claims +2. **Context Extraction** — Filter extracts `TenantContext` from JWT +3. **Explicit Threading** — `TenantContext` passed through all orchestrators and workers +4. **Row-Level Scoping** — Every domain table has `tenant_id` column; queries filter by it + +## Affected Tables + +All domain tables include `tenant_id`: + +- `connections`, `pipeline_designs`, `snapshot_jobs`, `sync_jobs` +- `dead_letter_events`, `processed_events`, `active_captures` +- `api_keys`, `audit_records`, `quotas` + +## Default Tenant + +The system tenant (UUID configured in migrations) is used for: +- Admin operations +- Agent registration +- System-level resources + +## Organization Hierarchy + +``` +Organization + └── Workspace + └── Project + └── Pipeline +``` + +Each level has its own RBAC scope. API keys can be scoped to specific organizations or projects. + +## Audit Trail + +Every state-changing operation is recorded in `audit_records`: + +```java +AuditRecord + ├── tenantId + ├── userId + ├── action (CREATE, UPDATE, DELETE, EXECUTE) + ├── resourceType (PIPELINE, CONNECTION, etc.) + ├── resourceId + ├── timestamp + └── details (JSONB) +``` + +GDPR compliance: right-to-delete anonymizes audit records instead of deleting them. diff --git a/wiki/Observability.md b/wiki/Observability.md new file mode 100644 index 0000000..f4ccb09 --- /dev/null +++ b/wiki/Observability.md @@ -0,0 +1,110 @@ +# Observability + +## Metrics (Micrometer + Prometheus) + +All metrics are exposed at `/actuator/prometheus`. + +### Application Metrics + +| Metric | Type | Tags | +|--------|------|------| +| `syncflow.snapshot.duration` | Timer | `pipeline` | +| `syncflow.snapshot.rows` | Counter | `pipeline` | +| `syncflow.snapshot.errors` | Counter | `pipeline` | +| `syncflow.cdc.events` | Counter | `pipeline`, `table` | +| `syncflow.cdc.lag` | Gauge | `pipeline` | +| `syncflow.sync.duration` | Timer | `pipeline` | +| `syncflow.sync.events` | Counter | `pipeline` | +| `syncflow.sync.errors` | Counter | `pipeline` | +| `syncflow.dlq.count` | Gauge | `pipeline` | + +### Infrastructure Metrics + +- JVM metrics (heap, GC, threads) +- HikariCP metrics (connections, pool usage) +- Tomcat metrics (requests, threads) +- Spring MVC metrics + +### Custom Tags + +All metrics include `application: syncflow` tag (configurable via `management.metrics.tags.application`). + +## Distributed Tracing (OpenTelemetry) + +Traces are exported to OTLP endpoint: + +```yaml +management: + otlp: + tracing: + endpoint: http://localhost:4318/v1/traces +``` + +### Trace Propagation + +- MDC integration: `traceId`, `correlationId`, `pipelineId` in all log lines +- HTTP header propagation for inter-service calls +- Debezium engine traces + +### Log Pattern + +``` +%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] [%X{traceId}] [%X{correlationId}] [%X{pipelineId}] %-5level %logger{36} - %msg%n +``` + +## Dashboards (Grafana) + +Pre-configured dashboards in `docker/grafana/`: + +| Dashboard | Panels | +|-----------|--------| +| Overview | Pipeline status, connection health, active captures | +| CDC | Event throughput, lag, error rate | +| Snapshot | Progress, duration, rows processed | +| System | JVM, CPU, memory, connections | + +## Health Checks + +```bash +# Basic health +GET /api/health + +# Detailed with connector status +GET /api/health?show-details=always + +# Kubernetes probes +GET /actuator/health/liveness +GET /actuator/health/readiness +``` + +## Diagnostics + +| Endpoint | Description | +|----------|-------------| +| `GET /api/diagnostics/system` | JVM/OS/Java info | +| `GET /api/diagnostics/connectors` | Connector health + latency | +| `GET /api/diagnostics/pipelines` | Pipeline status + metrics | +| `GET /api/diagnostics/executions` | Recent execution history | + +## Alerting + +Prometheus alerting rules in `k8s/base/prometheusrule.yaml`: + +- High error rate (> 1% of events) +- CDC lag exceeding threshold (> 30s) +- Snapshot duration exceeding SLA (> 1h) +- DLQ depth growing +- Agent offline +- Memory usage > 80% + +## SSE (Server-Sent Events) + +Live progress streaming for snapshots: + +``` +GET /api/snapshots/{id}/events +``` + +Events: +- `snapshot-status` — Progress updates (batches, rows, percentage) +- `snapshot-complete` — Terminal state (COMPLETED/CANCELLED/FAILED) diff --git a/wiki/Pipelines.md b/wiki/Pipelines.md new file mode 100644 index 0000000..f9f36c8 --- /dev/null +++ b/wiki/Pipelines.md @@ -0,0 +1,61 @@ +# Pipelines + +A pipeline defines how data flows from a source connection to a destination connection, including which tables to sync, column mappings, and transformation rules. + +## Pipeline Design + +A pipeline design is the declarative specification: + +``` +PipelineDesign + ├── source (ConnectionReference → connectionId) + ├── destination (ConnectionReference → connectionId) + ├── tableMappings[] + │ ├── sourceTable + │ ├── destinationTable (optional, defaults to sourceTable) + │ ├── primaryKey (destinationColumns) + │ └── columnMappings[] + │ ├── sourceColumn + │ ├── destinationColumn + │ └── transformation (optional SQL expression) + └── settings + ├── batchSize (default 100) + └── syncMode (SNAPSHOT, CDC, BOTH) +``` + +## API Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/pipelines` | Create pipeline design | +| `GET` | `/api/pipelines` | List all pipelines | +| `GET` | `/api/pipelines/{id}` | Get pipeline by ID | +| `PUT` | `/api/pipelines/{id}` | Update pipeline | +| `DELETE` | `/api/pipelines/{id}` | Delete pipeline | +| `POST` | `/api/pipelines/{id}/validate` | Validate pipeline | +| `POST` | `/api/pipelines/{id}/rollback` | Rollback to version | +| `GET` | `/api/pipelines/{id}/versions` | List pipeline versions | +| `GET` | `/api/pipelines/{id}/preview` | Preview pipeline output | +| `GET` | `/api/pipelines/{id}/conflicts` | Detect mapping conflicts | + +## Versioning + +Every pipeline update creates a version in `pipeline_design_versions`. Rollback restores a previous version without data loss. + +## Validation + +`POST /api/pipelines/{id}/validate` checks: + +- Source and destination connections exist and are healthy +- Source tables exist in the source schema +- Column types are compatible +- Primary key columns exist +- No circular transformations + +## Conflict Detection + +`GET /api/pipelines/{id}/conflicts` detects: + +- Multiple source columns mapped to the same destination column +- Primary key columns not included in the mapping +- Transformation expressions referencing non-existent columns diff --git a/wiki/Plugin-System.md b/wiki/Plugin-System.md new file mode 100644 index 0000000..e851606 --- /dev/null +++ b/wiki/Plugin-System.md @@ -0,0 +1,137 @@ +# Plugin System + +SyncFlow supports third-party connectors via a plugin SDK with isolated ClassLoader loading. + +## Architecture + +``` +syncflow-plugin-api (standalone JAR) + ├── PluginConnector (SPI interface) + ├── PluginDescriptor (manifest-driven metadata) + ├── PluginLifecycle (install/enable/disable/uninstall) + ├── ConfigurationSchema (settings schema) + └── Capability interfaces (CdcProvider, SnapshotProvider, DestinationWriterProvider) + +PluginManager (in syncflow-api) + ├── URLClassLoader per plugin (isolation) + ├── MANIFEST.MF scanning (Plugin-Connector-Class) + └── Versioned compatibility checks +``` + +## Plugin Structure + +A plugin is a JAR with: + +``` +my-connector-plugin.jar +├── META-INF/ +│ └── MANIFEST.MF +│ Plugin-Connector-Class: com.example.MyConnector +│ Plugin-Version: 1.0.0 +│ Plugin-Name: My Custom Connector +│ Plugin-Description: Connects to MyDatabase +│ Plugin-Platform-Version: 1 +├── com/example/MyConnector.class +└── lib/ + └── my-database-driver.jar +``` + +## Plugin SPI + +```java +public interface PluginConnector extends Connector { + // Standard connector methods from core SPI +} + +public interface CdcProvider { + CdcCapableConnector createCdcConnector(Map config); +} + +public interface SnapshotProvider { + SnapshotCapableConnector createSnapshotConnector(Map config); +} + +public interface DestinationWriterProvider { + DestinationWriter createWriter(Map config); +} +``` + +## API Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/plugins` | List installed plugins | +| `GET` | `/api/plugins/{id}` | Get plugin details | +| `POST` | `/api/plugins/install` | Install plugin JAR | +| `POST` | `/api/plugins/{id}/enable` | Enable plugin | +| `POST` | `/api/plugins/{id}/disable` | Disable plugin | +| `DELETE` | `/api/plugins/{id}` | Uninstall plugin | +| `GET` | `/api/plugins/{id}/capabilities` | Get plugin capabilities | + +## Isolation + +Each plugin runs in its own `URLClassLoader`: + +- Plugin classes are invisible to other plugins and the main application +- Plugin dependencies (in `lib/`) are loaded from the plugin JAR +- No classpath pollution +- Plugin can use different library versions than the host + +## Versioned Compatibility + +```java +public interface PluginDescriptor { + String name(); + String version(); + int minimumPlatformVersion(); // Minimum SyncFlow version required + int maximumPlatformVersion(); // Maximum SyncFlow version supported (0 = no limit) +} +``` + +The plugin manager rejects plugins that declare incompatible platform versions. + +## Lifecycle + +``` +INSTALLED → ENABLED → DISABLED → UNINSTALLED + ↑ ↓ + └──────────┘ +``` + +- **INSTALLED** — JAR loaded, classes available +- **ENABLED** — Plugin active, can handle connections +- **DISABLED** — Plugin paused, existing connections continue +- **UNINSTALLED** — JAR removed, ClassLoader closed + +## Writing a Plugin + +1. Create a new Java project with `syncflow-plugin-api` as `compileOnly` dependency +2. Implement `PluginConnector` (or subset of SPI) +3. Implement capability providers (`CdcProvider`, `SnapshotProvider`, `DestinationWriterProvider`) +4. Create `META-INF/MANIFEST.MF` with `Plugin-Connector-Class` +5. Package as JAR with dependencies in `lib/` +6. Install via `POST /api/plugins/install` or drop in plugin directory + +## Example + +```java +public class MyDatabaseConnector implements PluginConnector { + @Override + public ConnectorType type() { + return ConnectorType.GENERIC_JDBC; + } + + @Override + public ConnectorCapabilities capabilities() { + return new ConnectorCapabilities(false, true, true, false, false); + // CDC=no, Snapshot=yes, Metadata=yes, Transactions=no, Offsets=no + } + + @Override + public void connect(ConnectionConfiguration config) { + // Establish connection + } + + // ... other SPI methods +} +``` diff --git a/wiki/Quick-Start.md b/wiki/Quick-Start.md new file mode 100644 index 0000000..3ddd803 --- /dev/null +++ b/wiki/Quick-Start.md @@ -0,0 +1,84 @@ +# Quick Start + +## Prerequisites + +- Java 25 (JDK with `--enable-preview` support) +- Docker and Docker Compose +- PostgreSQL 16+ (or use the Docker Compose service) + +## 1. Clone and Build + +```bash +git clone https://github.com/lekhrocks/syncflow.git +cd syncflow +./gradlew clean build +``` + +## 2. Start Infrastructure + +```bash +docker compose -f docker/docker-compose.yml up -d postgres +``` + +This starts PostgreSQL on port 5432 with the `syncflow` database. + +## 3. Run the Application + +```bash +./gradlew :syncflow-api:bootRun +``` + +The API starts on port 8080. Default admin credentials: + +- **Username:** `admin` +- **Password:** `admin-test-password` + +## 4. Verify + +```bash +# Health check +curl http://localhost:8080/api/health + +# Login +curl -X POST http://localhost:8080/api/auth/login \ + -H "Content-Type: application/json" \ + -d '{"username":"admin","password":"admin-test-password"}' +``` + +## 5. Swagger UI + +Open http://localhost:8080/swagger-ui.html for interactive API exploration. + +--- + +## Full Stack (Optional) + +Start all services including MySQL, MongoDB, Kafka, and monitoring: + +```bash +docker compose -f docker/docker-compose.yml up --build +``` + +Services: + +| Service | Port | Profile | +|---------|------|---------| +| PostgreSQL | 5432 | always | +| MySQL | 3306 | mysql | +| MongoDB | 27017 | mongodb | +| Redis | 6379 | redis | +| Kafka | 9092 | kafka | +| Prometheus | 9090 | monitoring | +| Grafana | 3000 | monitoring | + +--- + +## Gradle Tasks + +| Task | Description | +|------|-------------| +| `./gradlew verify` | Format check + compile + unit tests | +| `./gradlew integrationTest` | Testcontainers integration tests (needs Docker) | +| `./gradlew smokeTest` | Docker up, migrate, integration tests, tear down | +| `./gradlew e2eTest` | Full end-to-end with all services | +| `./gradlew benchmark` | JMH microbenchmarks | diff --git a/wiki/Roadmap.md b/wiki/Roadmap.md new file mode 100644 index 0000000..314ab54 --- /dev/null +++ b/wiki/Roadmap.md @@ -0,0 +1,85 @@ +# Roadmap + +Feature status as of 2026-09-06. Tracked in `ARCHITECTURE_ANALYSIS.md`. + +## Completed + +### P0 — Security/Correctness + +| ID | Feature | Status | +|----|---------|--------| +| F1 | Replace ThreadLocal tenant context with explicit passing | ✅ Done | +| F2 | Fix multi-table CDC processing in SyncOrchestrator | ✅ Done | +| F3 | Implement DELETE in DestinationRouter / writers | ✅ Done | +| F4 | Persist runtime state to DB (not in-memory maps) | ✅ Done | +| F5 | Add connection pooling to writers | ✅ Done | + +### P1 — Scalability/Performance + +| ID | Feature | Status | +|----|---------|--------| +| F6 | Batch writes in DestinationRouter | ✅ Done | +| F7 | Add circuit breaker + backpressure to event queues | ✅ Done | +| F8 | Make all hardcoded constants configurable | ✅ Done | +| F9 | Unify ValidationResult and ProcessingContext | ✅ Done | +| F10 | Extract ConnectionConfiguration mapper | ✅ Done | + +### P2 — Architecture + +| ID | Feature | Status | +|----|---------|--------| +| F11 | Create syncflow-runtime module | Deferred | +| F12 | Create syncflow-persistence module | ✅ Done | +| F13 | Implement distributed locking | ✅ Done | +| F14 | Add exactly-once CDC | ✅ Done | +| F15 | Parallel snapshot (PK-range chunking) | ✅ Done | + +### P3 — Platform + +| ID | Feature | Status | +|----|---------|--------| +| F16 | Migrate to reactive or structured concurrency | ✅ Done | + +--- + +## In Progress + +| ID | Feature | Notes | +|----|---------|-------| +| P4 | Jackson ObjectMapper reuse | ✅ Done (PR #73) | + +--- + +## Upcoming + +### P3 — Platform + +| ID | Feature | Priority | +|----|---------|----------| +| F17 | Multi-region / geo-replication support | High | +| F18 | Implement connector plugin system (dynamic loading) | Medium | +| F19 | Add SQL-based transformation engine (push down to DB) | Medium | + +### Deferred + +| ID | Feature | Reason | +|----|---------|--------| +| F11 | Create syncflow-runtime module | Single consumer today; 490+ tests at risk | +| Reactive/WebFlux path | Equivalent to virtual threads; revisit if backpressure needed | +| ADR-006 | pgvector for semantic search | Wait until doc count > 1000 | + +--- + +## SLOs + +Defined in `docs/sre/slo.md`: + +| Metric | Target | +|--------|--------| +| API availability | 99.95% | +| Agent heartbeat | 99.9% | +| REST P95 latency | < 250ms | +| CDC lag | < 10s | +| Sync lag | < 60s | +| RTO | < 5min | +| RPO | < 30s | diff --git a/wiki/Runbooks.md b/wiki/Runbooks.md new file mode 100644 index 0000000..0013de2 --- /dev/null +++ b/wiki/Runbooks.md @@ -0,0 +1,174 @@ +# Runbooks + +Troubleshooting guides for common failure scenarios. Full runbooks in `docs/runbooks/`. + +## Pipeline Failure + +**Symptoms:** Pipeline status = FAILED, events not flowing + +**Diagnosis:** +```bash +# Check pipeline status +curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/pipelines/{id} + +# Check DLQ +curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/dashboard/errors + +# Check logs +kubectl logs -l app=syncflow --tail=100 | grep -i "pipeline.*error" +``` + +**Resolution:** +1. Verify source/destination connections are healthy +2. Check for schema changes in source database +3. Review DLQ for failed events +4. Restart capture: `POST /api/pipelines/{id}/capture/stop` then `start` + +## CDC Lag + +**Symptoms:** CDC lag > 30 seconds, events delayed + +**Diagnosis:** +```bash +# Check CDC metrics +curl http://localhost:8080/api/diagnostics/connectors + +# Check capture status +curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/pipelines/{id}/capture/status +``` + +**Resolution:** +1. Increase `syncflow.runtime.sync.batch-size` +2. Check destination write latency +3. Verify network between source and SyncFlow +4. Consider adding Kafka transport for cross-pod distribution + +## Checkpoint Corruption + +**Symptoms:** Snapshot resumes from wrong position, duplicate data + +**Diagnosis:** +```bash +# Check snapshot checkpoints +psql -d syncflow -c "SELECT * FROM snapshot_checkpoints WHERE pipeline_id = '{id}'" +``` + +**Resolution:** +1. Delete corrupted checkpoints: `DELETE FROM snapshot_checkpoints WHERE pipeline_id = '{id}'` +2. Restart snapshot from beginning +3. Verify `chunk_index` values are sequential + +## Agent Offline + +**Symptoms:** Agent not sending heartbeats, jobs failing + +**Diagnosis:** +```bash +# Check agent status +curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/agents + +# Check agent logs +kubectl logs -l app=syncflow-agent --tail=100 +``` + +**Resolution:** +1. Verify agent can reach control plane URL +2. Check agent token is valid +3. Restart agent pod +4. Drain agent before shutdown: `POST /api/agents/{id}/drain` + +## Out of Memory + +**Symptoms:** OOMKilled, heap exhaustion + +**Diagnosis:** +```bash +# Check JVM metrics +curl http://localhost:8080/api/diagnostics/system + +# Check heap usage +curl http://localhost:9090/actuator/metrics/jvm.memory.used +``` + +**Resolution:** +1. Increase `-Xmx` in JVM args +2. Reduce `syncflow.runtime.sync.queue-capacity` +3. Reduce `syncflow.runtime.snapshot.parallelism` +4. Check for memory leaks in connector clones + +## Database Full + +**Symptoms:** Write failures, Flyway migration errors + +**Diagnosis:** +```bash +# Check database size +psql -d syncflow -c "SELECT pg_size_pretty(pg_database_size('syncflow'))" + +# Check table sizes +psql -d syncflow -c "SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) FROM pg_tables WHERE schemaname='public' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC" +``` + +**Resolution:** +1. Clean DLQ: `DELETE FROM dead_letter_events WHERE created_at < NOW() - INTERVAL '7 days'` +2. Clean processed events: `DELETE FROM processed_events WHERE created_at < NOW() - INTERVAL '30 days'` +3. Vacuum: `VACUUM FULL ANALYZE` +4. Add storage or archival policy + +## Slow Sync + +**Symptoms:** Sync latency > 60 seconds + +**Diagnosis:** +```bash +# Check sync metrics +curl http://localhost:8080/api/dashboard/metrics + +# Check writer performance +curl http://localhost:8080/api/diagnostics/connectors +``` + +**Resolution:** +1. Increase `syncflow.runtime.sync.batch-size` +2. Check destination database performance +3. Verify network latency +4. Consider connection pooling (already uses HikariCP) +5. Check for lock contention in destination + +## High Retry Count + +**Symptoms:** Events retrying frequently, approaching DLQ threshold + +**Diagnosis:** +```bash +# Check retry metrics +curl http://localhost:8080/api/dashboard/errors + +# Check specific pipeline errors +curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/pipelines/{id}/capture/status +``` + +**Resolution:** +1. Identify root cause from error messages +2. Fix transient issues (network, locks) +3. Adjust `syncflow.runtime.retry.max-attempts` if needed +4. Review DLQ for permanent failures + +## High DLQ Depth + +**Symptoms:** DLQ growing, events not processing + +**Diagnosis:** +```bash +# Check DLQ count +curl http://localhost:8080/api/dashboard/errors + +# List DLQ events +curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/dead-letter?pipelineId={id} +``` + +**Resolution:** +1. Fix root cause of failures +2. Replay resolved events: `POST /api/dead-letter/{id}/replay` +3. Purge old events: `DELETE /api/dead-letter?olderThan=7d` +4. Set up alerts for DLQ depth threshold diff --git a/wiki/Security.md b/wiki/Security.md new file mode 100644 index 0000000..e542c27 --- /dev/null +++ b/wiki/Security.md @@ -0,0 +1,121 @@ +# Security + +## Authentication + +### JWT Tokens + +```bash +POST /api/auth/login +{ + "username": "admin", + "password": "admin-test-password" +} +``` + +Response: +```json +{ + "token": "eyJhbGciOi...", + "mustChangePassword": false +} +``` + +Include in subsequent requests: +``` +Authorization: Bearer eyJhbGciOi... +``` + +### Configuration + +| Variable | Description | Default | +|----------|-------------|---------| +| `SYNCFLOW_JWT_SECRET` | HMAC signing secret (>= 32 bytes) | *(required)* | +| `SYNCFLOW_JWT_ISSUER` | Token issuer claim | `syncflow` | +| `SYNCFLOW_JWT_EXPIRY_MINUTES` | Token lifetime | `60` | + +### Password Policy + +- BCrypt hashing +- `mustChangePassword` flag for first login +- `POST /api/auth/change-password` for password rotation + +## Authorization (RBAC) + +Roles and permissions managed via `AuthorizationService`: + +| Permission | Scope | +|------------|-------| +| `CONNECTION_READ` | View connections | +| `CONNECTION_WRITE` | Create/update connections | +| `CONNECTION_DELETE` | Delete connections | +| `PIPELINE_READ` | View pipelines | +| `PIPELINE_WRITE` | Create/update pipelines | +| `PIPELINE_DELETE` | Delete pipelines | +| `PIPELINE_EXECUTE` | Start snapshots/CDC | +| `ORG_READ` | View organization | +| `ORG_WRITE` | Manage organization | +| `AI_USE` | Access AI copilot | +| `AUDIT_READ` | View audit records | +| `APIKEY_REVOKE` | Revoke API keys | +| `EXECUTION_READ` | View execution logs | + +## Credential Encryption + +All stored credentials are encrypted with AES-256: + +```bash +SYNCFLOW_ENCRYPTION_KEY= +``` + +The `EncryptionService` handles encrypt/decrypt transparently. Credentials are never stored in plaintext. + +## API Keys + +```bash +# Issue API key +POST /api/admin/apikeys +{ + "name": "ci-pipeline", + "scope": "PIPELINE_READ,PIPELINE_EXECUTE", + "expiresAt": "2026-12-31T23:59:59Z" +} + +# Use API key +X-Api-Key: sf_key_abc123... + +# Revoke +DELETE /api/admin/apikeys/{id} +``` + +Keys are hashed (SHA-256) before storage. Only the prefix is returned on creation. + +## Agent Authentication + +Agents authenticate via token header: + +``` +X-Agent-Token: +``` + +Agent tokens are validated by `AgentTokenFilter` before reaching controllers. + +## Public Paths + +These endpoints do not require authentication: + +``` +/api/health/** +/api/auth/** +/api/agents/register +/api/agents/heartbeat +/actuator/** +/v3/api-docs/** +/swagger-ui/** +/graphiql/** +``` + +## Network Security + +- Kubernetes `NetworkPolicy` manifests restrict pod-to-pod communication +- TLS termination at ingress (NGINX) +- No sensitive data in logs (credentials masked) diff --git a/wiki/Snapshots.md b/wiki/Snapshots.md new file mode 100644 index 0000000..f613621 --- /dev/null +++ b/wiki/Snapshots.md @@ -0,0 +1,73 @@ +# Snapshots + +A snapshot performs the initial bulk load of data from a source to a destination. It uses parallel PK-range chunking for high throughput and checkpoints for resume-on-failure. + +## How It Works + +``` +1. Estimate total rows across all mapped tables +2. Split each table into PK-range chunks (via rangeChunks()) +3. Fork one StructuredTaskScope task per chunk +4. Each task: + a. Reads a batch of rows (keyset pagination) + b. Applies filter/transform pipeline + c. Writes to destination (serialized on writerLock) + d. Checkpoints every N batches + e. Publishes progress via SSE +5. On completion: flush + commit (or rollback on cancel) +6. Durable state in snapshot_jobs table +``` + +## API Endpoints + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/pipelines/{id}/snapshot` | Start snapshot | +| `GET` | `/api/snapshots` | List snapshot jobs | +| `GET` | `/api/snapshots/{id}` | Get snapshot job | +| `GET` | `/api/snapshots/{id}/progress` | Get progress | +| `GET` | `/api/snapshots/{id}/events` | SSE live progress stream | +| `POST` | `/api/snapshots/{id}/cancel` | Cancel snapshot | + +## Parallel Execution + +Configured via `syncflow.runtime.snapshot`: + +| Property | Default | Description | +|----------|---------|-------------| +| `parallelism` | 4 | Max concurrent chunk workers | +| `max-chunks` | 64 | Max chunks per table | +| `checkpoint-interval-batches` | 5 | Checkpoint frequency | +| `progress-publish-interval-batches` | 10 | SSE progress frequency | + +Each worker gets its own connector clone (JDBC connections are not thread-safe). A 64-chunk table with 4 parallelism opens only 4 DB connections. + +## Checkpoint and Resume + +Every N batches, the snapshot persists a checkpoint: + +``` +SnapshotCheckpoint + ├── pipelineId + ├── sourceTable + ├── chunkIndex + ├── lastBatchNumber + ├── rowsProcessed + └── cursor (keyset pagination cursor) +``` + +On restart, each chunk resumes from its last checkpoint cursor instead of re-reading from the start. + +## Cancellation + +`POST /api/snapshots/{id}/cancel` sets a cancellation flag. The snapshot: + +1. Checks the flag before each batch +2. Under `progressLock`, rolls back partial writes +3. Sets terminal status to CANCELLED (not FAILED) + +This prevents duplicate rows on resume. + +## Distributed Locking + +`SnapshotExecutor.start()` acquires a Postgres advisory lock (`snapshot:{pipelineId}`) before starting. Two pods cannot start the same pipeline's snapshot concurrently.