Skip to content

docs: add wiki pages for GitHub wiki #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lekhrocks merged 1 commit into from
Sep 6, 2026
Merged

docs: add wiki pages for GitHub wiki #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions wiki/AI-Copilot.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# AI Copilot

SyncFlow includes an LLM-powered AI copilot for pipeline assistance, review, and optimization.

## Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/ai/chat` | Chat with AI copilot |
| `POST` | `/api/ai/plan` | Create reasoning plan |
| `POST` | `/api/ai/analyze` | Multi-agent analysis |
| `POST` | `/api/ai/document` | Search knowledge base |
| `POST` | `/api/ai/review` | AI pipeline review |
| `POST` | `/api/ai/recommend` | Optimization recommendations |
| `GET` | `/api/ai/history` | Conversation history |

## Features

### Chat

Conversational interface for pipeline questions:

```json
POST /api/ai/chat
{
"message": "How do I set up CDC from PostgreSQL to MySQL?",
"sessionId": "optional-session-id"
}
```

### Pipeline Review

AI analyzes a pipeline design and suggests improvements:

```json
POST /api/ai/review
{
"pipelineId": "pipeline-uuid"
}
```

Review covers:
- Column type compatibility
- Missing transformations
- Performance bottlenecks
- Security concerns
- Best practices

### Optimization Recommendations

```json
POST /api/ai/recommend
{
"pipelineId": "pipeline-uuid",
"metrics": {
"throughput": 1000,
"latency": 50,
"errorRate": 0.01
}
}
```

### Knowledge Base Search

Semantic search across pipeline documentation:

```json
POST /api/ai/document
{
"query": "how to handle schema changes in CDC"
}
```

## Configuration

```yaml
syncflow:
ai:
endpoint: ${SYNCFLOW_AI_ENDPOINT:https://api.openai.com/v1/chat/completions}
model: ${SYNCFLOW_AI_MODEL:gpt-4o}
api-key: ${SYNCFLOW_AI_API_KEY:}
max-tokens: 4096
temperature: 0.3
```

## Authentication

AI endpoints require `AI_USE` permission. Check via `AuthorizationService`.

## Privacy

- Prompts are not stored permanently
- Conversation history is session-scoped
- No sensitive data (credentials, tokens) included in prompts
174 changes: 174 additions & 0 deletions wiki/API-Reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
# API Reference

Base URL: `http://localhost:8080`

All endpoints except auth require `Authorization: Bearer <jwt-token>` header.

Swagger UI: `http://localhost:8080/swagger-ui.html`
OpenAPI docs: `http://localhost:8080/v3/api-docs`

---

## Authentication

| Method | Path | Description | Auth |
|--------|------|-------------|------|
| `POST` | `/api/auth/login` | Login (returns JWT) | No |
| `POST` | `/api/auth/change-password` | Change password | Yes |
| `GET` | `/api/auth/me` | Current user info | Yes |

---

## Connections

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/connections` | Create connection |
| `GET` | `/api/connections` | List connections |
| `GET` | `/api/connections/{id}` | Get connection |
| `PUT` | `/api/connections/{id}` | Update connection |
| `DELETE` | `/api/connections/{id}` | Delete connection |
| `POST` | `/api/connections/test` | Test connection |
| `GET` | `/api/connections/{id}/health` | Health check |

---

## Metadata Discovery

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/connections/{id}/metadata` | Discover schemas |
| `GET` | `/api/connections/{id}/schemas/{schema}/tables` | Discover tables |
| `GET` | `/api/connections/{id}/schemas/{schema}/tables/{table}/columns` | Discover columns |
| `GET` | `/api/connections/{id}/schemas/{schema}/tables/{table}/indexes` | Discover indexes |
| `GET` | `/api/connections/{id}/schemas/{schema}/tables/{table}/constraints` | Discover constraints |
| `POST` | `/api/connections/{id}/metadata/refresh` | Refresh cache |

---

## Pipelines

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/pipelines` | Create pipeline |
| `GET` | `/api/pipelines` | List pipelines |
| `GET` | `/api/pipelines/{id}` | Get pipeline |
| `PUT` | `/api/pipelines/{id}` | Update pipeline |
| `DELETE` | `/api/pipelines/{id}` | Delete pipeline |
| `POST` | `/api/pipelines/{id}/validate` | Validate pipeline |
| `POST` | `/api/pipelines/{id}/rollback` | Rollback to version |
| `GET` | `/api/pipelines/{id}/versions` | List versions |
| `GET` | `/api/pipelines/{id}/preview` | Preview output |
| `GET` | `/api/pipelines/{id}/conflicts` | Detect conflicts |

---

## Snapshots

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/pipelines/{id}/snapshot` | Start snapshot |
| `GET` | `/api/snapshots` | List snapshots |
| `GET` | `/api/snapshots/{id}` | Get snapshot |
| `GET` | `/api/snapshots/{id}/progress` | Get progress |
| `GET` | `/api/snapshots/{id}/events` | SSE progress stream |
| `POST` | `/api/snapshots/{id}/cancel` | Cancel snapshot |

---

## CDC Capture

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/pipelines/{id}/capture/start` | Start CDC |
| `POST` | `/api/pipelines/{id}/capture/stop` | Stop CDC |
| `POST` | `/api/pipelines/{id}/capture/pause` | Pause CDC |
| `POST` | `/api/pipelines/{id}/capture/resume` | Resume CDC |
| `GET` | `/api/pipelines/{id}/capture/status` | Capture status |

---

## Agent Fleet

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/agents/register` | Register agent |
| `POST` | `/api/agents/heartbeat` | Agent heartbeat |
| `GET` | `/api/agents` | List agents |
| `GET` | `/api/agents/{id}` | Get agent |
| `POST` | `/api/agents/{id}/drain` | Drain agent |
| `POST` | `/api/agents/{id}/restart` | Restart agent |
| `GET` | `/api/agents/{id}/metrics` | Agent metrics |

---

## Plugins

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/plugins` | List plugins |
| `GET` | `/api/plugins/{id}` | Get plugin |
| `POST` | `/api/plugins/install` | Install plugin |
| `POST` | `/api/plugins/{id}/enable` | Enable plugin |
| `POST` | `/api/plugins/{id}/disable` | Disable plugin |
| `DELETE` | `/api/plugins/{id}` | Uninstall plugin |
| `GET` | `/api/plugins/{id}/capabilities` | Plugin capabilities |

---

## AI Copilot

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/ai/chat` | Chat with copilot |
| `POST` | `/api/ai/plan` | Create reasoning plan |
| `POST` | `/api/ai/analyze` | Multi-agent analysis |
| `POST` | `/api/ai/document` | Search knowledge base |
| `POST` | `/api/ai/review` | Pipeline review |
| `POST` | `/api/ai/recommend` | Optimization recommendations |
| `GET` | `/api/ai/history` | Conversation history |

---

## Admin & Multi-Tenancy

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/admin/organizations` | Create organization |
| `POST` | `/api/admin/workspaces` | Create workspace |
| `POST` | `/api/admin/projects` | Create project |
| `POST` | `/api/admin/apikeys` | Issue API key |
| `DELETE` | `/api/admin/apikeys/{id}` | Revoke API key |
| `GET` | `/api/admin/quotas` | Get tenant quota |
| `GET` | `/api/admin/audit` | List audit records |
| `GET` | `/api/admin/tenants` | Current tenant context |

---

## Dashboard & Diagnostics

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/dashboard/overview` | Dashboard overview |
| `GET` | `/api/dashboard/pipelines` | Pipeline dashboard |
| `GET` | `/api/dashboard/connections` | Connection dashboard |
| `GET` | `/api/dashboard/connectors` | Connector dashboard |
| `GET` | `/api/dashboard/jobs` | Job dashboard |
| `GET` | `/api/dashboard/metrics` | Metrics dashboard |
| `GET` | `/api/dashboard/errors` | Error dashboard |
| `GET` | `/api/diagnostics/system` | System diagnostics |
| `GET` | `/api/diagnostics/connectors` | Connector diagnostics |
| `GET` | `/api/diagnostics/pipelines` | Pipeline diagnostics |
| `GET` | `/api/diagnostics/executions` | Execution diagnostics |

---

## Health & Operations

| Method | Path | Description | Auth |
|--------|------|-------------|------|
| `GET` | `/api/health` | Health check | No |
| `GET` | `/actuator/health` | Spring health | No |
| `GET` | `/actuator/prometheus` | Prometheus metrics | No |
| `GET` | `/actuator/metrics` | Micrometer metrics | No |
| `POST` | `/actuator/shutdown` | Graceful shutdown | No |
92 changes: 92 additions & 0 deletions wiki/Agent-Data-Plane.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Agent / Data Plane

The agent module is a standalone Spring Boot application that runs in the data plane, executing snapshot and CDC operations close to the databases.

## Architecture

```
Control Plane (syncflow-api :8080)
|
+-- POST /api/agents/register (agent registers on startup)
+-- POST /api/agents/heartbeat (every 15s, with HW metrics)
+-- GET /api/agents (list all agents)
+-- POST /api/agents/{id}/drain (drain agent)
+-- POST /api/agents/{id}/restart (restart agent)
|
Data Plane (syncflow-agent :9090)
├── AgentRegistrar (registers with control plane)
└── HeartbeatSender (sends hardware metrics)
```

## Running the Agent

```bash
# Build
./gradlew :syncflow-agent:bootRun

# Or with Docker
docker build -f docker/Dockerfile.agent -t syncflow-agent .
docker run -e SYNCFLOW_AGENT_CONTROL_PLANE=http://control-plane:8080 syncflow-agent
```

## Registration

On startup, the agent registers with the control plane:

```json
POST /api/agents/register
{
"version": "0.1.0",
"capabilities": ["SNAPSHOT", "CDC", "SYNCHRONIZATION", "METADATA"],
"labels": {"type": "standard"},
"environment": "customer",
"region": "us-east-1",
"hostname": "agent-1.example.com"
}
```

## Heartbeat

Every 15 seconds, the agent sends hardware metrics:

```json
POST /api/agents/heartbeat
{
"agentId": "agent-uuid",
"cpuPercent": 45.2,
"memoryUsed": 2147483648,
"memoryTotal": 4294967296,
"runningJobs": 2
}
```

## Agent States

| State | Description |
|-------|-------------|
| `ONLINE` | Agent is registered and sending heartbeats |
| `DRAINING` | Agent is finishing current jobs, not accepting new ones |
| `OFFLINE` | Agent missed heartbeats (detected by control plane) |

## Fleet Management

`FleetManager` in the control plane manages the agent fleet:

- Tracks online/offline status
- Assigns jobs to agents based on capability and load
- Handles agent drain (move jobs before shutdown)
- Monitors hardware metrics for capacity planning

## Configuration

| Variable | Description | Default |
|----------|-------------|---------|
| `SYNCFLOW_AGENT_CONTROL_PLANE` | Control plane URL | `http://localhost:8080` |
| `SYNCFLOW_AGENT_VERSION` | Agent version | `0.1.0` |

## Use Cases

1. **Remote Execution** — Run snapshots/CDC on machines close to source databases
2. **Load Distribution** — Spread workload across multiple agents
3. **Network Isolation** — Agents in VPC with database access; control plane in DMZ
4. **Edge Deployment** — Agents at edge locations syncing to central control plane
Loading
Loading