Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 42 additions & 17 deletions packages/spec/docs/SYNC_ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,10 +210,29 @@ Complete, production-grade integration with external systems. Includes authentic

### Example

> **`ConnectorInput` is the AUTHOR shape.** It is `z.input` of
> `ConnectorSchema`, so every key carrying a `.default()` — `enabled`,
> `status`, `connectionTimeoutMs`, `requestTimeoutMs`, all of `syncConfig`'s
> `strategy` / `direction` / `realtimeSync` / `conflictResolution` /
> `batchSize` / `deleteMode`, a mapping's `required` / `syncMode`, a webhook's
> `method` / `timeoutMs` / `isActive` / `signatureAlgorithm` — is optional when
> you write a connector, and `syncConfig.schedule` takes the bare cron string
> the schema wraps for you. Annotate the **result** of
> `ConnectorSchema.parse(…)` with the bare **`Connector`**, which is `z.infer`:
> there those keys are all present and `schedule` is already the
> `{ dialect: 'cron', source }` envelope. Note the asymmetry with L2 above,
> where the bare `ETLPipeline` *is* the author shape and the parse result is
> `ETLPipelineParsed` — `integration/connector.zod.ts` has not been moved onto
> that house convention yet (#5551). The example below states the defaulted
> keys anyway, because it is a tour of the surface; the Migration Guide's
> sketches omit them, because that is what ordinary authoring looks like.
> To have the literal validated as you write it, prefer `defineConnector(…)`,
> which takes this same input shape and returns the parsed one.

```typescript
import { Connector } from '@objectstack/spec/integration';
import type { ConnectorInput } from '@objectstack/spec/integration';

const sapConnector: Connector = {
const sapConnector: ConnectorInput = {
name: 'sap_erp_connector',
label: 'SAP ERP Integration',
type: 'saas',
Expand Down Expand Up @@ -241,22 +260,28 @@ const sapConnector: Connector = {
deleteMode: 'soft_delete'
},

// Field Mappings with Transformations
// Field Mappings with Transformations.
// The keys are `source` / `target` — the canonical spelling of the base
// protocol in `shared/mapping.zod.ts`, which every mapping surface extends.
fieldMappings: [
{
sourceField: 'customer_number',
targetField: 'customer_id',
source: 'customer_number',
target: 'customer_id',
dataType: 'string',
required: true,
syncMode: 'bidirectional'
},
{
sourceField: 'order_value',
targetField: 'order_total',
source: 'order_value',
target: 'order_total',
dataType: 'number',
// `transform.type` is a discriminated union with exactly five members:
// `constant` / `cast` / `lookup` / `javascript` / `map`. The bare string
// below is `ExpressionInput` shorthand — the schema wraps it into an
// `{ dialect, source }` envelope on parse.
transform: {
type: 'custom',
function: 'value => parseFloat(value) / 100' // Convert cents to dollars
type: 'javascript',
expression: 'value / 100' // Convert cents to dollars
},
syncMode: 'bidirectional'
}
Expand All @@ -270,11 +295,11 @@ const sapConnector: Connector = {
events: ['record.created', 'record.updated'],
secret: process.env.WEBHOOK_SECRET!,
signatureAlgorithm: 'hmac_sha256',
retryPolicy: {
maxRetries: 3,
backoffStrategy: 'exponential',
initialDelayMs: 1000
},
// (`retryPolicy` sat here until #3494 retired it — webhook delivery
// retries are owned by the messaging outbox on a fixed schedule, and the
// authored policy was never read. There is no replacement, and it is a
// different thing from `retryConfig` below, which governs the calls this
// connector MAKES.)
timeoutMs: 30000,
isActive: true
}
Expand All @@ -283,7 +308,7 @@ const sapConnector: Connector = {
// (`rateLimitConfig` sat here until #4911 retired it — no outbound
// rate-limiting engine ever existed. Throttle at the provider/gateway.)

// Retry Configuration
// Retry Configuration — for the connector's own outbound requests
retryConfig: {
strategy: 'exponential_backoff',
maxAttempts: 5,
Expand Down Expand Up @@ -378,7 +403,7 @@ When a connector's declarative sync needs complex transformations:

**Before (L3 `syncConfig`):**
```typescript
const connector: Connector = {
const connector: ConnectorInput = {
name: 'orders',
type: 'saas',
authentication: { type: 'api-key', ... },
Expand Down Expand Up @@ -423,7 +448,7 @@ const pipeline: ETLPipeline = {

**After (L3):**
```typescript
const connector: Connector = {
const connector: ConnectorInput = {
authentication: { type: 'oauth2', ... },
webhooks: [...],
retryConfig: { ... }
Expand Down
20 changes: 11 additions & 9 deletions packages/spec/src/automation/etl-author-shape.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,15 +113,17 @@ describe('[#4963] SYNC_ARCHITECTURE.md pipeline examples compile', () => {
// assertion below pass over an empty program — the way a gate goes dormant.
expect(pipelineBlocks.length, 'ETLPipeline examples in SYNC_ARCHITECTURE.md').toBe(3);
// The other three are the L3 `Connector` examples, out of this gate's scope
// because they belong to `integration/connector.zod.ts`. Two of them are
// Migration-Guide sketches that elide with a bare `...`, which is not
// TypeScript. The third — the full `sapConnector` example — is NOT exempt on
// its merits: run through this same harness it reports four diagnostics, and
// three of them are keys or values the schema REJECTS (`sourceField` /
// `targetField` for `source` / `target`, `transform.type: 'custom'`,
// `webhooks[].retryPolicy`). That is filed as #5515, not fixed here, because
// the fourth diagnostic is `Connector` being `z.infer` — this issue's twin
// on a file whose migration surface is NOT empty, so it needs its own ruling.
// because they belong to `integration/connector.zod.ts` — and covered, since
// #5515, by that file's own gate: `integration/connector-author-shape.test.ts`
// classifies the same three (two Migration-Guide sketches that elide with a
// bare `...`, which is not TypeScript; one full `sapConnector` example) and
// compiles the third. When this pin was written that example reported four
// diagnostics, three of them keys or values the schema REJECTS (`sourceField`
// / `targetField` for `source` / `target`, `transform.type: 'custom'`,
// `webhooks[].retryPolicy`); those are fixed in the document. The fourth was
// `Connector` being `z.infer` — this issue's twin on a file whose migration
// surface is NOT empty — and it is solved there by annotating the example
// with `ConnectorInput`; the alias flip itself is still open as #5551.
// The total is pinned rather than left open so that ADDING a block to this
// document is a decision someone has to make on purpose: a new ETL example
// is picked up automatically by the selector above, and anything else
Expand Down
Loading
Loading