feat(service-datasource-admin): open-source the runtime datasource admin mechanism (ADR-0015 Addendum)

.objectui-sha

@@ -1 +1 @@
-1508a8d32b881335f5820dd47fbb91470c13cbad
+fdd083657e2da9832059492d4c88e818a5990a8d

packages/services/service-datasource-admin/README.md

@@ -0,0 +1,85 @@
+# @objectstack/service-datasource-admin
+
+Runtime **UI-created datasource lifecycle** — the "Add Datasource" wizard backend
+from **ADR-0015 Addendum**. This package is the open-source **mechanism**:
+list / test-connection / create / update / remove datasources that an admin
+defines *in the UI* at runtime (as opposed to code-defined `*.datasource.ts`).
+
+It deliberately ships **no managed credential vault and no multi-tenant overlay**.
+Both are injected through stable seams, so a private host can layer enterprise
+behaviour on without forking:
+
+- **credentials** → a host-provided `SecretBinder` over any `ICryptoProvider`.
+  The framework default uses `InMemoryCryptoProvider` (dev / self-host, single
+  node). A managed host swaps in a KMS/Vault-backed `ICryptoProvider` for
+  rotation, per-tenant isolation and compliance.
+- **drivers** → a swappable `IDatasourceDriverFactory`. The default factory
+  covers `postgres` / `sqlite` / `mongodb` / `memory`; a host can register a
+  factory that adds premium drivers (Salesforce / SAP / Oracle / …).
+
+The tier line therefore falls on *which `ICryptoProvider` / driver factory you
+inject* — a neutral, technical seam — not on whether the UI can manage
+datasources at all.
+
+## Scope
+
+Two orthogonal axes of "datasource":
+
+| Axis | `origin: 'code'` | `origin: 'runtime'` |
+|------|------------------|---------------------|
+| Defined by | `*.datasource.ts` (GitOps) | the UI wizard, at runtime |
+| Mutable at runtime | no (read-only) | yes |
+| Stored in | code / artefacts | `sys_metadata` + secret in `sys_secret` |
+
+**This package owns only the `origin: 'runtime'` lifecycle.** *Federation*
+(introspect / draft / import / validate of external tables, ADR-0015 main body)
+lives in `@objectstack/service-external-datasource`.
+
+## Contents
+
+- `DatasourceAdminService` + `DatasourceAdminServicePlugin` — registers the
+  `'datasource-admin'` kernel service.
+- `createDefaultDatasourceDriverFactory` — postgres / sqlite / mongodb / memory
+  factory used to probe a connection and hot-register a pool.
+- `createDatasourceSecretBinder` — fail-closed `sys_secret` binder over an
+  `ICryptoProvider`; only an opaque `credentialsRef` is ever persisted, never
+  cleartext. With no binder wired, secret-bearing create/update throws.
+- `registerDatasourceAdminRoutes` — REST routes under `/api/v1/datasources`.
+- `contracts/` — `IDatasourceAdminService`, `IDatasourceDriverFactory` + DTOs.
+
+## Host wiring
+
+```ts
+import {
+  DatasourceAdminServicePlugin,
+  createDefaultDatasourceDriverFactory,
+  createDatasourceSecretBinder,
+  registerDatasourceAdminRoutes,
+} from '@objectstack/service-datasource-admin';
+import { InMemoryCryptoProvider } from '@objectstack/service-settings';
+
+// 1. secret binder (fail-closed: no crypto provider ⇒ secret-bearing
+//    create/update throws instead of persisting cleartext).
+//    A managed host swaps InMemoryCryptoProvider for a KMS/Vault provider.
+const cryptoProvider = new InMemoryCryptoProvider();
+const lazyEngine = {
+  insert: (o, d, opt) => kernel.getService('data').insert(o, d, opt),
+  delete: (o, opt) => kernel.getService('data').delete(o, opt),
+  find:   (o, q)   => kernel.getService('data').find(o, q),
+};
+const secrets = createDatasourceSecretBinder({ engine: lazyEngine, cryptoProvider });
+
+// 2. plugin (driverFactory + secrets are the enterprise injection points)
+await kernel.use(
+  new DatasourceAdminServicePlugin({
+    driverFactory: createDefaultDatasourceDriverFactory(),
+    secrets,
+  }),
+);
+
+// 3. REST routes (mount alongside the federation routes)
+registerDatasourceAdminRoutes(httpServer, pluginContext, '/api/v1');
+```
+
+`@objectstack/core` and `@objectstack/spec` are required deps; the driver
+packages are optional peers (imported lazily only for the drivers you use).

packages/services/service-datasource-admin/package.json

@@ -0,0 +1,62 @@
+{
+  "name": "@objectstack/service-datasource-admin",
+  "version": "7.5.0",
+  "license": "Apache-2.0",
+  "description": "Runtime UI-created datasource lifecycle (ADR-0015 Addendum) — list/test/create/update/remove datasources defined in the UI. Mechanism only: credential storage is delegated to a host-provided SecretBinder (CryptoProvider) and drivers to a swappable factory, so a managed credential vault / multi-tenant overlay can be layered on without forking.",
+  "type": "module",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "require": "./dist/index.cjs"
+    },
+    "./contracts": {
+      "types": "./dist/contracts/index.d.ts",
+      "import": "./dist/contracts/index.js",
+      "require": "./dist/contracts/index.cjs"
+    }
+  },
+  "scripts": {
+    "build": "tsup",
+    "dev": "tsup --watch",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "typecheck": "tsc --noEmit"
+  },
+  "dependencies": {
+    "@objectstack/core": "workspace:*",
+    "@objectstack/spec": "workspace:*"
+  },
+  "peerDependencies": {
+    "@objectstack/driver-memory": "workspace:*",
+    "@objectstack/driver-mongodb": "workspace:*",
+    "@objectstack/driver-sql": "workspace:*"
+  },
+  "peerDependenciesMeta": {
+    "@objectstack/driver-sql": { "optional": true },
+    "@objectstack/driver-mongodb": { "optional": true },
+    "@objectstack/driver-memory": { "optional": true }
+  },
+  "devDependencies": {
+    "@objectstack/driver-memory": "workspace:*",
+    "@objectstack/driver-sql": "workspace:*",
+    "@types/node": "^25.9.1",
+    "tsup": "^8.5.1",
+    "typescript": "^6.0.3",
+    "vitest": "^4.1.8"
+  },
+  "keywords": [
+    "objectstack",
+    "datasource",
+    "datasource-admin",
+    "runtime-datasource"
+  ],
+  "author": "ObjectStack",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/objectstack-ai/framework.git",
+    "directory": "packages/services/service-datasource-admin"
+  }
+}

packages/services/service-datasource-admin/src/__tests__/datasource-admin-plugin.test.ts

@@ -0,0 +1,231 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+
+import { describe, it, expect } from 'vitest';
+import type { IDatasourceAdminService, IDatasourceDriverFactory } from '../contracts/index.js';
+import {
+  DatasourceAdminServicePlugin,
+  type DatasourceAdminServicePluginOptions,
+} from '../datasource-admin-plugin.js';
+
+/**
+ * Minimal PluginContext + in-memory metadata service. Boots the plugin and
+ * returns the registered `datasource-admin` service so we can exercise the
+ * plugin's glue (probe via factory, fail-closed secret) end to end.
+ */
+async function boot(opts: DatasourceAdminServicePluginOptions & {
+  services?: Record<string, unknown>;
+} = {}) {
+  const registry = new Map<string, Map<string, unknown>>();
+  const metadata = {
+    get: async (t: string, n: string) => registry.get(t)?.get(n),
+    list: async (t: string) => [...(registry.get(t)?.values() ?? [])],
+    register: async (t: string, n: string, d: unknown) => {
+      if (!registry.has(t)) registry.set(t, new Map());
+      registry.get(t)!.set(n, d);
+    },
+    unregister: async (t: string, n: string) => {
+      registry.get(t)?.delete(n);
+    },
+    listObjects: async () => [...(registry.get('object')?.values() ?? [])],
+  };
+
+  const services: Record<string, unknown> = { metadata, ...(opts.services ?? {}) };
+  let registered: IDatasourceAdminService | undefined;
+  const ctx: any = {
+    getService: (name: string) => {
+      if (name in services) return services[name];
+      throw new Error(`no service ${name}`);
+    },
+    registerService: (name: string, svc: unknown) => {
+      if (name === 'datasource-admin') registered = svc as IDatasourceAdminService;
+    },
+    trigger: async () => {},
+    logger: { warn() {}, info() {} },
+  };
+
+  const { services: _omit, ...pluginOpts } = opts;
+  const plugin = new DatasourceAdminServicePlugin(pluginOpts);
+  await plugin.init(ctx);
+  return { service: registered!, registry, metadata, plugin, ctx };
+}
+
+/** A driver factory whose handle records connect/ping/disconnect calls. */
+function fakeFactory(over?: Partial<IDatasourceDriverFactory> & { onProbe?: () => void }): IDatasourceDriverFactory {
+  return {
+    supports: (id: string) => id === 'postgres',
+    create: async (spec) => ({
+      connect: async () => {},
+      ping: async () => {
+        over?.onProbe?.();
+        // expose the secret the factory received for assertions
+        (globalThis as any).__lastProbeSecret = spec.secret;
+      },
+      disconnect: async () => {},
+      serverVersion: async () => 'PostgreSQL 16.1',
+    }),
+    ...over,
+  };
+}
+
+describe('DatasourceAdminServicePlugin: probe', () => {
+  it('tests a connection through the driver factory (latency + version)', async () => {
+    const { service } = await boot({
+      driverFactory: fakeFactory(),
+    });
+    const res = await service.testConnection(
+      { name: 'reporting', driver: 'postgres', config: { host: 'db' } },
+      { value: 's3cret' },
+    );
+    expect(res.ok).toBe(true);
+    expect(res.serverVersion).toBe('PostgreSQL 16.1');
+    expect(typeof res.latencyMs).toBe('number');
+    expect((globalThis as any).__lastProbeSecret).toBe('s3cret');
+  });
+
+  it('returns ok:false when no factory supports the driver', async () => {
+    const { service } = await boot({ driverFactory: fakeFactory() });
+    const res = await service.testConnection({ name: 'x', driver: 'oracle', config: {} });
+    expect(res.ok).toBe(false);
+    expect(res.error).toMatch(/no driver factory supports/i);
+  });
+
+  it('returns ok:false when no factory is registered at all', async () => {
+    const { service } = await boot();
+    const res = await service.testConnection({ name: 'x', driver: 'postgres', config: {} });
+    expect(res.ok).toBe(false);
+    expect(res.error).toMatch(/no driver factory is registered/i);
+  });
+});
+
+describe('DatasourceAdminServicePlugin: secret fail-closed', () => {
+  it('refuses to create a secret-bearing datasource without a secret binder', async () => {
+    const { service, registry } = await boot({ driverFactory: fakeFactory() });
+    await expect(
+      service.createDatasource({ name: 'reporting', driver: 'postgres', config: {} }, { value: 'pw' }),
+    ).rejects.toThrow(/no secret store configured/i);
+    // nothing persisted
+    expect(registry.get('datasource')?.size ?? 0).toBe(0);
+  });
+
+  it('persists a credentialsRef (not cleartext) when a binder is wired', async () => {
+    const bound: string[] = [];
+    const { service, registry } = await boot({
+      driverFactory: fakeFactory(),
+      secrets: {
+        bind: async (input, hint) => {
+          bound.push(input.value);
+          return `sys_secret://datasource/${hint.name}#1`;
+        },
+      },
+    });
+    await service.createDatasource({ name: 'reporting', driver: 'postgres', config: {} }, { value: 'pw' });
+    const rec = registry.get('datasource')?.get('reporting') as any;
+    expect(rec.origin).toBe('runtime');
+    expect(rec.external?.credentialsRef).toBe('sys_secret://datasource/reporting#1');
+    expect(JSON.stringify(rec)).not.toContain('pw');
+    expect(bound).toEqual(['pw']);
+  });
+});
+
+describe('DatasourceAdminServicePlugin: boot rehydration', () => {
+  /** Fake engine ('data') that records hot-registered drivers. */
+  function fakeEngine() {
+    const drivers: any[] = [];
+    return {
+      drivers,
+      registerDriver: (d: any) => drivers.push(d),
+      registerDatasourceDef: () => {},
+      getDriverByName: (n: string) => drivers.find((d) => d.name === n),
+    };
+  }
+
+  /** Factory that records the spec (incl. resolved secret) of each create(). */
+  function recordingFactory() {
+    const specs: any[] = [];
+    const factory: IDatasourceDriverFactory = {
+      supports: (id: string) => id === 'postgres',
+      create: async (spec) => {
+        specs.push(spec);
+        return { connect: async () => {}, disconnect: async () => {} };
+      },
+    };
+    return { factory, specs };
+  }
+
+  it('rebuilds runtime pools at start(), decrypting the credentialsRef', async () => {
+    const engine = fakeEngine();
+    const { factory, specs } = recordingFactory();
+    const resolved: string[] = [];
+
+    const { plugin, ctx, registry } = await boot({
+      driverFactory: factory,
+      services: { data: engine },
+      secrets: {
+        bind: async () => 'sys_secret:abc',
+        resolve: async (ref) => {
+          resolved.push(ref);
+          return ref === 'sys_secret:abc' ? 'super-secret-pw' : undefined;
+        },
+      },
+    });
+
+    // Simulate a persisted (DB-backed) runtime datasource that survived a restart.
+    registry.set(
+      'datasource',
+      new Map<string, unknown>([
+        ['crm_primary', { name: 'crm_primary', driver: 'sqlite', origin: 'code' }],
+        [
+          'reporting',
+          {
+            name: 'reporting',
+            driver: 'postgres',
+            origin: 'runtime',
+            active: true,
+            config: { host: 'db' },
+            external: { credentialsRef: 'sys_secret:abc' },
+          },
+        ],
+        [
+          'archived',
+          { name: 'archived', driver: 'postgres', origin: 'runtime', active: false },
+        ],
+      ]),
+    );
+
+    await plugin.start(ctx);
+
+    // Only the active runtime datasource is rehydrated — not the code one, not the inactive one.
+    expect(engine.drivers.map((d) => d.name)).toEqual(['reporting']);
+    // The credentialsRef was dereferenced and the cleartext handed to the factory.
+    expect(resolved).toEqual(['sys_secret:abc']);
+    expect(specs).toHaveLength(1);
+    expect(specs[0].secret).toBe('super-secret-pw');
+    expect(specs[0].name).toBe('reporting');
+  });
+
+  it('does not block boot when nothing is persisted (dev: in-memory store)', async () => {
+    const engine = fakeEngine();
+    const { factory } = recordingFactory();
+    const { plugin, ctx } = await boot({ driverFactory: factory, services: { data: engine } });
+    await expect(plugin.start(ctx)).resolves.toBeUndefined();
+    expect(engine.drivers).toHaveLength(0);
+  });
+});
+
+describe('DatasourceAdminServicePlugin: persistence + bound count', () => {
+  it('lists code (artefact) + runtime records with origin, blocks remove while bound', async () => {
+    const { service, registry } = await boot({ driverFactory: fakeFactory() });
+    // seed an artefact (code) datasource lacking explicit origin
+    registry.set('datasource', new Map([['crm_primary', { name: 'crm_primary', driver: 'sqlite' }]]));
+    // seed an object bound to a runtime datasource
+    registry.set('object', new Map([['lead', { name: 'lead', datasource: 'reporting' }]]));
+
+    await service.createDatasource({ name: 'reporting', driver: 'postgres', config: {} });
+
+    const list = await service.listDatasources();
+    expect(list.find((d) => d.name === 'crm_primary')?.origin).toBe('code');
+    expect(list.find((d) => d.name === 'reporting')?.origin).toBe('runtime');
+
+    await expect(service.removeDatasource('reporting')).rejects.toThrow(/1 object\(s\)/);
+  });
+});