Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .changeset/28209-skills-code-datasource-env-credential.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
'@objectstack/skills': patch
---

docs(skills): objectstack-data teaches the code-defined datasource credential route (`OS_DATASOURCE_<NAME>_PASSWORD`, behind the host switch)

Clause-②: no
Metadata-protocol: none

- **Before.** `rules/datasources.md` taught `credentialsRef` and `connector` as the only credential channels. A datasource defined in code has no `sys_secret:` handle to point at (the Setup form refuses a code-origin datasource, and nothing else mints one), so an agent following the rule wrote a reference the connect could never resolve.
- **Now.** The credential section says a code-declared datasource reads its password at connect from `OS_DATASOURCE_<NAME>_PASSWORD` (the name upper-cased) while the host switch `OS_DATASOURCE_ENV_CREDENTIALS` is `on`; that `os serve`, `os start` and `os dev` default it on, a library composition leaves it off, and the switch takes `on` or `off` only; that a declared `credentialsRef` or `connector` still wins and never falls back to the variable; and the operator's caution that an installed package may declare a datasource of any free name and so read that variable.
- The file stays under its token ceiling: the new text is paid for by deleting restatements in the same file (the field-reference pointer the references index already carries, the `writable` and `columnMap` restatements of their own sections, the `datasource` field default the SKILL.md field table carries, and two detail clauses). No ceiling moves, no frontmatter, schema or behaviour changes.
70 changes: 67 additions & 3 deletions docs/adr/0015-external-datasource-federation.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@
**Deciders**: ObjectStack Protocol Architects
**Builds on**: [ADR-0005](./0005-metadata-customization-overlay.md) (one Zod source of truth per metadata type)
**Consumers**: `@objectstack/spec`, `@objectstack/runtime`, `@objectstack/objectql`, `@objectstack/plugins/driver-sql`, `@objectstack/services/service-ai`, `@objectstack/cli`, `../objectui`
**Amended by**: [ADR-0143](./0143-outbound-connector-auth-is-open-tier.md) (2026-10-27), attribution only — see the note below. D7 (2026-10-29, ruling 6486136301 on #27298) — an API-backed datasource may borrow a declared connector instance's credential through `external.connector`; see the D7 note below. #27688 (2026-10-30), example only — §8's `credentialsRef` is a `sys_secret:` handle, the only form the shipped secret binder resolves; see the note under §8. #27917 (2026-10-30), dialect list only — §4.6 names the `odata` vocabulary the matrix gained with #27745.
**Amended by**: [ADR-0143](./0143-outbound-connector-auth-is-open-tier.md) (2026-10-27), attribution only — see the note below. D7 (2026-10-29, ruling 6486136301 on #27298) — an API-backed datasource may borrow a declared connector instance's credential through `external.connector`; see the D7 note below. #27688 (2026-10-30), example only — §8's `credentialsRef` is a `sys_secret:` handle, the only form the shipped secret binder resolves; see the note under §8. #27917 (2026-10-30), dialect list only — §4.6 names the `odata` vocabulary the matrix gained with #27745. #28208 (2026-10-31, ruling 6525963988 on #27687) — a datasource defined in code reads its credential at connect from `OS_DATASOURCE_<NAME>_PASSWORD`, behind the host switch `OS_DATASOURCE_ENV_CREDENTIALS`; see the note under §10 and the correction in the D7 note.

> **Amendment note D7 (2026-10-29, ruling 6486136301, letter A) — a second credential channel.**
> §10 says credentials reach a datasource only through `credentialsRef`. That stays true for a
> database. An **API-backed** datasource (a plugin-contributed driver over a vendor API, such as
> database *(amended 2026-10-31, #28208: a database datasource defined in code has a third route,
> the environment variable derived from its name — see the note under §10)*. An **API-backed**
> datasource (a plugin-contributed driver over a vendor API, such as
> OData) may instead name a declared connector instance in `external.connector` and borrow that
> instance's credential — whichever kind its `auth` declares (`ConnectorInstanceAuthSchema`:
> `bearer`, `api-key` and `basic` today; ADR-0143 D2 adds the grant-bearing `oauth2`,
Expand Down Expand Up @@ -766,7 +768,9 @@ managed Postgres for low-latency joins.
component that resolves the actual connection string. *(Amended by
D7, see the note at the top: an API-backed datasource may instead
borrow a declared connector instance's credential through
`external.connector`.)*
`external.connector`.)* *(Amended 2026-10-31, #28208, see the note below: a datasource
defined in code, which has no handle to point at, reads its credential at connect from the
environment variable derived from its name.)*
- **Connection pools** default conservatively for external mode
(`max: 5`) to protect the remote system; `pool` may be overridden.
- **Timeouts** are enforced at the driver. AI tool calls additionally
Expand All @@ -787,6 +791,66 @@ managed Postgres for low-latency joins.
otherwise federated objects fall back to poll-based change feeds.
This is out of scope for v1.

> **Amendment note (2026-10-31, #28208 — ruling 6525963988 on #27687, option 5; the failure text and
> the collision claim as PR #28325 left them) — a third credential route, for a datasource defined
> in code.** A datasource declared in code (`defineDatasource`, a package's `*.datasource.ts`) has
> no `sys_secret:` handle to point `credentialsRef` at: the Setup → Datasources form refuses a
> code-origin datasource, and nothing else mints one. Its route is the runtime environment. At
> connect, `DatasourceConnectionService` reads `OS_DATASOURCE_<NAME>_PASSWORD`, `<NAME>` being the
> datasource name upper-cased with every character outside `[A-Z0-9]` replaced by `_` (`warehouse`
> → `OS_DATASOURCE_WAREHOUSE_PASSWORD`;
> `packages/services/service-datasource/src/datasource-env-credential.ts#datasourceEnvCredentialVar`),
> and hands a non-empty value to the driver factory as the connection's `secret` — the password for
> `postgres`, `mysql` and `mongodb`, the `authToken` for `turso`. The value is never logged, stored
> or answered; every message names the variable only. The guards, each a ruled condition:
>
> - **A host switch, off unless the host opts in.** The slot is read only while
> `OS_DATASOURCE_ENV_CREDENTIALS` reads `on` (`resolveDatasourceEnvCredentialsSwitch`, same file:
> unset or empty is off; `on` / `off` in any letter case; any other value throws, and the
> datasource-admin plugin reads it at `init()`, so the boot is refused). `os serve` stamps `on`
> into its own environment when the operator left it unset or empty, just before it builds the
> admin plugin (`packages/cli/src/commands/serve.ts`, the `adminMod` block, through
> `packages/cli/src/utils/datasource-env-credentials-default.ts`); `os start` and `os dev` run
> `serve`, so they get it too. Every other composition — a host that builds its own kernel, a cloud
> runtime — never runs that line and stays off.
> - **Code origin only, by provenance.** The slot serves a name in the host's code-datasource set
> (the `code-datasource-names` service, filled by `@objectstack/runtime` from the boot artifact,
> from `default`, and from a package installed into the running kernel) on the boot's
> `declared-auto` connect — never a record's `origin`. A datasource created through Setup or the
> API under the same name connects through `registerPool` and never reads it; `default` keeps
> `OS_DATABASE_URL` (an `asDefault` connect never reads the slot).
> - **A declared channel wins and stays fail-closed.** The slot sits in `connect()` after the
> `credentialsRef` resolution (ADR-0062 D3) and the D7 connector borrow, and before the driver
> factory; it is consulted only when neither is declared, so a declared reference that does not
> resolve never falls back to the environment.
> - **An unset variable changes nothing about the connect.** The driver is built without a secret;
> when the database refuses, the verdict is the driver's own (`failed-degraded`) and its reason is
> the driver's error followed by `(note: datasource 'NAME' is defined in code and reads its
> credential from OS_DATASOURCE_NAME_PASSWORD at connect, which is not set on this host)`. A bound
> datasource still fails the boot (ADR-0062 D5), and the thrown message carries that note.
> - **Two live datasources that derive one variable are refused.** Valid names
> (`^[a-z_][a-z0-9_]*$`) cannot collide, so the guard covers names a host hands `connect()`
> directly: the second connect is refused `failed-credentials`, naming both datasources and the
> variable, and its variable is never read. A connect that failed releases its claim, so it never
> counts as the owner.
> - **A plugin driver that resolves its credential per request** (the OData driver resolves
> `credentialsRef` that way) does not read the slot: the slot feeds only the `secret` a driver
> receives when its connection is built.
>
> **The trust consequence, stated for operators.** The derived name is the only key, so any code
> datasource the host registers under that name reads it. Under `os serve` the code set includes
> every package installed into the environment, so an installed package can declare a datasource of
> any free name and so read the matching `OS_DATASOURCE_<NAME>_PASSWORD`. Install only packages you
> trust with the credentials in that process's environment. On a host where several tenants'
> artifacts share one process, a process-wide variable would be readable by every tenant artifact
> that declares the name — which is why the switch is off outside the CLI host. The spec's five
> credential refusals, the refused-key description and its two `password` guidance entries end
> with one shared sentence
> (`packages/spec/src/data/driver/common.zod.ts#CODE_DEFINED_CREDENTIAL_ROUTE`) that names this
> route. Not taken: a variable named in metadata (authored metadata could then point the runtime at
> an arbitrary host variable); an environment fallback behind a declared `credentialsRef`; a
> per-driver table of authentication error codes to decide when the note applies.

---

## 11. Rollout Plan
Expand Down
4 changes: 4 additions & 0 deletions docs/adr/0062-external-datasource-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@ Auto-connect must not change apps that today declare datasources that are *decor

`DatasourceConnectionService` resolves `external.credentialsRef` (and any `secret` config fields) through the host-provided `SecretBinder` over `ICryptoProvider` **before** building the driver. Open-core default is `InMemoryCryptoProvider`; a datasource that needs a secret the host cannot decrypt **fails closed** (clear boot error, datasource left unconnected — not a silent skip). This reuses the exact mechanism the runtime-admin "Add Datasource" wizard already uses, so code- and runtime-origin secrets converge.

> **Amendment (2026-10-31, #28208 — ruling 6525963988 on #27687, option 5; the failure text and the collision claim as PR #28325 left them) — a second resolution source at the same point, for a datasource defined in code.** `credentialsRef` and the ADR-0015 D7 connector borrow keep the point above and keep failing closed. A datasource the host registers from code can carry neither: the Setup → Datasources form refuses a code-origin datasource, so no `sys_secret:` handle is ever minted for one. For it, `DatasourceConnectionService.connect()` has a third step at that same point — after the `credentialsRef` resolution and the connector borrow, before the driver factory — taken only when neither channel is declared, the connect is not `asDefault`, the trigger is `declared-auto`, the name is in the host's code-datasource set (the `code-datasource-names` service: provenance, never a record's `origin`), and the host opted in. The step reads the environment variable `OS_DATASOURCE_<NAME>_PASSWORD` (`datasourceEnvCredentialVar`, `packages/services/service-datasource/src/datasource-env-credential.ts`: the name upper-cased, each character outside `[A-Z0-9]` replaced by `_`) and hands a non-empty value to the factory as the connection's `secret`.
>
> The opt-in is the host switch `OS_DATASOURCE_ENV_CREDENTIALS`. The datasource-admin plugin wires the slot into the service only when the switch reads `on` (unset or empty is off; any other value refuses the boot at `init()`), and only the CLI host stamps `on` as its default — `os serve`, which `os start` and `os dev` run (`packages/cli/src/utils/datasource-env-credentials-default.ts`); a library composition never sees the slot. An unset variable leaves the build as it was: a driver refusal is `failed-degraded` with the driver's own error, followed by `(note: datasource 'NAME' is defined in code and reads its credential from OS_DATASOURCE_NAME_PASSWORD at connect, which is not set on this host)`, and a declared reference that fails never falls through to the variable. Two live connects whose names derive one variable are refused at the second (`failed-credentials`, naming both datasources and the variable; a connect that failed releases its claim). The value reaches nothing but the factory, and no message carries it. **Default behavior** below gains one line: a datasource defined in code connects with the credential its derived variable holds when the host opted in. Who may then read that variable — any installed package declaring the name — is recorded as an operator's caution in ADR-0015 §10's amendment note. Not taken: a variable named in metadata, and an environment fallback behind a declared reference.

### D4 — Visibility converges on the metadata registry (shipped, ratified here)

Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta/datasource` and **Setup → Datasources** via the single in-memory metadata registry, stamped `origin: 'code'` (read-only in the admin UI). Shipped in #2157 (the in-memory metadata fallback was missing `registerInMemory`, silently skipping registration on the host-config boot path). This ADR ratifies that the metadata registry is the single source for datasource listing across all boot paths; the admin `listDatasources` and `protocol.getMetaItems` both read it.
Expand Down
27 changes: 13 additions & 14 deletions skills/objectstack-data/rules/datasources.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
# Datasources & Federation

A **datasource** (`defineDatasource`, a `*.datasource.ts` file) is a
connection to a data store. Objects route to one via their `datasource` field
(default: `'default'`). `default` is host-reserved: the host connects it from
connection to a data store. Objects route to one via their `datasource` field.
`default` is host-reserved: the host connects it from
`OS_DATABASE_URL`, and a stack that declares a datasource named `default` is
refused. Declare a datasource under its own name only to read/write a
**separate** or **external** store.

Full field reference: `node_modules/@objectstack/spec/src/data/datasource.zod.ts`.

## `schemaMode` — who owns the schema

| Mode | Meaning |
Expand All @@ -29,22 +27,18 @@ ObjectSchema.create({
remoteName: 'customers', // remote TABLE (object name may differ)
// remoteSchema: 'public', // optional schema/namespace (pg/mysql)
// columnMap: { cust_region: 'region' }, // remoteColumn → localField
// writable: true, // per-object write opt-in (see below)
},
fields: { id: { type: 'text' }, name: { type: 'text' }, region: { type: 'text' } },
});
```

### ✅ / ❌ Column mapping (ADR-0062 D7)

- ✅ Map remote columns with **`external.columnMap`**.
- ❌ **`field.columnName` does not exist — on ANY object.** Authoring it is a
parse error everywhere, not only on a federated object.

## Auto-connect (no `onEnable`)

A declared datasource is connected, and its federated objects registered,
**automatically at boot** when it is **meaningfully addressed**:
A declared datasource is connected **automatically at boot** when it is
**meaningfully addressed**:

1. it is **external** (`schemaMode !== 'managed'`), **or**
2. an object **explicitly** binds via `object.datasource === <name>`, **or**
Expand All @@ -60,9 +54,15 @@ object throws.
> `ctx.drivers.register(driver)` in `onEnable` is only an escape hatch for a
> runtime-built driver; idempotent with auto-connect.

## Credentials — one channel, fail-closed
## Credentials — fail-closed

Never inline a password. `external` names **one** credential channel:
Never inline a password. A code-declared datasource reads one at connect from
`OS_DATASOURCE_<NAME>_PASSWORD` (`NAME` = the name upper-cased) while the host
switch `OS_DATASOURCE_ENV_CREDENTIALS` is `on` (`os serve`/`start`/`dev` default
it on; a library composition is off; `on`/`off` only). Operator caution: an
installed package may declare a datasource of any free name and so read that
variable. `external` may instead name **one** channel, which wins and never
falls back to the variable:

- **`credentialsRef`** — a secret-store handle, resolved at connect, before the
driver is built.
Expand All @@ -71,7 +71,7 @@ Never inline a password. `external` names **one** credential channel:
with a `provider` and an `auth` other than `none`:
`schemaMode: 'external', external: { connector: 'erp_api' }`. Refused beside
`credentialsRef`, on a built-in driver, on a `managed` datasource, and naming
an undeclared instance (at `defineStack`, else at connect).
an undeclared instance.

No secret store, an unresolvable handle, a disabled instance or no automation
service leaves the datasource **unconnected with a clear error**. A connector
Expand All @@ -81,4 +81,3 @@ credential not set yet answers `503` per request.

Federation is read-only by default. To write, **both** gates must be on:
`datasource.external.allowWrites: true` **and** the object's `external.writable: true`.
Otherwise every write to the object is rejected.
Loading