From 17988ade2c9a3e191a12c51a7419669c612ce2a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 31 Oct 2026 15:25:39 +0000 Subject: [PATCH] docs(skills,adr): teach the code-defined datasource credential route (OS_DATASOURCE_NAME_PASSWORD behind the host switch) skills/objectstack-data/rules/datasources.md: the credential section now says a code-declared datasource reads its password at connect from the variable derived from its name while the host switch OS_DATASOURCE_ENV_CREDENTIALS is on (the CLI host defaults it on, a library composition is off, on/off only), that a declared credentialsRef or connector still wins with no fallback, and the operator's caution that an installed package may declare a datasource of any free name and so read that variable. The added text is paid for under the token ratchet by deleting restatements in the same file; the file lands at 910 of 911 tokens. ADR-0015: the Amended-by line, an inline correction of the D7 note's "that stays true for a database", the section 10 credentials bullet, and a dated amendment note under section 10 recording the route, its guards, the failure text and the trust consequence. ADR-0062 D3: a dated amendment recording the slot's place in the connect's resolution order. A patch changeset for @objectstack/skills, which ships skills/**. Claude-Session: https://claude.ai/code/session_01FEkQ5phX4SAaFQKww3TfKb Co-authored-by: Claude --- ...9-skills-code-datasource-env-credential.md | 12 ++++ .../0015-external-datasource-federation.md | 70 ++++++++++++++++++- docs/adr/0062-external-datasource-runtime.md | 4 ++ skills/objectstack-data/rules/datasources.md | 27 ++++--- 4 files changed, 96 insertions(+), 17 deletions(-) create mode 100644 .changeset/28209-skills-code-datasource-env-credential.md diff --git a/.changeset/28209-skills-code-datasource-env-credential.md b/.changeset/28209-skills-code-datasource-env-credential.md new file mode 100644 index 00000000000..633e895bf44 --- /dev/null +++ b/.changeset/28209-skills-code-datasource-env-credential.md @@ -0,0 +1,12 @@ +--- +'@objectstack/skills': patch +--- + +docs(skills): objectstack-data teaches the code-defined datasource credential route (`OS_DATASOURCE__PASSWORD`, behind the host switch) + +Clause-②: no +Metadata-protocol: none + +- **Before.** `rules/datasources.md` taught `credentialsRef` and `connector` as the only credential channels. A datasource defined in code has no `sys_secret:` handle to point at (the Setup form refuses a code-origin datasource, and nothing else mints one), so an agent following the rule wrote a reference the connect could never resolve. +- **Now.** The credential section says a code-declared datasource reads its password at connect from `OS_DATASOURCE__PASSWORD` (the name upper-cased) while the host switch `OS_DATASOURCE_ENV_CREDENTIALS` is `on`; that `os serve`, `os start` and `os dev` default it on, a library composition leaves it off, and the switch takes `on` or `off` only; that a declared `credentialsRef` or `connector` still wins and never falls back to the variable; and the operator's caution that an installed package may declare a datasource of any free name and so read that variable. +- The file stays under its token ceiling: the new text is paid for by deleting restatements in the same file (the field-reference pointer the references index already carries, the `writable` and `columnMap` restatements of their own sections, the `datasource` field default the SKILL.md field table carries, and two detail clauses). No ceiling moves, no frontmatter, schema or behaviour changes. diff --git a/docs/adr/0015-external-datasource-federation.md b/docs/adr/0015-external-datasource-federation.md index c60f78363ea..2fddd0594cd 100644 --- a/docs/adr/0015-external-datasource-federation.md +++ b/docs/adr/0015-external-datasource-federation.md @@ -4,11 +4,13 @@ **Deciders**: ObjectStack Protocol Architects **Builds on**: [ADR-0005](./0005-metadata-customization-overlay.md) (one Zod source of truth per metadata type) **Consumers**: `@objectstack/spec`, `@objectstack/runtime`, `@objectstack/objectql`, `@objectstack/plugins/driver-sql`, `@objectstack/services/service-ai`, `@objectstack/cli`, `../objectui` -**Amended by**: [ADR-0143](./0143-outbound-connector-auth-is-open-tier.md) (2026-10-27), attribution only — see the note below. D7 (2026-10-29, ruling 6486136301 on #27298) — an API-backed datasource may borrow a declared connector instance's credential through `external.connector`; see the D7 note below. #27688 (2026-10-30), example only — §8's `credentialsRef` is a `sys_secret:` handle, the only form the shipped secret binder resolves; see the note under §8. #27917 (2026-10-30), dialect list only — §4.6 names the `odata` vocabulary the matrix gained with #27745. +**Amended by**: [ADR-0143](./0143-outbound-connector-auth-is-open-tier.md) (2026-10-27), attribution only — see the note below. D7 (2026-10-29, ruling 6486136301 on #27298) — an API-backed datasource may borrow a declared connector instance's credential through `external.connector`; see the D7 note below. #27688 (2026-10-30), example only — §8's `credentialsRef` is a `sys_secret:` handle, the only form the shipped secret binder resolves; see the note under §8. #27917 (2026-10-30), dialect list only — §4.6 names the `odata` vocabulary the matrix gained with #27745. #28208 (2026-10-31, ruling 6525963988 on #27687) — a datasource defined in code reads its credential at connect from `OS_DATASOURCE__PASSWORD`, behind the host switch `OS_DATASOURCE_ENV_CREDENTIALS`; see the note under §10 and the correction in the D7 note. > **Amendment note D7 (2026-10-29, ruling 6486136301, letter A) — a second credential channel.** > §10 says credentials reach a datasource only through `credentialsRef`. That stays true for a -> database. An **API-backed** datasource (a plugin-contributed driver over a vendor API, such as +> database *(amended 2026-10-31, #28208: a database datasource defined in code has a third route, +> the environment variable derived from its name — see the note under §10)*. An **API-backed** +> datasource (a plugin-contributed driver over a vendor API, such as > OData) may instead name a declared connector instance in `external.connector` and borrow that > instance's credential — whichever kind its `auth` declares (`ConnectorInstanceAuthSchema`: > `bearer`, `api-key` and `basic` today; ADR-0143 D2 adds the grant-bearing `oauth2`, @@ -766,7 +768,9 @@ managed Postgres for low-latency joins. component that resolves the actual connection string. *(Amended by D7, see the note at the top: an API-backed datasource may instead borrow a declared connector instance's credential through - `external.connector`.)* + `external.connector`.)* *(Amended 2026-10-31, #28208, see the note below: a datasource + defined in code, which has no handle to point at, reads its credential at connect from the + environment variable derived from its name.)* - **Connection pools** default conservatively for external mode (`max: 5`) to protect the remote system; `pool` may be overridden. - **Timeouts** are enforced at the driver. AI tool calls additionally @@ -787,6 +791,66 @@ managed Postgres for low-latency joins. otherwise federated objects fall back to poll-based change feeds. This is out of scope for v1. +> **Amendment note (2026-10-31, #28208 — ruling 6525963988 on #27687, option 5; the failure text and +> the collision claim as PR #28325 left them) — a third credential route, for a datasource defined +> in code.** A datasource declared in code (`defineDatasource`, a package's `*.datasource.ts`) has +> no `sys_secret:` handle to point `credentialsRef` at: the Setup → Datasources form refuses a +> code-origin datasource, and nothing else mints one. Its route is the runtime environment. At +> connect, `DatasourceConnectionService` reads `OS_DATASOURCE__PASSWORD`, `` being the +> datasource name upper-cased with every character outside `[A-Z0-9]` replaced by `_` (`warehouse` +> → `OS_DATASOURCE_WAREHOUSE_PASSWORD`; +> `packages/services/service-datasource/src/datasource-env-credential.ts#datasourceEnvCredentialVar`), +> and hands a non-empty value to the driver factory as the connection's `secret` — the password for +> `postgres`, `mysql` and `mongodb`, the `authToken` for `turso`. The value is never logged, stored +> or answered; every message names the variable only. The guards, each a ruled condition: +> +> - **A host switch, off unless the host opts in.** The slot is read only while +> `OS_DATASOURCE_ENV_CREDENTIALS` reads `on` (`resolveDatasourceEnvCredentialsSwitch`, same file: +> unset or empty is off; `on` / `off` in any letter case; any other value throws, and the +> datasource-admin plugin reads it at `init()`, so the boot is refused). `os serve` stamps `on` +> into its own environment when the operator left it unset or empty, just before it builds the +> admin plugin (`packages/cli/src/commands/serve.ts`, the `adminMod` block, through +> `packages/cli/src/utils/datasource-env-credentials-default.ts`); `os start` and `os dev` run +> `serve`, so they get it too. Every other composition — a host that builds its own kernel, a cloud +> runtime — never runs that line and stays off. +> - **Code origin only, by provenance.** The slot serves a name in the host's code-datasource set +> (the `code-datasource-names` service, filled by `@objectstack/runtime` from the boot artifact, +> from `default`, and from a package installed into the running kernel) on the boot's +> `declared-auto` connect — never a record's `origin`. A datasource created through Setup or the +> API under the same name connects through `registerPool` and never reads it; `default` keeps +> `OS_DATABASE_URL` (an `asDefault` connect never reads the slot). +> - **A declared channel wins and stays fail-closed.** The slot sits in `connect()` after the +> `credentialsRef` resolution (ADR-0062 D3) and the D7 connector borrow, and before the driver +> factory; it is consulted only when neither is declared, so a declared reference that does not +> resolve never falls back to the environment. +> - **An unset variable changes nothing about the connect.** The driver is built without a secret; +> when the database refuses, the verdict is the driver's own (`failed-degraded`) and its reason is +> the driver's error followed by `(note: datasource 'NAME' is defined in code and reads its +> credential from OS_DATASOURCE_NAME_PASSWORD at connect, which is not set on this host)`. A bound +> datasource still fails the boot (ADR-0062 D5), and the thrown message carries that note. +> - **Two live datasources that derive one variable are refused.** Valid names +> (`^[a-z_][a-z0-9_]*$`) cannot collide, so the guard covers names a host hands `connect()` +> directly: the second connect is refused `failed-credentials`, naming both datasources and the +> variable, and its variable is never read. A connect that failed releases its claim, so it never +> counts as the owner. +> - **A plugin driver that resolves its credential per request** (the OData driver resolves +> `credentialsRef` that way) does not read the slot: the slot feeds only the `secret` a driver +> receives when its connection is built. +> +> **The trust consequence, stated for operators.** The derived name is the only key, so any code +> datasource the host registers under that name reads it. Under `os serve` the code set includes +> every package installed into the environment, so an installed package can declare a datasource of +> any free name and so read the matching `OS_DATASOURCE__PASSWORD`. Install only packages you +> trust with the credentials in that process's environment. On a host where several tenants' +> artifacts share one process, a process-wide variable would be readable by every tenant artifact +> that declares the name — which is why the switch is off outside the CLI host. The spec's five +> credential refusals, the refused-key description and its two `password` guidance entries end +> with one shared sentence +> (`packages/spec/src/data/driver/common.zod.ts#CODE_DEFINED_CREDENTIAL_ROUTE`) that names this +> route. Not taken: a variable named in metadata (authored metadata could then point the runtime at +> an arbitrary host variable); an environment fallback behind a declared `credentialsRef`; a +> per-driver table of authentication error codes to decide when the note applies. + --- ## 11. Rollout Plan diff --git a/docs/adr/0062-external-datasource-runtime.md b/docs/adr/0062-external-datasource-runtime.md index 99533a32c73..f3aa9feda99 100644 --- a/docs/adr/0062-external-datasource-runtime.md +++ b/docs/adr/0062-external-datasource-runtime.md @@ -91,6 +91,10 @@ Auto-connect must not change apps that today declare datasources that are *decor `DatasourceConnectionService` resolves `external.credentialsRef` (and any `secret` config fields) through the host-provided `SecretBinder` over `ICryptoProvider` **before** building the driver. Open-core default is `InMemoryCryptoProvider`; a datasource that needs a secret the host cannot decrypt **fails closed** (clear boot error, datasource left unconnected — not a silent skip). This reuses the exact mechanism the runtime-admin "Add Datasource" wizard already uses, so code- and runtime-origin secrets converge. +> **Amendment (2026-10-31, #28208 — ruling 6525963988 on #27687, option 5; the failure text and the collision claim as PR #28325 left them) — a second resolution source at the same point, for a datasource defined in code.** `credentialsRef` and the ADR-0015 D7 connector borrow keep the point above and keep failing closed. A datasource the host registers from code can carry neither: the Setup → Datasources form refuses a code-origin datasource, so no `sys_secret:` handle is ever minted for one. For it, `DatasourceConnectionService.connect()` has a third step at that same point — after the `credentialsRef` resolution and the connector borrow, before the driver factory — taken only when neither channel is declared, the connect is not `asDefault`, the trigger is `declared-auto`, the name is in the host's code-datasource set (the `code-datasource-names` service: provenance, never a record's `origin`), and the host opted in. The step reads the environment variable `OS_DATASOURCE__PASSWORD` (`datasourceEnvCredentialVar`, `packages/services/service-datasource/src/datasource-env-credential.ts`: the name upper-cased, each character outside `[A-Z0-9]` replaced by `_`) and hands a non-empty value to the factory as the connection's `secret`. +> +> The opt-in is the host switch `OS_DATASOURCE_ENV_CREDENTIALS`. The datasource-admin plugin wires the slot into the service only when the switch reads `on` (unset or empty is off; any other value refuses the boot at `init()`), and only the CLI host stamps `on` as its default — `os serve`, which `os start` and `os dev` run (`packages/cli/src/utils/datasource-env-credentials-default.ts`); a library composition never sees the slot. An unset variable leaves the build as it was: a driver refusal is `failed-degraded` with the driver's own error, followed by `(note: datasource 'NAME' is defined in code and reads its credential from OS_DATASOURCE_NAME_PASSWORD at connect, which is not set on this host)`, and a declared reference that fails never falls through to the variable. Two live connects whose names derive one variable are refused at the second (`failed-credentials`, naming both datasources and the variable; a connect that failed releases its claim). The value reaches nothing but the factory, and no message carries it. **Default behavior** below gains one line: a datasource defined in code connects with the credential its derived variable holds when the host opted in. Who may then read that variable — any installed package declaring the name — is recorded as an operator's caution in ADR-0015 §10's amendment note. Not taken: a variable named in metadata, and an environment fallback behind a declared reference. + ### D4 — Visibility converges on the metadata registry (shipped, ratified here) Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta/datasource` and **Setup → Datasources** via the single in-memory metadata registry, stamped `origin: 'code'` (read-only in the admin UI). Shipped in #2157 (the in-memory metadata fallback was missing `registerInMemory`, silently skipping registration on the host-config boot path). This ADR ratifies that the metadata registry is the single source for datasource listing across all boot paths; the admin `listDatasources` and `protocol.getMetaItems` both read it. diff --git a/skills/objectstack-data/rules/datasources.md b/skills/objectstack-data/rules/datasources.md index 4f6005a77f3..5fae75cc172 100644 --- a/skills/objectstack-data/rules/datasources.md +++ b/skills/objectstack-data/rules/datasources.md @@ -1,14 +1,12 @@ # Datasources & Federation A **datasource** (`defineDatasource`, a `*.datasource.ts` file) is a -connection to a data store. Objects route to one via their `datasource` field -(default: `'default'`). `default` is host-reserved: the host connects it from +connection to a data store. Objects route to one via their `datasource` field. +`default` is host-reserved: the host connects it from `OS_DATABASE_URL`, and a stack that declares a datasource named `default` is refused. Declare a datasource under its own name only to read/write a **separate** or **external** store. -Full field reference: `node_modules/@objectstack/spec/src/data/datasource.zod.ts`. - ## `schemaMode` — who owns the schema | Mode | Meaning | @@ -29,22 +27,18 @@ ObjectSchema.create({ remoteName: 'customers', // remote TABLE (object name may differ) // remoteSchema: 'public', // optional schema/namespace (pg/mysql) // columnMap: { cust_region: 'region' }, // remoteColumn → localField - // writable: true, // per-object write opt-in (see below) }, fields: { id: { type: 'text' }, name: { type: 'text' }, region: { type: 'text' } }, }); ``` -### ✅ / ❌ Column mapping (ADR-0062 D7) - -- ✅ Map remote columns with **`external.columnMap`**. - ❌ **`field.columnName` does not exist — on ANY object.** Authoring it is a parse error everywhere, not only on a federated object. ## Auto-connect (no `onEnable`) -A declared datasource is connected, and its federated objects registered, -**automatically at boot** when it is **meaningfully addressed**: +A declared datasource is connected **automatically at boot** when it is +**meaningfully addressed**: 1. it is **external** (`schemaMode !== 'managed'`), **or** 2. an object **explicitly** binds via `object.datasource === `, **or** @@ -60,9 +54,15 @@ object throws. > `ctx.drivers.register(driver)` in `onEnable` is only an escape hatch for a > runtime-built driver; idempotent with auto-connect. -## Credentials — one channel, fail-closed +## Credentials — fail-closed -Never inline a password. `external` names **one** credential channel: +Never inline a password. A code-declared datasource reads one at connect from +`OS_DATASOURCE__PASSWORD` (`NAME` = the name upper-cased) while the host +switch `OS_DATASOURCE_ENV_CREDENTIALS` is `on` (`os serve`/`start`/`dev` default +it on; a library composition is off; `on`/`off` only). Operator caution: an +installed package may declare a datasource of any free name and so read that +variable. `external` may instead name **one** channel, which wins and never +falls back to the variable: - **`credentialsRef`** — a secret-store handle, resolved at connect, before the driver is built. @@ -71,7 +71,7 @@ Never inline a password. `external` names **one** credential channel: with a `provider` and an `auth` other than `none`: `schemaMode: 'external', external: { connector: 'erp_api' }`. Refused beside `credentialsRef`, on a built-in driver, on a `managed` datasource, and naming - an undeclared instance (at `defineStack`, else at connect). + an undeclared instance. No secret store, an unresolvable handle, a disabled instance or no automation service leaves the datasource **unconnected with a clear error**. A connector @@ -81,4 +81,3 @@ credential not set yet answers `503` per request. Federation is read-only by default. To write, **both** gates must be on: `datasource.external.allowWrites: true` **and** the object's `external.writable: true`. -Otherwise every write to the object is rejected.