fix(crypto): fail loud instead of silently minting an ephemeral key (#1507)

.changeset/local-crypto-provider-fail-loud-1507.md

@@ -0,0 +1,38 @@
+---
+"@objectstack/service-settings": minor
+"@objectstack/cli": patch
+"@objectstack/spec": patch
+---
+
+Fail loud instead of silently minting an ephemeral encryption key; ship a persistent env-master-key provider as the default (#1507).
+
+The default `ICryptoProvider` backs every secret-at-rest in the platform —
+encrypted settings (`sys_setting.value_enc`), ObjectQL `secret` fields, and
+runtime datasource credentials. Its key resolution previously fell back,
+**silently**, to a fresh per-process `randomBytes(32)` key (or auto-minted a
+new on-disk key on every boot) when no stable key was available. In an
+ephemeral-FS container or a multi-node cluster, each restart / each node then
+encrypts under a different key, and every previously-written `sys_secret` value
+becomes undecryptable. The failure was invisible at encrypt and boot time and
+only surfaced later as "all my saved passwords / API keys / DB credentials
+fail to decrypt".
+
+- **Renamed `InMemoryCryptoProvider` → `LocalCryptoProvider`.** The old name
+  implied an ephemeral key when the provider in fact persists one.
+  `InMemoryCryptoProvider` stays as a deprecated alias for backward
+  compatibility.
+- **Added `OS_SECRET_KEY`** as the canonical production master key (32-byte
+  hex or base64), the documented production default. `OS_DEV_CRYPTO_KEY`
+  remains the dev convenience key.
+- **Fail-loud in production.** When `NODE_ENV=production` and no stable key
+  source (env var or a pre-existing persisted file) is available, the provider
+  now throws an actionable error at construction instead of generating a key —
+  turning silent data-loss into a config error at boot. It never auto-mints a
+  key in production. Development and test keep the ergonomic fallback
+  (persisted dev key / ephemeral test key).
+- `serve` surfaces the production-key error verbatim and refuses to wire an
+  unstable provider for `secret` fields.
+
+KMS / Vault providers (managed custody, per-tenant keys, automatic rotation)
+remain future/enterprise plug-ins behind the same `ICryptoProvider` seam;
+"your stored secret is still there after a reboot" stays open-source.

packages/cli/src/commands/serve.ts

@@ -1684,15 +1684,24 @@ export default class Serve extends Command {
           // cleartext (by design).
           let secrets: any = undefined;
           try {
-            const { InMemoryCryptoProvider } = await import(
+            const { LocalCryptoProvider } = await import(
               /* webpackIgnore: true */ '@objectstack/service-settings'
             );
-            sharedCryptoProvider = sharedCryptoProvider ?? new InMemoryCryptoProvider();
+            // First block to touch `sharedCryptoProvider` (still undefined
+            // here), so create it directly; the secret-field wiring below
+            // reuses this instance so every sys_secret shares one key.
+            sharedCryptoProvider = new LocalCryptoProvider();
             secrets = createDatasourceSecretBinder({
               engine: lazySecretEngine,
               cryptoProvider: sharedCryptoProvider,
             });
           } catch (cryptoErr: any) {
+            // Best-effort fail-closed: leave `secrets` undefined so the plugin
+            // rejects secret-bearing create/update rather than storing
+            // cleartext. A production deployment with no stable key still
+            // aborts boot loudly at the secret-field wiring below (where
+            // LocalCryptoProvider's "Refusing to start in production" error is
+            // rethrown), so we don't duplicate that abort here.
             console.warn(
               chalk.yellow(
                 `  ⚠ datasource admin: no CryptoProvider (${cryptoErr?.message ?? cryptoErr}); secret-bearing datasource create/update will fail closed`,
@@ -1791,40 +1800,51 @@ export default class Serve extends Command {
       // objectql's `secret` field type encrypts on write to `sys_secret`
       // and fails closed when no ICryptoProvider is registered. objectql
       // must NOT depend on a crypto implementation (layering), so the
-      // host injects one here. Dev/self-host gets an independent
-      // InMemoryCryptoProvider; production hosts swap this for a
-      // KMS/Vault-backed provider (e.g. via an env-gated branch or a
-      // dedicated plugin) before secrets are written. We resolve the
-      // data engine by its registered service name and feature-detect
-      // `setCryptoProvider` so older engines / alternate data services
-      // degrade gracefully (writing a secret then fails closed, as
-      // designed, rather than silently storing cleartext).
+      // host injects one here. Dev/self-host gets a LocalCryptoProvider
+      // (AES-256-GCM keyed off `OS_SECRET_KEY` or a persisted dev key);
+      // production hosts swap this for a KMS/Vault-backed provider (e.g.
+      // via an env-gated branch or a dedicated plugin) before secrets are
+      // written. We resolve the data engine by its registered service name
+      // and feature-detect `setCryptoProvider` so older engines / alternate
+      // data services degrade gracefully (writing a secret then fails
+      // closed, as designed, rather than silently storing cleartext).
       try {
         const dataEngine: any =
           kernel.getService?.('data') ?? kernel.getService?.('objectql');
         if (dataEngine && typeof dataEngine.setCryptoProvider === 'function') {
           if (!sharedCryptoProvider) {
-            const { InMemoryCryptoProvider } = await import(
+            const { LocalCryptoProvider } = await import(
               /* webpackIgnore: true */ '@objectstack/service-settings'
             );
-            sharedCryptoProvider = new InMemoryCryptoProvider();
+            // In production LocalCryptoProvider throws when no stable key
+            // (OS_SECRET_KEY / persisted file) is available — the fail-loud
+            // guard against silently minting an ephemeral key and losing
+            // every sys_secret value after a restart. Let that error be loud:
+            // secret writes must not proceed under an unstable key.
+            sharedCryptoProvider = new LocalCryptoProvider();
           }
           dataEngine.setCryptoProvider(sharedCryptoProvider);
           if (isDev) {
             console.log(
               chalk.dim(
-                '  ↪ secret fields: InMemoryCryptoProvider wired (dev) — swap for KMS/Vault in production',
+                '  ↪ secret fields: LocalCryptoProvider wired (dev) — set OS_SECRET_KEY and swap for KMS/Vault in production',
               ),
             );
           }
         }
       } catch (err: any) {
-        // Non-fatal: without a provider, secret writes fail closed by
-        // design. Surface a hint so operators know why a `secret` field
+        const msg = String(err?.message ?? err);
+        if (msg.includes('Refusing to start in production')) {
+          // Fail-loud config error: print the actionable guidance verbatim.
+          console.error(chalk.red(msg));
+          throw err;
+        }
+        // Otherwise non-fatal: without a provider, secret writes fail closed
+        // by design. Surface a hint so operators know why a `secret` field
         // write might reject.
         console.warn(
           chalk.yellow(
-            `  ⚠ secret fields: no CryptoProvider wired (${err?.message ?? err}); writing a secret field will fail closed`,
+            `  ⚠ secret fields: no CryptoProvider wired (${msg}); writing a secret field will fail closed`,
           ),
         );
       }

packages/objectql/src/engine.ts

@@ -1076,7 +1076,7 @@ export class ObjectQL implements IDataEngine {
    * **fail-closed** — the write throws rather than persist cleartext.
    *
    * Mirrors the Settings subsystem's ICryptoProvider wiring; the host (e.g.
-   * `serve`) injects `InMemoryCryptoProvider` in dev and a KMS/Vault-backed
+   * `serve`) injects `LocalCryptoProvider` in dev and a KMS/Vault-backed
    * provider in production.
    */
   setCryptoProvider(provider: ICryptoProvider): void {
@@ -1126,7 +1126,7 @@ export class ObjectQL implements IDataEngine {
       if (!this.cryptoProvider) {
         throw new Error(
           `Cannot persist secret field "${object}.${field}": no CryptoProvider is registered. `
-            + 'Wire one via engine.setCryptoProvider(...) (e.g. InMemoryCryptoProvider in dev, '
+            + 'Wire one via engine.setCryptoProvider(...) (e.g. LocalCryptoProvider in dev, '
             + 'a KMS/Vault provider in production). Refusing to store cleartext (fail-closed).',
         );
       }

packages/services/service-datasource-admin/src/datasource-secret-binder.ts

@@ -12,7 +12,7 @@
  * touches metadata.
  *
  * This is the dev/self-host wiring; production hosts swap the
- * `InMemoryCryptoProvider` for a KMS-backed `ICryptoProvider` and pass it here.
+ * `LocalCryptoProvider` for a KMS-backed `ICryptoProvider` and pass it here.
  */
 
 import type { CryptoHandle, ICryptoProvider } from '@objectstack/spec/contracts';

packages/services/service-settings/README.md

@@ -32,10 +32,44 @@ true` and writes (service or REST) fail with HTTP 409.
 
 ## Encryption
 
-`Specifier.encrypted: true` (implicit for `password`) round-trips the
-value through a pluggable `CryptoAdapter`. The default
-`NoopCryptoAdapter` is a base64 wrapper — production deployments must
-provide a real KMS adapter via plugin options.
+`Specifier.encrypted: true` (implicit for `password`) routes the value
+through a pluggable `ICryptoProvider` into `sys_secret`; only an opaque
+handle id lands in `sys_setting.value_enc`. The same provider backs every
+secret-at-rest in the platform: encrypted settings, ObjectQL `secret`
+fields, and runtime datasource credentials.
+
+### Default provider: `LocalCryptoProvider`
+
+The default is `LocalCryptoProvider` — AES-256-GCM keyed off a single
+32-byte data key. It resolves its key in order:
+
+1. **`OS_SECRET_KEY`** — the canonical production master key (32-byte hex
+   or base64). Set this in any container / multi-node deployment.
+   Generate one with `openssl rand -hex 32`. It **must be identical**
+   across every restart and every node, or previously-encrypted secrets
+   become undecryptable.
+2. `OS_DEV_CRYPTO_KEY` — dev convenience key (same format).
+3. A persisted file at `~/.objectstack/dev-crypto-key` (mode 0600). In
+   development this is auto-created so single-host dev loops survive
+   restarts; in production it is only *read*, never minted.
+
+**Fail-loud in production.** When `NODE_ENV=production` and no stable key
+source (env var or pre-existing file) is available, the provider refuses
+to start with an actionable error instead of silently generating an
+ephemeral key. This turns the old silent-data-loss footgun — every
+`sys_secret` value becoming undecryptable after a container restart or on
+a second node — into a config error at boot.
+
+Secrets surviving a restart is **correctness, not a premium feature**, so
+`LocalCryptoProvider` and the env-key path are open-source. KMS / Vault
+providers (managed custody, per-tenant keys, automatic rotation) plug in
+behind the same `ICryptoProvider` seam via `cryptoProvider` plugin option.
+
+> `InMemoryCryptoProvider` is a deprecated alias for `LocalCryptoProvider`
+> (the old name wrongly implied an ephemeral key).
+
+The legacy `CryptoAdapter` / `NoopCryptoAdapter` (a base64 wrapper) remains
+only as a pre-Phase-3 backward-compat path when no `cryptoProvider` is wired.
 
 ## Audit