Default ICryptoProvider can silently degrade to an ephemeral key — all sys_secret data (settings, secret fields, datasource creds) becomes undecryptable after restart in containers / multi-node

[xuyushun441-sys]

Summary

The platform's default ICryptoProvider is InMemoryCryptoProvider, wired in by SettingsServicePlugin and the serve command. Its key resolution falls back, silently, to a fresh per-process random key when neither a stable env key nor a readable on-disk key file is available. When that happens, every secret in sys_secret encrypted under the previous key becomes undecryptable on the next boot.

This is a platform-wide secret-at-rest concern, not specific to any one feature. The same provider + sys_secret table backs:

• SettingsService encrypted settings (sys_setting.value_enc, e.g. storage.s3_secret_access_key)
• any ObjectQL Field.secret() field
• runtime datasource credentials (@objectstack/service-datasource-admin, added in feat(service-datasource-admin): open-source the runtime datasource admin mechanism (ADR-0015 Addendum) #1502 — it rides on this exact layer)

So if the default is fragile, it is fragile for all of the above at once.

Current behaviour

packages/services/service-settings/src/in-memory-crypto-provider.ts resolves its AES-256-GCM key in three tiers:

1. OS_DEV_CRYPTO_KEY env var (stable) →
2. on-disk file ~/.objectstack/dev-crypto-key (auto-created on first run, mode 0600 — survives restart on the same host/FS) →
3. ephemeral randomBytes(32) when the file can't be read/written, or in test mode.

Default wiring (no explicit provider passed):

• packages/services/service-settings/src/settings-service-plugin.ts: cryptoProvider: this.opts.cryptoProvider ?? new InMemoryCryptoProvider()
• packages/cli/src/commands/serve.ts: dataEngine.setCryptoProvider(new InMemoryCryptoProvider())

No KMS/Vault/env-master-key provider ships today — InMemoryCryptoProvider is the de-facto production default; the contract (packages/spec/src/contracts/crypto-provider.ts) mentions AwsKmsCryptoProvider / GcpKmsCryptoProvider / HashicorpVaultCryptoProvider only as future vision.

Why this is a footgun

Single-node bare-metal dev is actually fine (the file fallback persists the key). The danger is the silent degradation to tier 3 in exactly the environments people deploy to:

Scenario	Key persistence	Secrets survive restart?
Single-node dev, writable $HOME	file fallback	✅
OS_DEV_CRYPTO_KEY set	stable env key	✅
Container with ephemeral FS	new file each container	❌
Multi-node (no shared/identical key)	per-node file	❌ (node A's ciphertext ≠ node B)
$HOME unwritable / IO error	ephemeral random	❌

The failure mode is the worst kind: no error at encrypt or boot time — the system happily mints a new key and only surfaces later as "all my saved passwords/API keys/DB credentials decrypt-fail" after a restart or on a second node.

Proposed changes

1. Fail loud instead of silently going ephemeral. When not in dev/test and no stable key source (env var or readable persisted file) is available, refuse to start (or refuse secret writes) with an explicit, actionable error — turn silent data-loss into a config error at boot.

2. Ship a persistent, env-master-key provider as the documented production default. An AES-256-GCM provider keyed off something like OS_SECRET_KEY (32-byte hex/base64). Far cheaper to adopt than KMS, and it closes the "no production provider exists" gap. KMS/Vault stay as future/enterprise plug-ins behind the same ICryptoProvider seam.

3. Reframe InMemoryCryptoProvider. The name reads as "ephemeral / insecure" when in practice it persists to disk. Consider renaming (e.g. LocalCryptoProvider) and/or documenting the env-key path as the recommended default, so operators don't assume it's dev-only and reach for a missing KMS provider.

Boundary note (open-source vs paid)

Secrets surviving a restart is correctness, not a premium feature — the persistent local / env-key provider must stay open-source. This mirrors the existing tiering principle (don't charge for correctness; charge for managed custody + scale): KMS, automatic key rotation, and per-tenant key isolation are the paid layer; "your stored password is still there after a reboot" is not.

Out of scope

• Actual KMS/Vault provider implementations (future).
• Key-rotation tooling beyond the existing rotateKey contract.

Evidence

• packages/services/service-settings/src/in-memory-crypto-provider.ts (key resolution + ephemeral fallback)
• packages/services/service-settings/src/settings-service-plugin.ts (default wiring)
• packages/cli/src/commands/serve.ts (engine crypto wiring)
• packages/spec/src/contracts/crypto-provider.ts (ICryptoProvider / CryptoHandle)
• sys_secret object definition + ObjectQL secret-field handling