Skip to content

feat(core)!: migrate legacy configuration and require encryption - #2914

Open
Innei wants to merge 26 commits into
masterfrom
codex/v18-preparation
Open

Innei wants to merge 26 commits into
masterfrom
codex/v18-preparation

Conversation

@Innei

@Innei Innei commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Summary

Shared migration helpers now live in database/app-migrations/helpers (configuration conversion and OAuth unwrapping). Release migrations depend on these helpers directly; src/upgrade retains CLI orchestration only.

Require encrypted credential storage and finish the legacy configuration cutover in the database before the application starts.

  • Add mandatory release-phase app-data migration 20261109-canonical-config. It converts old GitHub/membership/image/TTS/SMTP settings, decrypts credentials at original paths and encrypts at new paths, removes retired option rows, and fails transactionally on conflicting credentials, invalid shapes or unreadable ciphertext.
  • Fix Admin settings field casing at the schema boundary: serialized acronym names (single_file_size_mb / singleFileSizeMb) resolve to declared singleFileSizeMB, and the form restores declared keys before display/edit. Unknown and retired fields remain rejected; DB and wire names are unchanged.
  • Delete configuration-service legacy conversion and virtual image/TTS sections. Current schemas/defaults and SDK configuration models use the canonical fields. Retired sections return CONFIG_NOT_FOUND; old/unknown declared PATCH fields return CONFIG_VALIDATION_FAILED. Partial updates, blank secret retention and freeform record keys remain supported.
  • Require the configuration migration ledger before configuration initialization. Historical conversion lives only in release migrations and read-only diagnostics.
  • Delete plaintext/ECB write modes, encryption-disable branches, machine-ID fallback and short-key derivation. Production builds require explicit hex64 field keys; JWT secrets of at least 32 characters remain required in every runtime. All new credentials use authenticated encryption or API-key hashes; historical reads/recovery remain.
  • Remove algorithm selection from runtime/codec configuration, ENCRYPT_ALGORITHM, CLI --encrypt_algorithm and YAML encrypt_algorithm. Removed env/YAML options fail explicitly and read-only check reports REMOVED_ENCRYPTION_OPTION. Field writes use fixed AES-256-GCM (96-bit random nonce, 128-bit tag, field AAD); historical default AES-256-ECB remains read-only.
  • Let source development startup omit the field key: generate and atomically publish a random key at tmp/dev-encryption.key (mode 0600), reuse it across restarts/workers and prefer explicit keys. Bad local/explicit keys fail without replacement. Release builds remain strict even under NODE_ENV=development, and standalone diagnostics still require the effective key in environment configuration.
  • Delete independent prepare, its dry-run/replica flags and preparation marker. Register mandatory 20261109-required-secret-storage after configuration migration, reusing transactional credential conversion with a new ID so historical opt-in ledger rows cannot skip the v18 backfill. Remove migrate:secrets and its activation flag.
  • Retain standalone read-only check and file-based maintenance JWT rotation; fix historical OAuth ciphertext rewrapping. Preserve credentials and publish configuration cache/events only after commit.
  • Backfill AI metrics and stop runtime references to retired metadata columns. Add compatible POST maintenance routes and migrate Admin/SDK request transport while retaining the remaining legacy GET routes.

Deployment contract

Stop every old application replica and background writer, rotate a short JWT if needed, then run node migrate.mjs (source: pnpm -C apps/core migrate). This single release runner handles configuration, credentials and AI metrics. Run read-only check, then start the new runtime. This configuration JSON contract change requires a maintenance cutover; it is not safe to let old-format writers overlap. A short existing JWT must be rewrapped before starting this runtime. App and migration services share persisted secrets.

Remove retired algorithm settings before this cutover, including old default values. Instances using a custom historical field algorithm must first convert affected credentials to V2 on the prior version with the original algorithm/key and verify them; preserve those settings for isolated old-snapshot recovery.

SDK configuration model removals belong to the matching major release and must not be published as an SDK 5.x patch. Other API removals, full SDK 6 cleanup and physical column deletion remain later work. No version bump, npm publication, production migration or deployment was performed.

Companion documentation: mx-space/docs#52

Validation

  • Admin settings acceptance: real owner login against migrated legacy data in isolated PostgreSQL/Redis; all 18 generated configuration sections round-trip through the actual Admin client. GitHub, membership, SMTP and AI edits persist after browser reload; five credential values remain encrypted and unchanged. Comment-upload limits save and reload as 7/70 after the casing fix.

  • Three new RED→GREEN regressions; 73 focused Admin tests, 40 focused Core tests, both production builds/typechecks, scoped lint/format and API guards pass. This covers settings display/submission/persistence with fixture credentials, not external email/payment/model-provider connectivity.

  • Real before/after screenshots and save/reload recording are available at https://ryugi.innei.dev/a/48IrQ95Vj1VecQTZtWKtf ; visual acceptance was confirmed by the user in the verification page and conversation.

  • Final Core regression after development-key support: 434 files passed, 3 skipped; 3594 tests passed, 11 intentionally skipped. Command: TZ=UTC pnpm -C apps/core test run --maxWorkers=3 --exclude '**/admin-agent.e2e-spec.ts' (exit 0).

  • Development-key startup regression first failed for the missing key, then passed. All 22 startup boundary tests pass, including concurrent first boots, restart decryption, mode 0600, explicit-key precedence, invalid-key preservation, non-development modes and the built release refusing fallback. A real PostgreSQL source-development migration without a configured field key preserves V2 credentials and is idempotent on restart.

  • Seven new algorithm-removal RED→GREEN regressions; 25 focused tests pass. Runtime refuses removed env/CLI/YAML options, check remains read-only, and default ECB conversion is independently decryptable as AES-256-GCM. Core build, Core/E2E typechecks, API guards, scoped lint/format and entrypoint shell syntax pass.

  • The unfiltered run still fails the pre-existing Agent fixture: missing AiAgentConversationRepository during setup, followed by cleanup failure (7 tests blocked). Its initial archive fixture failure was corrected by aligning both inherited PG connection aliases, verified in isolation and in the final regression. No claim that the unfiltered command is green.

  • New regressions first failed for omitted credential backfill, stale historical migration ledgers, the removed command and the obsolete preparation-marker requirement. Focused migration/config/JWT/check coverage and built release-entry archive restore pass.

  • Prior independent configuration review: two data-preservation findings fixed through 5 real-PG RED→GREEN regressions (all raw known sections validated before defaults; canonical/mixed third-party siblings preserved).

  • Real private PostgreSQL release runner automatically migrates legacy data, preserves credential values/AAD, removes retired rows and records completion. Conflict, damaged ciphertext and malformed credential shape roll back without a completion marker.

  • Current-only service regressions: startup/missing-marker refusal, retired GET/PATCH refusal, strict unknown-field rejection, partial updates and freeform OAuth keys.

  • Historical ZIP/v1 archive restored into private PostgreSQL, migrated through built migrate.mjs and verified through built read-only check; original credentials retained.

  • Core and Admin production builds; Core/Admin/SDK/E2E typechecks; API envelope/validation guards; migration lint over all 50 SQL files.

  • SDK 207 tests. Both Compose templates validate required shared secrets.

Database migrations checklist

  • App-data migration is registered unconditionally and runs in the existing release phase, before configuration startup.
  • JSON configuration contract is migrated under the shared configuration lock and a transaction; stop old writers for this major-version cutover.
  • No new DDL or physical column deletion; migration schema retains the older AI columns.
  • Required v18 credential migration reuses the historical transactional converter under a new ledger ID. The old opt-in registration is removed; historical ledger rows are preserved. Failed conversion does not record completion.
  • Further column contraction remains a separate release after all old column references retire.

@safedep

safedep Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

SafeDep policy: failed

Policy: SafeDep default policy revision 1. The plan of this tenant has no policy management.

Setting Value Set by Coverage
scan_gate.fail_on HIGH SafeDep default policy ENFORCE
Package Finding Result Rule
vite 6.0.5 2 vulnerabilities (2 high): GHSA-587h-7wmj-h5pm, GHSA-p9ff-h696-f583 Failing scan_gate.fail_on

Fixed rule of the check: risky-oss-licenses

SafeDep Report Summary

Green Malicious Packages Badge Red Vulnerable Packages Badge Green Risky License Badge

Package Details
Package Malware Vulnerability Risky License Report
icon vite @ 6.0.5
apps/core/package.json
ok icon
red cross icon
ok icon
🔗
icon @compodoc/compodoc @ 1.1.26
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon bump @ 0.0.0
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon cross-env @ 7.0.3
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon mime-types @ 3.0.2
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon mkdirp @ 3.0.1
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon nanoid @ 6.0.2
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon nodemailer @ 10.2.1
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon nodemon @ 3.1.4
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon object-scan @ 20.0.4
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon octokit @ 5.0.5
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon rimraf @ 6.0.2
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon tsx @ 4.19.2
apps/core/package.json
ok icon
ok icon
ok icon
🔗
icon vite-node @ 2.1.8
apps/core/package.json
ok icon
ok icon
ok icon
🔗

View complete scan results →

This report is generated by SafeDep GitHub App

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T14:35:22.693672Z dde2289 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@gitguardian

gitguardian Bot commented Nov 8, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
35876281 Triggered Generic Encryption Key 565ebb9 .github/workflows/ci.yml View secret
35876281 Triggered Generic Encryption Key 565ebb9 .github/workflows/ci.yml View secret
35876281 Triggered Generic Encryption Key 565ebb9 .github/workflows/release.yml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@Innei Innei changed the title feat(core): prepare v17 installations for v18 compatibility changes feat(core): require encrypted storage and prepare v18 upgrades Nov 8, 2026
@Innei Innei changed the title feat(core): require encrypted storage and prepare v18 upgrades feat(core)!: migrate legacy configuration and require encryption Nov 8, 2026
Innei added 2 commits November 9, 2026 02:24
BREAKING CHANGE: Remove ENCRYPT_ALGORITHM, --encrypt_algorithm and YAML encrypt_algorithm. Field writes use fixed AES-256-GCM; only the default ECB legacy reader remains. Custom legacy algorithms require conversion on the prior version before upgrading.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant