From cfb0cf205012e1556caa835d410990000f80db28 Mon Sep 17 00:00:00 2001 From: Rob Royce Date: Tue, 6 Oct 2026 16:16:16 -0700 Subject: [PATCH 1/4] feat(desktop): bundle the storage service and a pinned Node.js runtime The packaged app now starts the storage service without an installed Node.js, without the repository, and without the working directory. - scripts/prepare-node-runtime.mjs downloads the official Node.js 24.21.0 archive, checks the pinned SHA-256 and the upstream SHASUMS256.txt, and extracts bin/node and LICENSE to vendor/node/-/. - electron-builder copies the runtime, its license, kc_storage, and kc_contracts to Contents/Resources, outside app.asar. - Packaged runs use only Resources/node/bin/node and pass only the token to the service. Unpackaged runs keep KC_NODE or node from PATH. - yarn package-local builds an unsigned local package. - docs/desktop-storage-completion.md: design for this milestone. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 3 + docs/desktop-storage-completion.md | 107 +++++++++++++++++++ package.json | 32 +++++- scripts/prepare-node-runtime.mjs | 113 +++++++++++++++++++++ src/kc_electron/src/app/storage.process.ts | 74 ++++++++++---- 5 files changed, 306 insertions(+), 23 deletions(-) create mode 100644 docs/desktop-storage-completion.md create mode 100644 scripts/prepare-node-runtime.mjs diff --git a/.gitignore b/.gitignore index 9882c7e2..b4f21942 100755 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,6 @@ auth.json # End-to-end test output e2e/.output/ + +# Prepared Node.js runtime for packaging +vendor/ diff --git a/docs/desktop-storage-completion.md b/docs/desktop-storage-completion.md new file mode 100644 index 00000000..ed49139e --- /dev/null +++ b/docs/desktop-storage-completion.md @@ -0,0 +1,107 @@ +# Desktop Storage Completion (Milestone 2) + +## Purpose + +Make the desktop app self-contained for local use: + +- The unsigned package starts the storage service without an installed + Node.js. +- A user can restore a library backup through Settings > Backup. + +Not in this milestone: signing, notarization, publishing, cloud hosting, +browser client. + +## Findings from the current code + +| Area | Current behavior | Problem | +| --- | --- | --- | +| Runtime | Electron runs `node` from `PATH` or `KC_NODE`. | A packaged app on a machine without Node.js cannot start the service. | +| Service files | Read from `/src/kc_storage`. | The package has no `src/kc_storage`. Files inside `app.asar` cannot be read by a separate Node.js process. | +| Restore | `restore` command only. It writes records and moves files in one step. It checks only `.tar` entry names it needs and ignores other entries. | No UI. Unknown entry types are not rejected. Validation and activation are not separate. | +| Locks | One process per data directory (`storage.lock`). | The running service owns the library, so the UI cannot use the command line restore. | + +## Packaging the service + +- `scripts/prepare-node-runtime.mjs` downloads the official Node.js + archive for one platform from `nodejs.org/dist`. The version and the + archive SHA-256 are pinned in the script. The script also checks the hash + against the upstream `SHASUMS256.txt`. It extracts `bin/node` and + `LICENSE` to `vendor/node/-/`. Git ignores `vendor/`. +- Pinned runtime: Node.js 24.21.0 (LTS), `darwin-arm64`. +- electron-builder copies these into the app, outside `app.asar`: + + | Package path | Source | + | --- | --- | + | `Contents/Resources/node/bin/node` | `vendor/node/darwin-/bin/node` | + | `Contents/Resources/node/LICENSE` | Node.js license notices | + | `Contents/Resources/kc_storage/` | `src/kc_storage/src/*.ts`, `package.json` | + | `Contents/Resources/kc_contracts/` | `src/kc_contracts/*.ts`, `package.json` | + +- Runtime selection in Electron main: + + | Run | Node.js | Service entry | Working directory | + | --- | --- | --- | --- | + | Packaged | `process.resourcesPath/node/bin/node` (only) | `process.resourcesPath/kc_storage/src/main.ts` | `process.resourcesPath` | + | Unpackaged | `KC_NODE` or `node` from `PATH` | `/src/kc_storage/src/main.ts` | repository | + +- A packaged run gives the service only `KC_STORAGE_TOKEN`. It does not pass + `PATH` or other variables. +- The service stays independent of Electron. `yarn storage ...` still works. + +## Restore through the interface + +The storage service does all validation and activation. The renderer +uploads the selected file. A future browser client can use the same API. + +| Step | Request | Service action | +| --- | --- | --- | +| Check | `GET /v1/library` | Return record counts and `empty`. The UI disables restore and explains why if the library is not empty. | +| Upload and validate | `POST /v1/restores` (body: the `.tar`) | Refuse at once if the library is not empty. Stream the file to `restore//archive.tar` (size limit). Validate the archive, manifest, records, relationships, and every asset hash. Copy assets to `restore//files/`. Return a preview. | +| Preview | — | UI shows counts, project names, backup date, warnings, and what the backup does not contain. | +| Confirm | `POST /v1/restores//activate` | Activate (below). Return counts. | +| Cancel | `DELETE /v1/restores/` | Remove the staging directory. | + +After activation the UI shows the result and reloads. + +## No concurrent writes + +Activation is one synchronous function in the service's single thread: +check that the library is empty, start a transaction, insert records, move +staged files into place, commit. No other request runs during it. +Validation runs before activation, and other writes can happen then. So +activation checks again that the library is empty, inside the transaction. +The restore never merges into or replaces user records. + +## Failure and interruption + +| Failure | Result | +| --- | --- | +| Invalid archive, version, record, relationship, or hash | Error before activation. Staging directory removed. Library unchanged. | +| Error during activation | Transaction rolled back. Moved files removed. Staging removed. | +| Process stops during activation, before commit | SQLite discards the transaction. On the next start, the service removes asset files without records, and all `restore/` staging directories. Library is empty again. | +| Process stops after commit | Restore is complete. Files were moved before the commit. | +| Process stops before activation | Staging is removed at the next start. | + +## Archive safety + +- Allowed entries: `manifest.json` and `assets/` only. + Regular files only. No prefix field, no duplicates. +- Every manifest asset must have an entry with the same size. Every asset + entry must be in the manifest. +- The service never uses an entry name as a path. Files are written by + asset ID. +- Limits: archive 8 GiB, manifest 64 MiB, 1,000,000 entries, each asset at + most the upload limit (2 GiB). + +The command line `restore` uses the same validation and activation code. + +## Tests + +- Service tests (no Electron): restore through HTTP and the command line, + non-empty refusal, unsupported version, missing and damaged assets, + unsafe entries (path traversal, symbolic link, unknown entry), failure + during activation, process stop during activation and restart. +- Packaged app (`yarn e2e-packaged`): scratch profiles, working directory + outside the repository, `PATH=/usr/bin:/bin:/usr/sbin:/sbin`, no + credentials. The full workflow with UI restore, the service command line + checked for the bundled runtime, two instances, process cleanup. diff --git a/package.json b/package.json index 7f4c0f46..e628fc8c 100755 --- a/package.json +++ b/package.json @@ -27,6 +27,8 @@ "clean": "rm -rf dist && yarn workspaces foreach run clean", "purge": "yarn workspaces foreach run purge && rm -rf node_modules && yarn cache clean", "pack": "electron-builder --dir", + "prepare-runtime": "node scripts/prepare-node-runtime.mjs", + "package-local": "yarn build && yarn prepare-runtime && CSC_IDENTITY_AUTO_DISCOVERY=false electron-builder --dir", "dist": "electron-builder --publish never --x64 --arm64", "dist-all": "electron-builder -mwl --x64 --arm64 --publish never", "interactive-upgrade": "yarn upgrade-interactive", @@ -39,6 +41,7 @@ "start-debug": "electron . --inspect=5858", "test": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --test \"tests/*.test.mts\"", "e2e": "node --test --test-concurrency=1 \"e2e/*.e2e.mjs\"", + "e2e-packaged": "node --test --test-concurrency=1 \"e2e/packaged/*.e2e.mjs\"", "storage": "node src/kc_storage/src/main.ts", "test-storage": "yarn workspace kc_storage test", "watch-main": "cd src/kc_angular && yarn run watch-main", @@ -122,6 +125,23 @@ { "from": "node_modules/@dqbd/tiktoken/tiktoken_bg.wasm", "to": "tiktoken_bg.wasm" + }, + { + "from": "src/kc_storage", + "to": "kc_storage", + "filter": [ + "package.json", + "src/**/*.ts" + ] + }, + { + "from": "src/kc_contracts", + "to": "kc_contracts", + "filter": [ + "package.json", + "*.ts", + "!*.d.ts" + ] } ], "publish": { @@ -139,7 +159,17 @@ "dmg", "zip" ], - "darkModeSupport": true + "darkModeSupport": true, + "extraResources": [ + { + "from": "vendor/node/darwin-${arch}/bin/node", + "to": "node/bin/node" + }, + { + "from": "vendor/node/darwin-${arch}/LICENSE", + "to": "node/LICENSE" + } + ] }, "linux": { "target": [ diff --git a/scripts/prepare-node-runtime.mjs b/scripts/prepare-node-runtime.mjs new file mode 100644 index 00000000..94fdc00a --- /dev/null +++ b/scripts/prepare-node-runtime.mjs @@ -0,0 +1,113 @@ +/* + * Prepare the Node.js runtime that the desktop package uses to run the + * storage service. + * + * node scripts/prepare-node-runtime.mjs [--platform darwin] [--arch arm64] + * + * Downloads the official archive from nodejs.org, checks its SHA-256 + * against the value pinned here and against the upstream SHASUMS256.txt, + * and extracts bin/node and LICENSE to vendor/node/-/. + * The archive is cached in vendor/node/cache/. Run it again: it does + * nothing when the runtime is already prepared. + */ + +import crypto from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import { execFileSync } from "node:child_process"; +import { parseArgs } from "node:util"; + +const VERSION = "24.21.0"; + +/** SHA-256 of node-v--.tar.gz, from SHASUMS256.txt. */ +const PINNED_SHA256 = { + "darwin-arm64": + "bed7eea5325e1108f32ce5228ddd6a5f0f08a499ee42aa7442aea583702f6057", + "darwin-x64": + "1462cb3b3046b815cf8ea436d3da450ec1a9f11dac7e5a46b0ada5305d7e8097", +}; + +const ROOT = path.resolve(import.meta.dirname, ".."); +const { values } = parseArgs({ + options: { + platform: { type: "string", default: process.platform }, + arch: { type: "string", default: process.arch }, + }, +}); + +const target = `${values.platform}-${values.arch}`; +const expected = PINNED_SHA256[target]; +if (!expected) { + console.error( + `No pinned Node.js runtime for ${target}. Supported: ${Object.keys(PINNED_SHA256).join(", ")}` + ); + process.exit(1); +} + +const name = `node-v${VERSION}-${target}`; +const archiveName = `${name}.tar.gz`; +const baseUrl = `https://nodejs.org/dist/v${VERSION}`; +const outDir = path.join(ROOT, "vendor", "node", target); +const cacheDir = path.join(ROOT, "vendor", "node", "cache"); +const stamp = path.join(outDir, "VERSION"); + +function sha256(file) { + return crypto.createHash("sha256").update(fs.readFileSync(file)).digest("hex"); +} + +async function download(url, file) { + const res = await fetch(url); + if (!res.ok) { + throw new Error(`Download failed: ${url} (${res.status})`); + } + fs.writeFileSync(file, Buffer.from(await res.arrayBuffer())); +} + +if (fs.existsSync(stamp) && fs.readFileSync(stamp, "utf8").trim() === `${VERSION} ${expected}`) { + console.log(`Node.js ${VERSION} for ${target} is ready: ${outDir}`); + process.exit(0); +} + +fs.mkdirSync(cacheDir, { recursive: true }); +const archive = path.join(cacheDir, archiveName); +if (!fs.existsSync(archive) || sha256(archive) !== expected) { + console.log(`Downloading ${baseUrl}/${archiveName}`); + await download(`${baseUrl}/${archiveName}`, archive); +} + +const actual = sha256(archive); +if (actual !== expected) { + fs.rmSync(archive, { force: true }); + throw new Error(`SHA-256 mismatch for ${archiveName}: got ${actual}, pinned ${expected}`); +} + +const sums = await (await fetch(`${baseUrl}/SHASUMS256.txt`)).text(); +const upstream = sums + .split("\n") + .map((line) => line.trim().split(/\s+/)) + .find(([, file]) => file === archiveName)?.[0]; +if (upstream !== expected) { + throw new Error(`Upstream SHASUMS256.txt does not match the pinned hash for ${archiveName}.`); +} + +fs.rmSync(outDir, { recursive: true, force: true }); +fs.mkdirSync(path.join(outDir, "bin"), { recursive: true }); +execFileSync("tar", [ + "-xzf", + archive, + "-C", + outDir, + "--strip-components=1", + `${name}/bin/node`, + `${name}/LICENSE`, +]); + +const reported = execFileSync(path.join(outDir, "bin", "node"), ["--version"], { + encoding: "utf8", +}).trim(); +if (reported !== `v${VERSION}`) { + throw new Error(`Extracted runtime reports ${reported}, expected v${VERSION}.`); +} + +fs.writeFileSync(stamp, `${VERSION} ${expected}\n`); +console.log(`Node.js ${VERSION} for ${target} is ready: ${outDir} (sha256 ${expected})`); diff --git a/src/kc_electron/src/app/storage.process.ts b/src/kc_electron/src/app/storage.process.ts index 7f22ddaa..0a5ef8f0 100644 --- a/src/kc_electron/src/app/storage.process.ts +++ b/src/kc_electron/src/app/storage.process.ts @@ -15,11 +15,17 @@ */ /* - * Starts the standalone storage service (src/kc_storage) with Node.js. + * Starts the standalone storage service (kc_storage) with Node.js. * - * Electron's own Node.js is too old for node:sqlite, so the service runs - * with the Node.js on PATH, or with KC_NODE. The service stops when this - * process closes its standard input, including after a crash. + * Electron's own Node.js is too old for node:sqlite, so the service runs in + * a separate Node.js 24 process: + * + * Packaged: Resources/node/bin/node (bundled) and Resources/kc_storage. + * PATH, KC_NODE, and the working directory are not used. + * Unpackaged: KC_NODE or node from PATH, and /src/kc_storage. + * + * The service stops when this process closes its standard input, including + * after a crash. */ import { spawn, ChildProcess } from "child_process"; @@ -32,22 +38,50 @@ const START_TIMEOUT_MS = 20000; let child: ChildProcess | undefined; +interface ServiceLocation { + node: string; + entry: string; + cwd: string; + env: NodeJS.ProcessEnv; +} + +function serviceLocation(token: string): ServiceLocation { + if (app.isPackaged) { + const resources = process.resourcesPath; + return { + node: path.join(resources, "node", "bin", "node"), + entry: path.join(resources, "kc_storage", "src", "main.ts"), + cwd: resources, + env: { KC_STORAGE_TOKEN: token }, + }; + } + // Only the variables the service needs. ELECTRON_RUN_AS_NODE and + // test-runner variables must not reach it. + return { + node: process.env.KC_NODE || "node", + entry: path.join(app.getAppPath(), "src", "kc_storage", "src", "main.ts"), + cwd: app.getAppPath(), + env: { + PATH: process.env.PATH, + HOME: process.env.HOME, + KC_STORAGE_TOKEN: token, + }, + }; +} + export function startStorageService(dataDir: string): Promise { - const entry = path.join( - app.getAppPath(), - "src", - "kc_storage", - "src", - "main.ts" - ); - const node = process.env.KC_NODE || "node"; const token = newToken(); + const { node, entry, cwd, env } = serviceLocation(token); return new Promise((resolve) => { if (!fs.existsSync(entry)) { resolve({ error: `Storage service not found at ${entry}.` }); return; } + if (app.isPackaged && !fs.existsSync(node)) { + resolve({ error: `Bundled Node.js runtime not found at ${node}.` }); + return; + } let stderr = ""; let settled = false; @@ -58,14 +92,6 @@ export function startStorageService(dataDir: string): Promise { } }; - // Only the variables the service needs. ELECTRON_RUN_AS_NODE and - // test-runner variables must not reach it. - const env: NodeJS.ProcessEnv = { - PATH: process.env.PATH, - HOME: process.env.HOME, - KC_STORAGE_TOKEN: token, - }; - child = spawn( node, [ @@ -79,9 +105,11 @@ export function startStorageService(dataDir: string): Promise { "null", "--exit-on-stdin-close", ], - { env, stdio: ["pipe", "pipe", "pipe"] } + { cwd, env, stdio: ["pipe", "pipe", "pipe"] } ); + console.log(`[Knowledge]: storage service runtime: ${node} ${entry}`); + const timer = setTimeout(() => { finish({ error: `Storage service did not start in time. ${stderr}`.trim(), @@ -92,7 +120,9 @@ export function startStorageService(dataDir: string): Promise { child.on("error", (e) => { clearTimeout(timer); finish({ - error: `Could not run "${node}": ${e.message}. Install Node.js 24 or set KC_NODE.`, + error: app.isPackaged + ? `Could not run the bundled Node.js runtime (${node}): ${e.message}.` + : `Could not run "${node}": ${e.message}. Install Node.js 24 or set KC_NODE.`, }); }); From 8d41e5645e4e5826cec92b4bd259b24bf6d1af09 Mon Sep 17 00:00:00 2001 From: Rob Royce Date: Tue, 6 Oct 2026 16:19:23 -0700 Subject: [PATCH 2/4] feat(storage): staged library restore with validation and safe activation Restore is now two steps in the service, used by both the command line and the HTTP API: - stageRestore validates the whole backup (format, supported version, records, duplicate IDs, project and asset references, sizes, SHA-256 of every file) and copies the files to restore//. The library does not change. - activateRestore runs as one synchronous step: refuse a library with records, insert records, move files, commit. A stop before commit leaves files without records, which the next start removes with old staging directories. The tar reader now accepts only manifest.json and assets/ as regular files, and rejects other entry types, prefixes, duplicates, and more than 1,000,000 entries. Entry names are never used as paths. API: GET /v1/library, POST /v1/restores (upload and validate), POST /v1/restores//activate, DELETE /v1/restores/. Tests: API and command line restore, non-empty refusal, writes after validation, cancel, unsupported version, missing and damaged files, unsafe entries, failure before commit, and process stop with restart. Co-Authored-By: Claude Opus 5.5 --- src/kc_contracts/storage.ts | 19 ++ src/kc_storage/src/assets.ts | 13 +- src/kc_storage/src/backup.ts | 123 +-------- src/kc_storage/src/cli.ts | 24 +- src/kc_storage/src/datadir.ts | 10 + src/kc_storage/src/http.ts | 104 +++++++- src/kc_storage/src/library.ts | 16 +- src/kc_storage/src/restore.ts | 373 ++++++++++++++++++++++++++++ src/kc_storage/src/tar.ts | 59 ++++- src/kc_storage/test/backup.test.ts | 4 +- src/kc_storage/test/restore.test.ts | 296 ++++++++++++++++++++++ 11 files changed, 896 insertions(+), 145 deletions(-) create mode 100644 src/kc_storage/src/restore.ts create mode 100644 src/kc_storage/test/restore.test.ts diff --git a/src/kc_contracts/storage.ts b/src/kc_contracts/storage.ts index 97e458c6..82d82993 100644 --- a/src/kc_contracts/storage.ts +++ b/src/kc_contracts/storage.ts @@ -90,6 +90,25 @@ export interface SourceList { export const ASSET_FILENAME_HEADER = "x-knowledge-filename"; export const ASSET_ORIGINAL_PATH_HEADER = "x-knowledge-original-path"; +/** GET /v1/library */ +export interface LibraryStatus { + empty: boolean; + counts: { projects: number; sources: number; assets: number }; +} + +/** Result of POST /v1/restores: a validated backup that is not active yet. */ +export interface RestorePreview { + id: string; + backupCreatedAt: string | null; + backupVersion: number; + counts: { projects: number; sources: number; assets: number; bytes: number }; + /** First 50 project names. */ + projectNames: string[]; + warnings: string[]; + /** Data that a library backup never contains. */ + notIncluded: string[]; +} + export interface LibraryBackupManifest { format: "knowledge-library-backup"; version: 1; diff --git a/src/kc_storage/src/assets.ts b/src/kc_storage/src/assets.ts index e83d2d42..c8973f8c 100644 --- a/src/kc_storage/src/assets.ts +++ b/src/kc_storage/src/assets.ts @@ -34,9 +34,16 @@ export class AssetStore { return path.join(this.root, id.slice(0, 2), id); } - /** Write a stream to a temporary file and hash it. */ - async stage(input: Readable, maxBytes: number): Promise { - const tmpPath = path.join(this.tmpDir, crypto.randomUUID()); + /** + * Write a stream to a temporary file and hash it. The directory must be + * on the same filesystem as the store (default: assets/tmp). + */ + async stage( + input: Readable, + maxBytes: number, + dir = this.tmpDir + ): Promise { + const tmpPath = path.join(dir, crypto.randomUUID()); const hash = crypto.createHash("sha256"); let size = 0; diff --git a/src/kc_storage/src/backup.ts b/src/kc_storage/src/backup.ts index 95c7be80..d765fcf9 100644 --- a/src/kc_storage/src/backup.ts +++ b/src/kc_storage/src/backup.ts @@ -4,20 +4,11 @@ * history, UI preferences, settings, or credentials. */ -import fs from "node:fs"; -import path from "node:path"; import type { Writable } from "node:stream"; import type { LibraryBackupManifest } from "../../kc_contracts/storage.ts"; -import { DATABASE_FILE, openDataDir, type DataDir } from "./datadir.ts"; -import { SCHEMA_VERSION, transaction } from "./db.ts"; -import type { StagedFile } from "./assets.ts"; -import { - readTarEntry, - readTarIndex, - streamTarEntry, - TarWriter, -} from "./tar.ts"; -import { validAssetId, validProject, validSource } from "./validate.ts"; +import type { DataDir } from "./datadir.ts"; +import { SCHEMA_VERSION } from "./db.ts"; +import { TarWriter } from "./tar.ts"; import { VERSION } from "./version.ts"; export const BACKUP_FORMAT = "knowledge-library-backup"; @@ -51,111 +42,3 @@ export async function writeBackup(dir: DataDir, out: Writable) { await tar.finish(); return manifest; } - -export interface RestoreSummary { - projects: number; - sources: number; - assets: number; -} - -/** - * Restore a backup into a new or empty data directory. Validates the - * manifest and every file hash before the database changes. - */ -export async function restoreBackup( - archive: string, - root: string -): Promise { - if (fs.existsSync(path.join(root, DATABASE_FILE))) { - const probe = openDataDir(root); - const empty = probe.library.isEmpty(); - probe.close(); - if (!empty) { - throw new Error( - `The data directory ${root} already has a library. Use an empty directory.` - ); - } - } - - const index = readTarIndex(archive); - const manifestEntry = index.get("manifest.json"); - if (!manifestEntry) { - throw new Error("The backup has no manifest.json."); - } - const manifest = JSON.parse( - readTarEntry(archive, manifestEntry).toString("utf8") - ); - if (manifest?.format !== BACKUP_FORMAT || manifest.version !== 1) { - throw new Error("The file is not a knowledge-library-backup version 1."); - } - - const projects = (manifest.projects ?? []).map((p: any) => - validProject(p.id, p) - ); - const sources = (manifest.sources ?? []).map((s: any) => - validSource(s.id, s) - ); - const assetList = (manifest.assets ?? []).map((a: any) => { - validAssetId(a.id); - const entry = index.get(`assets/${a.id}`); - if (!entry || entry.size !== a.size) { - throw new Error(`The backup has no valid file for asset ${a.id}.`); - } - return { record: a, entry }; - }); - - const dir = openDataDir(root); - const staged: { id: string; file: StagedFile }[] = []; - try { - for (const { record, entry } of assetList) { - const file = await dir.assets.stage( - streamTarEntry(archive, entry), - Number.MAX_SAFE_INTEGER - ); - staged.push({ id: record.id, file }); - if (file.sha256 !== record.sha256) { - throw new Error( - `Hash mismatch for asset ${record.id}. The backup is damaged.` - ); - } - } - - const createdAt = new Map(); - for (const r of [ - ...(manifest.projects ?? []), - ...(manifest.sources ?? []), - ]) { - if (r.createdAt) createdAt.set(r.id, r.createdAt); - } - - transaction(dir.db, () => { - for (const { record } of assetList) { - dir.library.insertAsset({ - id: record.id, - sha256: record.sha256, - size: record.size, - mediaType: record.mediaType, - filename: record.filename, - originalPath: record.originalPath ?? null, - createdAt: record.createdAt, - }); - } - for (const p of projects) dir.library.putProject(p, createdAt.get(p.id)); - for (const s of sources) dir.library.putSource(s, createdAt.get(s.id)); - for (const { id, file } of staged) dir.assets.commit(file, id); - }); - return { - projects: projects.length, - sources: sources.length, - assets: assetList.length, - }; - } catch (e) { - for (const { id, file } of staged) { - dir.assets.discard(file); - if (!dir.library.getAsset(id)) dir.assets.remove(id); - } - throw e; - } finally { - dir.close(); - } -} diff --git a/src/kc_storage/src/cli.ts b/src/kc_storage/src/cli.ts index 1c552102..4db1aafa 100644 --- a/src/kc_storage/src/cli.ts +++ b/src/kc_storage/src/cli.ts @@ -6,7 +6,8 @@ import fs from "node:fs"; import path from "node:path"; import { parseArgs } from "node:util"; import type { AddressInfo } from "node:net"; -import { restoreBackup, writeBackup } from "./backup.ts"; +import { writeBackup } from "./backup.ts"; +import { activateRestore, assertEmpty, stageRestore } from "./restore.ts"; import { openDataDir } from "./datadir.ts"; import { createServer } from "./http.ts"; import { migrate } from "./migrate.ts"; @@ -106,10 +107,23 @@ export async function main(argv: string[]) { } case "restore": { if (!values.from) fail("--from is required."); - const result = await restoreBackup(values.from, dataDir); - console.log( - JSON.stringify({ restored: result, dataDir: path.resolve(dataDir) }) - ); + // Same validation and activation as the HTTP API + const dir = openDataDir(dataDir); + try { + assertEmpty(dir); + const staged = await stageRestore(dir, values.from); + const result = activateRestore(dir, staged); + console.log( + JSON.stringify({ + restored: result, + warnings: staged.preview.warnings, + notIncluded: staged.preview.notIncluded, + dataDir: dir.root, + }) + ); + } finally { + dir.close(); + } return; } default: diff --git a/src/kc_storage/src/datadir.ts b/src/kc_storage/src/datadir.ts index c4265ebd..7eab6b72 100644 --- a/src/kc_storage/src/datadir.ts +++ b/src/kc_storage/src/datadir.ts @@ -22,6 +22,9 @@ export interface DataDir { export const DATABASE_FILE = "library.sqlite"; +/** Staging directories of restores that are not active yet. */ +export const RESTORE_DIR = "restore"; + function alive(pid: number) { try { process.kill(pid, 0); @@ -84,6 +87,13 @@ export function openDataDir(root: string): DataDir { // Recover from an interrupted upload, migration, or restore const swept = assets.sweep(library.assetIds()); + const staging = path.join(resolved, RESTORE_DIR); + if (fs.existsSync(staging)) { + for (const entry of fs.readdirSync(staging)) { + fs.rmSync(path.join(staging, entry), { recursive: true, force: true }); + swept.push(`${RESTORE_DIR}/${entry}`); + } + } return { root: resolved, diff --git a/src/kc_storage/src/http.ts b/src/kc_storage/src/http.ts index f03c8db2..992b092b 100644 --- a/src/kc_storage/src/http.ts +++ b/src/kc_storage/src/http.ts @@ -11,8 +11,22 @@ import { ASSET_ORIGINAL_PATH_HEADER, type ErrorResponse, type HealthResponse, + type LibraryStatus, } from "../../kc_contracts/storage.ts"; import { writeBackup } from "./backup.ts"; +import { + activateRestore, + assertEmpty, + discardRestore, + newStagingDir, + RESTORE_LIMITS, + RestoreError, + stageRestore, + type StagedRestore, +} from "./restore.ts"; +import path from "node:path"; +import { pipeline } from "node:stream/promises"; +import { Transform } from "node:stream"; import type { DataDir } from "./datadir.ts"; import { now, SCHEMA_VERSION } from "./db.ts"; import { @@ -33,6 +47,9 @@ import { } from "./validate.ts"; import { VERSION } from "./version.ts"; +/** How long a validated restore waits for confirmation. */ +const RESTORE_PENDING_MS = 30 * 60 * 1000; + export interface ServerOptions { dir: DataDir; token: string; @@ -99,6 +116,41 @@ export function createServer(options: ServerOptions): http.Server { const maxJson = options.maxJsonBytes ?? 10 * 1024 * 1024; const maxUpload = options.maxUploadBytes ?? 2 * 1024 ** 3; + // A validated restore waits here for confirmation. One at a time. + let pending: { staged: StagedRestore; timer: NodeJS.Timeout } | undefined; + const dropPending = () => { + if (pending) { + clearTimeout(pending.timer); + discardRestore(pending.staged); + pending = undefined; + } + }; + + async function receiveArchive(req: IncomingMessage, file: string) { + const declared = Number(req.headers["content-length"] ?? 0); + if (declared > RESTORE_LIMITS.archiveBytes) { + throw tooLarge( + `The backup is larger than ${RESTORE_LIMITS.archiveBytes} bytes.` + ); + } + let size = 0; + const meter = new Transform({ + transform(chunk: Buffer, _enc, done) { + size += chunk.length; + if (size > RESTORE_LIMITS.archiveBytes) { + done( + tooLarge( + `The backup is larger than ${RESTORE_LIMITS.archiveBytes} bytes.` + ) + ); + return; + } + done(null, chunk); + }, + }); + await pipeline(req, meter, fs.createWriteStream(file, { flags: "wx" })); + } + async function route(req: IncomingMessage, res: ServerResponse) { const url = new URL(req.url ?? "/", "http://localhost"); let parts: string[]; @@ -231,6 +283,54 @@ export function createServer(options: ServerOptions): http.Server { } } + if (resource === "library" && !id && method === "GET") { + const counts = library.counts(); + const status: LibraryStatus = { + empty: counts.projects + counts.sources + counts.assets === 0, + counts, + }; + return send(res, 200, status); + } + + if (resource === "restores") { + // Upload and validate. Nothing becomes active. + if (method === "POST" && !id) { + assertEmpty(dir); + dropPending(); + const staging = newStagingDir(dir); + const archive = path.join(staging.stagingDir, "archive.tar"); + try { + await receiveArchive(req, archive); + } catch (e) { + fs.rmSync(staging.stagingDir, { recursive: true, force: true }); + throw e; + } + const staged = await stageRestore(dir, archive, staging); + fs.rmSync(archive, { force: true }); + const timer = setTimeout(dropPending, RESTORE_PENDING_MS); + timer.unref(); + pending = { staged, timer }; + return send(res, 201, { restore: staged.preview }); + } + + const current = pending && pending.staged.id === id ? pending : undefined; + if (id && !sub && method === "DELETE") { + if (current) dropPending(); + return send(res, 204); + } + if (id && sub === "activate" && method === "POST") { + if (!current) { + throw notFound( + "This restore is not pending. It expired or the service restarted. Select the backup again." + ); + } + clearTimeout(current.timer); + pending = undefined; + const restored = activateRestore(dir, current.staged); + return send(res, 200, { restored }); + } + } + if (resource === "backup" && !id && method === "GET") { const date = new Date().toISOString().slice(0, 10); res.writeHead(200, { @@ -277,12 +377,14 @@ export function createServer(options: ServerOptions): http.Server { const error = e instanceof StorageError ? e + : e instanceof RestoreError + ? badRequest(e.message) : new StorageError( 500, "internal", "Internal error. See the service log." ); - if (!(e instanceof StorageError)) { + if (!(e instanceof StorageError) && !(e instanceof RestoreError)) { console.error( `[kc-storage] ${req.method} ${req.url}: ${e?.stack ?? e}` ); diff --git a/src/kc_storage/src/library.ts b/src/kc_storage/src/library.ts index 2f7dbbe9..369dfc1b 100644 --- a/src/kc_storage/src/library.ts +++ b/src/kc_storage/src/library.ts @@ -280,12 +280,18 @@ export class Library { return new Set(rows.map((r) => r.id)); } - isEmpty(): boolean { - const row = this.db + counts(): { projects: number; sources: number; assets: number } { + return this.db .prepare( - "SELECT (SELECT COUNT(*) FROM projects) + (SELECT COUNT(*) FROM sources) + (SELECT COUNT(*) FROM assets) AS n" + `SELECT (SELECT COUNT(*) FROM projects) AS projects, + (SELECT COUNT(*) FROM sources) AS sources, + (SELECT COUNT(*) FROM assets) AS assets` ) - .get() as { n: number }; - return row.n === 0; + .get() as { projects: number; sources: number; assets: number }; + } + + isEmpty(): boolean { + const c = this.counts(); + return c.projects + c.sources + c.assets === 0; } } diff --git a/src/kc_storage/src/restore.ts b/src/kc_storage/src/restore.ts new file mode 100644 index 00000000..23994fed --- /dev/null +++ b/src/kc_storage/src/restore.ts @@ -0,0 +1,373 @@ +/* + * Library restore in two steps. + * + * stageRestore() Validate the whole backup and copy its files into a + * staging directory. The active library does not change. + * activateRestore() One synchronous step: check that the library is + * empty, insert the records, move the staged files into + * place, commit. No other request runs during it. + * + * If the process stops during activation, SQLite discards the transaction + * and the next start removes files without records and old staging + * directories. The library is empty again. + * + * The command line and the HTTP API both use this module. + */ + +import crypto from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import type { + AssetRecord, + ProjectRecord, + RestorePreview, + SourceRecord, +} from "../../kc_contracts/storage.ts"; +import type { StagedFile } from "./assets.ts"; +import { RESTORE_DIR, type DataDir } from "./datadir.ts"; +import { transaction } from "./db.ts"; +import { conflict } from "./errors.ts"; +import { readTarEntry, readTarIndex, streamTarEntry } from "./tar.ts"; +import { validAssetId, validProject, validSource } from "./validate.ts"; +import { BACKUP_FORMAT } from "./backup.ts"; + +export const SUPPORTED_BACKUP_VERSIONS = [1]; + +export const RESTORE_LIMITS = { + archiveBytes: 8 * 1024 ** 3, + manifestBytes: 64 * 1024 ** 2, + entries: 1_000_000, + assetBytes: 2 * 1024 ** 3, +}; + +/** What a library backup never contains. Shown before confirmation. */ +export const NOT_IN_LIBRARY_BACKUP = [ + "Chat history", + "Inbox and UI preferences", + "Application settings", + "API keys", + "Extracted-text cache", +]; + +const ASSET_ENTRY = + /^assets\/[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; + +/** A validated backup with its files in a staging directory. */ +export interface StagedRestore { + id: string; + dir: string; + preview: RestorePreview; + projects: ProjectRecord[]; + sources: SourceRecord[]; + assets: { record: AssetRecord; file: StagedFile }[]; + createdAt: Map; +} + +export interface RestoreResult { + projects: number; + sources: number; + assets: number; +} + +export class RestoreError extends Error {} + +function fail(message: string): never { + throw new RestoreError(message); +} + +/** Refuse a library that has user records. Never merge or replace them. */ +export function assertEmpty(dir: DataDir) { + const counts = dir.library.counts(); + if (counts.projects + counts.sources + counts.assets > 0) { + throw conflict( + `Restore needs an empty library. This library has ${counts.projects} projects, ` + + `${counts.sources} sources, and ${counts.assets} files. ` + + "Restore into a new profile instead. Existing data is not changed." + ); + } +} + +/** New staging directory inside the data directory (same filesystem). */ +export function newStagingDir(dir: DataDir) { + const id = crypto.randomUUID(); + const stagingDir = path.join(dir.root, RESTORE_DIR, id); + fs.mkdirSync(path.join(stagingDir, "files"), { recursive: true }); + return { id, stagingDir }; +} + +/** + * Validate a backup archive and stage its files. Throws RestoreError with + * a user-facing message. Removes the staging directory on failure. + */ +export async function stageRestore( + dir: DataDir, + archive: string, + staging = newStagingDir(dir) +): Promise { + const { id, stagingDir } = staging; + try { + const size = fs.statSync(archive).size; + if (size > RESTORE_LIMITS.archiveBytes) { + fail(`The backup is larger than ${RESTORE_LIMITS.archiveBytes} bytes.`); + } + + let index; + try { + index = readTarIndex(archive, { + allowName: (name) => name === "manifest.json" || ASSET_ENTRY.test(name), + maxEntries: RESTORE_LIMITS.entries, + }); + } catch (e: any) { + fail(`The file is not a valid library backup: ${e.message}`); + } + + const manifestEntry = index.get("manifest.json"); + if (!manifestEntry) fail("The backup has no manifest.json."); + if (manifestEntry.size > RESTORE_LIMITS.manifestBytes) { + fail("The backup manifest is too large."); + } + let manifest: any; + try { + manifest = JSON.parse( + readTarEntry(archive, manifestEntry).toString("utf8") + ); + } catch { + fail("The backup manifest is not valid JSON."); + } + if (manifest?.format !== BACKUP_FORMAT) { + fail("The file is not a Knowledge library backup."); + } + if (!SUPPORTED_BACKUP_VERSIONS.includes(manifest.version)) { + fail( + `Backup version ${ + manifest.version + } is not supported. Supported versions: ${SUPPORTED_BACKUP_VERSIONS.join( + ", " + )}.` + ); + } + for (const list of ["projects", "sources", "assets"]) { + if (!Array.isArray(manifest[list])) + fail(`The manifest has no ${list} list.`); + } + + // Records + const projects: ProjectRecord[] = []; + const sources: SourceRecord[] = []; + const createdAt = new Map(); + const unique = (seen: Set, id: string, kind: string) => { + if (seen.has(id)) fail(`Duplicate ${kind} ID ${id} in the backup.`); + seen.add(id); + }; + const projectIds = new Set(); + for (const p of manifest.projects) { + let record: ProjectRecord; + try { + record = validProject(p?.id, p); + } catch (e: any) { + fail(`Invalid project ${JSON.stringify(p?.id)}: ${e.message}`); + } + unique(projectIds, record.id, "project"); + projects.push(record); + if (typeof p.createdAt === "string") + createdAt.set(`p:${record.id}`, p.createdAt); + } + + // Assets: manifest and archive entries must match one to one + const assets: StagedRestore["assets"] = []; + const assetIds = new Set(); + let bytes = 0; + for (const a of manifest.assets) { + try { + validAssetId(a?.id); + } catch { + fail(`Invalid asset ID ${JSON.stringify(a?.id)}.`); + } + unique(assetIds, a.id, "asset"); + if ( + !Number.isSafeInteger(a.size) || + a.size < 0 || + a.size > RESTORE_LIMITS.assetBytes + ) { + fail(`Invalid size for asset ${a.id}.`); + } + if (typeof a.sha256 !== "string" || !/^[0-9a-f]{64}$/.test(a.sha256)) { + fail(`Invalid SHA-256 for asset ${a.id}.`); + } + if ( + typeof a.filename !== "string" || + !a.filename || + typeof a.mediaType !== "string" + ) { + fail(`Invalid filename or media type for asset ${a.id}.`); + } + const entry = index.get(`assets/${a.id}`); + if (!entry) + fail( + `The file for asset ${a.id} (${a.filename}) is missing from the backup.` + ); + if (entry.size !== a.size) + fail(`The file for asset ${a.id} has the wrong size.`); + bytes += a.size; + } + for (const name of index.keys()) { + if ( + name !== "manifest.json" && + !assetIds.has(name.slice("assets/".length)) + ) { + fail(`The backup has a file that the manifest does not list: ${name}.`); + } + } + + const sourceIds = new Set(); + const usedAssets = new Set(); + for (const s of manifest.sources) { + let record: SourceRecord; + try { + record = validSource(s?.id, s); + } catch (e: any) { + fail(`Invalid source ${JSON.stringify(s?.id)}: ${e.message}`); + } + unique(sourceIds, record.id, "source"); + if (!projectIds.has(record.projectId)) { + fail( + `Source ${record.id} refers to project ${record.projectId}, which is not in the backup.` + ); + } + if (record.assetId) { + if (!assetIds.has(record.assetId)) { + fail( + `Source ${record.id} refers to asset ${record.assetId}, which is not in the backup.` + ); + } + usedAssets.add(record.assetId); + } + sources.push(record); + if (typeof s.createdAt === "string") + createdAt.set(`s:${record.id}`, s.createdAt); + } + + const warnings: string[] = []; + const dangling = projects.filter( + (p) => p.parentId && !projectIds.has(p.parentId) + ); + if (dangling.length > 0) { + warnings.push( + `${dangling.length} projects refer to a parent project that is not in the backup.` + ); + } + const unused = [...assetIds].filter((id) => !usedAssets.has(id)); + if (unused.length > 0) { + warnings.push(`${unused.length} files are not used by any source.`); + } + + // Copy and hash every file before anything becomes active + const filesDir = path.join(stagingDir, "files"); + for (const a of manifest.assets) { + const file = await dir.assets.stage( + streamTarEntry(archive, index.get(`assets/${a.id}`)!), + RESTORE_LIMITS.assetBytes, + filesDir + ); + if (file.sha256 !== a.sha256 || file.size !== a.size) { + fail( + `The file for asset ${a.id} (${a.filename}) is damaged. Its SHA-256 does not match.` + ); + } + assets.push({ + record: { + id: a.id, + sha256: a.sha256, + size: a.size, + mediaType: a.mediaType, + filename: a.filename, + originalPath: + typeof a.originalPath === "string" ? a.originalPath : null, + createdAt: + typeof a.createdAt === "string" + ? a.createdAt + : new Date().toISOString(), + }, + file, + }); + } + + const preview: RestorePreview = { + id, + backupCreatedAt: + typeof manifest.createdAt === "string" ? manifest.createdAt : null, + backupVersion: manifest.version, + counts: { + projects: projects.length, + sources: sources.length, + assets: assets.length, + bytes, + }, + projectNames: projects.slice(0, 50).map((p) => p.name), + warnings, + notIncluded: NOT_IN_LIBRARY_BACKUP, + }; + return { + id, + dir: stagingDir, + preview, + projects, + sources, + assets, + createdAt, + }; + } catch (e) { + fs.rmSync(stagingDir, { recursive: true, force: true }); + throw e; + } +} + +/** + * Make a staged restore active. Synchronous: no other request runs during + * it. Refuses a library that has records. Removes the staging directory. + */ +export function activateRestore( + dir: DataDir, + staged: StagedRestore +): RestoreResult { + const moved: string[] = []; + try { + transaction(dir.db, () => { + assertEmpty(dir); + for (const { record } of staged.assets) dir.library.insertAsset(record); + for (const p of staged.projects) + dir.library.putProject(p, staged.createdAt.get(`p:${p.id}`)); + for (const s of staged.sources) + dir.library.putSource(s, staged.createdAt.get(`s:${s.id}`)); + + // Files move before the commit. A stop before the commit leaves files + // without records, which the next start removes. + for (const { record, file } of staged.assets) { + dir.assets.commit(file, record.id); + moved.push(record.id); + if ( + process.env.KC_STORAGE_TEST_FAULT === "restore-exit-during-activation" + ) { + process.exit(70); + } + } + if (process.env.KC_STORAGE_TEST_FAULT === "restore-fail-before-commit") { + throw new RestoreError("Injected test fault before commit."); + } + }); + return { + projects: staged.projects.length, + sources: staged.sources.length, + assets: staged.assets.length, + }; + } catch (e) { + for (const id of moved) dir.assets.remove(id); + throw e; + } finally { + discardRestore(staged); + } +} + +export function discardRestore(staged: { dir: string }) { + fs.rmSync(staged.dir, { recursive: true, force: true }); +} diff --git a/src/kc_storage/src/tar.ts b/src/kc_storage/src/tar.ts index cba222e8..8c7d35a6 100644 --- a/src/kc_storage/src/tar.ts +++ b/src/kc_storage/src/tar.ts @@ -8,7 +8,13 @@ import { Readable, type Writable } from "node:stream"; const BLOCK = 512; -function header(name: string, size: number, mtime: Date): Buffer { +/** A ustar header. Type "0" is a regular file. Tests use other types. */ +export function tarHeader( + name: string, + size: number, + mtime: Date, + type = "0" +): Buffer { if (Buffer.byteLength(name) > 100) { throw new Error(`Tar entry name is too long: ${name}`); } @@ -23,7 +29,7 @@ function header(name: string, size: number, mtime: Date): Buffer { h.write(octal(size, 12), 124, 12, "ascii"); h.write(octal(Math.floor(mtime.getTime() / 1000), 12), 136, 12, "ascii"); h.write(" ", 148, 8, "ascii"); - h.write("0", 156, 1, "ascii"); + h.write(type, 156, 1, "ascii"); h.write("ustar\0", 257, 6, "ascii"); h.write("00", 263, 2, "ascii"); @@ -51,14 +57,14 @@ export class TarWriter { } async addBuffer(name: string, data: Buffer) { - await this.write(header(name, data.length, new Date())); + await this.write(tarHeader(name, data.length, new Date())); await this.write(data); await this.write(padding(data.length)); } async addFile(name: string, file: string) { const { size, mtime } = fs.statSync(file); - await this.write(header(name, size, mtime)); + await this.write(tarHeader(name, size, mtime)); let written = 0; for await (const chunk of fs.createReadStream(file)) { written += chunk.length; @@ -81,8 +87,21 @@ export interface TarEntry { size: number; } -/** Read the entry index of a tar file. Validates every header checksum. */ -export function readTarIndex(file: string): Map { +export interface TarLimits { + /** Accept only these entry names. */ + allowName: (name: string) => boolean; + maxEntries: number; +} + +/** + * Read the entry index of a tar file. Rejects bad checksums, entry types + * other than regular files, the ustar prefix field, duplicate names, and + * names that allowName refuses. Entry names are never used as paths. + */ +export function readTarIndex( + file: string, + limits: TarLimits +): Map { const fd = fs.openSync(file, "r"); const entries = new Map(); try { @@ -106,12 +125,34 @@ export function readTarIndex(file: string): Map { const name = h.toString("utf8", 0, 100).replace(/\0.*$/s, ""); const size = parseInt(h.toString("ascii", 124, 136).trim(), 8); const type = h.toString("ascii", 156, 157); + const prefix = h.toString("utf8", 345, 500).replace(/\0.*$/s, ""); + + if (type !== "0" && type !== "\0") { + throw new Error( + `Unsupported tar entry type "${type}" for ${JSON.stringify(name)}.` + ); + } + if (prefix) { + throw new Error( + `Unsupported tar name prefix for ${JSON.stringify(name)}.` + ); + } + if (!limits.allowName(name)) { + throw new Error(`Unexpected tar entry ${JSON.stringify(name)}.`); + } + if (entries.has(name)) { + throw new Error(`Duplicate tar entry ${JSON.stringify(name)}.`); + } if (!Number.isFinite(size) || size < 0 || offset + BLOCK + size > total) { - throw new Error(`Invalid tar entry size for ${name}.`); + throw new Error(`Invalid tar entry size for ${JSON.stringify(name)}.`); } - if (type === "0" || type === "\0") { - entries.set(name, { name, offset: offset + BLOCK, size }); + if (entries.size >= limits.maxEntries) { + throw new Error( + `The archive has more than ${limits.maxEntries} entries.` + ); } + + entries.set(name, { name, offset: offset + BLOCK, size }); offset += BLOCK + Math.ceil(size / BLOCK) * BLOCK; } throw new Error("The tar file ends without an end-of-archive marker."); diff --git a/src/kc_storage/test/backup.test.ts b/src/kc_storage/test/backup.test.ts index 02f053ce..152ac279 100644 --- a/src/kc_storage/test/backup.test.ts +++ b/src/kc_storage/test/backup.test.ts @@ -124,7 +124,7 @@ test("restore refuses a directory that already has a library", async () => { const result = cli(["restore", "--data-dir", source, "--from", tar]); assert.equal(result.status, 1); - assert.match(result.stderr, /already has a library/); + assert.match(result.stderr, /Restore needs an empty library/); }); test("a damaged backup is rejected and leaves the target empty", async () => { @@ -145,7 +145,7 @@ test("a damaged backup is rejected and leaves the target empty", async () => { const target = tempDir("bk3-target"); const result = cli(["restore", "--data-dir", target, "--from", tar]); assert.equal(result.status, 1); - assert.match(result.stderr, /Hash mismatch/); + assert.match(result.stderr, /is damaged/); const svc2 = await startService(target); const api2 = client(svc2); diff --git a/src/kc_storage/test/restore.test.ts b/src/kc_storage/test/restore.test.ts new file mode 100644 index 00000000..a61de0a8 --- /dev/null +++ b/src/kc_storage/test/restore.test.ts @@ -0,0 +1,296 @@ +import { afterEach, test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; +import { readTarEntry, readTarIndex, tarHeader, TarWriter } from "../src/tar.ts"; +import { FIXTURES, cli, client, startService, stopAll, tempDir, type Service } from "./helpers.ts"; + +afterEach(stopAll); + +/** A library with two projects, three sources, and two files, and its backup. */ +async function makeBackup() { + const work = tempDir("rs-work"); + const source = tempDir("rs-source"); + const svc = await startService(source); + const api = client(svc); + await api.put("/v1/projects/p1", { name: "Restore Project", parentId: null, data: { description: "d" } }); + await api.put("/v1/projects/p2", { name: "Child", parentId: "p1", data: {} }); + const pdf = (await api.upload(path.join(FIXTURES, "recovery-fixture.pdf"), "paper.pdf", "application/pdf")).body.asset; + const txt = (await api.upload(path.join(FIXTURES, "recovery-note.txt"), "note.txt", "text/plain")).body.asset; + const file = (title: string, assetId: string) => ({ + projectId: "p1", + title, + ingestType: "file", + assetId, + data: { topics: ["t"], meta: [{ key: "annotation", value: title }] }, + }); + await api.put("/v1/sources/s-pdf", file("paper.pdf", pdf.id)); + await api.put("/v1/sources/s-txt", file("note.txt", txt.id)); + await api.put("/v1/sources/s-web", { + projectId: "p2", + title: "Site", + ingestType: "website", + assetId: null, + data: { accessLink: "https://example.com/" }, + }); + const expected = { + projects: (await api.get("/v1/projects")).body.projects.map(({ updatedAt, ...r }: any) => r), + sources: (await api.get("/v1/sources")).body.sources.map(({ updatedAt, ...r }: any) => r), + }; + await svc.stop(); + const tar = path.join(work, "library.tar"); + assert.equal(cli(["backup", "--data-dir", source, "--out", tar]).status, 0); + return { work, tar, expected, assets: { pdf, txt } }; +} + +/** Read a backup into a manifest and its asset files. */ +function unpack(tar: string) { + const index = readTarIndex(tar, { allowName: () => true, maxEntries: 100 }); + const manifest = JSON.parse(readTarEntry(tar, index.get("manifest.json")!).toString()); + const files = new Map(); + for (const [name, entry] of index) if (name !== "manifest.json") files.set(name, readTarEntry(tar, entry)); + return { manifest, files }; +} + +/** Write a tar from entries. type "0" is a regular file. */ +async function pack(file: string, entries: { name: string; data?: Buffer; type?: string }[]) { + const out = fs.createWriteStream(file); + const tar = new TarWriter(out); + for (const e of entries) { + if (!e.type || e.type === "0") { + await tar.addBuffer(e.name, e.data ?? Buffer.alloc(0)); + } else { + out.write(tarHeader(e.name, 0, new Date(), e.type)); + } + } + await tar.finish(); + await new Promise((r) => out.end(r)); + return file; +} + +async function modified(tar: string, out: string, change: (m: any, f: Map) => void, extra: any[] = []) { + const { manifest, files } = unpack(tar); + change(manifest, files); + return pack(out, [ + { name: "manifest.json", data: Buffer.from(JSON.stringify(manifest)) }, + ...[...files].map(([name, data]) => ({ name, data })), + ...extra, + ]); +} + +async function upload(svc: Service, tar: string) { + const res = await fetch(`${svc.url}/v1/restores`, { + method: "POST", + headers: { Authorization: `Bearer ${svc.token}`, "Content-Type": "application/x-tar" }, + body: fs.readFileSync(tar), + }); + return { status: res.status, body: (await res.json()) as any }; +} + +async function activate(svc: Service, id: string) { + const res = await fetch(`${svc.url}/v1/restores/${id}/activate`, { + method: "POST", + headers: { Authorization: `Bearer ${svc.token}` }, + }); + return { status: res.status, body: (await res.json()) as any }; +} + +function allFiles(dir: string): string[] { + if (!fs.existsSync(dir)) return []; + return fs.readdirSync(dir, { recursive: true, withFileTypes: true }) + .filter((d) => d.isFile()) + .map((d) => path.relative(dir, path.join(d.parentPath, d.name))); +} + +/** Managed files and staging files in a data directory. */ +function storedFiles(dataDir: string) { + return [ + ...allFiles(path.join(dataDir, "assets")).filter((f) => !f.startsWith("tmp")), + ...allFiles(path.join(dataDir, "restore")), + ]; +} + +test("restore through the API: validate, preview, activate, refuse a second time", async () => { + const { tar, expected, assets } = await makeBackup(); + const target = tempDir("rs-target"); + const svc = await startService(target); + const api = client(svc); + + // An initialized but empty library is accepted + assert.deepEqual((await api.get("/v1/library")).body, { + empty: true, + counts: { projects: 0, sources: 0, assets: 0 }, + }); + + const up = await upload(svc, tar); + assert.equal(up.status, 201, JSON.stringify(up.body)); + const preview = up.body.restore; + assert.deepEqual(preview.counts, { projects: 2, sources: 3, assets: 2, bytes: assets.pdf.size + assets.txt.size }); + assert.deepEqual(preview.projectNames.sort(), ["Child", "Restore Project"]); + assert.equal(preview.backupVersion, 1); + assert.ok(preview.notIncluded.includes("Chat history")); + assert.ok(preview.notIncluded.includes("API keys")); + + // Nothing is active before confirmation + assert.equal((await api.get("/v1/library")).body.empty, true); + + const done = await activate(svc, preview.id); + assert.equal(done.status, 200); + assert.deepEqual(done.body.restored, { projects: 2, sources: 3, assets: 2 }); + assert.deepEqual( + (await api.get("/v1/projects")).body.projects.map(({ updatedAt, ...r }: any) => r), + expected.projects + ); + assert.deepEqual( + (await api.get("/v1/sources")).body.sources.map(({ updatedAt, ...r }: any) => r), + expected.sources + ); + assert.ok((await api.content(assets.pdf.id)).bytes.equals(fs.readFileSync(path.join(FIXTURES, "recovery-fixture.pdf")))); + assert.deepEqual(allFiles(path.join(target, "restore")), []); + + // A library with records is refused before the upload is read + const again = await upload(svc, tar); + assert.equal(again.status, 409); + assert.match(again.body.error.message, /Restore needs an empty library\. This library has 2 projects, 3 sources, and 2 files/); + assert.equal((await api.get("/v1/sources")).body.sources.length, 3); + assert.equal((await activate(svc, preview.id)).status, 404); +}); + +test("a write after validation blocks activation; nothing is merged", async () => { + const { tar } = await makeBackup(); + const target = tempDir("rs-race"); + const svc = await startService(target); + const api = client(svc); + const up = await upload(svc, tar); + assert.equal(up.status, 201); + + await api.put("/v1/projects/mine", { name: "Created meanwhile", parentId: null, data: {} }); + const done = await activate(svc, up.body.restore.id); + assert.equal(done.status, 409); + assert.deepEqual((await api.get("/v1/projects")).body.projects.map((p: any) => p.id), ["mine"]); + assert.deepEqual(storedFiles(target), []); +}); + +test("cancel removes the staged files", async () => { + const { tar } = await makeBackup(); + const target = tempDir("rs-cancel"); + const svc = await startService(target); + const up = await upload(svc, tar); + assert.ok(allFiles(path.join(target, "restore")).length > 0); + const res = await fetch(`${svc.url}/v1/restores/${up.body.restore.id}`, { + method: "DELETE", + headers: { Authorization: `Bearer ${svc.token}` }, + }); + assert.equal(res.status, 204); + assert.deepEqual(allFiles(path.join(target, "restore")), []); + assert.equal((await activate(svc, up.body.restore.id)).status, 404); +}); + +test("invalid backups are rejected before activation and leave nothing behind", async () => { + const { tar, work, assets } = await makeBackup(); + const target = tempDir("rs-invalid"); + const svc = await startService(target); + const api = client(svc); + const out = (name: string) => path.join(work, name); + + const cases: [string, Promise, RegExp][] = [ + [ + "unsupported version", + modified(tar, out("v2.tar"), (m) => (m.version = 2)), + /Backup version 2 is not supported/, + ], + [ + "missing asset file", + modified(tar, out("missing.tar"), (_m, f) => f.delete(`assets/${assets.txt.id}`)), + /is missing from the backup/, + ], + [ + "damaged asset file", + modified(tar, out("damaged.tar"), (_m, f) => { + const data = Buffer.from(f.get(`assets/${assets.txt.id}`)!); + data[0] ^= 0xff; + f.set(`assets/${assets.txt.id}`, data); + }), + /is damaged/, + ], + [ + "source refers to a missing project", + modified(tar, out("orphan.tar"), (m) => (m.sources[0].projectId = "nope")), + /not in the backup/, + ], + [ + "path traversal entry", + modified(tar, out("traversal.tar"), () => {}, [{ name: "../escaped.txt", data: Buffer.from("x") }]), + /Unexpected tar entry/, + ], + [ + "absolute path entry", + modified(tar, out("absolute.tar"), () => {}, [{ name: "/tmp/kc-escaped.txt", data: Buffer.from("x") }]), + /Unexpected tar entry/, + ], + [ + "symbolic link entry", + modified(tar, out("symlink.tar"), () => {}, [{ name: "assets/link", type: "2" }]), + /Unsupported tar entry type "2"/, + ], + [ + "unlisted asset entry", + modified(tar, out("unlisted.tar"), () => {}, [ + { name: "assets/00000000-0000-4000-8000-000000000000", data: Buffer.from("x") }, + ]), + /does not list/, + ], + ["not a tar file", Promise.resolve(path.join(FIXTURES, "recovery-note.txt")), /not a valid library backup/], + ]; + + for (const [label, file, message] of cases) { + const res = await upload(svc, await file); + assert.equal(res.status, 400, `${label}: ${JSON.stringify(res.body)}`); + assert.match(res.body.error.message, message, label); + assert.deepEqual(storedFiles(target), [], `${label}: files left behind`); + assert.equal((await api.get("/v1/library")).body.empty, true, label); + } + assert.ok(!fs.existsSync(path.join(work, "escaped.txt"))); + assert.ok(!fs.existsSync("/tmp/kc-escaped.txt")); + + // The command line uses the same validation + const viaCli = cli(["restore", "--data-dir", tempDir("rs-cli"), "--from", out("v2.tar")]); + assert.equal(viaCli.status, 1); + assert.match(viaCli.stderr, /Backup version 2 is not supported/); +}); + +test("a failure during activation rolls back records and files", async () => { + const { tar } = await makeBackup(); + const target = tempDir("rs-fail"); + const result = cli(["restore", "--data-dir", target, "--from", tar], { + KC_STORAGE_TEST_FAULT: "restore-fail-before-commit", + }); + assert.equal(result.status, 1); + assert.match(result.stderr, /Injected test fault/); + assert.deepEqual(storedFiles(target), []); + + const svc = await startService(target); + assert.equal((await client(svc).get("/v1/library")).body.empty, true); +}); + +test("a process stop during activation leaves an empty library after restart", async () => { + const { tar, expected } = await makeBackup(); + const target = tempDir("rs-crash"); + const result = cli(["restore", "--data-dir", target, "--from", tar], { + KC_STORAGE_TEST_FAULT: "restore-exit-during-activation", + }); + assert.equal(result.status, 70); + // The process stopped after it moved a file and before it committed + assert.ok(storedFiles(target).length > 0); + + const svc = await startService(target); + const api = client(svc); + assert.equal((await api.get("/v1/library")).body.empty, true); + assert.deepEqual(storedFiles(target), []); + assert.match(svc.stderr(), /removed \d+ unreferenced or temporary files/); + + // A later restore succeeds + const up = await upload(svc, tar); + assert.equal((await activate(svc, up.body.restore.id)).status, 200); + assert.equal((await api.get("/v1/sources")).body.sources.length, expected.sources.length); +}); From e98f7f3d9deeed5beb541bad043751cc8a793302 Mon Sep 17 00:00:00 2001 From: Rob Royce Date: Tue, 6 Oct 2026 21:42:20 -0700 Subject: [PATCH 3/4] feat(ui): restore a library backup from Settings > Backup - The Library Backup panel states what the backup contains and what it does not contain. - Restore Library uploads the selected .tar to the storage service, which validates it. The panel shows the backup date, counts, project names, warnings, and the data that is not in the backup, then asks for confirmation. After activation it shows the result and reloads. - A library with records is refused with the reason and the counts. No data is replaced, merged, or deleted. - Counts in restore messages use correct plurals. Tests: - e2e/workflow.e2e.mjs restores through the UI instead of the command line. - e2e/packaged/packaged.e2e.mjs (yarn e2e-packaged) runs the unsigned package from a copy outside the repository, with scratch profiles, a working directory outside the repository, and PATH without Node.js. It checks the bundled runtime in the service command line, the full workflow with UI restore, an unsupported backup version, the refusal for a non-empty library, unchanged chat history, two instances, and service cleanup. Co-Authored-By: Claude Opus 5.5 --- e2e/lib.mjs | 35 +- e2e/packaged/packaged.e2e.mjs | 335 ++++++++++++++++++ e2e/steps.mjs | 36 +- e2e/workflow.e2e.mjs | 24 +- .../settings/storage-settings.component.ts | 220 +++++++++++- .../services/ipc-services/storage.service.ts | 36 ++ src/kc_storage/src/restore.ts | 9 +- 7 files changed, 662 insertions(+), 33 deletions(-) create mode 100644 e2e/packaged/packaged.e2e.mjs diff --git a/e2e/lib.mjs b/e2e/lib.mjs index e1568f63..eba6a7a6 100644 --- a/e2e/lib.mjs +++ b/e2e/lib.mjs @@ -39,18 +39,37 @@ export function newRun(name) { } /** Launch the app with the given profile directory. */ -export async function launch(profileDir, logFile, env = {}) { +/** + * Launch the app with the given profile directory. + * + * options.packaged: path to a packaged Knowledge.app. The app then gets + * only options.env (no inherited variables) and runs in options.cwd. + */ +export async function launch(profileDir, logFile, env = {}, options = {}) { if (!profileDir) { throw new Error("launch() needs an explicit profile directory"); } const log = fs.createWriteStream(logFile, { flags: "a" }); - const app = await electron.launch({ - executablePath: ELECTRON, - args: [REPO], - cwd: REPO, - env: { ...appEnv(), ...env, KC_PROFILE_DIR: profileDir }, - timeout: 60000, - }); + const app = await electron.launch( + options.packaged + ? { + executablePath: path.join( + options.packaged, + "Contents/MacOS/Knowledge" + ), + args: [], + cwd: options.cwd, + env: { ...env, KC_PROFILE_DIR: profileDir }, + timeout: 60000, + } + : { + executablePath: ELECTRON, + args: [REPO], + cwd: REPO, + env: { ...appEnv(), ...env, KC_PROFILE_DIR: profileDir }, + timeout: 60000, + } + ); app.process().stdout?.on("data", (d) => log.write(`[main] ${d}`)); app.process().stderr?.on("data", (d) => log.write(`[main:err] ${d}`)); const page = await app.firstWindow(); diff --git a/e2e/packaged/packaged.e2e.mjs b/e2e/packaged/packaged.e2e.mjs new file mode 100644 index 00000000..a39417b1 --- /dev/null +++ b/e2e/packaged/packaged.e2e.mjs @@ -0,0 +1,335 @@ +/* + * The unsigned packaged app, self-contained: + * + * - The app is copied to a temporary directory outside the repository. + * - It runs with only HOME, TMPDIR, and PATH=/usr/bin:/bin:/usr/sbin:/sbin + * (no Node.js, no AWS, Apple, or AI credentials). + * - The working directory is outside the repository. + * - Every launch uses an explicit scratch profile. + * + * Build first: yarn package-local + * Run: yarn e2e-packaged + * App: KC_PACKAGED_APP= (default: dist/mac-arm64) + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFileSync, spawnSync } from "node:child_process"; +import { + FIXTURES, + REPO, + backendInfo, + captureDownloads, + close, + launch, + readAsset, + readLibrary, + shot, + startFixtureSite, + waitForFile, +} from "../lib.mjs"; +import * as ui from "../steps.mjs"; +import { + readTarEntry, + readTarIndex, + TarWriter, +} from "../../src/kc_storage/src/tar.ts"; + +const BUILT_APP = + process.env.KC_PACKAGED_APP ?? + path.join(REPO, "dist/mac-arm64/Knowledge.app"); +const RESTRICTED_PATH = "/usr/bin:/bin:/usr/sbin:/sbin"; +const PROJECT = "Packaged Project"; +const SITE_TITLE = "Knowledge Fixture Site"; +const byTitle = (list) => + [...list].sort((a, b) => a.title.localeCompare(b.title)); + +/** Storage service processes that serve a data directory under root. */ +function serviceProcesses(root) { + const out = spawnSync("/usr/bin/pgrep", ["-f", `serve --data-dir ${root}`], { + encoding: "utf8", + }); + return out.stdout + .split("\n") + .filter(Boolean) + .map((pid) => ({ + pid, + command: execFileSync("/bin/ps", ["-o", "command=", "-p", pid], { + encoding: "utf8", + }).trim(), + })); +} + +async function waitForNoServices(root) { + for (let i = 0; i < 40 && serviceProcesses(root).length > 0; i++) { + await new Promise((r) => setTimeout(r, 250)); + } + return serviceProcesses(root); +} + +/** Copy a backup and set its manifest version. */ +async function withVersion(tar, out, version) { + const index = readTarIndex(tar, { allowName: () => true, maxEntries: 100 }); + const manifest = JSON.parse( + readTarEntry(tar, index.get("manifest.json")).toString() + ); + manifest.version = version; + const stream = fs.createWriteStream(out); + const writer = new TarWriter(stream); + await writer.addBuffer( + "manifest.json", + Buffer.from(JSON.stringify(manifest)) + ); + for (const [name, entry] of index) { + if (name !== "manifest.json") + await writer.addBuffer(name, readTarEntry(tar, entry)); + } + await writer.finish(); + await new Promise((r) => stream.end(r)); + return out; +} + +test( + "packaged app works without Node.js, the repository, or credentials", + { timeout: 600000 }, + async () => { + assert.ok( + fs.existsSync(BUILT_APP), + `Packaged app not found: ${BUILT_APP}. Run yarn package-local.` + ); + + // Everything outside the repository + // realpath: macOS reports /var/folders as /private/var/folders + const root = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), "kc-packaged-")) + ); + const app = path.join(root, "Knowledge.app"); + execFileSync("/usr/bin/ditto", [BUILT_APP, app]); + const cwd = path.join(root, "cwd"); + fs.mkdirSync(cwd); + const env = { + HOME: os.homedir(), + TMPDIR: os.tmpdir(), + PATH: RESTRICTED_PATH, + }; + const options = { packaged: app, cwd }; + const log = path.join(root, "app.log"); + const shots = path.join(root, "shots"); + fs.mkdirSync(shots); + console.log(`packaged test directory: ${root}`); + + // No Node.js on the restricted PATH + assert.notEqual(spawnSync("/usr/bin/which", ["node"], { env }).status, 0); + + const profileA = path.join(root, "profile-a"); + const profileB = path.join(root, "profile-b"); + const originals = path.join(root, "originals"); + fs.mkdirSync(originals); + const pdf = path.join(originals, "recovery-fixture.pdf"); + const txt = path.join(originals, "recovery-note.txt"); + fs.copyFileSync(path.join(FIXTURES, "recovery-fixture.pdf"), pdf); + fs.copyFileSync(path.join(FIXTURES, "recovery-note.txt"), txt); + const pdfBytes = fs.readFileSync(pdf); + const txtBytes = fs.readFileSync(txt); + const site = await startFixtureSite(); + + const expectedRuntime = `${app}/Contents/Resources/node/bin/node ${app}/Contents/Resources/kc_storage/src/main.ts serve`; + let a; + let b; + try { + // 1. Start. The service runs with the bundled runtime from the package. + a = await launch(profileA, log, env, options); + const services = serviceProcesses(profileA); + assert.equal(services.length, 1, JSON.stringify(services)); + assert.ok( + services[0].command.startsWith(expectedRuntime), + services[0].command + ); + const cwdOfService = execFileSync( + "/usr/sbin/lsof", + ["-a", "-p", services[0].pid, "-d", "cwd", "-Fn"], + { + encoding: "utf8", + } + ); + assert.match( + cwdOfService, + new RegExp( + `n${app.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}/Contents/Resources` + ) + ); + + // 2-5. Create, save a link, import files, annotate + await ui.createProject(a.page, PROJECT, "Packaged workflow"); + await ui.addLinkToProject(a.page, site.url, SITE_TITLE); + await ui.importFilesToProject(a.page, [pdf, txt]); + await ui.openTable(a.page); + await ui.openSource(a.page, "recovery-note.txt"); + await ui.addAnnotation(a.page, "annotation", "packaged note"); + await ui.addTopic(a.page, "packagedtopic"); + await a.page.waitForTimeout(1500); + await ui.closeDialog(a.page); + await a.page.waitForTimeout(1000); + const library = await readLibrary(a.page); + assert.equal(library.length, 1); + assert.equal(library[0].sources.length, 3); + await close(a); + a = undefined; + assert.deepEqual(await waitForNoServices(profileA), []); + + // 6. Restart and retrieve + a = await launch(profileA, log, env, options); + const afterRestart = await readLibrary(a.page); + assert.deepEqual( + byTitle(afterRestart[0].sources), + byTitle(library[0].sources) + ); + const note = afterRestart[0].sources.find( + (s) => s.title === "recovery-note.txt" + ); + assert.deepEqual(note.topics, ["packagedtopic"]); + assert.deepEqual(note.annotations, ["packaged note"]); + + // 7. Export the library through the UI + const downloads = path.join(root, "downloads"); + await captureDownloads(a.app, downloads); + await ui.openSettings(a.page, "Backup"); + await a.page.locator("button", { hasText: "Export Library" }).click(); + const date = new Date().toISOString().slice(0, 10); + const tar = await waitForFile( + path.join(downloads, `knowledge-library-${date}.tar`) + ); + + // 8. Make the original files unavailable + fs.rmSync(originals, { recursive: true }); + + // 9. Second isolated profile, while the first instance still runs + b = await launch(profileB, log, env, options); + const infoA = await backendInfo(a.page); + const infoB = await backendInfo(b.page); + assert.notEqual(infoA.storage.url, infoB.storage.url); + assert.notEqual(infoA.storage.token, infoB.storage.token); + assert.notEqual(infoA.chat.url, infoB.chat.url); + const crossStatus = await b.page.evaluate( + async ({ url, token }) => + ( + await fetch(`${url}/v1/projects`, { + headers: { Authorization: `Bearer ${token}` }, + }) + ).status, + { url: infoA.storage.url, token: infoB.storage.token } + ); + assert.equal(crossStatus, 401); + for (const s of [ + ...serviceProcesses(profileA), + ...serviceProcesses(profileB), + ]) { + assert.ok(s.command.startsWith(expectedRuntime), s.command); + } + await close(a); + a = undefined; + assert.deepEqual(await waitForNoServices(profileA), []); + + // Chat history and preferences must not change during a library restore + await b.page.evaluate(() => + localStorage.setItem("chat-sentinel", "keep me") + ); + + // An unsupported backup version is refused with a clear message + const v2 = await withVersion(tar, path.join(root, "v2.tar"), 2); + const refused = await ui.restoreLibraryError(b.page, v2); + assert.match(refused, /Backup version 2 is not supported/); + assert.deepEqual(await readLibrary(b.page), []); + + // 10. Restore through the interface + const restore = await ui.restoreLibrary(b.page, tar); + assert.match(restore.preview, /1 project, 3 sources, 2 files/); + assert.match(restore.preview, /Chat history/); + assert.match(restore.preview, /API keys/); + assert.match( + restore.result, + /Restored 1 project, 3 sources, and 2 files/ + ); + assert.equal( + await b.page.evaluate(() => localStorage.getItem("chat-sentinel")), + "keep me" + ); + + // The interface now refuses another restore and says why + await ui.openSettings(b.page, "Backup"); + const refusal = await b.page.locator("#restore-refused").innerText(); + assert.match( + refusal, + /Restore works only into an empty library\. This library has 1 project, 3 sources, and 2 files/ + ); + await shot(b.page, shots, "b-restore-refused"); + await close(b); + b = undefined; + + // 11. Restart + b = await launch(profileB, log, env, options); + await ui.selectProject(b.page, PROJECT); + + // 12. Open the restored files without the originals + const restored = await readLibrary(b.page); + assert.deepEqual( + byTitle(restored[0].sources), + byTitle(afterRestart[0].sources) + ); + const ids = Object.fromEntries( + restored[0].sources.map((s) => [s.title, s.assetId]) + ); + assert.ok( + (await readAsset(b.page, ids["recovery-fixture.pdf"])).equals(pdfBytes) + ); + assert.ok( + (await readAsset(b.page, ids["recovery-note.txt"])).equals(txtBytes) + ); + + await ui.openTable(b.page); + await ui.openSource(b.page, "recovery-fixture.pdf"); + await ui.openDocumentTab(b.page); + const embed = b.page.locator("source-document embed"); + await embed.waitFor({ timeout: 15000 }); + assert.match(await embed.getAttribute("src"), /^blob:/); + await b.page.waitForTimeout(2000); + await shot(b.page, shots, "b-restored-pdf"); + await ui.closeDialog(b.page); + + const thumbnail = await b.page.evaluate( + (assetId) => + new Promise((resolve) => { + window.api.receive("E2A:FileSystem:FileThumbnail", (responses) => { + const mine = responses.find( + (r) => r.success?.data?.id === "probe" + ); + if (mine) resolve(mine.success.data.thumbnail); + }); + window.api.send("A2E:FileSystem:FileThumbnail", [ + { + path: "/nonexistent/recovery-fixture.pdf", + assetId, + id: "probe", + }, + ]); + }), + ids["recovery-fixture.pdf"] + ); + assert.match(thumbnail, /^data:image\/png;base64,.{100,}/); + await close(b); + b = undefined; + + // No service process remains after the app exits + assert.deepEqual(await waitForNoServices(root), []); + } finally { + if (a) await close(a); + if (b) await close(b); + site.close(); + // Keep logs, profiles, and screenshots. Remove the 440 MB app copy. + fs.rmSync(app, { recursive: true, force: true }); + } + } +); diff --git a/e2e/steps.mjs b/e2e/steps.mjs index 89adf758..7e85f7a0 100644 --- a/e2e/steps.mjs +++ b/e2e/steps.mjs @@ -147,7 +147,39 @@ export async function search(page, query) { ); } +/** Open a Settings section. Settings is a dialog; reuse it if it is open. */ export async function openSettings(page, section) { - await page.locator("i.pi-cog").first().click(); - await page.locator(".p-menuitem-link", { hasText: section }).click(); + const item = page.locator(".p-menuitem-link", { hasText: section }); + if (!(await item.isVisible())) { + await page.locator("i.pi-cog").first().click(); + } + await item.click(); +} + +/** + * Restore a library backup through Settings > Backup. Returns the preview + * text that the user sees before confirmation. + */ +export async function restoreLibrary(page, tar) { + await openSettings(page, "Backup"); + await page.locator("#restore-library-input").setInputFiles(tar); + const preview = page.locator("#restore-preview"); + await preview.waitFor({ timeout: 60000 }); + const text = await preview.innerText(); + await page.locator("#restore-confirm").click(); + await page.locator("#restore-result").waitFor({ timeout: 60000 }); + const result = await page.locator("#restore-result").innerText(); + // The app reloads after a restore + await page.waitForEvent("load", { timeout: 15000 }); + await page.waitForSelector("app-create button", { timeout: 30000 }); + return { preview: text, result }; +} + +/** Upload a library backup that the service must refuse. Returns the error. */ +export async function restoreLibraryError(page, tar) { + await openSettings(page, "Backup"); + await page.locator("#restore-library-input").setInputFiles(tar); + const error = page.locator("#restore-error"); + await error.waitFor({ timeout: 60000 }); + return error.innerText(); } diff --git a/e2e/workflow.e2e.mjs b/e2e/workflow.e2e.mjs index 749cb9e1..67d44959 100644 --- a/e2e/workflow.e2e.mjs +++ b/e2e/workflow.e2e.mjs @@ -11,10 +11,8 @@ import { test } from "node:test"; import assert from "node:assert/strict"; import fs from "node:fs"; import path from "node:path"; -import { spawnSync } from "node:child_process"; import { FIXTURES, - REPO, captureDownloads, close, launch, @@ -155,20 +153,16 @@ test( await close(ctx); ctx = undefined; - // Restore into the storage directory of a new profile, then open it - const restored = spawnSync( - process.execPath, - [ - path.join(REPO, "src/kc_storage/src/main.ts"), - "restore", - "--data-dir", - path.join(profileB, "data", "library"), - "--from", - tar, - ], - { encoding: "utf8" } + // Restore through the UI into a new profile, then restart and open it + ctx = await launch(profileB, log); + const restore = await ui.restoreLibrary(ctx.page, tar); + assert.match(restore.preview, /1 project, 3 sources, 2 files/); + assert.match( + restore.result, + /Restored 1 project, 3 sources, and 2 files/ ); - assert.equal(restored.status, 0, restored.stderr); + await close(ctx); + ctx = undefined; ctx = await launch(profileB, log); await ui.selectProject(ctx.page, PROJECT); diff --git a/src/kc_angular/src/app/components/settings/storage-settings.component.ts b/src/kc_angular/src/app/components/settings/storage-settings.component.ts index 01866f04..0b74b336 100755 --- a/src/kc_angular/src/app/components/settings/storage-settings.component.ts +++ b/src/kc_angular/src/app/components/settings/storage-settings.component.ts @@ -14,7 +14,8 @@ * limitations under the License. */ -import { Component } from '@angular/core'; +import { Component, OnInit } from '@angular/core'; +import type { RestorePreview } from '@contracts/storage'; import { StorageService } from '@services/ipc-services/storage.service'; import { FormBuilder, FormGroup } from '@angular/forms'; import { NotificationsService } from '@services/user-services/notifications.service'; @@ -35,10 +36,9 @@ import { SettingsService } from '@services/ipc-services/settings.service';
- Projects, sources, topics, metadata, and copies of imported - files. Restore it with the storage service command line (see - DEVELOPMENT.md). It does not hold chat history, preferences, - or settings. + Contains projects, sources, topics, metadata, and copies of + imported files. Does not contain chat history, the inbox, UI + preferences, settings, or API keys.
@@ -50,6 +50,99 @@ import { SettingsService } from '@services/ipc-services/settings.service'; >
+ + +
+ + +
+ + +
+ {{ refusal }} +
+ +
+
Restore this library?
+
+ Backup created: + {{ preview.backupCreatedAt | date : 'medium' }} (format + version {{ preview.backupVersion }}) +
+
+ {{ count(preview.counts.projects, 'project') }}, + {{ count(preview.counts.sources, 'source') }}, + {{ count(preview.counts.assets, 'file') }} ({{ + preview.counts.bytes / 1024 | number : '1.0-0' + }} + KB) +
+
+ Projects: {{ preview.projectNames.join(', ') }} +
+
+ Warning: {{ warning }} +
+
+ Not in this backup, and not changed: + {{ preview.notIncluded.join(', ') }}. +
+
+ + +
+
+ +
+ {{ restoreError }} +
+ +
+ {{ restoreResult }} +
@@ -103,13 +196,26 @@ import { SettingsService } from '@services/ipc-services/settings.service'; `, styles: [], }) -export class StorageSettingsComponent { +export class StorageSettingsComponent implements OnInit { exportType = 'Everything'; exporting = false; exportingLibrary = false; + /** Restore is allowed only into a library without records. */ + libraryEmpty = false; + + refusal = ''; + + preview?: RestorePreview; + + restoreBusy = false; + + restoreError = ''; + + restoreResult = ''; + form: FormGroup; constructor( @@ -121,6 +227,108 @@ export class StorageSettingsComponent { this.form = formBuilder.group({}); } + count(n: number, noun: string) { + return `${n} ${noun}${n === 1 ? '' : 's'}`; + } + + ngOnInit() { + this.refreshLibraryStatus(); + } + + private message(e: any): string { + return e?.error?.error?.message ?? e?.message ?? `${e}`; + } + + async refreshLibraryStatus() { + try { + const status = await this.storage.libraryStatus(); + this.libraryEmpty = status.empty; + const c = status.counts; + this.refusal = status.empty + ? '' + : `Restore works only into an empty library. This library has ${this.count( + c.projects, + 'project' + )}, ` + + `${this.count(c.sources, 'source')}, and ${this.count( + c.assets, + 'file' + )}. To restore, start Knowledge with a new ` + + 'profile. Existing data is not changed or deleted.'; + } catch (e) { + this.libraryEmpty = false; + this.refusal = `Library status unavailable: ${this.message(e)}`; + } + } + + /** Upload the selected file. The service validates it. Nothing changes yet. */ + async onSelectLibraryBackup($event: any) { + const input: HTMLInputElement = $event.target; + const file = input.files?.[0]; + input.value = ''; + if (!file) { + return; + } + this.restoreError = ''; + this.restoreResult = ''; + this.restoreBusy = true; + try { + this.preview = await this.storage.stageRestore(file); + } catch (e) { + this.restoreError = `The backup cannot be restored. ${this.message(e)}`; + await this.refreshLibraryStatus(); + } finally { + this.restoreBusy = false; + } + } + + async onConfirmRestore() { + if (!this.preview) { + return; + } + this.restoreBusy = true; + this.restoreError = ''; + try { + const r = await this.storage.activateRestore(this.preview.id); + this.preview = undefined; + this.restoreResult = + `Restored ${this.count(r.projects, 'project')}, ${this.count( + r.sources, + 'source' + )}, ` + + `and ${this.count(r.assets, 'file')}. ` + + 'Knowledge reloads now.'; + this.notifications.success( + 'Backup', + 'Library Restored', + this.restoreResult + ); + + // Reload so every service reads the restored library + setTimeout(() => window.location.reload(), 2500); + } catch (e) { + this.preview = undefined; + this.restoreError = `Restore failed. The library is not changed. ${this.message( + e + )}`; + await this.refreshLibraryStatus(); + } finally { + this.restoreBusy = false; + } + } + + async onCancelRestore() { + const id = this.preview?.id; + this.preview = undefined; + if (id) { + try { + await this.storage.cancelRestore(id); + } catch { + // The staged files are removed at the next service start + } + } + } + onExport() { this.exporting = true; try { diff --git a/src/kc_angular/src/app/services/ipc-services/storage.service.ts b/src/kc_angular/src/app/services/ipc-services/storage.service.ts index 0fddb462..5e134235 100755 --- a/src/kc_angular/src/app/services/ipc-services/storage.service.ts +++ b/src/kc_angular/src/app/services/ipc-services/storage.service.ts @@ -39,8 +39,10 @@ import { sourceToRecord, } from '@contracts/mapping'; import type { + LibraryStatus, ProjectList, ProjectRecord, + RestorePreview, SourceList, SourceRecord, } from '@contracts/storage'; @@ -264,6 +266,40 @@ export class StorageService { return blob.size; } + libraryStatus(): Promise { + return firstValueFrom(this.http.get(`${this.api}/library`)); + } + + /** + * Upload a library backup. The service validates it and keeps it staged. + * Nothing becomes active until activateRestore(). + */ + async stageRestore(file: Blob): Promise { + const { restore } = await firstValueFrom( + this.http.post<{ restore: RestorePreview }>( + `${this.api}/restores`, + file, + { + headers: { 'Content-Type': 'application/x-tar' }, + } + ) + ); + return restore; + } + + async activateRestore(id: string) { + const { restored } = await firstValueFrom( + this.http.post<{ + restored: { projects: number; sources: number; assets: number }; + }>(`${this.api}/restores/${id}/activate`, null) + ); + return restored; + } + + async cancelRestore(id: string) { + await firstValueFrom(this.http.delete(`${this.api}/restores/${id}`)); + } + /** * Download a backup of renderer storage: chat history and preferences. * Projects and sources are not in it. They are in the library backup. diff --git a/src/kc_storage/src/restore.ts b/src/kc_storage/src/restore.ts index 23994fed..87fe842a 100644 --- a/src/kc_storage/src/restore.ts +++ b/src/kc_storage/src/restore.ts @@ -79,9 +79,14 @@ function fail(message: string): never { export function assertEmpty(dir: DataDir) { const counts = dir.library.counts(); if (counts.projects + counts.sources + counts.assets > 0) { + const n = (count: number, noun: string) => + `${count} ${noun}${count === 1 ? "" : "s"}`; throw conflict( - `Restore needs an empty library. This library has ${counts.projects} projects, ` + - `${counts.sources} sources, and ${counts.assets} files. ` + + `Restore needs an empty library. This library has ${n( + counts.projects, + "project" + )}, ` + + `${n(counts.sources, "source")}, and ${n(counts.assets, "file")}. ` + "Restore into a new profile instead. Existing data is not changed." ); } From f067b49d8bcfa00b3a309283b4e46a15c5331394 Mon Sep 17 00:00:00 2001 From: Rob Royce Date: Tue, 6 Oct 2026 21:44:19 -0700 Subject: [PATCH 4/4] docs: packaging, library restore, access boundary, and follow-ups - DEVELOPMENT.md: runtime selection for packaged and unpackaged runs, yarn package-local, package contents, restore through Settings > Backup and the command line, what each backup contains, packaged tests, and current limits. - Correct the security statements: the token stops requests without it, and the servers accept loopback connections only. Programs that run as the same OS user can read the token. The access boundary is the OS user. - docs/storage-service.md: restore API and packaged runtime. - docs/desktop-storage-completion.md: verification record. - docs/follow-ups.md: remove completed items; add signing, other platforms, package size, and restore follow-ups. Co-Authored-By: Claude Opus 5.5 --- DEVELOPMENT.md | 150 ++++++++++++++++++++++------- docs/desktop-storage-completion.md | 31 ++++++ docs/follow-ups.md | 21 +++- docs/storage-service.md | 23 ++++- 4 files changed, 182 insertions(+), 43 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 1fae8098..9ddef177 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -8,15 +8,19 @@ You do not need AWS, an Apple developer account, or an AI API key. | Tool | Version | Notes | | --- | --- | --- | | macOS | 26.6 (arm64) | Verified. Linux and Windows are not verified. | -| Node.js | 24.21.0 | Verified. The app starts the storage service with this Node.js. | +| Node.js | 24.21.0 | Verified. Development builds, tests, and the unpackaged app use it. | | Yarn | 3.2.4 | The repository contains this version (`.yarnrc.yml` `yarnPath`). Any global `yarn` starts it. | | Electron | 26.3.0 | Installed by Yarn. | No native compiler is necessary. The install does not build `canvas`. The storage service (`src/kc_storage`) needs Node.js 24 or later. It uses -the built-in `node:sqlite` module. The desktop app starts it with `node` -from `PATH`. To use a different binary, set `KC_NODE=/path/to/node`. +the built-in `node:sqlite` module. + +| Run | Node.js for the storage service | +| --- | --- | +| Unpackaged (`yarn start`) | `node` from `PATH`, or `KC_NODE=/path/to/node` | +| Packaged app | Bundled Node.js 24.21.0 in `Contents/Resources/node/bin/node`. `PATH` and `KC_NODE` are not used. | Compatibility note: Angular 14 officially supports Node.js 14.15+ and 16.10+ only. Node.js 24 builds and runs this branch, but it is outside the official range. @@ -60,10 +64,21 @@ The app does not read or write the old local-storage project keys ## Instances and local servers Each instance starts two local servers: the chat server (Electron) and the -storage service (Node.js). Each listens on a random loopback port and -requires its own random token. The renderer receives the addresses and -tokens through IPC (`A2E:Backend:Info`). One instance cannot use the -servers of another instance. +storage service (Node.js). Each listens on a random port on `127.0.0.1` +and requires its own random bearer token. The renderer receives the +addresses and tokens through IPC (`A2E:Backend:Info`). + +### Access boundary + +- Verified: a request without the correct token gets `401`. The window of + one instance does not know the address or token of another instance, so + it cannot use the other instance's servers by mistake. +- Verified: the servers do not accept connections from other computers. +- Not protected: other programs that run as the same macOS user. Such a + program can read the token (for example, from the storage service's + environment, `KC_STORAGE_TOKEN`), or read the profile files directly. + The real boundary is the operating-system user account. +- The browser extension server (port 9000, off by default) has no token. Only one instance can use a profile at a time. A second start with the same profile prints a message and exits. Different profiles can run at the same time. @@ -194,35 +209,62 @@ do not change them. Changes made after the migration are not in them. ## Backup and restore -Settings > Backup has two backups. Neither is a complete application backup. +Settings > Backup has two separate backups. Neither is a complete +application backup. -| Backup | Contents | Restore | -| --- | --- | --- | -| Library (`knowledge-library-.tar`) | Projects, sources, topics, metadata, managed files with SHA-256 hashes | `yarn storage restore` into an empty data directory | -| Chat and preferences (`knowledge-backup-.json`) | Renderer local storage: chat history, inbox, UI preferences | Settings > Backup > Restore | +| Backup | Contains | Does not contain | Restore | +| --- | --- | --- | --- | +| Library (`knowledge-library-.tar`) | Projects, sources, topics, metadata, copies of imported files (SHA-256 for each) | Chat history, inbox, UI preferences, settings, API keys, extracted-text cache | Settings > Backup > Restore Library, or `yarn storage restore` | +| Chat and preferences (`knowledge-backup-.json`) | Renderer local storage: chat history, inbox, UI preferences | Projects, sources, files, settings | Settings > Backup > Restore | + +### Restore a library in the app + +1. Start Knowledge with a new, empty profile. For example: + `KC_PROFILE_DIR=/path/to/new-profile yarn start`. +2. Open Settings > Backup. +3. Select Restore Library and choose the `.tar` file. +4. Read the preview: date, counts, project names, warnings, and the data + that is not in the backup. +5. Select Restore. Knowledge shows the result and reloads. + +The restored library does not need the original files. + +Restore works only into an empty library. An empty library can have a +database file. If the library has projects, sources, or files, the app +shows the counts and refuses. It never replaces, merges, or deletes data. + +The storage service does all checks before anything changes: + +- Format `knowledge-library-backup`, supported version (1). +- Archive entries: only `manifest.json` and `assets/`, regular + files only. Other names, links, and entry types are refused. +- Records, duplicate IDs, project and file references, sizes, and the + SHA-256 of every file. +- Limits: archive 8 GiB, manifest 64 MiB, 1,000,000 entries, 2 GiB for + each file. -Not in either backup: settings, the API key, the extracted-text cache. -File sources without a managed copy (missing at migration) still depend on -the original path. +Then it activates the restore in one step. If the step fails, or if the +process stops during it, the library stays empty. Unused staging files are +removed at the next start. A library restore does not change chat history +or preferences. -To restore a library into a new profile: +### Restore from the command line ```sh yarn storage restore --data-dir /data/library --from knowledge-library-.tar -KC_PROFILE_DIR= yarn start ``` -Restore checks every file hash and refuses a directory that already has a -library. The restored library does not need the original files. +Stop the app first. The command uses the same checks as the app. ## Tests | Command | Checks | Needs | | --- | --- | --- | | `yarn test` | Renderer backup, profile path isolation | Node.js 22.18+ | -| `yarn test-storage` | Storage service without Electron: persistence, files, validation and access limits, migration (repeat, missing files, failure, rollback), backup and restore | Node.js 24+ | +| `yarn test-storage` | Storage service without Electron: persistence, files, validation and access limits, migration (repeat, missing files, failure, rollback), backup, restore (API and command line, non-empty refusal, unsupported version, missing and damaged files, unsafe archive entries, failure and process stop during activation) | Node.js 24+ | | `yarn workspace kc_storage typecheck` | Storage service types (TypeScript 5) | | | `yarn e2e` | Desktop UI with real profiles (below) | macOS, `yarn build-dev` first | +| `yarn e2e-packaged` | The unsigned package (below) | macOS arm64, `yarn package-local` first | The end-to-end tests start the real app with new, empty profiles in `e2e/.output//`. They never use your normal profile. @@ -231,11 +273,24 @@ The end-to-end tests start the real app with new, empty profiles in import a PDF and a text file, annotate, search, restart, delete the original files, read the same records and files, show the PDF, make a thumbnail and extract text from the managed copy, export the library, - restore it into a second profile, and show the PDF there. + restore it into a second profile through Settings > Backup, and show the + PDF there. - `e2e/isolation.e2e.mjs`: two profiles at the same time get different chat and storage addresses and tokens. Each server rejects the other instance's requests. A second start of the same profile exits. +- `e2e/packaged/packaged.e2e.mjs`: copies `dist/mac-arm64/Knowledge.app` + to a temporary directory and runs it with scratch profiles, a working + directory outside the repository, only `HOME`, `TMPDIR`, and + `PATH=/usr/bin:/bin:/usr/sbin:/sbin` (no Node.js, no credentials). It + checks that the service command line uses the bundled runtime, then runs + the full workflow: create, link, import, annotate, restart, export, + delete the originals, restore through the UI into a second profile, + restart, and open the files. It also checks an unsupported backup + version, the refusal for a non-empty library, unchanged chat history, + two instances at the same time, and that no service process remains. + Output stays in `$TMPDIR/kc-packaged-*`. The test removes the app copy. + Test files are in `e2e/fixtures/`. To delete old test output, delete `e2e/.output/`. @@ -254,16 +309,18 @@ curl -s -X POST http://127.0.0.1:7777 -d '{"op":"close"}' no notes editor. The `notes` field is kept but not shown. - Search uses titles, topics, descriptions, and source type. It does not search file content. -- The library backup includes managed files. Restore needs the storage - service command line. The UI cannot restore a library. +- Library restore works only into an empty library. +- Chat history, UI preferences, and settings are separate from the library + and its backup. - A file source that had no original file at migration has no managed copy. It still depends on its original path. -- Packaged builds cannot start the storage service. The package does not - contain `src/kc_storage` or a Node.js 24 runtime. Not tested on this - branch. A packaged app is expected to show the start error and quit. -- Signed and notarized releases are not verified. +- Packaging is verified only for macOS arm64 (unsigned). A pinned runtime + hash exists for macOS x64, but that build is not tested. Linux and + Windows packages have no runtime configuration. +- Signed and notarized releases are not verified. Signing must also sign + the bundled Node.js binary. - AI chat with a real API key is not verified. -- Only macOS is verified. +- Electron 26 and Angular 14 are outside their supported versions. Follow-up work is in [docs/follow-ups.md](docs/follow-ups.md). @@ -285,14 +342,39 @@ plain text. ## Packaging ```sh -CSC_IDENTITY_AUTO_DISCOVERY=false yarn run pack +yarn package-local ``` -Use `yarn run pack`. The command `yarn pack` is a Yarn built-in. It makes an -npm archive and does not start electron-builder. +This command does these steps: + +1. `yarn build`: production builds of Angular and Electron. +2. `yarn prepare-runtime`: downloads the official Node.js 24.21.0 archive + for this computer (`darwin-arm64`) from `nodejs.org`, checks the pinned + SHA-256 and the upstream `SHASUMS256.txt`, and extracts `bin/node` and + `LICENSE` to `vendor/node/darwin-arm64/`. It needs the network only the + first time. Git ignores `vendor/`. +3. electron-builder without signing (`CSC_IDENTITY_AUTO_DISCOVERY=false`). + +The result is an unsigned app in `dist/mac-arm64/Knowledge.app` (about +440 MB). It contains, outside `app.asar`: + +| Path in `Contents/Resources` | Contents | +| --- | --- | +| `node/bin/node`, `node/LICENSE` | Node.js runtime and its license notices | +| `kc_storage/` | Storage service source | +| `kc_contracts/` | API contracts | + +The packaged app uses the normal per-user data locations unless +`KC_PROFILE_DIR` is set. For tests, always set `KC_PROFILE_DIR`. + +To change the runtime version, edit `VERSION` and `PINNED_SHA256` in +`scripts/prepare-node-runtime.mjs`. Take the hashes from +`https://nodejs.org/dist/v/SHASUMS256.txt`. + +Use `yarn package-local` or `yarn run pack`. The command `yarn pack` is a +Yarn built-in that makes an npm archive. -The result is an unsigned app in `dist/mac-arm64`. Notarization starts only -when `APPLEID`, `APPLEPWD`, and `TEAMID` are set. Signing needs a valid -Developer ID certificate. See the packaging limit above. +Notarization starts only when `APPLEID`, `APPLEPWD`, and `TEAMID` are set. +Signing needs a valid Developer ID certificate. Neither is verified. Do not use `yarn publish`. Its target S3 bucket does not exist. diff --git a/docs/desktop-storage-completion.md b/docs/desktop-storage-completion.md index ed49139e..8fe9a2cf 100644 --- a/docs/desktop-storage-completion.md +++ b/docs/desktop-storage-completion.md @@ -105,3 +105,34 @@ The command line `restore` uses the same validation and activation code. outside the repository, `PATH=/usr/bin:/bin:/usr/sbin:/sbin`, no credentials. The full workflow with UI restore, the service command line checked for the bundled runtime, two instances, process cleanup. + +## Verified (2026-10-06, macOS 26.6 arm64) + +| Command | Result | +| --- | --- | +| `yarn install --immutable` | Pass | +| `yarn build-dev` | Pass | +| `yarn test` | 8 of 8 pass | +| `yarn test-storage` | 22 of 22 pass | +| `yarn workspace kc_storage typecheck` | Pass | +| `yarn e2e` | 2 of 2 pass (UI restore in the development app) | +| `yarn package-local` | Pass. Unsigned `dist/mac-arm64/Knowledge.app`, about 440 MB. | +| `yarn e2e-packaged` | 1 of 1 pass | + +The packaged test ran the app from a copy in `$TMPDIR`, with +`PATH=/usr/bin:/bin:/usr/sbin:/sbin` (`which node` fails), and the working +directory outside the repository. The service command line was: + +``` +/Knowledge.app/Contents/Resources/node/bin/node + /Knowledge.app/Contents/Resources/kc_storage/src/main.ts serve + --data-dir /data/library --port 0 --allow-origin null --exit-on-stdin-close +``` + +The service working directory was `Contents/Resources`. No service process +remained after the app closed. + +Also verified by hand: the storage service accepts connections on +`127.0.0.1` only. A connection to the computer's network address is refused. + +Not verified: signed or notarized builds, macOS x64, Linux, Windows. diff --git a/docs/follow-ups.md b/docs/follow-ups.md index 4582559f..a544d3a2 100644 --- a/docs/follow-ups.md +++ b/docs/follow-ups.md @@ -1,18 +1,29 @@ # Follow-up Work -Work found during the recovery and storage milestones. Not started. +Work found during the recovery, storage, and desktop completion +milestones. Not started. ## Storage service | Item | Reason | | --- | --- | -| Package the service with a Node.js 24 runtime | Packaged builds cannot start the service now. | -| Restore a library from the UI | Restore needs the command line now. | -| Orphan asset cleanup policy | Deleting a source keeps its managed file. Only unreferenced files after a rollback or crash are removed. | +| Orphan asset cleanup policy | Deleting a source keeps its managed file. Only files without records after a rollback, crash, or failed restore are removed. | +| Restore into a library that has records (merge or replace) | Restore refuses a non-empty library now. | | Move the extracted-text cache into the service | The chat server keeps it in `data/storage/sources/`, outside the library. | | Thumbnails in the service | Electron makes thumbnails from a local copy of the managed file. | | Move favicon cache, inbox, and chat history out of local storage | They stay in the renderer for this milestone. | | Remove `kc_shared` dependency on Angular models | `kc_shared/models/project.model.ts` imports an Angular model. Electron needs the `@shared` alias because of it. | +| Stronger local access control | The token stops requests without it. Programs that run as the same OS user can read it. | + +## Packaging and release + +| Item | Reason | +| --- | --- | +| Signing and notarization | Not verified. Signing must include the bundled `node` binary. V8 may need the `allow-jit` entitlement under the hardened runtime. | +| macOS x64 package | A pinned runtime hash exists. The build is not tested. | +| Linux and Windows packages | No runtime configuration. Not tested. | +| Smaller package | The bundled Node.js adds about 120 MB. | +| Stop electron-builder rebuilding `canvas` | The app does not use `canvas`. The rebuild works now but is not needed. | ## Desktop app @@ -25,10 +36,10 @@ Work found during the recovery and storage milestones. Not started. | Root `postinstall` key is outside `scripts` | It never runs. Remove it or move it. | | Fresh-profile route error (`inbox/undefined`) | Cosmetic error in the log on first start. | | PDF viewer title shows the blob ID | Managed PDFs open from a `blob:` URL. | +| Select a project after a library restore | After restore, no project is selected. The user selects one in the tree. | ## Features (separate tasks) - Full-text search over extracted text. - Notes editor for the existing `notes` field. - Verification of AI chat with a real API key. -- Signed and notarized release builds. diff --git a/docs/storage-service.md b/docs/storage-service.md index 505e1009..aee99438 100644 --- a/docs/storage-service.md +++ b/docs/storage-service.md @@ -84,6 +84,10 @@ Base: `http://127.0.0.1:`. All `/v1` routes need | `GET /v1/assets/:id` | Asset metadata | | `GET /v1/assets/:id/content` | File bytes | | `GET /v1/backup` | Portable backup (tar) | +| `GET /v1/library` | Record counts and whether the library is empty | +| `POST /v1/restores` | Upload and validate a backup. Returns a preview. Nothing changes yet. | +| `POST /v1/restores/:id/activate` | Make the validated backup active (empty library only) | +| `DELETE /v1/restores/:id` | Discard a validated backup | Errors use `{ "error": { "code", "message" } }` and a matching HTTP status. @@ -93,7 +97,11 @@ Errors use `{ "error": { "code", "message" } }` and a matching HTTP status. - Token from the `KC_STORAGE_TOKEN` environment variable. Required. - The `Host` header must be `127.0.0.1:` or `localhost:`. - CORS allows only the configured origin (`null` for the desktop renderer). - CORS and loopback are not authentication. The token is. + CORS and loopback are not authentication. +- The token stops requests that do not have it. It does not stop programs + that run as the same operating-system user: they can read the token from + the service's environment or read the data directory directly. The + access boundary is the operating-system user account. - No route takes a filesystem path. Asset IDs must match a strict pattern. - One process per data directory (lock file). @@ -111,8 +119,11 @@ Errors use `{ "error": { "code", "message" } }` and a matching HTTP status. ## Desktop connection -Electron main starts `node src/kc_storage/src/main.ts serve` with -`--data-dir /data/library`, port `0`, and a new token. The service prints +Electron main starts ` /src/main.ts serve` with +`--data-dir /data/library`, port `0`, and a new token. A packaged +app uses the bundled runtime in `Contents/Resources/node/bin/node` and the +service files in `Contents/Resources/kc_storage` (see +[desktop-storage-completion.md](desktop-storage-completion.md)). The service prints one JSON line with its URL. The renderer receives the URL and token through `A2E:Backend:Info`. If the service does not start, the app shows an error and quits. The service stops when its standard input closes. @@ -144,7 +155,11 @@ or delete the data directory. The renderer data is not changed by migration. - `GET /v1/backup` or `backup --data-dir --out `. - Tar file: `manifest.json` (format `knowledge-library-backup`, version 1, projects, sources, assets with SHA-256) and `assets/`. -- `restore --data-dir --from ` checks every hash. +- Restore into an empty library through the API (`POST /v1/restores`, then + `POST /v1/restores//activate`), Settings > Backup, or + `restore --data-dir --from `. All use the same staged + validation and activation. See + [desktop-storage-completion.md](desktop-storage-completion.md). - Not included: chat history, UI preferences, settings, API key, text cache. ## UI workflow