diff --git a/.gitignore b/.gitignore index 9882c7e2..b4f21942 100755 --- a/.gitignore +++ b/.gitignore @@ -35,3 +35,6 @@ auth.json # End-to-end test output e2e/.output/ + +# Prepared Node.js runtime for packaging +vendor/ diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 1fae8098..9ddef177 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -8,15 +8,19 @@ You do not need AWS, an Apple developer account, or an AI API key. | Tool | Version | Notes | | --- | --- | --- | | macOS | 26.6 (arm64) | Verified. Linux and Windows are not verified. | -| Node.js | 24.21.0 | Verified. The app starts the storage service with this Node.js. | +| Node.js | 24.21.0 | Verified. Development builds, tests, and the unpackaged app use it. | | Yarn | 3.2.4 | The repository contains this version (`.yarnrc.yml` `yarnPath`). Any global `yarn` starts it. | | Electron | 26.3.0 | Installed by Yarn. | No native compiler is necessary. The install does not build `canvas`. The storage service (`src/kc_storage`) needs Node.js 24 or later. It uses -the built-in `node:sqlite` module. The desktop app starts it with `node` -from `PATH`. To use a different binary, set `KC_NODE=/path/to/node`. +the built-in `node:sqlite` module. + +| Run | Node.js for the storage service | +| --- | --- | +| Unpackaged (`yarn start`) | `node` from `PATH`, or `KC_NODE=/path/to/node` | +| Packaged app | Bundled Node.js 24.21.0 in `Contents/Resources/node/bin/node`. `PATH` and `KC_NODE` are not used. | Compatibility note: Angular 14 officially supports Node.js 14.15+ and 16.10+ only. Node.js 24 builds and runs this branch, but it is outside the official range. @@ -60,10 +64,21 @@ The app does not read or write the old local-storage project keys ## Instances and local servers Each instance starts two local servers: the chat server (Electron) and the -storage service (Node.js). Each listens on a random loopback port and -requires its own random token. The renderer receives the addresses and -tokens through IPC (`A2E:Backend:Info`). One instance cannot use the -servers of another instance. +storage service (Node.js). Each listens on a random port on `127.0.0.1` +and requires its own random bearer token. The renderer receives the +addresses and tokens through IPC (`A2E:Backend:Info`). + +### Access boundary + +- Verified: a request without the correct token gets `401`. The window of + one instance does not know the address or token of another instance, so + it cannot use the other instance's servers by mistake. +- Verified: the servers do not accept connections from other computers. +- Not protected: other programs that run as the same macOS user. Such a + program can read the token (for example, from the storage service's + environment, `KC_STORAGE_TOKEN`), or read the profile files directly. + The real boundary is the operating-system user account. +- The browser extension server (port 9000, off by default) has no token. Only one instance can use a profile at a time. A second start with the same profile prints a message and exits. Different profiles can run at the same time. @@ -194,35 +209,62 @@ do not change them. Changes made after the migration are not in them. ## Backup and restore -Settings > Backup has two backups. Neither is a complete application backup. +Settings > Backup has two separate backups. Neither is a complete +application backup. -| Backup | Contents | Restore | -| --- | --- | --- | -| Library (`knowledge-library-.tar`) | Projects, sources, topics, metadata, managed files with SHA-256 hashes | `yarn storage restore` into an empty data directory | -| Chat and preferences (`knowledge-backup-.json`) | Renderer local storage: chat history, inbox, UI preferences | Settings > Backup > Restore | +| Backup | Contains | Does not contain | Restore | +| --- | --- | --- | --- | +| Library (`knowledge-library-.tar`) | Projects, sources, topics, metadata, copies of imported files (SHA-256 for each) | Chat history, inbox, UI preferences, settings, API keys, extracted-text cache | Settings > Backup > Restore Library, or `yarn storage restore` | +| Chat and preferences (`knowledge-backup-.json`) | Renderer local storage: chat history, inbox, UI preferences | Projects, sources, files, settings | Settings > Backup > Restore | + +### Restore a library in the app + +1. Start Knowledge with a new, empty profile. For example: + `KC_PROFILE_DIR=/path/to/new-profile yarn start`. +2. Open Settings > Backup. +3. Select Restore Library and choose the `.tar` file. +4. Read the preview: date, counts, project names, warnings, and the data + that is not in the backup. +5. Select Restore. Knowledge shows the result and reloads. + +The restored library does not need the original files. + +Restore works only into an empty library. An empty library can have a +database file. If the library has projects, sources, or files, the app +shows the counts and refuses. It never replaces, merges, or deletes data. + +The storage service does all checks before anything changes: + +- Format `knowledge-library-backup`, supported version (1). +- Archive entries: only `manifest.json` and `assets/`, regular + files only. Other names, links, and entry types are refused. +- Records, duplicate IDs, project and file references, sizes, and the + SHA-256 of every file. +- Limits: archive 8 GiB, manifest 64 MiB, 1,000,000 entries, 2 GiB for + each file. -Not in either backup: settings, the API key, the extracted-text cache. -File sources without a managed copy (missing at migration) still depend on -the original path. +Then it activates the restore in one step. If the step fails, or if the +process stops during it, the library stays empty. Unused staging files are +removed at the next start. A library restore does not change chat history +or preferences. -To restore a library into a new profile: +### Restore from the command line ```sh yarn storage restore --data-dir /data/library --from knowledge-library-.tar -KC_PROFILE_DIR= yarn start ``` -Restore checks every file hash and refuses a directory that already has a -library. The restored library does not need the original files. +Stop the app first. The command uses the same checks as the app. ## Tests | Command | Checks | Needs | | --- | --- | --- | | `yarn test` | Renderer backup, profile path isolation | Node.js 22.18+ | -| `yarn test-storage` | Storage service without Electron: persistence, files, validation and access limits, migration (repeat, missing files, failure, rollback), backup and restore | Node.js 24+ | +| `yarn test-storage` | Storage service without Electron: persistence, files, validation and access limits, migration (repeat, missing files, failure, rollback), backup, restore (API and command line, non-empty refusal, unsupported version, missing and damaged files, unsafe archive entries, failure and process stop during activation) | Node.js 24+ | | `yarn workspace kc_storage typecheck` | Storage service types (TypeScript 5) | | | `yarn e2e` | Desktop UI with real profiles (below) | macOS, `yarn build-dev` first | +| `yarn e2e-packaged` | The unsigned package (below) | macOS arm64, `yarn package-local` first | The end-to-end tests start the real app with new, empty profiles in `e2e/.output//`. They never use your normal profile. @@ -231,11 +273,24 @@ The end-to-end tests start the real app with new, empty profiles in import a PDF and a text file, annotate, search, restart, delete the original files, read the same records and files, show the PDF, make a thumbnail and extract text from the managed copy, export the library, - restore it into a second profile, and show the PDF there. + restore it into a second profile through Settings > Backup, and show the + PDF there. - `e2e/isolation.e2e.mjs`: two profiles at the same time get different chat and storage addresses and tokens. Each server rejects the other instance's requests. A second start of the same profile exits. +- `e2e/packaged/packaged.e2e.mjs`: copies `dist/mac-arm64/Knowledge.app` + to a temporary directory and runs it with scratch profiles, a working + directory outside the repository, only `HOME`, `TMPDIR`, and + `PATH=/usr/bin:/bin:/usr/sbin:/sbin` (no Node.js, no credentials). It + checks that the service command line uses the bundled runtime, then runs + the full workflow: create, link, import, annotate, restart, export, + delete the originals, restore through the UI into a second profile, + restart, and open the files. It also checks an unsupported backup + version, the refusal for a non-empty library, unchanged chat history, + two instances at the same time, and that no service process remains. + Output stays in `$TMPDIR/kc-packaged-*`. The test removes the app copy. + Test files are in `e2e/fixtures/`. To delete old test output, delete `e2e/.output/`. @@ -254,16 +309,18 @@ curl -s -X POST http://127.0.0.1:7777 -d '{"op":"close"}' no notes editor. The `notes` field is kept but not shown. - Search uses titles, topics, descriptions, and source type. It does not search file content. -- The library backup includes managed files. Restore needs the storage - service command line. The UI cannot restore a library. +- Library restore works only into an empty library. +- Chat history, UI preferences, and settings are separate from the library + and its backup. - A file source that had no original file at migration has no managed copy. It still depends on its original path. -- Packaged builds cannot start the storage service. The package does not - contain `src/kc_storage` or a Node.js 24 runtime. Not tested on this - branch. A packaged app is expected to show the start error and quit. -- Signed and notarized releases are not verified. +- Packaging is verified only for macOS arm64 (unsigned). A pinned runtime + hash exists for macOS x64, but that build is not tested. Linux and + Windows packages have no runtime configuration. +- Signed and notarized releases are not verified. Signing must also sign + the bundled Node.js binary. - AI chat with a real API key is not verified. -- Only macOS is verified. +- Electron 26 and Angular 14 are outside their supported versions. Follow-up work is in [docs/follow-ups.md](docs/follow-ups.md). @@ -285,14 +342,39 @@ plain text. ## Packaging ```sh -CSC_IDENTITY_AUTO_DISCOVERY=false yarn run pack +yarn package-local ``` -Use `yarn run pack`. The command `yarn pack` is a Yarn built-in. It makes an -npm archive and does not start electron-builder. +This command does these steps: + +1. `yarn build`: production builds of Angular and Electron. +2. `yarn prepare-runtime`: downloads the official Node.js 24.21.0 archive + for this computer (`darwin-arm64`) from `nodejs.org`, checks the pinned + SHA-256 and the upstream `SHASUMS256.txt`, and extracts `bin/node` and + `LICENSE` to `vendor/node/darwin-arm64/`. It needs the network only the + first time. Git ignores `vendor/`. +3. electron-builder without signing (`CSC_IDENTITY_AUTO_DISCOVERY=false`). + +The result is an unsigned app in `dist/mac-arm64/Knowledge.app` (about +440 MB). It contains, outside `app.asar`: + +| Path in `Contents/Resources` | Contents | +| --- | --- | +| `node/bin/node`, `node/LICENSE` | Node.js runtime and its license notices | +| `kc_storage/` | Storage service source | +| `kc_contracts/` | API contracts | + +The packaged app uses the normal per-user data locations unless +`KC_PROFILE_DIR` is set. For tests, always set `KC_PROFILE_DIR`. + +To change the runtime version, edit `VERSION` and `PINNED_SHA256` in +`scripts/prepare-node-runtime.mjs`. Take the hashes from +`https://nodejs.org/dist/v/SHASUMS256.txt`. + +Use `yarn package-local` or `yarn run pack`. The command `yarn pack` is a +Yarn built-in that makes an npm archive. -The result is an unsigned app in `dist/mac-arm64`. Notarization starts only -when `APPLEID`, `APPLEPWD`, and `TEAMID` are set. Signing needs a valid -Developer ID certificate. See the packaging limit above. +Notarization starts only when `APPLEID`, `APPLEPWD`, and `TEAMID` are set. +Signing needs a valid Developer ID certificate. Neither is verified. Do not use `yarn publish`. Its target S3 bucket does not exist. diff --git a/docs/desktop-storage-completion.md b/docs/desktop-storage-completion.md new file mode 100644 index 00000000..8fe9a2cf --- /dev/null +++ b/docs/desktop-storage-completion.md @@ -0,0 +1,138 @@ +# Desktop Storage Completion (Milestone 2) + +## Purpose + +Make the desktop app self-contained for local use: + +- The unsigned package starts the storage service without an installed + Node.js. +- A user can restore a library backup through Settings > Backup. + +Not in this milestone: signing, notarization, publishing, cloud hosting, +browser client. + +## Findings from the current code + +| Area | Current behavior | Problem | +| --- | --- | --- | +| Runtime | Electron runs `node` from `PATH` or `KC_NODE`. | A packaged app on a machine without Node.js cannot start the service. | +| Service files | Read from `/src/kc_storage`. | The package has no `src/kc_storage`. Files inside `app.asar` cannot be read by a separate Node.js process. | +| Restore | `restore` command only. It writes records and moves files in one step. It checks only `.tar` entry names it needs and ignores other entries. | No UI. Unknown entry types are not rejected. Validation and activation are not separate. | +| Locks | One process per data directory (`storage.lock`). | The running service owns the library, so the UI cannot use the command line restore. | + +## Packaging the service + +- `scripts/prepare-node-runtime.mjs` downloads the official Node.js + archive for one platform from `nodejs.org/dist`. The version and the + archive SHA-256 are pinned in the script. The script also checks the hash + against the upstream `SHASUMS256.txt`. It extracts `bin/node` and + `LICENSE` to `vendor/node/-/`. Git ignores `vendor/`. +- Pinned runtime: Node.js 24.21.0 (LTS), `darwin-arm64`. +- electron-builder copies these into the app, outside `app.asar`: + + | Package path | Source | + | --- | --- | + | `Contents/Resources/node/bin/node` | `vendor/node/darwin-/bin/node` | + | `Contents/Resources/node/LICENSE` | Node.js license notices | + | `Contents/Resources/kc_storage/` | `src/kc_storage/src/*.ts`, `package.json` | + | `Contents/Resources/kc_contracts/` | `src/kc_contracts/*.ts`, `package.json` | + +- Runtime selection in Electron main: + + | Run | Node.js | Service entry | Working directory | + | --- | --- | --- | --- | + | Packaged | `process.resourcesPath/node/bin/node` (only) | `process.resourcesPath/kc_storage/src/main.ts` | `process.resourcesPath` | + | Unpackaged | `KC_NODE` or `node` from `PATH` | `/src/kc_storage/src/main.ts` | repository | + +- A packaged run gives the service only `KC_STORAGE_TOKEN`. It does not pass + `PATH` or other variables. +- The service stays independent of Electron. `yarn storage ...` still works. + +## Restore through the interface + +The storage service does all validation and activation. The renderer +uploads the selected file. A future browser client can use the same API. + +| Step | Request | Service action | +| --- | --- | --- | +| Check | `GET /v1/library` | Return record counts and `empty`. The UI disables restore and explains why if the library is not empty. | +| Upload and validate | `POST /v1/restores` (body: the `.tar`) | Refuse at once if the library is not empty. Stream the file to `restore//archive.tar` (size limit). Validate the archive, manifest, records, relationships, and every asset hash. Copy assets to `restore//files/`. Return a preview. | +| Preview | — | UI shows counts, project names, backup date, warnings, and what the backup does not contain. | +| Confirm | `POST /v1/restores//activate` | Activate (below). Return counts. | +| Cancel | `DELETE /v1/restores/` | Remove the staging directory. | + +After activation the UI shows the result and reloads. + +## No concurrent writes + +Activation is one synchronous function in the service's single thread: +check that the library is empty, start a transaction, insert records, move +staged files into place, commit. No other request runs during it. +Validation runs before activation, and other writes can happen then. So +activation checks again that the library is empty, inside the transaction. +The restore never merges into or replaces user records. + +## Failure and interruption + +| Failure | Result | +| --- | --- | +| Invalid archive, version, record, relationship, or hash | Error before activation. Staging directory removed. Library unchanged. | +| Error during activation | Transaction rolled back. Moved files removed. Staging removed. | +| Process stops during activation, before commit | SQLite discards the transaction. On the next start, the service removes asset files without records, and all `restore/` staging directories. Library is empty again. | +| Process stops after commit | Restore is complete. Files were moved before the commit. | +| Process stops before activation | Staging is removed at the next start. | + +## Archive safety + +- Allowed entries: `manifest.json` and `assets/` only. + Regular files only. No prefix field, no duplicates. +- Every manifest asset must have an entry with the same size. Every asset + entry must be in the manifest. +- The service never uses an entry name as a path. Files are written by + asset ID. +- Limits: archive 8 GiB, manifest 64 MiB, 1,000,000 entries, each asset at + most the upload limit (2 GiB). + +The command line `restore` uses the same validation and activation code. + +## Tests + +- Service tests (no Electron): restore through HTTP and the command line, + non-empty refusal, unsupported version, missing and damaged assets, + unsafe entries (path traversal, symbolic link, unknown entry), failure + during activation, process stop during activation and restart. +- Packaged app (`yarn e2e-packaged`): scratch profiles, working directory + outside the repository, `PATH=/usr/bin:/bin:/usr/sbin:/sbin`, no + credentials. The full workflow with UI restore, the service command line + checked for the bundled runtime, two instances, process cleanup. + +## Verified (2026-10-06, macOS 26.6 arm64) + +| Command | Result | +| --- | --- | +| `yarn install --immutable` | Pass | +| `yarn build-dev` | Pass | +| `yarn test` | 8 of 8 pass | +| `yarn test-storage` | 22 of 22 pass | +| `yarn workspace kc_storage typecheck` | Pass | +| `yarn e2e` | 2 of 2 pass (UI restore in the development app) | +| `yarn package-local` | Pass. Unsigned `dist/mac-arm64/Knowledge.app`, about 440 MB. | +| `yarn e2e-packaged` | 1 of 1 pass | + +The packaged test ran the app from a copy in `$TMPDIR`, with +`PATH=/usr/bin:/bin:/usr/sbin:/sbin` (`which node` fails), and the working +directory outside the repository. The service command line was: + +``` +/Knowledge.app/Contents/Resources/node/bin/node + /Knowledge.app/Contents/Resources/kc_storage/src/main.ts serve + --data-dir /data/library --port 0 --allow-origin null --exit-on-stdin-close +``` + +The service working directory was `Contents/Resources`. No service process +remained after the app closed. + +Also verified by hand: the storage service accepts connections on +`127.0.0.1` only. A connection to the computer's network address is refused. + +Not verified: signed or notarized builds, macOS x64, Linux, Windows. diff --git a/docs/follow-ups.md b/docs/follow-ups.md index 4582559f..a544d3a2 100644 --- a/docs/follow-ups.md +++ b/docs/follow-ups.md @@ -1,18 +1,29 @@ # Follow-up Work -Work found during the recovery and storage milestones. Not started. +Work found during the recovery, storage, and desktop completion +milestones. Not started. ## Storage service | Item | Reason | | --- | --- | -| Package the service with a Node.js 24 runtime | Packaged builds cannot start the service now. | -| Restore a library from the UI | Restore needs the command line now. | -| Orphan asset cleanup policy | Deleting a source keeps its managed file. Only unreferenced files after a rollback or crash are removed. | +| Orphan asset cleanup policy | Deleting a source keeps its managed file. Only files without records after a rollback, crash, or failed restore are removed. | +| Restore into a library that has records (merge or replace) | Restore refuses a non-empty library now. | | Move the extracted-text cache into the service | The chat server keeps it in `data/storage/sources/`, outside the library. | | Thumbnails in the service | Electron makes thumbnails from a local copy of the managed file. | | Move favicon cache, inbox, and chat history out of local storage | They stay in the renderer for this milestone. | | Remove `kc_shared` dependency on Angular models | `kc_shared/models/project.model.ts` imports an Angular model. Electron needs the `@shared` alias because of it. | +| Stronger local access control | The token stops requests without it. Programs that run as the same OS user can read it. | + +## Packaging and release + +| Item | Reason | +| --- | --- | +| Signing and notarization | Not verified. Signing must include the bundled `node` binary. V8 may need the `allow-jit` entitlement under the hardened runtime. | +| macOS x64 package | A pinned runtime hash exists. The build is not tested. | +| Linux and Windows packages | No runtime configuration. Not tested. | +| Smaller package | The bundled Node.js adds about 120 MB. | +| Stop electron-builder rebuilding `canvas` | The app does not use `canvas`. The rebuild works now but is not needed. | ## Desktop app @@ -25,10 +36,10 @@ Work found during the recovery and storage milestones. Not started. | Root `postinstall` key is outside `scripts` | It never runs. Remove it or move it. | | Fresh-profile route error (`inbox/undefined`) | Cosmetic error in the log on first start. | | PDF viewer title shows the blob ID | Managed PDFs open from a `blob:` URL. | +| Select a project after a library restore | After restore, no project is selected. The user selects one in the tree. | ## Features (separate tasks) - Full-text search over extracted text. - Notes editor for the existing `notes` field. - Verification of AI chat with a real API key. -- Signed and notarized release builds. diff --git a/docs/storage-service.md b/docs/storage-service.md index 505e1009..aee99438 100644 --- a/docs/storage-service.md +++ b/docs/storage-service.md @@ -84,6 +84,10 @@ Base: `http://127.0.0.1:`. All `/v1` routes need | `GET /v1/assets/:id` | Asset metadata | | `GET /v1/assets/:id/content` | File bytes | | `GET /v1/backup` | Portable backup (tar) | +| `GET /v1/library` | Record counts and whether the library is empty | +| `POST /v1/restores` | Upload and validate a backup. Returns a preview. Nothing changes yet. | +| `POST /v1/restores/:id/activate` | Make the validated backup active (empty library only) | +| `DELETE /v1/restores/:id` | Discard a validated backup | Errors use `{ "error": { "code", "message" } }` and a matching HTTP status. @@ -93,7 +97,11 @@ Errors use `{ "error": { "code", "message" } }` and a matching HTTP status. - Token from the `KC_STORAGE_TOKEN` environment variable. Required. - The `Host` header must be `127.0.0.1:` or `localhost:`. - CORS allows only the configured origin (`null` for the desktop renderer). - CORS and loopback are not authentication. The token is. + CORS and loopback are not authentication. +- The token stops requests that do not have it. It does not stop programs + that run as the same operating-system user: they can read the token from + the service's environment or read the data directory directly. The + access boundary is the operating-system user account. - No route takes a filesystem path. Asset IDs must match a strict pattern. - One process per data directory (lock file). @@ -111,8 +119,11 @@ Errors use `{ "error": { "code", "message" } }` and a matching HTTP status. ## Desktop connection -Electron main starts `node src/kc_storage/src/main.ts serve` with -`--data-dir /data/library`, port `0`, and a new token. The service prints +Electron main starts ` /src/main.ts serve` with +`--data-dir /data/library`, port `0`, and a new token. A packaged +app uses the bundled runtime in `Contents/Resources/node/bin/node` and the +service files in `Contents/Resources/kc_storage` (see +[desktop-storage-completion.md](desktop-storage-completion.md)). The service prints one JSON line with its URL. The renderer receives the URL and token through `A2E:Backend:Info`. If the service does not start, the app shows an error and quits. The service stops when its standard input closes. @@ -144,7 +155,11 @@ or delete the data directory. The renderer data is not changed by migration. - `GET /v1/backup` or `backup --data-dir --out `. - Tar file: `manifest.json` (format `knowledge-library-backup`, version 1, projects, sources, assets with SHA-256) and `assets/`. -- `restore --data-dir --from ` checks every hash. +- Restore into an empty library through the API (`POST /v1/restores`, then + `POST /v1/restores//activate`), Settings > Backup, or + `restore --data-dir --from `. All use the same staged + validation and activation. See + [desktop-storage-completion.md](desktop-storage-completion.md). - Not included: chat history, UI preferences, settings, API key, text cache. ## UI workflow diff --git a/e2e/lib.mjs b/e2e/lib.mjs index e1568f63..eba6a7a6 100644 --- a/e2e/lib.mjs +++ b/e2e/lib.mjs @@ -39,18 +39,37 @@ export function newRun(name) { } /** Launch the app with the given profile directory. */ -export async function launch(profileDir, logFile, env = {}) { +/** + * Launch the app with the given profile directory. + * + * options.packaged: path to a packaged Knowledge.app. The app then gets + * only options.env (no inherited variables) and runs in options.cwd. + */ +export async function launch(profileDir, logFile, env = {}, options = {}) { if (!profileDir) { throw new Error("launch() needs an explicit profile directory"); } const log = fs.createWriteStream(logFile, { flags: "a" }); - const app = await electron.launch({ - executablePath: ELECTRON, - args: [REPO], - cwd: REPO, - env: { ...appEnv(), ...env, KC_PROFILE_DIR: profileDir }, - timeout: 60000, - }); + const app = await electron.launch( + options.packaged + ? { + executablePath: path.join( + options.packaged, + "Contents/MacOS/Knowledge" + ), + args: [], + cwd: options.cwd, + env: { ...env, KC_PROFILE_DIR: profileDir }, + timeout: 60000, + } + : { + executablePath: ELECTRON, + args: [REPO], + cwd: REPO, + env: { ...appEnv(), ...env, KC_PROFILE_DIR: profileDir }, + timeout: 60000, + } + ); app.process().stdout?.on("data", (d) => log.write(`[main] ${d}`)); app.process().stderr?.on("data", (d) => log.write(`[main:err] ${d}`)); const page = await app.firstWindow(); diff --git a/e2e/packaged/packaged.e2e.mjs b/e2e/packaged/packaged.e2e.mjs new file mode 100644 index 00000000..a39417b1 --- /dev/null +++ b/e2e/packaged/packaged.e2e.mjs @@ -0,0 +1,335 @@ +/* + * The unsigned packaged app, self-contained: + * + * - The app is copied to a temporary directory outside the repository. + * - It runs with only HOME, TMPDIR, and PATH=/usr/bin:/bin:/usr/sbin:/sbin + * (no Node.js, no AWS, Apple, or AI credentials). + * - The working directory is outside the repository. + * - Every launch uses an explicit scratch profile. + * + * Build first: yarn package-local + * Run: yarn e2e-packaged + * App: KC_PACKAGED_APP= (default: dist/mac-arm64) + */ + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { execFileSync, spawnSync } from "node:child_process"; +import { + FIXTURES, + REPO, + backendInfo, + captureDownloads, + close, + launch, + readAsset, + readLibrary, + shot, + startFixtureSite, + waitForFile, +} from "../lib.mjs"; +import * as ui from "../steps.mjs"; +import { + readTarEntry, + readTarIndex, + TarWriter, +} from "../../src/kc_storage/src/tar.ts"; + +const BUILT_APP = + process.env.KC_PACKAGED_APP ?? + path.join(REPO, "dist/mac-arm64/Knowledge.app"); +const RESTRICTED_PATH = "/usr/bin:/bin:/usr/sbin:/sbin"; +const PROJECT = "Packaged Project"; +const SITE_TITLE = "Knowledge Fixture Site"; +const byTitle = (list) => + [...list].sort((a, b) => a.title.localeCompare(b.title)); + +/** Storage service processes that serve a data directory under root. */ +function serviceProcesses(root) { + const out = spawnSync("/usr/bin/pgrep", ["-f", `serve --data-dir ${root}`], { + encoding: "utf8", + }); + return out.stdout + .split("\n") + .filter(Boolean) + .map((pid) => ({ + pid, + command: execFileSync("/bin/ps", ["-o", "command=", "-p", pid], { + encoding: "utf8", + }).trim(), + })); +} + +async function waitForNoServices(root) { + for (let i = 0; i < 40 && serviceProcesses(root).length > 0; i++) { + await new Promise((r) => setTimeout(r, 250)); + } + return serviceProcesses(root); +} + +/** Copy a backup and set its manifest version. */ +async function withVersion(tar, out, version) { + const index = readTarIndex(tar, { allowName: () => true, maxEntries: 100 }); + const manifest = JSON.parse( + readTarEntry(tar, index.get("manifest.json")).toString() + ); + manifest.version = version; + const stream = fs.createWriteStream(out); + const writer = new TarWriter(stream); + await writer.addBuffer( + "manifest.json", + Buffer.from(JSON.stringify(manifest)) + ); + for (const [name, entry] of index) { + if (name !== "manifest.json") + await writer.addBuffer(name, readTarEntry(tar, entry)); + } + await writer.finish(); + await new Promise((r) => stream.end(r)); + return out; +} + +test( + "packaged app works without Node.js, the repository, or credentials", + { timeout: 600000 }, + async () => { + assert.ok( + fs.existsSync(BUILT_APP), + `Packaged app not found: ${BUILT_APP}. Run yarn package-local.` + ); + + // Everything outside the repository + // realpath: macOS reports /var/folders as /private/var/folders + const root = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), "kc-packaged-")) + ); + const app = path.join(root, "Knowledge.app"); + execFileSync("/usr/bin/ditto", [BUILT_APP, app]); + const cwd = path.join(root, "cwd"); + fs.mkdirSync(cwd); + const env = { + HOME: os.homedir(), + TMPDIR: os.tmpdir(), + PATH: RESTRICTED_PATH, + }; + const options = { packaged: app, cwd }; + const log = path.join(root, "app.log"); + const shots = path.join(root, "shots"); + fs.mkdirSync(shots); + console.log(`packaged test directory: ${root}`); + + // No Node.js on the restricted PATH + assert.notEqual(spawnSync("/usr/bin/which", ["node"], { env }).status, 0); + + const profileA = path.join(root, "profile-a"); + const profileB = path.join(root, "profile-b"); + const originals = path.join(root, "originals"); + fs.mkdirSync(originals); + const pdf = path.join(originals, "recovery-fixture.pdf"); + const txt = path.join(originals, "recovery-note.txt"); + fs.copyFileSync(path.join(FIXTURES, "recovery-fixture.pdf"), pdf); + fs.copyFileSync(path.join(FIXTURES, "recovery-note.txt"), txt); + const pdfBytes = fs.readFileSync(pdf); + const txtBytes = fs.readFileSync(txt); + const site = await startFixtureSite(); + + const expectedRuntime = `${app}/Contents/Resources/node/bin/node ${app}/Contents/Resources/kc_storage/src/main.ts serve`; + let a; + let b; + try { + // 1. Start. The service runs with the bundled runtime from the package. + a = await launch(profileA, log, env, options); + const services = serviceProcesses(profileA); + assert.equal(services.length, 1, JSON.stringify(services)); + assert.ok( + services[0].command.startsWith(expectedRuntime), + services[0].command + ); + const cwdOfService = execFileSync( + "/usr/sbin/lsof", + ["-a", "-p", services[0].pid, "-d", "cwd", "-Fn"], + { + encoding: "utf8", + } + ); + assert.match( + cwdOfService, + new RegExp( + `n${app.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}/Contents/Resources` + ) + ); + + // 2-5. Create, save a link, import files, annotate + await ui.createProject(a.page, PROJECT, "Packaged workflow"); + await ui.addLinkToProject(a.page, site.url, SITE_TITLE); + await ui.importFilesToProject(a.page, [pdf, txt]); + await ui.openTable(a.page); + await ui.openSource(a.page, "recovery-note.txt"); + await ui.addAnnotation(a.page, "annotation", "packaged note"); + await ui.addTopic(a.page, "packagedtopic"); + await a.page.waitForTimeout(1500); + await ui.closeDialog(a.page); + await a.page.waitForTimeout(1000); + const library = await readLibrary(a.page); + assert.equal(library.length, 1); + assert.equal(library[0].sources.length, 3); + await close(a); + a = undefined; + assert.deepEqual(await waitForNoServices(profileA), []); + + // 6. Restart and retrieve + a = await launch(profileA, log, env, options); + const afterRestart = await readLibrary(a.page); + assert.deepEqual( + byTitle(afterRestart[0].sources), + byTitle(library[0].sources) + ); + const note = afterRestart[0].sources.find( + (s) => s.title === "recovery-note.txt" + ); + assert.deepEqual(note.topics, ["packagedtopic"]); + assert.deepEqual(note.annotations, ["packaged note"]); + + // 7. Export the library through the UI + const downloads = path.join(root, "downloads"); + await captureDownloads(a.app, downloads); + await ui.openSettings(a.page, "Backup"); + await a.page.locator("button", { hasText: "Export Library" }).click(); + const date = new Date().toISOString().slice(0, 10); + const tar = await waitForFile( + path.join(downloads, `knowledge-library-${date}.tar`) + ); + + // 8. Make the original files unavailable + fs.rmSync(originals, { recursive: true }); + + // 9. Second isolated profile, while the first instance still runs + b = await launch(profileB, log, env, options); + const infoA = await backendInfo(a.page); + const infoB = await backendInfo(b.page); + assert.notEqual(infoA.storage.url, infoB.storage.url); + assert.notEqual(infoA.storage.token, infoB.storage.token); + assert.notEqual(infoA.chat.url, infoB.chat.url); + const crossStatus = await b.page.evaluate( + async ({ url, token }) => + ( + await fetch(`${url}/v1/projects`, { + headers: { Authorization: `Bearer ${token}` }, + }) + ).status, + { url: infoA.storage.url, token: infoB.storage.token } + ); + assert.equal(crossStatus, 401); + for (const s of [ + ...serviceProcesses(profileA), + ...serviceProcesses(profileB), + ]) { + assert.ok(s.command.startsWith(expectedRuntime), s.command); + } + await close(a); + a = undefined; + assert.deepEqual(await waitForNoServices(profileA), []); + + // Chat history and preferences must not change during a library restore + await b.page.evaluate(() => + localStorage.setItem("chat-sentinel", "keep me") + ); + + // An unsupported backup version is refused with a clear message + const v2 = await withVersion(tar, path.join(root, "v2.tar"), 2); + const refused = await ui.restoreLibraryError(b.page, v2); + assert.match(refused, /Backup version 2 is not supported/); + assert.deepEqual(await readLibrary(b.page), []); + + // 10. Restore through the interface + const restore = await ui.restoreLibrary(b.page, tar); + assert.match(restore.preview, /1 project, 3 sources, 2 files/); + assert.match(restore.preview, /Chat history/); + assert.match(restore.preview, /API keys/); + assert.match( + restore.result, + /Restored 1 project, 3 sources, and 2 files/ + ); + assert.equal( + await b.page.evaluate(() => localStorage.getItem("chat-sentinel")), + "keep me" + ); + + // The interface now refuses another restore and says why + await ui.openSettings(b.page, "Backup"); + const refusal = await b.page.locator("#restore-refused").innerText(); + assert.match( + refusal, + /Restore works only into an empty library\. This library has 1 project, 3 sources, and 2 files/ + ); + await shot(b.page, shots, "b-restore-refused"); + await close(b); + b = undefined; + + // 11. Restart + b = await launch(profileB, log, env, options); + await ui.selectProject(b.page, PROJECT); + + // 12. Open the restored files without the originals + const restored = await readLibrary(b.page); + assert.deepEqual( + byTitle(restored[0].sources), + byTitle(afterRestart[0].sources) + ); + const ids = Object.fromEntries( + restored[0].sources.map((s) => [s.title, s.assetId]) + ); + assert.ok( + (await readAsset(b.page, ids["recovery-fixture.pdf"])).equals(pdfBytes) + ); + assert.ok( + (await readAsset(b.page, ids["recovery-note.txt"])).equals(txtBytes) + ); + + await ui.openTable(b.page); + await ui.openSource(b.page, "recovery-fixture.pdf"); + await ui.openDocumentTab(b.page); + const embed = b.page.locator("source-document embed"); + await embed.waitFor({ timeout: 15000 }); + assert.match(await embed.getAttribute("src"), /^blob:/); + await b.page.waitForTimeout(2000); + await shot(b.page, shots, "b-restored-pdf"); + await ui.closeDialog(b.page); + + const thumbnail = await b.page.evaluate( + (assetId) => + new Promise((resolve) => { + window.api.receive("E2A:FileSystem:FileThumbnail", (responses) => { + const mine = responses.find( + (r) => r.success?.data?.id === "probe" + ); + if (mine) resolve(mine.success.data.thumbnail); + }); + window.api.send("A2E:FileSystem:FileThumbnail", [ + { + path: "/nonexistent/recovery-fixture.pdf", + assetId, + id: "probe", + }, + ]); + }), + ids["recovery-fixture.pdf"] + ); + assert.match(thumbnail, /^data:image\/png;base64,.{100,}/); + await close(b); + b = undefined; + + // No service process remains after the app exits + assert.deepEqual(await waitForNoServices(root), []); + } finally { + if (a) await close(a); + if (b) await close(b); + site.close(); + // Keep logs, profiles, and screenshots. Remove the 440 MB app copy. + fs.rmSync(app, { recursive: true, force: true }); + } + } +); diff --git a/e2e/steps.mjs b/e2e/steps.mjs index 89adf758..7e85f7a0 100644 --- a/e2e/steps.mjs +++ b/e2e/steps.mjs @@ -147,7 +147,39 @@ export async function search(page, query) { ); } +/** Open a Settings section. Settings is a dialog; reuse it if it is open. */ export async function openSettings(page, section) { - await page.locator("i.pi-cog").first().click(); - await page.locator(".p-menuitem-link", { hasText: section }).click(); + const item = page.locator(".p-menuitem-link", { hasText: section }); + if (!(await item.isVisible())) { + await page.locator("i.pi-cog").first().click(); + } + await item.click(); +} + +/** + * Restore a library backup through Settings > Backup. Returns the preview + * text that the user sees before confirmation. + */ +export async function restoreLibrary(page, tar) { + await openSettings(page, "Backup"); + await page.locator("#restore-library-input").setInputFiles(tar); + const preview = page.locator("#restore-preview"); + await preview.waitFor({ timeout: 60000 }); + const text = await preview.innerText(); + await page.locator("#restore-confirm").click(); + await page.locator("#restore-result").waitFor({ timeout: 60000 }); + const result = await page.locator("#restore-result").innerText(); + // The app reloads after a restore + await page.waitForEvent("load", { timeout: 15000 }); + await page.waitForSelector("app-create button", { timeout: 30000 }); + return { preview: text, result }; +} + +/** Upload a library backup that the service must refuse. Returns the error. */ +export async function restoreLibraryError(page, tar) { + await openSettings(page, "Backup"); + await page.locator("#restore-library-input").setInputFiles(tar); + const error = page.locator("#restore-error"); + await error.waitFor({ timeout: 60000 }); + return error.innerText(); } diff --git a/e2e/workflow.e2e.mjs b/e2e/workflow.e2e.mjs index 749cb9e1..67d44959 100644 --- a/e2e/workflow.e2e.mjs +++ b/e2e/workflow.e2e.mjs @@ -11,10 +11,8 @@ import { test } from "node:test"; import assert from "node:assert/strict"; import fs from "node:fs"; import path from "node:path"; -import { spawnSync } from "node:child_process"; import { FIXTURES, - REPO, captureDownloads, close, launch, @@ -155,20 +153,16 @@ test( await close(ctx); ctx = undefined; - // Restore into the storage directory of a new profile, then open it - const restored = spawnSync( - process.execPath, - [ - path.join(REPO, "src/kc_storage/src/main.ts"), - "restore", - "--data-dir", - path.join(profileB, "data", "library"), - "--from", - tar, - ], - { encoding: "utf8" } + // Restore through the UI into a new profile, then restart and open it + ctx = await launch(profileB, log); + const restore = await ui.restoreLibrary(ctx.page, tar); + assert.match(restore.preview, /1 project, 3 sources, 2 files/); + assert.match( + restore.result, + /Restored 1 project, 3 sources, and 2 files/ ); - assert.equal(restored.status, 0, restored.stderr); + await close(ctx); + ctx = undefined; ctx = await launch(profileB, log); await ui.selectProject(ctx.page, PROJECT); diff --git a/package.json b/package.json index 7f4c0f46..e628fc8c 100644 --- a/package.json +++ b/package.json @@ -27,6 +27,8 @@ "clean": "rm -rf dist && yarn workspaces foreach run clean", "purge": "yarn workspaces foreach run purge && rm -rf node_modules && yarn cache clean", "pack": "electron-builder --dir", + "prepare-runtime": "node scripts/prepare-node-runtime.mjs", + "package-local": "yarn build && yarn prepare-runtime && CSC_IDENTITY_AUTO_DISCOVERY=false electron-builder --dir", "dist": "electron-builder --publish never --x64 --arm64", "dist-all": "electron-builder -mwl --x64 --arm64 --publish never", "interactive-upgrade": "yarn upgrade-interactive", @@ -39,6 +41,7 @@ "start-debug": "electron . --inspect=5858", "test": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --test \"tests/*.test.mts\"", "e2e": "node --test --test-concurrency=1 \"e2e/*.e2e.mjs\"", + "e2e-packaged": "node --test --test-concurrency=1 \"e2e/packaged/*.e2e.mjs\"", "storage": "node src/kc_storage/src/main.ts", "test-storage": "yarn workspace kc_storage test", "watch-main": "cd src/kc_angular && yarn run watch-main", @@ -122,6 +125,23 @@ { "from": "node_modules/@dqbd/tiktoken/tiktoken_bg.wasm", "to": "tiktoken_bg.wasm" + }, + { + "from": "src/kc_storage", + "to": "kc_storage", + "filter": [ + "package.json", + "src/**/*.ts" + ] + }, + { + "from": "src/kc_contracts", + "to": "kc_contracts", + "filter": [ + "package.json", + "*.ts", + "!*.d.ts" + ] } ], "publish": { @@ -139,7 +159,17 @@ "dmg", "zip" ], - "darkModeSupport": true + "darkModeSupport": true, + "extraResources": [ + { + "from": "vendor/node/darwin-${arch}/bin/node", + "to": "node/bin/node" + }, + { + "from": "vendor/node/darwin-${arch}/LICENSE", + "to": "node/LICENSE" + } + ] }, "linux": { "target": [ diff --git a/scripts/prepare-node-runtime.mjs b/scripts/prepare-node-runtime.mjs new file mode 100644 index 00000000..94fdc00a --- /dev/null +++ b/scripts/prepare-node-runtime.mjs @@ -0,0 +1,113 @@ +/* + * Prepare the Node.js runtime that the desktop package uses to run the + * storage service. + * + * node scripts/prepare-node-runtime.mjs [--platform darwin] [--arch arm64] + * + * Downloads the official archive from nodejs.org, checks its SHA-256 + * against the value pinned here and against the upstream SHASUMS256.txt, + * and extracts bin/node and LICENSE to vendor/node/-/. + * The archive is cached in vendor/node/cache/. Run it again: it does + * nothing when the runtime is already prepared. + */ + +import crypto from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import { execFileSync } from "node:child_process"; +import { parseArgs } from "node:util"; + +const VERSION = "24.21.0"; + +/** SHA-256 of node-v--.tar.gz, from SHASUMS256.txt. */ +const PINNED_SHA256 = { + "darwin-arm64": + "bed7eea5325e1108f32ce5228ddd6a5f0f08a499ee42aa7442aea583702f6057", + "darwin-x64": + "1462cb3b3046b815cf8ea436d3da450ec1a9f11dac7e5a46b0ada5305d7e8097", +}; + +const ROOT = path.resolve(import.meta.dirname, ".."); +const { values } = parseArgs({ + options: { + platform: { type: "string", default: process.platform }, + arch: { type: "string", default: process.arch }, + }, +}); + +const target = `${values.platform}-${values.arch}`; +const expected = PINNED_SHA256[target]; +if (!expected) { + console.error( + `No pinned Node.js runtime for ${target}. Supported: ${Object.keys(PINNED_SHA256).join(", ")}` + ); + process.exit(1); +} + +const name = `node-v${VERSION}-${target}`; +const archiveName = `${name}.tar.gz`; +const baseUrl = `https://nodejs.org/dist/v${VERSION}`; +const outDir = path.join(ROOT, "vendor", "node", target); +const cacheDir = path.join(ROOT, "vendor", "node", "cache"); +const stamp = path.join(outDir, "VERSION"); + +function sha256(file) { + return crypto.createHash("sha256").update(fs.readFileSync(file)).digest("hex"); +} + +async function download(url, file) { + const res = await fetch(url); + if (!res.ok) { + throw new Error(`Download failed: ${url} (${res.status})`); + } + fs.writeFileSync(file, Buffer.from(await res.arrayBuffer())); +} + +if (fs.existsSync(stamp) && fs.readFileSync(stamp, "utf8").trim() === `${VERSION} ${expected}`) { + console.log(`Node.js ${VERSION} for ${target} is ready: ${outDir}`); + process.exit(0); +} + +fs.mkdirSync(cacheDir, { recursive: true }); +const archive = path.join(cacheDir, archiveName); +if (!fs.existsSync(archive) || sha256(archive) !== expected) { + console.log(`Downloading ${baseUrl}/${archiveName}`); + await download(`${baseUrl}/${archiveName}`, archive); +} + +const actual = sha256(archive); +if (actual !== expected) { + fs.rmSync(archive, { force: true }); + throw new Error(`SHA-256 mismatch for ${archiveName}: got ${actual}, pinned ${expected}`); +} + +const sums = await (await fetch(`${baseUrl}/SHASUMS256.txt`)).text(); +const upstream = sums + .split("\n") + .map((line) => line.trim().split(/\s+/)) + .find(([, file]) => file === archiveName)?.[0]; +if (upstream !== expected) { + throw new Error(`Upstream SHASUMS256.txt does not match the pinned hash for ${archiveName}.`); +} + +fs.rmSync(outDir, { recursive: true, force: true }); +fs.mkdirSync(path.join(outDir, "bin"), { recursive: true }); +execFileSync("tar", [ + "-xzf", + archive, + "-C", + outDir, + "--strip-components=1", + `${name}/bin/node`, + `${name}/LICENSE`, +]); + +const reported = execFileSync(path.join(outDir, "bin", "node"), ["--version"], { + encoding: "utf8", +}).trim(); +if (reported !== `v${VERSION}`) { + throw new Error(`Extracted runtime reports ${reported}, expected v${VERSION}.`); +} + +fs.writeFileSync(stamp, `${VERSION} ${expected}\n`); +console.log(`Node.js ${VERSION} for ${target} is ready: ${outDir} (sha256 ${expected})`); diff --git a/src/kc_angular/src/app/components/settings/storage-settings.component.ts b/src/kc_angular/src/app/components/settings/storage-settings.component.ts index 01866f04..0b74b336 100755 --- a/src/kc_angular/src/app/components/settings/storage-settings.component.ts +++ b/src/kc_angular/src/app/components/settings/storage-settings.component.ts @@ -14,7 +14,8 @@ * limitations under the License. */ -import { Component } from '@angular/core'; +import { Component, OnInit } from '@angular/core'; +import type { RestorePreview } from '@contracts/storage'; import { StorageService } from '@services/ipc-services/storage.service'; import { FormBuilder, FormGroup } from '@angular/forms'; import { NotificationsService } from '@services/user-services/notifications.service'; @@ -35,10 +36,9 @@ import { SettingsService } from '@services/ipc-services/settings.service';
- Projects, sources, topics, metadata, and copies of imported - files. Restore it with the storage service command line (see - DEVELOPMENT.md). It does not hold chat history, preferences, - or settings. + Contains projects, sources, topics, metadata, and copies of + imported files. Does not contain chat history, the inbox, UI + preferences, settings, or API keys.
@@ -50,6 +50,99 @@ import { SettingsService } from '@services/ipc-services/settings.service'; >
+ + +
+ + +
+ + +
+ {{ refusal }} +
+ +
+
Restore this library?
+
+ Backup created: + {{ preview.backupCreatedAt | date : 'medium' }} (format + version {{ preview.backupVersion }}) +
+
+ {{ count(preview.counts.projects, 'project') }}, + {{ count(preview.counts.sources, 'source') }}, + {{ count(preview.counts.assets, 'file') }} ({{ + preview.counts.bytes / 1024 | number : '1.0-0' + }} + KB) +
+
+ Projects: {{ preview.projectNames.join(', ') }} +
+
+ Warning: {{ warning }} +
+
+ Not in this backup, and not changed: + {{ preview.notIncluded.join(', ') }}. +
+
+ + +
+
+ +
+ {{ restoreError }} +
+ +
+ {{ restoreResult }} +
@@ -103,13 +196,26 @@ import { SettingsService } from '@services/ipc-services/settings.service'; `, styles: [], }) -export class StorageSettingsComponent { +export class StorageSettingsComponent implements OnInit { exportType = 'Everything'; exporting = false; exportingLibrary = false; + /** Restore is allowed only into a library without records. */ + libraryEmpty = false; + + refusal = ''; + + preview?: RestorePreview; + + restoreBusy = false; + + restoreError = ''; + + restoreResult = ''; + form: FormGroup; constructor( @@ -121,6 +227,108 @@ export class StorageSettingsComponent { this.form = formBuilder.group({}); } + count(n: number, noun: string) { + return `${n} ${noun}${n === 1 ? '' : 's'}`; + } + + ngOnInit() { + this.refreshLibraryStatus(); + } + + private message(e: any): string { + return e?.error?.error?.message ?? e?.message ?? `${e}`; + } + + async refreshLibraryStatus() { + try { + const status = await this.storage.libraryStatus(); + this.libraryEmpty = status.empty; + const c = status.counts; + this.refusal = status.empty + ? '' + : `Restore works only into an empty library. This library has ${this.count( + c.projects, + 'project' + )}, ` + + `${this.count(c.sources, 'source')}, and ${this.count( + c.assets, + 'file' + )}. To restore, start Knowledge with a new ` + + 'profile. Existing data is not changed or deleted.'; + } catch (e) { + this.libraryEmpty = false; + this.refusal = `Library status unavailable: ${this.message(e)}`; + } + } + + /** Upload the selected file. The service validates it. Nothing changes yet. */ + async onSelectLibraryBackup($event: any) { + const input: HTMLInputElement = $event.target; + const file = input.files?.[0]; + input.value = ''; + if (!file) { + return; + } + this.restoreError = ''; + this.restoreResult = ''; + this.restoreBusy = true; + try { + this.preview = await this.storage.stageRestore(file); + } catch (e) { + this.restoreError = `The backup cannot be restored. ${this.message(e)}`; + await this.refreshLibraryStatus(); + } finally { + this.restoreBusy = false; + } + } + + async onConfirmRestore() { + if (!this.preview) { + return; + } + this.restoreBusy = true; + this.restoreError = ''; + try { + const r = await this.storage.activateRestore(this.preview.id); + this.preview = undefined; + this.restoreResult = + `Restored ${this.count(r.projects, 'project')}, ${this.count( + r.sources, + 'source' + )}, ` + + `and ${this.count(r.assets, 'file')}. ` + + 'Knowledge reloads now.'; + this.notifications.success( + 'Backup', + 'Library Restored', + this.restoreResult + ); + + // Reload so every service reads the restored library + setTimeout(() => window.location.reload(), 2500); + } catch (e) { + this.preview = undefined; + this.restoreError = `Restore failed. The library is not changed. ${this.message( + e + )}`; + await this.refreshLibraryStatus(); + } finally { + this.restoreBusy = false; + } + } + + async onCancelRestore() { + const id = this.preview?.id; + this.preview = undefined; + if (id) { + try { + await this.storage.cancelRestore(id); + } catch { + // The staged files are removed at the next service start + } + } + } + onExport() { this.exporting = true; try { diff --git a/src/kc_angular/src/app/services/ipc-services/storage.service.ts b/src/kc_angular/src/app/services/ipc-services/storage.service.ts index 0fddb462..5e134235 100755 --- a/src/kc_angular/src/app/services/ipc-services/storage.service.ts +++ b/src/kc_angular/src/app/services/ipc-services/storage.service.ts @@ -39,8 +39,10 @@ import { sourceToRecord, } from '@contracts/mapping'; import type { + LibraryStatus, ProjectList, ProjectRecord, + RestorePreview, SourceList, SourceRecord, } from '@contracts/storage'; @@ -264,6 +266,40 @@ export class StorageService { return blob.size; } + libraryStatus(): Promise { + return firstValueFrom(this.http.get(`${this.api}/library`)); + } + + /** + * Upload a library backup. The service validates it and keeps it staged. + * Nothing becomes active until activateRestore(). + */ + async stageRestore(file: Blob): Promise { + const { restore } = await firstValueFrom( + this.http.post<{ restore: RestorePreview }>( + `${this.api}/restores`, + file, + { + headers: { 'Content-Type': 'application/x-tar' }, + } + ) + ); + return restore; + } + + async activateRestore(id: string) { + const { restored } = await firstValueFrom( + this.http.post<{ + restored: { projects: number; sources: number; assets: number }; + }>(`${this.api}/restores/${id}/activate`, null) + ); + return restored; + } + + async cancelRestore(id: string) { + await firstValueFrom(this.http.delete(`${this.api}/restores/${id}`)); + } + /** * Download a backup of renderer storage: chat history and preferences. * Projects and sources are not in it. They are in the library backup. diff --git a/src/kc_contracts/storage.ts b/src/kc_contracts/storage.ts index 97e458c6..82d82993 100644 --- a/src/kc_contracts/storage.ts +++ b/src/kc_contracts/storage.ts @@ -90,6 +90,25 @@ export interface SourceList { export const ASSET_FILENAME_HEADER = "x-knowledge-filename"; export const ASSET_ORIGINAL_PATH_HEADER = "x-knowledge-original-path"; +/** GET /v1/library */ +export interface LibraryStatus { + empty: boolean; + counts: { projects: number; sources: number; assets: number }; +} + +/** Result of POST /v1/restores: a validated backup that is not active yet. */ +export interface RestorePreview { + id: string; + backupCreatedAt: string | null; + backupVersion: number; + counts: { projects: number; sources: number; assets: number; bytes: number }; + /** First 50 project names. */ + projectNames: string[]; + warnings: string[]; + /** Data that a library backup never contains. */ + notIncluded: string[]; +} + export interface LibraryBackupManifest { format: "knowledge-library-backup"; version: 1; diff --git a/src/kc_electron/src/app/storage.process.ts b/src/kc_electron/src/app/storage.process.ts index 7f22ddaa..0a5ef8f0 100644 --- a/src/kc_electron/src/app/storage.process.ts +++ b/src/kc_electron/src/app/storage.process.ts @@ -15,11 +15,17 @@ */ /* - * Starts the standalone storage service (src/kc_storage) with Node.js. + * Starts the standalone storage service (kc_storage) with Node.js. * - * Electron's own Node.js is too old for node:sqlite, so the service runs - * with the Node.js on PATH, or with KC_NODE. The service stops when this - * process closes its standard input, including after a crash. + * Electron's own Node.js is too old for node:sqlite, so the service runs in + * a separate Node.js 24 process: + * + * Packaged: Resources/node/bin/node (bundled) and Resources/kc_storage. + * PATH, KC_NODE, and the working directory are not used. + * Unpackaged: KC_NODE or node from PATH, and /src/kc_storage. + * + * The service stops when this process closes its standard input, including + * after a crash. */ import { spawn, ChildProcess } from "child_process"; @@ -32,22 +38,50 @@ const START_TIMEOUT_MS = 20000; let child: ChildProcess | undefined; +interface ServiceLocation { + node: string; + entry: string; + cwd: string; + env: NodeJS.ProcessEnv; +} + +function serviceLocation(token: string): ServiceLocation { + if (app.isPackaged) { + const resources = process.resourcesPath; + return { + node: path.join(resources, "node", "bin", "node"), + entry: path.join(resources, "kc_storage", "src", "main.ts"), + cwd: resources, + env: { KC_STORAGE_TOKEN: token }, + }; + } + // Only the variables the service needs. ELECTRON_RUN_AS_NODE and + // test-runner variables must not reach it. + return { + node: process.env.KC_NODE || "node", + entry: path.join(app.getAppPath(), "src", "kc_storage", "src", "main.ts"), + cwd: app.getAppPath(), + env: { + PATH: process.env.PATH, + HOME: process.env.HOME, + KC_STORAGE_TOKEN: token, + }, + }; +} + export function startStorageService(dataDir: string): Promise { - const entry = path.join( - app.getAppPath(), - "src", - "kc_storage", - "src", - "main.ts" - ); - const node = process.env.KC_NODE || "node"; const token = newToken(); + const { node, entry, cwd, env } = serviceLocation(token); return new Promise((resolve) => { if (!fs.existsSync(entry)) { resolve({ error: `Storage service not found at ${entry}.` }); return; } + if (app.isPackaged && !fs.existsSync(node)) { + resolve({ error: `Bundled Node.js runtime not found at ${node}.` }); + return; + } let stderr = ""; let settled = false; @@ -58,14 +92,6 @@ export function startStorageService(dataDir: string): Promise { } }; - // Only the variables the service needs. ELECTRON_RUN_AS_NODE and - // test-runner variables must not reach it. - const env: NodeJS.ProcessEnv = { - PATH: process.env.PATH, - HOME: process.env.HOME, - KC_STORAGE_TOKEN: token, - }; - child = spawn( node, [ @@ -79,9 +105,11 @@ export function startStorageService(dataDir: string): Promise { "null", "--exit-on-stdin-close", ], - { env, stdio: ["pipe", "pipe", "pipe"] } + { cwd, env, stdio: ["pipe", "pipe", "pipe"] } ); + console.log(`[Knowledge]: storage service runtime: ${node} ${entry}`); + const timer = setTimeout(() => { finish({ error: `Storage service did not start in time. ${stderr}`.trim(), @@ -92,7 +120,9 @@ export function startStorageService(dataDir: string): Promise { child.on("error", (e) => { clearTimeout(timer); finish({ - error: `Could not run "${node}": ${e.message}. Install Node.js 24 or set KC_NODE.`, + error: app.isPackaged + ? `Could not run the bundled Node.js runtime (${node}): ${e.message}.` + : `Could not run "${node}": ${e.message}. Install Node.js 24 or set KC_NODE.`, }); }); diff --git a/src/kc_storage/src/assets.ts b/src/kc_storage/src/assets.ts index e83d2d42..c8973f8c 100644 --- a/src/kc_storage/src/assets.ts +++ b/src/kc_storage/src/assets.ts @@ -34,9 +34,16 @@ export class AssetStore { return path.join(this.root, id.slice(0, 2), id); } - /** Write a stream to a temporary file and hash it. */ - async stage(input: Readable, maxBytes: number): Promise { - const tmpPath = path.join(this.tmpDir, crypto.randomUUID()); + /** + * Write a stream to a temporary file and hash it. The directory must be + * on the same filesystem as the store (default: assets/tmp). + */ + async stage( + input: Readable, + maxBytes: number, + dir = this.tmpDir + ): Promise { + const tmpPath = path.join(dir, crypto.randomUUID()); const hash = crypto.createHash("sha256"); let size = 0; diff --git a/src/kc_storage/src/backup.ts b/src/kc_storage/src/backup.ts index 95c7be80..d765fcf9 100644 --- a/src/kc_storage/src/backup.ts +++ b/src/kc_storage/src/backup.ts @@ -4,20 +4,11 @@ * history, UI preferences, settings, or credentials. */ -import fs from "node:fs"; -import path from "node:path"; import type { Writable } from "node:stream"; import type { LibraryBackupManifest } from "../../kc_contracts/storage.ts"; -import { DATABASE_FILE, openDataDir, type DataDir } from "./datadir.ts"; -import { SCHEMA_VERSION, transaction } from "./db.ts"; -import type { StagedFile } from "./assets.ts"; -import { - readTarEntry, - readTarIndex, - streamTarEntry, - TarWriter, -} from "./tar.ts"; -import { validAssetId, validProject, validSource } from "./validate.ts"; +import type { DataDir } from "./datadir.ts"; +import { SCHEMA_VERSION } from "./db.ts"; +import { TarWriter } from "./tar.ts"; import { VERSION } from "./version.ts"; export const BACKUP_FORMAT = "knowledge-library-backup"; @@ -51,111 +42,3 @@ export async function writeBackup(dir: DataDir, out: Writable) { await tar.finish(); return manifest; } - -export interface RestoreSummary { - projects: number; - sources: number; - assets: number; -} - -/** - * Restore a backup into a new or empty data directory. Validates the - * manifest and every file hash before the database changes. - */ -export async function restoreBackup( - archive: string, - root: string -): Promise { - if (fs.existsSync(path.join(root, DATABASE_FILE))) { - const probe = openDataDir(root); - const empty = probe.library.isEmpty(); - probe.close(); - if (!empty) { - throw new Error( - `The data directory ${root} already has a library. Use an empty directory.` - ); - } - } - - const index = readTarIndex(archive); - const manifestEntry = index.get("manifest.json"); - if (!manifestEntry) { - throw new Error("The backup has no manifest.json."); - } - const manifest = JSON.parse( - readTarEntry(archive, manifestEntry).toString("utf8") - ); - if (manifest?.format !== BACKUP_FORMAT || manifest.version !== 1) { - throw new Error("The file is not a knowledge-library-backup version 1."); - } - - const projects = (manifest.projects ?? []).map((p: any) => - validProject(p.id, p) - ); - const sources = (manifest.sources ?? []).map((s: any) => - validSource(s.id, s) - ); - const assetList = (manifest.assets ?? []).map((a: any) => { - validAssetId(a.id); - const entry = index.get(`assets/${a.id}`); - if (!entry || entry.size !== a.size) { - throw new Error(`The backup has no valid file for asset ${a.id}.`); - } - return { record: a, entry }; - }); - - const dir = openDataDir(root); - const staged: { id: string; file: StagedFile }[] = []; - try { - for (const { record, entry } of assetList) { - const file = await dir.assets.stage( - streamTarEntry(archive, entry), - Number.MAX_SAFE_INTEGER - ); - staged.push({ id: record.id, file }); - if (file.sha256 !== record.sha256) { - throw new Error( - `Hash mismatch for asset ${record.id}. The backup is damaged.` - ); - } - } - - const createdAt = new Map(); - for (const r of [ - ...(manifest.projects ?? []), - ...(manifest.sources ?? []), - ]) { - if (r.createdAt) createdAt.set(r.id, r.createdAt); - } - - transaction(dir.db, () => { - for (const { record } of assetList) { - dir.library.insertAsset({ - id: record.id, - sha256: record.sha256, - size: record.size, - mediaType: record.mediaType, - filename: record.filename, - originalPath: record.originalPath ?? null, - createdAt: record.createdAt, - }); - } - for (const p of projects) dir.library.putProject(p, createdAt.get(p.id)); - for (const s of sources) dir.library.putSource(s, createdAt.get(s.id)); - for (const { id, file } of staged) dir.assets.commit(file, id); - }); - return { - projects: projects.length, - sources: sources.length, - assets: assetList.length, - }; - } catch (e) { - for (const { id, file } of staged) { - dir.assets.discard(file); - if (!dir.library.getAsset(id)) dir.assets.remove(id); - } - throw e; - } finally { - dir.close(); - } -} diff --git a/src/kc_storage/src/cli.ts b/src/kc_storage/src/cli.ts index 1c552102..4db1aafa 100644 --- a/src/kc_storage/src/cli.ts +++ b/src/kc_storage/src/cli.ts @@ -6,7 +6,8 @@ import fs from "node:fs"; import path from "node:path"; import { parseArgs } from "node:util"; import type { AddressInfo } from "node:net"; -import { restoreBackup, writeBackup } from "./backup.ts"; +import { writeBackup } from "./backup.ts"; +import { activateRestore, assertEmpty, stageRestore } from "./restore.ts"; import { openDataDir } from "./datadir.ts"; import { createServer } from "./http.ts"; import { migrate } from "./migrate.ts"; @@ -106,10 +107,23 @@ export async function main(argv: string[]) { } case "restore": { if (!values.from) fail("--from is required."); - const result = await restoreBackup(values.from, dataDir); - console.log( - JSON.stringify({ restored: result, dataDir: path.resolve(dataDir) }) - ); + // Same validation and activation as the HTTP API + const dir = openDataDir(dataDir); + try { + assertEmpty(dir); + const staged = await stageRestore(dir, values.from); + const result = activateRestore(dir, staged); + console.log( + JSON.stringify({ + restored: result, + warnings: staged.preview.warnings, + notIncluded: staged.preview.notIncluded, + dataDir: dir.root, + }) + ); + } finally { + dir.close(); + } return; } default: diff --git a/src/kc_storage/src/datadir.ts b/src/kc_storage/src/datadir.ts index c4265ebd..7eab6b72 100644 --- a/src/kc_storage/src/datadir.ts +++ b/src/kc_storage/src/datadir.ts @@ -22,6 +22,9 @@ export interface DataDir { export const DATABASE_FILE = "library.sqlite"; +/** Staging directories of restores that are not active yet. */ +export const RESTORE_DIR = "restore"; + function alive(pid: number) { try { process.kill(pid, 0); @@ -84,6 +87,13 @@ export function openDataDir(root: string): DataDir { // Recover from an interrupted upload, migration, or restore const swept = assets.sweep(library.assetIds()); + const staging = path.join(resolved, RESTORE_DIR); + if (fs.existsSync(staging)) { + for (const entry of fs.readdirSync(staging)) { + fs.rmSync(path.join(staging, entry), { recursive: true, force: true }); + swept.push(`${RESTORE_DIR}/${entry}`); + } + } return { root: resolved, diff --git a/src/kc_storage/src/http.ts b/src/kc_storage/src/http.ts index f03c8db2..992b092b 100644 --- a/src/kc_storage/src/http.ts +++ b/src/kc_storage/src/http.ts @@ -11,8 +11,22 @@ import { ASSET_ORIGINAL_PATH_HEADER, type ErrorResponse, type HealthResponse, + type LibraryStatus, } from "../../kc_contracts/storage.ts"; import { writeBackup } from "./backup.ts"; +import { + activateRestore, + assertEmpty, + discardRestore, + newStagingDir, + RESTORE_LIMITS, + RestoreError, + stageRestore, + type StagedRestore, +} from "./restore.ts"; +import path from "node:path"; +import { pipeline } from "node:stream/promises"; +import { Transform } from "node:stream"; import type { DataDir } from "./datadir.ts"; import { now, SCHEMA_VERSION } from "./db.ts"; import { @@ -33,6 +47,9 @@ import { } from "./validate.ts"; import { VERSION } from "./version.ts"; +/** How long a validated restore waits for confirmation. */ +const RESTORE_PENDING_MS = 30 * 60 * 1000; + export interface ServerOptions { dir: DataDir; token: string; @@ -99,6 +116,41 @@ export function createServer(options: ServerOptions): http.Server { const maxJson = options.maxJsonBytes ?? 10 * 1024 * 1024; const maxUpload = options.maxUploadBytes ?? 2 * 1024 ** 3; + // A validated restore waits here for confirmation. One at a time. + let pending: { staged: StagedRestore; timer: NodeJS.Timeout } | undefined; + const dropPending = () => { + if (pending) { + clearTimeout(pending.timer); + discardRestore(pending.staged); + pending = undefined; + } + }; + + async function receiveArchive(req: IncomingMessage, file: string) { + const declared = Number(req.headers["content-length"] ?? 0); + if (declared > RESTORE_LIMITS.archiveBytes) { + throw tooLarge( + `The backup is larger than ${RESTORE_LIMITS.archiveBytes} bytes.` + ); + } + let size = 0; + const meter = new Transform({ + transform(chunk: Buffer, _enc, done) { + size += chunk.length; + if (size > RESTORE_LIMITS.archiveBytes) { + done( + tooLarge( + `The backup is larger than ${RESTORE_LIMITS.archiveBytes} bytes.` + ) + ); + return; + } + done(null, chunk); + }, + }); + await pipeline(req, meter, fs.createWriteStream(file, { flags: "wx" })); + } + async function route(req: IncomingMessage, res: ServerResponse) { const url = new URL(req.url ?? "/", "http://localhost"); let parts: string[]; @@ -231,6 +283,54 @@ export function createServer(options: ServerOptions): http.Server { } } + if (resource === "library" && !id && method === "GET") { + const counts = library.counts(); + const status: LibraryStatus = { + empty: counts.projects + counts.sources + counts.assets === 0, + counts, + }; + return send(res, 200, status); + } + + if (resource === "restores") { + // Upload and validate. Nothing becomes active. + if (method === "POST" && !id) { + assertEmpty(dir); + dropPending(); + const staging = newStagingDir(dir); + const archive = path.join(staging.stagingDir, "archive.tar"); + try { + await receiveArchive(req, archive); + } catch (e) { + fs.rmSync(staging.stagingDir, { recursive: true, force: true }); + throw e; + } + const staged = await stageRestore(dir, archive, staging); + fs.rmSync(archive, { force: true }); + const timer = setTimeout(dropPending, RESTORE_PENDING_MS); + timer.unref(); + pending = { staged, timer }; + return send(res, 201, { restore: staged.preview }); + } + + const current = pending && pending.staged.id === id ? pending : undefined; + if (id && !sub && method === "DELETE") { + if (current) dropPending(); + return send(res, 204); + } + if (id && sub === "activate" && method === "POST") { + if (!current) { + throw notFound( + "This restore is not pending. It expired or the service restarted. Select the backup again." + ); + } + clearTimeout(current.timer); + pending = undefined; + const restored = activateRestore(dir, current.staged); + return send(res, 200, { restored }); + } + } + if (resource === "backup" && !id && method === "GET") { const date = new Date().toISOString().slice(0, 10); res.writeHead(200, { @@ -277,12 +377,14 @@ export function createServer(options: ServerOptions): http.Server { const error = e instanceof StorageError ? e + : e instanceof RestoreError + ? badRequest(e.message) : new StorageError( 500, "internal", "Internal error. See the service log." ); - if (!(e instanceof StorageError)) { + if (!(e instanceof StorageError) && !(e instanceof RestoreError)) { console.error( `[kc-storage] ${req.method} ${req.url}: ${e?.stack ?? e}` ); diff --git a/src/kc_storage/src/library.ts b/src/kc_storage/src/library.ts index 2f7dbbe9..369dfc1b 100644 --- a/src/kc_storage/src/library.ts +++ b/src/kc_storage/src/library.ts @@ -280,12 +280,18 @@ export class Library { return new Set(rows.map((r) => r.id)); } - isEmpty(): boolean { - const row = this.db + counts(): { projects: number; sources: number; assets: number } { + return this.db .prepare( - "SELECT (SELECT COUNT(*) FROM projects) + (SELECT COUNT(*) FROM sources) + (SELECT COUNT(*) FROM assets) AS n" + `SELECT (SELECT COUNT(*) FROM projects) AS projects, + (SELECT COUNT(*) FROM sources) AS sources, + (SELECT COUNT(*) FROM assets) AS assets` ) - .get() as { n: number }; - return row.n === 0; + .get() as { projects: number; sources: number; assets: number }; + } + + isEmpty(): boolean { + const c = this.counts(); + return c.projects + c.sources + c.assets === 0; } } diff --git a/src/kc_storage/src/restore.ts b/src/kc_storage/src/restore.ts new file mode 100644 index 00000000..87fe842a --- /dev/null +++ b/src/kc_storage/src/restore.ts @@ -0,0 +1,378 @@ +/* + * Library restore in two steps. + * + * stageRestore() Validate the whole backup and copy its files into a + * staging directory. The active library does not change. + * activateRestore() One synchronous step: check that the library is + * empty, insert the records, move the staged files into + * place, commit. No other request runs during it. + * + * If the process stops during activation, SQLite discards the transaction + * and the next start removes files without records and old staging + * directories. The library is empty again. + * + * The command line and the HTTP API both use this module. + */ + +import crypto from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import type { + AssetRecord, + ProjectRecord, + RestorePreview, + SourceRecord, +} from "../../kc_contracts/storage.ts"; +import type { StagedFile } from "./assets.ts"; +import { RESTORE_DIR, type DataDir } from "./datadir.ts"; +import { transaction } from "./db.ts"; +import { conflict } from "./errors.ts"; +import { readTarEntry, readTarIndex, streamTarEntry } from "./tar.ts"; +import { validAssetId, validProject, validSource } from "./validate.ts"; +import { BACKUP_FORMAT } from "./backup.ts"; + +export const SUPPORTED_BACKUP_VERSIONS = [1]; + +export const RESTORE_LIMITS = { + archiveBytes: 8 * 1024 ** 3, + manifestBytes: 64 * 1024 ** 2, + entries: 1_000_000, + assetBytes: 2 * 1024 ** 3, +}; + +/** What a library backup never contains. Shown before confirmation. */ +export const NOT_IN_LIBRARY_BACKUP = [ + "Chat history", + "Inbox and UI preferences", + "Application settings", + "API keys", + "Extracted-text cache", +]; + +const ASSET_ENTRY = + /^assets\/[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; + +/** A validated backup with its files in a staging directory. */ +export interface StagedRestore { + id: string; + dir: string; + preview: RestorePreview; + projects: ProjectRecord[]; + sources: SourceRecord[]; + assets: { record: AssetRecord; file: StagedFile }[]; + createdAt: Map; +} + +export interface RestoreResult { + projects: number; + sources: number; + assets: number; +} + +export class RestoreError extends Error {} + +function fail(message: string): never { + throw new RestoreError(message); +} + +/** Refuse a library that has user records. Never merge or replace them. */ +export function assertEmpty(dir: DataDir) { + const counts = dir.library.counts(); + if (counts.projects + counts.sources + counts.assets > 0) { + const n = (count: number, noun: string) => + `${count} ${noun}${count === 1 ? "" : "s"}`; + throw conflict( + `Restore needs an empty library. This library has ${n( + counts.projects, + "project" + )}, ` + + `${n(counts.sources, "source")}, and ${n(counts.assets, "file")}. ` + + "Restore into a new profile instead. Existing data is not changed." + ); + } +} + +/** New staging directory inside the data directory (same filesystem). */ +export function newStagingDir(dir: DataDir) { + const id = crypto.randomUUID(); + const stagingDir = path.join(dir.root, RESTORE_DIR, id); + fs.mkdirSync(path.join(stagingDir, "files"), { recursive: true }); + return { id, stagingDir }; +} + +/** + * Validate a backup archive and stage its files. Throws RestoreError with + * a user-facing message. Removes the staging directory on failure. + */ +export async function stageRestore( + dir: DataDir, + archive: string, + staging = newStagingDir(dir) +): Promise { + const { id, stagingDir } = staging; + try { + const size = fs.statSync(archive).size; + if (size > RESTORE_LIMITS.archiveBytes) { + fail(`The backup is larger than ${RESTORE_LIMITS.archiveBytes} bytes.`); + } + + let index; + try { + index = readTarIndex(archive, { + allowName: (name) => name === "manifest.json" || ASSET_ENTRY.test(name), + maxEntries: RESTORE_LIMITS.entries, + }); + } catch (e: any) { + fail(`The file is not a valid library backup: ${e.message}`); + } + + const manifestEntry = index.get("manifest.json"); + if (!manifestEntry) fail("The backup has no manifest.json."); + if (manifestEntry.size > RESTORE_LIMITS.manifestBytes) { + fail("The backup manifest is too large."); + } + let manifest: any; + try { + manifest = JSON.parse( + readTarEntry(archive, manifestEntry).toString("utf8") + ); + } catch { + fail("The backup manifest is not valid JSON."); + } + if (manifest?.format !== BACKUP_FORMAT) { + fail("The file is not a Knowledge library backup."); + } + if (!SUPPORTED_BACKUP_VERSIONS.includes(manifest.version)) { + fail( + `Backup version ${ + manifest.version + } is not supported. Supported versions: ${SUPPORTED_BACKUP_VERSIONS.join( + ", " + )}.` + ); + } + for (const list of ["projects", "sources", "assets"]) { + if (!Array.isArray(manifest[list])) + fail(`The manifest has no ${list} list.`); + } + + // Records + const projects: ProjectRecord[] = []; + const sources: SourceRecord[] = []; + const createdAt = new Map(); + const unique = (seen: Set, id: string, kind: string) => { + if (seen.has(id)) fail(`Duplicate ${kind} ID ${id} in the backup.`); + seen.add(id); + }; + const projectIds = new Set(); + for (const p of manifest.projects) { + let record: ProjectRecord; + try { + record = validProject(p?.id, p); + } catch (e: any) { + fail(`Invalid project ${JSON.stringify(p?.id)}: ${e.message}`); + } + unique(projectIds, record.id, "project"); + projects.push(record); + if (typeof p.createdAt === "string") + createdAt.set(`p:${record.id}`, p.createdAt); + } + + // Assets: manifest and archive entries must match one to one + const assets: StagedRestore["assets"] = []; + const assetIds = new Set(); + let bytes = 0; + for (const a of manifest.assets) { + try { + validAssetId(a?.id); + } catch { + fail(`Invalid asset ID ${JSON.stringify(a?.id)}.`); + } + unique(assetIds, a.id, "asset"); + if ( + !Number.isSafeInteger(a.size) || + a.size < 0 || + a.size > RESTORE_LIMITS.assetBytes + ) { + fail(`Invalid size for asset ${a.id}.`); + } + if (typeof a.sha256 !== "string" || !/^[0-9a-f]{64}$/.test(a.sha256)) { + fail(`Invalid SHA-256 for asset ${a.id}.`); + } + if ( + typeof a.filename !== "string" || + !a.filename || + typeof a.mediaType !== "string" + ) { + fail(`Invalid filename or media type for asset ${a.id}.`); + } + const entry = index.get(`assets/${a.id}`); + if (!entry) + fail( + `The file for asset ${a.id} (${a.filename}) is missing from the backup.` + ); + if (entry.size !== a.size) + fail(`The file for asset ${a.id} has the wrong size.`); + bytes += a.size; + } + for (const name of index.keys()) { + if ( + name !== "manifest.json" && + !assetIds.has(name.slice("assets/".length)) + ) { + fail(`The backup has a file that the manifest does not list: ${name}.`); + } + } + + const sourceIds = new Set(); + const usedAssets = new Set(); + for (const s of manifest.sources) { + let record: SourceRecord; + try { + record = validSource(s?.id, s); + } catch (e: any) { + fail(`Invalid source ${JSON.stringify(s?.id)}: ${e.message}`); + } + unique(sourceIds, record.id, "source"); + if (!projectIds.has(record.projectId)) { + fail( + `Source ${record.id} refers to project ${record.projectId}, which is not in the backup.` + ); + } + if (record.assetId) { + if (!assetIds.has(record.assetId)) { + fail( + `Source ${record.id} refers to asset ${record.assetId}, which is not in the backup.` + ); + } + usedAssets.add(record.assetId); + } + sources.push(record); + if (typeof s.createdAt === "string") + createdAt.set(`s:${record.id}`, s.createdAt); + } + + const warnings: string[] = []; + const dangling = projects.filter( + (p) => p.parentId && !projectIds.has(p.parentId) + ); + if (dangling.length > 0) { + warnings.push( + `${dangling.length} projects refer to a parent project that is not in the backup.` + ); + } + const unused = [...assetIds].filter((id) => !usedAssets.has(id)); + if (unused.length > 0) { + warnings.push(`${unused.length} files are not used by any source.`); + } + + // Copy and hash every file before anything becomes active + const filesDir = path.join(stagingDir, "files"); + for (const a of manifest.assets) { + const file = await dir.assets.stage( + streamTarEntry(archive, index.get(`assets/${a.id}`)!), + RESTORE_LIMITS.assetBytes, + filesDir + ); + if (file.sha256 !== a.sha256 || file.size !== a.size) { + fail( + `The file for asset ${a.id} (${a.filename}) is damaged. Its SHA-256 does not match.` + ); + } + assets.push({ + record: { + id: a.id, + sha256: a.sha256, + size: a.size, + mediaType: a.mediaType, + filename: a.filename, + originalPath: + typeof a.originalPath === "string" ? a.originalPath : null, + createdAt: + typeof a.createdAt === "string" + ? a.createdAt + : new Date().toISOString(), + }, + file, + }); + } + + const preview: RestorePreview = { + id, + backupCreatedAt: + typeof manifest.createdAt === "string" ? manifest.createdAt : null, + backupVersion: manifest.version, + counts: { + projects: projects.length, + sources: sources.length, + assets: assets.length, + bytes, + }, + projectNames: projects.slice(0, 50).map((p) => p.name), + warnings, + notIncluded: NOT_IN_LIBRARY_BACKUP, + }; + return { + id, + dir: stagingDir, + preview, + projects, + sources, + assets, + createdAt, + }; + } catch (e) { + fs.rmSync(stagingDir, { recursive: true, force: true }); + throw e; + } +} + +/** + * Make a staged restore active. Synchronous: no other request runs during + * it. Refuses a library that has records. Removes the staging directory. + */ +export function activateRestore( + dir: DataDir, + staged: StagedRestore +): RestoreResult { + const moved: string[] = []; + try { + transaction(dir.db, () => { + assertEmpty(dir); + for (const { record } of staged.assets) dir.library.insertAsset(record); + for (const p of staged.projects) + dir.library.putProject(p, staged.createdAt.get(`p:${p.id}`)); + for (const s of staged.sources) + dir.library.putSource(s, staged.createdAt.get(`s:${s.id}`)); + + // Files move before the commit. A stop before the commit leaves files + // without records, which the next start removes. + for (const { record, file } of staged.assets) { + dir.assets.commit(file, record.id); + moved.push(record.id); + if ( + process.env.KC_STORAGE_TEST_FAULT === "restore-exit-during-activation" + ) { + process.exit(70); + } + } + if (process.env.KC_STORAGE_TEST_FAULT === "restore-fail-before-commit") { + throw new RestoreError("Injected test fault before commit."); + } + }); + return { + projects: staged.projects.length, + sources: staged.sources.length, + assets: staged.assets.length, + }; + } catch (e) { + for (const id of moved) dir.assets.remove(id); + throw e; + } finally { + discardRestore(staged); + } +} + +export function discardRestore(staged: { dir: string }) { + fs.rmSync(staged.dir, { recursive: true, force: true }); +} diff --git a/src/kc_storage/src/tar.ts b/src/kc_storage/src/tar.ts index cba222e8..8c7d35a6 100644 --- a/src/kc_storage/src/tar.ts +++ b/src/kc_storage/src/tar.ts @@ -8,7 +8,13 @@ import { Readable, type Writable } from "node:stream"; const BLOCK = 512; -function header(name: string, size: number, mtime: Date): Buffer { +/** A ustar header. Type "0" is a regular file. Tests use other types. */ +export function tarHeader( + name: string, + size: number, + mtime: Date, + type = "0" +): Buffer { if (Buffer.byteLength(name) > 100) { throw new Error(`Tar entry name is too long: ${name}`); } @@ -23,7 +29,7 @@ function header(name: string, size: number, mtime: Date): Buffer { h.write(octal(size, 12), 124, 12, "ascii"); h.write(octal(Math.floor(mtime.getTime() / 1000), 12), 136, 12, "ascii"); h.write(" ", 148, 8, "ascii"); - h.write("0", 156, 1, "ascii"); + h.write(type, 156, 1, "ascii"); h.write("ustar\0", 257, 6, "ascii"); h.write("00", 263, 2, "ascii"); @@ -51,14 +57,14 @@ export class TarWriter { } async addBuffer(name: string, data: Buffer) { - await this.write(header(name, data.length, new Date())); + await this.write(tarHeader(name, data.length, new Date())); await this.write(data); await this.write(padding(data.length)); } async addFile(name: string, file: string) { const { size, mtime } = fs.statSync(file); - await this.write(header(name, size, mtime)); + await this.write(tarHeader(name, size, mtime)); let written = 0; for await (const chunk of fs.createReadStream(file)) { written += chunk.length; @@ -81,8 +87,21 @@ export interface TarEntry { size: number; } -/** Read the entry index of a tar file. Validates every header checksum. */ -export function readTarIndex(file: string): Map { +export interface TarLimits { + /** Accept only these entry names. */ + allowName: (name: string) => boolean; + maxEntries: number; +} + +/** + * Read the entry index of a tar file. Rejects bad checksums, entry types + * other than regular files, the ustar prefix field, duplicate names, and + * names that allowName refuses. Entry names are never used as paths. + */ +export function readTarIndex( + file: string, + limits: TarLimits +): Map { const fd = fs.openSync(file, "r"); const entries = new Map(); try { @@ -106,12 +125,34 @@ export function readTarIndex(file: string): Map { const name = h.toString("utf8", 0, 100).replace(/\0.*$/s, ""); const size = parseInt(h.toString("ascii", 124, 136).trim(), 8); const type = h.toString("ascii", 156, 157); + const prefix = h.toString("utf8", 345, 500).replace(/\0.*$/s, ""); + + if (type !== "0" && type !== "\0") { + throw new Error( + `Unsupported tar entry type "${type}" for ${JSON.stringify(name)}.` + ); + } + if (prefix) { + throw new Error( + `Unsupported tar name prefix for ${JSON.stringify(name)}.` + ); + } + if (!limits.allowName(name)) { + throw new Error(`Unexpected tar entry ${JSON.stringify(name)}.`); + } + if (entries.has(name)) { + throw new Error(`Duplicate tar entry ${JSON.stringify(name)}.`); + } if (!Number.isFinite(size) || size < 0 || offset + BLOCK + size > total) { - throw new Error(`Invalid tar entry size for ${name}.`); + throw new Error(`Invalid tar entry size for ${JSON.stringify(name)}.`); } - if (type === "0" || type === "\0") { - entries.set(name, { name, offset: offset + BLOCK, size }); + if (entries.size >= limits.maxEntries) { + throw new Error( + `The archive has more than ${limits.maxEntries} entries.` + ); } + + entries.set(name, { name, offset: offset + BLOCK, size }); offset += BLOCK + Math.ceil(size / BLOCK) * BLOCK; } throw new Error("The tar file ends without an end-of-archive marker."); diff --git a/src/kc_storage/test/backup.test.ts b/src/kc_storage/test/backup.test.ts index 02f053ce..152ac279 100644 --- a/src/kc_storage/test/backup.test.ts +++ b/src/kc_storage/test/backup.test.ts @@ -124,7 +124,7 @@ test("restore refuses a directory that already has a library", async () => { const result = cli(["restore", "--data-dir", source, "--from", tar]); assert.equal(result.status, 1); - assert.match(result.stderr, /already has a library/); + assert.match(result.stderr, /Restore needs an empty library/); }); test("a damaged backup is rejected and leaves the target empty", async () => { @@ -145,7 +145,7 @@ test("a damaged backup is rejected and leaves the target empty", async () => { const target = tempDir("bk3-target"); const result = cli(["restore", "--data-dir", target, "--from", tar]); assert.equal(result.status, 1); - assert.match(result.stderr, /Hash mismatch/); + assert.match(result.stderr, /is damaged/); const svc2 = await startService(target); const api2 = client(svc2); diff --git a/src/kc_storage/test/restore.test.ts b/src/kc_storage/test/restore.test.ts new file mode 100644 index 00000000..a61de0a8 --- /dev/null +++ b/src/kc_storage/test/restore.test.ts @@ -0,0 +1,296 @@ +import { afterEach, test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; +import { readTarEntry, readTarIndex, tarHeader, TarWriter } from "../src/tar.ts"; +import { FIXTURES, cli, client, startService, stopAll, tempDir, type Service } from "./helpers.ts"; + +afterEach(stopAll); + +/** A library with two projects, three sources, and two files, and its backup. */ +async function makeBackup() { + const work = tempDir("rs-work"); + const source = tempDir("rs-source"); + const svc = await startService(source); + const api = client(svc); + await api.put("/v1/projects/p1", { name: "Restore Project", parentId: null, data: { description: "d" } }); + await api.put("/v1/projects/p2", { name: "Child", parentId: "p1", data: {} }); + const pdf = (await api.upload(path.join(FIXTURES, "recovery-fixture.pdf"), "paper.pdf", "application/pdf")).body.asset; + const txt = (await api.upload(path.join(FIXTURES, "recovery-note.txt"), "note.txt", "text/plain")).body.asset; + const file = (title: string, assetId: string) => ({ + projectId: "p1", + title, + ingestType: "file", + assetId, + data: { topics: ["t"], meta: [{ key: "annotation", value: title }] }, + }); + await api.put("/v1/sources/s-pdf", file("paper.pdf", pdf.id)); + await api.put("/v1/sources/s-txt", file("note.txt", txt.id)); + await api.put("/v1/sources/s-web", { + projectId: "p2", + title: "Site", + ingestType: "website", + assetId: null, + data: { accessLink: "https://example.com/" }, + }); + const expected = { + projects: (await api.get("/v1/projects")).body.projects.map(({ updatedAt, ...r }: any) => r), + sources: (await api.get("/v1/sources")).body.sources.map(({ updatedAt, ...r }: any) => r), + }; + await svc.stop(); + const tar = path.join(work, "library.tar"); + assert.equal(cli(["backup", "--data-dir", source, "--out", tar]).status, 0); + return { work, tar, expected, assets: { pdf, txt } }; +} + +/** Read a backup into a manifest and its asset files. */ +function unpack(tar: string) { + const index = readTarIndex(tar, { allowName: () => true, maxEntries: 100 }); + const manifest = JSON.parse(readTarEntry(tar, index.get("manifest.json")!).toString()); + const files = new Map(); + for (const [name, entry] of index) if (name !== "manifest.json") files.set(name, readTarEntry(tar, entry)); + return { manifest, files }; +} + +/** Write a tar from entries. type "0" is a regular file. */ +async function pack(file: string, entries: { name: string; data?: Buffer; type?: string }[]) { + const out = fs.createWriteStream(file); + const tar = new TarWriter(out); + for (const e of entries) { + if (!e.type || e.type === "0") { + await tar.addBuffer(e.name, e.data ?? Buffer.alloc(0)); + } else { + out.write(tarHeader(e.name, 0, new Date(), e.type)); + } + } + await tar.finish(); + await new Promise((r) => out.end(r)); + return file; +} + +async function modified(tar: string, out: string, change: (m: any, f: Map) => void, extra: any[] = []) { + const { manifest, files } = unpack(tar); + change(manifest, files); + return pack(out, [ + { name: "manifest.json", data: Buffer.from(JSON.stringify(manifest)) }, + ...[...files].map(([name, data]) => ({ name, data })), + ...extra, + ]); +} + +async function upload(svc: Service, tar: string) { + const res = await fetch(`${svc.url}/v1/restores`, { + method: "POST", + headers: { Authorization: `Bearer ${svc.token}`, "Content-Type": "application/x-tar" }, + body: fs.readFileSync(tar), + }); + return { status: res.status, body: (await res.json()) as any }; +} + +async function activate(svc: Service, id: string) { + const res = await fetch(`${svc.url}/v1/restores/${id}/activate`, { + method: "POST", + headers: { Authorization: `Bearer ${svc.token}` }, + }); + return { status: res.status, body: (await res.json()) as any }; +} + +function allFiles(dir: string): string[] { + if (!fs.existsSync(dir)) return []; + return fs.readdirSync(dir, { recursive: true, withFileTypes: true }) + .filter((d) => d.isFile()) + .map((d) => path.relative(dir, path.join(d.parentPath, d.name))); +} + +/** Managed files and staging files in a data directory. */ +function storedFiles(dataDir: string) { + return [ + ...allFiles(path.join(dataDir, "assets")).filter((f) => !f.startsWith("tmp")), + ...allFiles(path.join(dataDir, "restore")), + ]; +} + +test("restore through the API: validate, preview, activate, refuse a second time", async () => { + const { tar, expected, assets } = await makeBackup(); + const target = tempDir("rs-target"); + const svc = await startService(target); + const api = client(svc); + + // An initialized but empty library is accepted + assert.deepEqual((await api.get("/v1/library")).body, { + empty: true, + counts: { projects: 0, sources: 0, assets: 0 }, + }); + + const up = await upload(svc, tar); + assert.equal(up.status, 201, JSON.stringify(up.body)); + const preview = up.body.restore; + assert.deepEqual(preview.counts, { projects: 2, sources: 3, assets: 2, bytes: assets.pdf.size + assets.txt.size }); + assert.deepEqual(preview.projectNames.sort(), ["Child", "Restore Project"]); + assert.equal(preview.backupVersion, 1); + assert.ok(preview.notIncluded.includes("Chat history")); + assert.ok(preview.notIncluded.includes("API keys")); + + // Nothing is active before confirmation + assert.equal((await api.get("/v1/library")).body.empty, true); + + const done = await activate(svc, preview.id); + assert.equal(done.status, 200); + assert.deepEqual(done.body.restored, { projects: 2, sources: 3, assets: 2 }); + assert.deepEqual( + (await api.get("/v1/projects")).body.projects.map(({ updatedAt, ...r }: any) => r), + expected.projects + ); + assert.deepEqual( + (await api.get("/v1/sources")).body.sources.map(({ updatedAt, ...r }: any) => r), + expected.sources + ); + assert.ok((await api.content(assets.pdf.id)).bytes.equals(fs.readFileSync(path.join(FIXTURES, "recovery-fixture.pdf")))); + assert.deepEqual(allFiles(path.join(target, "restore")), []); + + // A library with records is refused before the upload is read + const again = await upload(svc, tar); + assert.equal(again.status, 409); + assert.match(again.body.error.message, /Restore needs an empty library\. This library has 2 projects, 3 sources, and 2 files/); + assert.equal((await api.get("/v1/sources")).body.sources.length, 3); + assert.equal((await activate(svc, preview.id)).status, 404); +}); + +test("a write after validation blocks activation; nothing is merged", async () => { + const { tar } = await makeBackup(); + const target = tempDir("rs-race"); + const svc = await startService(target); + const api = client(svc); + const up = await upload(svc, tar); + assert.equal(up.status, 201); + + await api.put("/v1/projects/mine", { name: "Created meanwhile", parentId: null, data: {} }); + const done = await activate(svc, up.body.restore.id); + assert.equal(done.status, 409); + assert.deepEqual((await api.get("/v1/projects")).body.projects.map((p: any) => p.id), ["mine"]); + assert.deepEqual(storedFiles(target), []); +}); + +test("cancel removes the staged files", async () => { + const { tar } = await makeBackup(); + const target = tempDir("rs-cancel"); + const svc = await startService(target); + const up = await upload(svc, tar); + assert.ok(allFiles(path.join(target, "restore")).length > 0); + const res = await fetch(`${svc.url}/v1/restores/${up.body.restore.id}`, { + method: "DELETE", + headers: { Authorization: `Bearer ${svc.token}` }, + }); + assert.equal(res.status, 204); + assert.deepEqual(allFiles(path.join(target, "restore")), []); + assert.equal((await activate(svc, up.body.restore.id)).status, 404); +}); + +test("invalid backups are rejected before activation and leave nothing behind", async () => { + const { tar, work, assets } = await makeBackup(); + const target = tempDir("rs-invalid"); + const svc = await startService(target); + const api = client(svc); + const out = (name: string) => path.join(work, name); + + const cases: [string, Promise, RegExp][] = [ + [ + "unsupported version", + modified(tar, out("v2.tar"), (m) => (m.version = 2)), + /Backup version 2 is not supported/, + ], + [ + "missing asset file", + modified(tar, out("missing.tar"), (_m, f) => f.delete(`assets/${assets.txt.id}`)), + /is missing from the backup/, + ], + [ + "damaged asset file", + modified(tar, out("damaged.tar"), (_m, f) => { + const data = Buffer.from(f.get(`assets/${assets.txt.id}`)!); + data[0] ^= 0xff; + f.set(`assets/${assets.txt.id}`, data); + }), + /is damaged/, + ], + [ + "source refers to a missing project", + modified(tar, out("orphan.tar"), (m) => (m.sources[0].projectId = "nope")), + /not in the backup/, + ], + [ + "path traversal entry", + modified(tar, out("traversal.tar"), () => {}, [{ name: "../escaped.txt", data: Buffer.from("x") }]), + /Unexpected tar entry/, + ], + [ + "absolute path entry", + modified(tar, out("absolute.tar"), () => {}, [{ name: "/tmp/kc-escaped.txt", data: Buffer.from("x") }]), + /Unexpected tar entry/, + ], + [ + "symbolic link entry", + modified(tar, out("symlink.tar"), () => {}, [{ name: "assets/link", type: "2" }]), + /Unsupported tar entry type "2"/, + ], + [ + "unlisted asset entry", + modified(tar, out("unlisted.tar"), () => {}, [ + { name: "assets/00000000-0000-4000-8000-000000000000", data: Buffer.from("x") }, + ]), + /does not list/, + ], + ["not a tar file", Promise.resolve(path.join(FIXTURES, "recovery-note.txt")), /not a valid library backup/], + ]; + + for (const [label, file, message] of cases) { + const res = await upload(svc, await file); + assert.equal(res.status, 400, `${label}: ${JSON.stringify(res.body)}`); + assert.match(res.body.error.message, message, label); + assert.deepEqual(storedFiles(target), [], `${label}: files left behind`); + assert.equal((await api.get("/v1/library")).body.empty, true, label); + } + assert.ok(!fs.existsSync(path.join(work, "escaped.txt"))); + assert.ok(!fs.existsSync("/tmp/kc-escaped.txt")); + + // The command line uses the same validation + const viaCli = cli(["restore", "--data-dir", tempDir("rs-cli"), "--from", out("v2.tar")]); + assert.equal(viaCli.status, 1); + assert.match(viaCli.stderr, /Backup version 2 is not supported/); +}); + +test("a failure during activation rolls back records and files", async () => { + const { tar } = await makeBackup(); + const target = tempDir("rs-fail"); + const result = cli(["restore", "--data-dir", target, "--from", tar], { + KC_STORAGE_TEST_FAULT: "restore-fail-before-commit", + }); + assert.equal(result.status, 1); + assert.match(result.stderr, /Injected test fault/); + assert.deepEqual(storedFiles(target), []); + + const svc = await startService(target); + assert.equal((await client(svc).get("/v1/library")).body.empty, true); +}); + +test("a process stop during activation leaves an empty library after restart", async () => { + const { tar, expected } = await makeBackup(); + const target = tempDir("rs-crash"); + const result = cli(["restore", "--data-dir", target, "--from", tar], { + KC_STORAGE_TEST_FAULT: "restore-exit-during-activation", + }); + assert.equal(result.status, 70); + // The process stopped after it moved a file and before it committed + assert.ok(storedFiles(target).length > 0); + + const svc = await startService(target); + const api = client(svc); + assert.equal((await api.get("/v1/library")).body.empty, true); + assert.deepEqual(storedFiles(target), []); + assert.match(svc.stderr(), /removed \d+ unreferenced or temporary files/); + + // A later restore succeeds + const up = await upload(svc, tar); + assert.equal((await activate(svc, up.body.restore.id)).status, 200); + assert.equal((await api.get("/v1/sources")).body.sources.length, expected.sources.length); +});