diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index e9cdfe86..1fae8098 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -8,12 +8,16 @@ You do not need AWS, an Apple developer account, or an AI API key. | Tool | Version | Notes | | --- | --- | --- | | macOS | 26.6 (arm64) | Verified. Linux and Windows are not verified. | -| Node.js | 24.21.0 | Verified. See the compatibility note below. | +| Node.js | 24.21.0 | Verified. The app starts the storage service with this Node.js. | | Yarn | 3.2.4 | The repository contains this version (`.yarnrc.yml` `yarnPath`). Any global `yarn` starts it. | | Electron | 26.3.0 | Installed by Yarn. | No native compiler is necessary. The install does not build `canvas`. +The storage service (`src/kc_storage`) needs Node.js 24 or later. It uses +the built-in `node:sqlite` module. The desktop app starts it with `node` +from `PATH`. To use a different binary, set `KC_NODE=/path/to/node`. + Compatibility note: Angular 14 officially supports Node.js 14.15+ and 16.10+ only. Node.js 24 builds and runs this branch, but it is outside the official range. Electron 26 is end-of-life. Upgrade these in a separate step. @@ -28,16 +32,45 @@ yarn start `yarn start-dev` does the build and the start in one command. +## Architecture + +| Part | Location | Runs in | +| --- | --- | --- | +| UI | `src/kc_angular` | Electron renderer | +| Desktop functions, chat server | `src/kc_electron` | Electron main process | +| Storage service | `src/kc_storage` | Separate Node.js process | +| API contracts | `src/kc_contracts` | Shared. No application imports. | + +The design of the storage service is in +[docs/storage-service.md](docs/storage-service.md). + +### Data ownership + +| Data | Owner | Location in a profile | +| --- | --- | --- | +| Projects, sources, topics, metadata | Storage service | `data/library/library.sqlite` | +| Copies of imported files | Storage service | `data/library/assets/` | +| Chat history, inbox, current project, UI preferences, favicon cache | Renderer | `userData` (local storage) | +| Settings | Electron | `settings/knowledge.settings.json` | +| Extracted-text cache, API key file | Electron chat server | `data/storage/sources/`, `data/openai.encrypted` | + +The app does not read or write the old local-storage project keys +(`kc-projects`, ``, `ks-`). Use the migration to import them. + ## Instances and local servers -Each instance starts its own chat server on a random loopback port. Each -server requires a random token. The renderer receives the address and the -token through IPC (`A2E:Backend:Info`). One instance cannot use the server of -another instance. +Each instance starts two local servers: the chat server (Electron) and the +storage service (Node.js). Each listens on a random loopback port and +requires its own random token. The renderer receives the addresses and +tokens through IPC (`A2E:Backend:Info`). One instance cannot use the +servers of another instance. Only one instance can use a profile at a time. A second start with the same profile prints a message and exits. Different profiles can run at the same time. +If the storage service does not start, the app shows an error and quits. +The storage service stops when the app quits or crashes. + ## Data profiles A development run never uses your normal Knowledge data. @@ -53,8 +86,8 @@ A profile contains these directories: | Directory | Contents | | --- | --- | -| `userData` | Chromium storage: local storage (projects, sources, chat history), cache, cookies | -| `data` | Source text cache (`storage/sources`), OpenAI key file, temporary files | +| `userData` | Chromium storage: local storage, cache, cookies | +| `data` | `library/` (storage service), text cache, API key file, temporary files | | `settings` | `knowledge.settings.json` | | `downloads` | Default save location. The autoscan folder is `downloads/Knowledge`. | @@ -76,38 +109,132 @@ out of the autoscan folder. Make a backup first. | --- | --- | | Chromium storage | `~/Library/Application Support/knowledge-canvas` | | Settings | `~/Library/Preferences/Knowledge/knowledge.settings.json` | -| Source text cache, API key | `~/.Knowledge` | +| Library, text cache, API key | `~/.Knowledge` | | Autoscan folder | `~/Downloads/Knowledge` | Do not confuse `~/Library/Application Support/Knowledge`. macOS owns that directory (`knowledgeC.db`). Knowledge does not use it. -## Tests +## Storage service without Electron + +```sh +export KC_STORAGE_TOKEN=$(node -e 'console.log(require("crypto").randomBytes(32).toString("hex"))') +yarn storage serve --data-dir /tmp/kc-library +``` -Unit tests (Node.js 22.18 or later): +The service prints one line with its address, for example +`{"event":"ready","url":"http://127.0.0.1:53124",...}`. Then: ```sh -yarn test +curl -s http://127.0.0.1:53124/health +curl -s -H "Authorization: Bearer $KC_STORAGE_TOKEN" http://127.0.0.1:53124/v1/projects ``` -These check backup and restore, and profile path isolation. +Stop it with Ctrl+C. Other commands: + +```sh +yarn storage migrate --data-dir --from [--dry-run] +yarn storage backup --data-dir --out +yarn storage restore --data-dir --from +``` + +Stop the app (or the service) before you use `migrate`, `backup`, or +`restore` on its data directory. The commands refuse a directory that is in use. + +## Migration from local storage + +Use a copy of development data. Do not migrate your normal profile. + +1. Get a renderer backup file (`knowledge-backup` version 1). Make it with + Settings > Backup > Chat and Preferences Backup > Export, or use one + made by the recovery baseline (commit `278a02a`). The file must contain + the old project keys. +2. Stop the app. +3. Check the plan without writing: + + ```sh + yarn storage migrate --data-dir /data/library --from --dry-run + ``` + +4. Run the migration: + + ```sh + yarn storage migrate --data-dir /data/library --from + ``` + +5. Read `/data/library/migrations//report.json`. It lists + imported, updated, unchanged, skipped, and failed items, missing files, + `ks-` records that differ from the project copy, and the local-storage + keys that stay in the renderer. + +The migration: -End-to-end tests (macOS): +- Keeps project and source IDs, and all fields (unknown fields stay in `data`). +- Copies each file source's original file into managed storage. A missing + file is reported. The source is imported without a managed copy. +- Can run again. Unchanged records are skipped. Files are not copied twice. + A source keeps its managed copy if the original file is gone later. +- Does not change the input file or the renderer data. +- Writes all records in one transaction. If it fails, nothing is written and + copied files are removed. + +### Rollback + +Each run saves a database copy at `migrations//pre-migration.sqlite`. + +1. Stop the app. +2. Delete `library.sqlite`, `library.sqlite-wal`, and `library.sqlite-shm` + in the data directory. +3. Copy `migrations//pre-migration.sqlite` to `library.sqlite`. +4. Start the app. The service removes copied files that no record uses. + +To go back to the recovery baseline instead, check out `recovery/local-dev`. +The renderer still has the old keys, because the migration and the new app +do not change them. Changes made after the migration are not in them. + +## Backup and restore + +Settings > Backup has two backups. Neither is a complete application backup. + +| Backup | Contents | Restore | +| --- | --- | --- | +| Library (`knowledge-library-.tar`) | Projects, sources, topics, metadata, managed files with SHA-256 hashes | `yarn storage restore` into an empty data directory | +| Chat and preferences (`knowledge-backup-.json`) | Renderer local storage: chat history, inbox, UI preferences | Settings > Backup > Restore | + +Not in either backup: settings, the API key, the extracted-text cache. +File sources without a managed copy (missing at migration) still depend on +the original path. + +To restore a library into a new profile: ```sh -yarn build-dev -yarn e2e +yarn storage restore --data-dir /data/library --from knowledge-library-.tar +KC_PROFILE_DIR= yarn start ``` +Restore checks every file hash and refuses a directory that already has a +library. The restored library does not need the original files. + +## Tests + +| Command | Checks | Needs | +| --- | --- | --- | +| `yarn test` | Renderer backup, profile path isolation | Node.js 22.18+ | +| `yarn test-storage` | Storage service without Electron: persistence, files, validation and access limits, migration (repeat, missing files, failure, rollback), backup and restore | Node.js 24+ | +| `yarn workspace kc_storage typecheck` | Storage service types (TypeScript 5) | | +| `yarn e2e` | Desktop UI with real profiles (below) | macOS, `yarn build-dev` first | + The end-to-end tests start the real app with new, empty profiles in -`e2e/.output//`. They never use your normal profile. They do these checks: +`e2e/.output//`. They never use your normal profile. - `e2e/workflow.e2e.mjs`: create a project, save a link (local fixture site), - import a PDF and a text file, annotate, search, restart, export a backup, - and restore it into a second profile. -- `e2e/isolation.e2e.mjs`: two profiles at the same time get different server - addresses and tokens. Each instance rejects the other instance's requests. - A second start of the same profile exits. + import a PDF and a text file, annotate, search, restart, delete the + original files, read the same records and files, show the PDF, make a + thumbnail and extract text from the managed copy, export the library, + restore it into a second profile, and show the PDF there. +- `e2e/isolation.e2e.mjs`: two profiles at the same time get different chat + and storage addresses and tokens. Each server rejects the other + instance's requests. A second start of the same profile exits. Test files are in `e2e/fixtures/`. To delete old test output, delete `e2e/.output/`. @@ -124,16 +251,22 @@ curl -s -X POST http://127.0.0.1:7777 -d '{"op":"close"}' ## Current limits - Annotations are topics and metadata entries (tag and value). The app has - no notes editor. + no notes editor. The `notes` field is kept but not shown. - Search uses titles, topics, descriptions, and source type. It does not search file content. -- A backup does not include imported files. -- File sources keep the absolute path of the original file. After a restore, - file sources work only if the original files are still at those paths. +- The library backup includes managed files. Restore needs the storage + service command line. The UI cannot restore a library. +- A file source that had no original file at migration has no managed copy. + It still depends on its original path. +- Packaged builds cannot start the storage service. The package does not + contain `src/kc_storage` or a Node.js 24 runtime. Not tested on this + branch. A packaged app is expected to show the start error and quit. - Signed and notarized releases are not verified. - AI chat with a real API key is not verified. - Only macOS is verified. +Follow-up work is in [docs/follow-ups.md](docs/follow-ups.md). + ## Optional services | Service | Default | How to enable | @@ -149,29 +282,6 @@ AI topics do not work. The app shows an API key dialog. The key file (`openai.encrypted`) uses a fixed passphrase. Treat it as plain text. -## Backup and restore - -Use Settings > Backup. - -A backup contains all renderer local storage: - -- Projects and their sources -- Source topics and metadata annotations -- Chat history -- The inbox -- Theme and table preferences - -A backup does not contain: - -- Settings (`knowledge.settings.json`) -- Imported files. File sources keep the absolute path of the original file. -- The source text cache. The app extracts the text again when necessary. -- The API key - -Restore merges the project list and replaces items that have the same ID. -The app reloads after a restore. Restore accepts only files that Settings > -Backup made (`format: knowledge-backup`, version 1). - ## Packaging ```sh @@ -183,6 +293,6 @@ npm archive and does not start electron-builder. The result is an unsigned app in `dist/mac-arm64`. Notarization starts only when `APPLEID`, `APPLEPWD`, and `TEAMID` are set. Signing needs a valid -Developer ID certificate. +Developer ID certificate. See the packaging limit above. Do not use `yarn publish`. Its target S3 bucket does not exist. diff --git a/docs/follow-ups.md b/docs/follow-ups.md new file mode 100644 index 00000000..4582559f --- /dev/null +++ b/docs/follow-ups.md @@ -0,0 +1,34 @@ +# Follow-up Work + +Work found during the recovery and storage milestones. Not started. + +## Storage service + +| Item | Reason | +| --- | --- | +| Package the service with a Node.js 24 runtime | Packaged builds cannot start the service now. | +| Restore a library from the UI | Restore needs the command line now. | +| Orphan asset cleanup policy | Deleting a source keeps its managed file. Only unreferenced files after a rollback or crash are removed. | +| Move the extracted-text cache into the service | The chat server keeps it in `data/storage/sources/`, outside the library. | +| Thumbnails in the service | Electron makes thumbnails from a local copy of the managed file. | +| Move favicon cache, inbox, and chat history out of local storage | They stay in the renderer for this milestone. | +| Remove `kc_shared` dependency on Angular models | `kc_shared/models/project.model.ts` imports an Angular model. Electron needs the `@shared` alias because of it. | + +## Desktop app + +| Item | Reason | +| --- | --- | +| Upgrade Electron (26 is end-of-life) | Replace `File.path` with `webUtils.getPathForFile` (removed in Electron 32). | +| Upgrade Angular 14 and TypeScript 4.8 | Node.js 24 is outside Angular 14's supported range. | +| Bind the browser extension server to loopback with a token | It listens on port 9000 on all interfaces when enabled. | +| Replace the fixed API key passphrase | The key file is obfuscated, not encrypted. | +| Root `postinstall` key is outside `scripts` | It never runs. Remove it or move it. | +| Fresh-profile route error (`inbox/undefined`) | Cosmetic error in the log on first start. | +| PDF viewer title shows the blob ID | Managed PDFs open from a `blob:` URL. | + +## Features (separate tasks) + +- Full-text search over extracted text. +- Notes editor for the existing `notes` field. +- Verification of AI chat with a real API key. +- Signed and notarized release builds. diff --git a/docs/storage-service.md b/docs/storage-service.md new file mode 100644 index 00000000..505e1009 --- /dev/null +++ b/docs/storage-service.md @@ -0,0 +1,178 @@ +# Storage Service Design (Milestone 1) + +## Purpose + +Move projects, sources, and imported files out of renderer `localStorage` +into a standalone service. The service runs with Node.js and does not need +Electron. The Angular UI stays. This is the first backend extraction step. + +## Current storage (evidence) + +Inspected from a backup of a real development profile (`e2e/.output`). + +| Item | Current behavior | +| --- | --- | +| Project record | `localStorage[]`. Fields: `id`, `name`, `parentId`, `subprojects`, `description`, `topics`, `calendar`, `events`, `authors`, `type`, `icon`, `expanded`, dates, `sources` (always empty), `knowledgeSource`. | +| Project list | `localStorage['kc-projects']`: array of project IDs. | +| Sources | Full copies inside `project.knowledgeSource`. The UI reads only these. | +| Separate source records | `localStorage['ks-']`. Written only by `DataService.sources.update`. Exist for some sources only. Can differ from the embedded copy (`meta`, `icon`). | +| Association | `source.associatedProject.value` plus membership in `project.knowledgeSource`. | +| Annotations | `source.topics` (strings) and `source.meta` (`{key, value}`). `source.notes` exists but no UI writes it. | +| File sources | `accessLink` and `reference.source.file.path` hold the absolute original path. No copy exists. | +| Derived fields | `source.icon` (Angular `SafeUrl` with an expired `blob:` URL), `source.thumbnail` (PNG data URL, about 57 KB), `localStorage['icon-']`. | +| Text cache | Electron chat server writes `/data/storage/sources/.json`. | +| Other renderer keys | `chat-`, `ingest-queue`, `current-project`, `theme`, `ks-table-rows`, card and graph preferences, topics. | + +All project and source writes go through `StorageService` (`saveProject`, +`updateProject`, `deleteProject`, `deleteKnowledgeSource`) and +`DataService.sources` (`ks-*` keys). + +## Service boundary + +- Package `src/kc_storage`. Node.js 24 or later. TypeScript run directly by + Node (type stripping). No runtime dependencies. SQLite from `node:sqlite`. +- Contracts in `src/kc_contracts/`. `storage.ts` has types and header names. + `mapping.ts` converts between UI objects and records. Neither file has + imports. +- The service owns the database and the managed files. No other process + opens them. +- Commands: `serve`, `migrate`, `backup`, `restore`. + +## Ownership after this milestone + +| Data | Owner | +| --- | --- | +| Projects, sources, topics, metadata, notes field | Storage service | +| Imported files | Storage service (managed assets) | +| Chat history, inbox, current project, UI preferences, favicon cache | Renderer `localStorage` | +| Settings | Electron (`knowledge.settings.json`) | +| Extracted-text cache, API key | Electron chat server (`/data/storage`, `/data`) | + +The app does not read or write the old project keys (`kc-projects`, +``, `ks-*`). It does not delete them. + +## Database (schema version 1) + +| Table | Columns | +| --- | --- | +| `schema_migrations` | `version`, `applied_at` | +| `projects` | `id` PK, `parent_id`, `name`, `data` (JSON), `created_at`, `updated_at` | +| `sources` | `id` PK, `project_id` FK → `projects` (cascade delete), `title`, `ingest_type`, `asset_id` FK → `assets`, `position`, `data` (JSON), `created_at`, `updated_at` | +| `assets` | `id` PK, `sha256`, `size`, `media_type`, `filename`, `original_path`, `created_at` | +| `import_runs` | `id` PK, `input_sha256`, `started_at`, `finished_at`, `status`, `report` (JSON) | + +`data` keeps every field that has no column, so no field is lost. The +service stores `data` as it receives it. Clients and the migration remove +the derived source fields `icon` and `thumbnail` before they write. The UI +derives them again. + +## API (version 1) + +Base: `http://127.0.0.1:`. All `/v1` routes need +`Authorization: Bearer `. + +| Method and path | Result | +| --- | --- | +| `GET /health` | Status, service version, schema version. No token. | +| `GET /v1/projects` | All projects | +| `PUT /v1/projects/:id` | Create or replace a project | +| `DELETE /v1/projects/:id` | Delete a project and its sources | +| `GET /v1/sources[?projectId=]` | Sources, in project order | +| `PUT /v1/sources/:id` | Create or replace a source. The project and asset must exist. | +| `DELETE /v1/sources/:id` | Delete a source | +| `POST /v1/assets` | Upload file bytes. Headers give filename and original path. | +| `GET /v1/assets/:id` | Asset metadata | +| `GET /v1/assets/:id/content` | File bytes | +| `GET /v1/backup` | Portable backup (tar) | + +Errors use `{ "error": { "code", "message" } }` and a matching HTTP status. + +## Local access + +- Listens on `127.0.0.1` only. Port `0` by default (ephemeral). +- Token from the `KC_STORAGE_TOKEN` environment variable. Required. +- The `Host` header must be `127.0.0.1:` or `localhost:`. +- CORS allows only the configured origin (`null` for the desktop renderer). + CORS and loopback are not authentication. The token is. +- No route takes a filesystem path. Asset IDs must match a strict pattern. +- One process per data directory (lock file). + +## Managed files + +- Upload streams to `assets/tmp/`, computes SHA-256, then renames to + `assets//`. +- The original filename, original path, size, media type, and hash are + stored as metadata. The original file is not changed. +- `source.asset_id` is the file identity. `accessLink` keeps the original + path as information only. +- The desktop app uploads through Electron main (`A2E:Storage:ImportFile`), + because the renderer cannot read local paths. A web client would upload + the same way with a `File`. + +## Desktop connection + +Electron main starts `node src/kc_storage/src/main.ts serve` with +`--data-dir /data/library`, port `0`, and a new token. The service prints +one JSON line with its URL. The renderer receives the URL and token through +`A2E:Backend:Info`. If the service does not start, the app shows an error +and quits. The service stops when its standard input closes. + +## Migration + +`node src/kc_storage/src/main.ts migrate --data-dir --from ` + +- Input: a Settings > Backup file (`knowledge-backup` version 1). Use a copy + of development data only. +- Validate the whole input before any write. +- Projects: every key with a project record. Sources: the embedded copies. + A `ks-*` record without an embedded copy is imported if its project + exists. Differences between `ks-*` and embedded copies are reported. +- File sources: copy the file at the original path into managed storage. + Report missing files. The source is still imported, without an asset. +- Keep existing IDs. Repeated runs update or skip. No duplicates. +- Before writing, snapshot the database (`VACUUM INTO`). Write all records + in one transaction. Remove copied files if the transaction fails. +- Write the report and a copy of the input to `migrations//`. + The report lists imported, updated, unchanged, skipped, missing, and + failed items, and the renderer keys that stay in `localStorage`. + +Rollback: stop the app, then restore `migrations//pre-migration.sqlite` +or delete the data directory. The renderer data is not changed by migration. + +## Backup and restore + +- `GET /v1/backup` or `backup --data-dir --out `. +- Tar file: `manifest.json` (format `knowledge-library-backup`, version 1, + projects, sources, assets with SHA-256) and `assets/`. +- `restore --data-dir --from ` checks every hash. +- Not included: chat history, UI preferences, settings, API key, text cache. + +## UI workflow + +Create project, save link, import PDF and text file, show sources, add topic +and metadata, search, restart, read the same records and file. The +Document tab, thumbnails, and "open file" use the asset. + +## Acceptance tests + +- Service (`yarn workspace kc_storage test`, no Electron): persistence across + restart, asset upload and retrieval, validation and access limits, + repeated migration, missing files, interrupted migration, backup and + restore into a new directory with the original files deleted. +- UI (`yarn e2e`): the workflow above, restart, file shown after the + original file is deleted. + +## Verified (2026-10-06, macOS 26.6, Node.js 24.21.0) + +- `yarn test-storage`: 16 tests, service started as a separate Node.js process. +- `yarn e2e`: the UI workflow above, restart, deleted originals, thumbnail + and text extraction from the managed copy, library export, restore into a + second profile, and instance isolation. +- Migration of a copy of a development profile backup: 1 project, 3 sources, + 2 files copied, 3 differing `ks-` records reported, 7 renderer keys left. + A second run: all unchanged, files reused. + +## Not in this milestone + +Full-text search, notes editor, thumbnail service, orphan asset cleanup +policy, packaged-app Node runtime, web client, authentication accounts. diff --git a/e2e/isolation.e2e.mjs b/e2e/isolation.e2e.mjs index 22de334a..19b21f5f 100644 --- a/e2e/isolation.e2e.mjs +++ b/e2e/isolation.e2e.mjs @@ -71,6 +71,32 @@ test( 401 ); + // Storage services: one per profile, each with its own token + assert.match(infoA.storage.url, /^http:\/\/127\.0\.0\.1:\d+$/); + assert.notEqual(infoA.storage.url, infoB.storage.url); + assert.notEqual(infoA.storage.token, infoB.storage.token); + const projects = (page, url, token) => + page.evaluate( + async ({ url, token }) => { + const headers = token ? { Authorization: `Bearer ${token}` } : {}; + return (await fetch(`${url}/v1/projects`, { headers })).status; + }, + { url, token } + ); + assert.equal( + await projects(a.page, infoA.storage.url, infoA.storage.token), + 200 + ); + assert.equal( + await projects(b.page, infoB.storage.url, infoB.storage.token), + 200 + ); + assert.equal(await projects(b.page, infoA.storage.url), 401); + assert.equal( + await projects(b.page, infoA.storage.url, infoB.storage.token), + 401 + ); + // A second launch of profile A exits and leaves the first instance running const second = spawnSync(ELECTRON, [REPO], { cwd: REPO, diff --git a/e2e/lib.mjs b/e2e/lib.mjs index 35aae002..e1568f63 100644 --- a/e2e/lib.mjs +++ b/e2e/lib.mjs @@ -121,24 +121,51 @@ export async function startFixtureSite() { }; } -/** Read projects and their sources as the renderer sees them. */ +/** Backend addresses and tokens of the instance that owns this page. */ +export function backendInfo(page) { + return page.evaluate(() => window.api.invoke("A2E:Backend:Info")); +} + +/** Read projects and sources from this instance's storage service. */ export async function readLibrary(page) { - return page.evaluate(() => { - const ids = JSON.parse(localStorage.getItem("kc-projects") ?? "[]"); - return ids.map((id) => { - const p = JSON.parse(localStorage.getItem(id)); - return { - id, - name: p.name, - sources: p.knowledgeSource.map((k) => ({ - title: k.title, - type: k.ingestType, - topics: k.topics ?? [], - annotations: (k.meta ?? []) + return page.evaluate(async () => { + const { storage } = await window.api.invoke("A2E:Backend:Info"); + const get = (route) => + fetch(`${storage.url}/v1/${route}`, { + headers: { Authorization: `Bearer ${storage.token}` }, + }).then((r) => r.json()); + const { projects } = await get("projects"); + const { sources } = await get("sources"); + return projects.map((p) => ({ + id: p.id, + name: p.name, + sources: sources + .filter((s) => s.projectId === p.id) + .map((s) => ({ + id: s.id, + title: s.title, + type: s.ingestType, + assetId: s.assetId, + topics: s.data.topics ?? [], + annotations: (s.data.meta ?? []) .filter((m) => m.key === "annotation") .map((m) => m.value), })), - }; - }); + })); }); } + +/** Bytes of a managed file, read through this instance's storage service. */ +export async function readAsset(page, assetId) { + const base64 = await page.evaluate(async (id) => { + const { storage } = await window.api.invoke("A2E:Backend:Info"); + const res = await fetch(`${storage.url}/v1/assets/${id}/content`, { + headers: { Authorization: `Bearer ${storage.token}` }, + }); + const bytes = new Uint8Array(await res.arrayBuffer()); + let text = ""; + for (const b of bytes) text += String.fromCharCode(b); + return btoa(text); + }, assetId); + return Buffer.from(base64, "base64"); +} diff --git a/e2e/steps.mjs b/e2e/steps.mjs index 1d872248..89adf758 100644 --- a/e2e/steps.mjs +++ b/e2e/steps.mjs @@ -59,6 +59,16 @@ export async function importFilesToProject(page, files) { await page.locator("button", { hasText: "Import" }).first().click(); } +/** Select a project in the project tree. */ +export async function selectProject(page, name) { + await page + .locator("p-tree, .p-tree") + .getByText(name, { exact: true }) + .first() + .click(); + await page.locator("i.pi-table").first().waitFor({ timeout: TIMEOUT }); +} + export async function openTable(page) { await page.locator("i.pi-table").first().click(); await page @@ -80,9 +90,24 @@ export async function openSource(page, title) { .waitFor({ timeout: TIMEOUT }); } +/** + * Close the open source dialog. An embedded PDF viewer keeps keyboard + * focus, so take focus away from it before pressing Escape. Use the close + * icon if Escape does not work. + */ export async function closeDialog(page) { + const dialog = page.locator("p-dynamicdialog"); + await page.evaluate(() => { + document.activeElement?.blur?.(); + document.body.focus(); + }); await page.keyboard.press("Escape"); - await page.waitForTimeout(500); + try { + await dialog.first().waitFor({ state: "detached", timeout: 3000 }); + } catch { + await dialog.locator(".pi-times").first().click(); + await dialog.first().waitFor({ state: "detached", timeout: 5000 }); + } } export async function addAnnotation(page, key, value) { diff --git a/e2e/workflow.e2e.mjs b/e2e/workflow.e2e.mjs index 40e08786..749cb9e1 100644 --- a/e2e/workflow.e2e.mjs +++ b/e2e/workflow.e2e.mjs @@ -1,20 +1,25 @@ /* - * Basic local workflow on fresh, isolated profiles: + * The library workflow through the existing UI, with the storage service: * create a project, save a link, import a PDF and a text file, annotate, - * search, restart, export a backup, and restore it into a second profile. + * search, restart, read the same records and file after the original files + * are deleted, export the library, and restore it into a second profile. * * Run: yarn build-dev && yarn e2e */ import { test } from "node:test"; import assert from "node:assert/strict"; +import fs from "node:fs"; import path from "node:path"; +import { spawnSync } from "node:child_process"; import { FIXTURES, + REPO, captureDownloads, close, launch, newRun, + readAsset, readLibrary, shot, startFixtureSite, @@ -22,19 +27,19 @@ import { } from "./lib.mjs"; import * as ui from "./steps.mjs"; -const PDF = path.join(FIXTURES, "recovery-fixture.pdf"); -const TXT = path.join(FIXTURES, "recovery-note.txt"); -const PROJECT = "Recovery Project"; +const PROJECT = "Library Project"; const SITE_TITLE = "Knowledge Fixture Site"; +const sortByTitle = (list) => + [...list].sort((a, b) => a.title.localeCompare(b.title)); -const expected = [ - { title: SITE_TITLE, type: "website" }, - { title: "recovery-fixture.pdf", type: "file" }, - { title: "recovery-note.txt", type: "file" }, -]; +async function documentSource(page) { + const embed = page.locator("source-document embed"); + await embed.waitFor({ timeout: 15000 }); + return embed.getAttribute("src"); +} test( - "basic workflow persists across restart and restores from backup", + "library workflow uses the storage service and survives restart, deletion, and restore", { timeout: 300000 }, async () => { const run = newRun("workflow"); @@ -43,17 +48,28 @@ test( const log = path.join(run, "app.log"); const site = await startFixtureSite(); + // Originals in the run directory, so the test can delete them later + const originals = path.join(run, "originals"); + fs.mkdirSync(originals); + const pdf = path.join(originals, "recovery-fixture.pdf"); + const txt = path.join(originals, "recovery-note.txt"); + fs.copyFileSync(path.join(FIXTURES, "recovery-fixture.pdf"), pdf); + fs.copyFileSync(path.join(FIXTURES, "recovery-note.txt"), txt); + const pdfBytes = fs.readFileSync(pdf); + const txtBytes = fs.readFileSync(txt); + + let ctx; try { - // 1-6: create, save, import, view, annotate, search - let ctx = await launch(profileA, log); + // 1-6: create, save, import, show, annotate, search + ctx = await launch(profileA, log); await ui.createProject(ctx.page, PROJECT, "Created by the e2e workflow."); await ui.addLinkToProject(ctx.page, site.url, SITE_TITLE); - await ui.importFilesToProject(ctx.page, [PDF, TXT]); + await ui.importFilesToProject(ctx.page, [pdf, txt]); await ui.openTable(ctx.page); assert.deepEqual( (await ui.tableTitles(ctx.page)).sort(), - expected.map((e) => e.title).sort() + [SITE_TITLE, "recovery-fixture.pdf", "recovery-note.txt"].sort() ); await ui.openSource(ctx.page, "recovery-note.txt"); @@ -62,68 +78,169 @@ test( await ctx.page.waitForTimeout(1500); await ui.closeDialog(ctx.page); - await ui.openSource(ctx.page, "recovery-fixture.pdf"); - await ui.openDocumentTab(ctx.page); - await ctx.page.waitForTimeout(2000); - await shot(ctx.page, run, "a-pdf-document"); - await ui.closeDialog(ctx.page); - const results = await ui.search(ctx.page, "Fixture Site"); assert.ok( results.some((r) => r.includes(SITE_TITLE)), `search results: ${results}` ); - await close(ctx); - // 7-9: restart and confirm the data remains - ctx = await launch(profileA, log); + // The records are in the storage service, with managed file copies + await ctx.page.waitForTimeout(1000); const library = await readLibrary(ctx.page); assert.equal(library.length, 1); assert.equal(library[0].name, PROJECT); - assert.deepEqual( - library[0].sources - .map(({ title, type }) => ({ title, type })) - .sort((a, b) => a.title.localeCompare(b.title)), - [...expected].sort((a, b) => a.title.localeCompare(b.title)) + const byTitle = Object.fromEntries( + library[0].sources.map((s) => [s.title, s]) ); - const note = library[0].sources.find( - (s) => s.title === "recovery-note.txt" + assert.equal(byTitle[SITE_TITLE].type, "website"); + assert.equal(byTitle[SITE_TITLE].assetId, null); + assert.ok(byTitle["recovery-fixture.pdf"].assetId); + assert.ok(byTitle["recovery-note.txt"].assetId); + assert.deepEqual(byTitle["recovery-note.txt"].topics, ["recoverytopic"]); + assert.deepEqual(byTitle["recovery-note.txt"].annotations, [ + "mentions chlorophyll", + ]); + + // The renderer no longer stores projects or sources + const legacyKeys = await ctx.page.evaluate(() => + Object.keys(localStorage).filter( + (k) => + k === "kc-projects" || + /^ks-[0-9a-f-]{36}$/.test(k) || + /^[0-9a-f-]{36}$/.test(k) + ) ); - assert.deepEqual(note.topics, ["recoverytopic"]); - assert.deepEqual(note.annotations, ["mentions chlorophyll"]); + assert.deepEqual(legacyKeys, []); + await close(ctx); + ctx = undefined; + + // The service stopped with the app and released its data directory + const storageDir = path.join(profileA, "data", "library"); + const lock = path.join(storageDir, "storage.lock"); + assert.ok(fs.existsSync(path.join(storageDir, "library.sqlite"))); + for (let i = 0; i < 20 && fs.existsSync(lock); i++) { + await new Promise((r) => setTimeout(r, 250)); + } + assert.ok(!fs.existsSync(lock), "storage service still running"); + + // 7-8: delete the originals, restart, and read the same records and files + fs.rmSync(originals, { recursive: true }); + ctx = await launch(profileA, log); + assert.deepEqual( + sortByTitle((await readLibrary(ctx.page))[0].sources), + sortByTitle(library[0].sources) + ); + const pdfAsset = byTitle["recovery-fixture.pdf"].assetId; + const txtAsset = byTitle["recovery-note.txt"].assetId; + assert.ok((await readAsset(ctx.page, pdfAsset)).equals(pdfBytes)); + assert.ok((await readAsset(ctx.page, txtAsset)).equals(txtBytes)); + + await ui.openTable(ctx.page); + await ui.openSource(ctx.page, "recovery-fixture.pdf"); + await ui.openDocumentTab(ctx.page); + assert.match(await documentSource(ctx.page), /^blob:/); + await ctx.page.waitForTimeout(2000); + await shot(ctx.page, run, "a-pdf-after-original-deleted"); + await ui.closeDialog(ctx.page); - // 10: export a backup + // Export the library through the UI const downloads = path.join(run, "downloads"); await captureDownloads(ctx.app, downloads); await ui.openSettings(ctx.page, "Backup"); - await ctx.page.locator("button", { hasText: "Export" }).click(); + await ctx.page.locator("button", { hasText: "Export Library" }).click(); const date = new Date().toISOString().slice(0, 10); - const backup = await waitForFile( - path.join(downloads, `knowledge-backup-${date}.json`) + const tar = await waitForFile( + path.join(downloads, `knowledge-library-${date}.tar`) ); await close(ctx); + ctx = undefined; - // 11: restore into an empty second profile, then restart it - ctx = await launch(profileB, log); - assert.deepEqual(await readLibrary(ctx.page), []); - await ui.openSettings(ctx.page, "Backup"); - await ctx.page - .locator("app-storage-settings input[type=file]") - .setInputFiles(backup); - await ctx.page.waitForFunction( - () => - JSON.parse(localStorage.getItem("kc-projects") ?? "[]").length === 1, - null, - { timeout: 15000 } + // Restore into the storage directory of a new profile, then open it + const restored = spawnSync( + process.execPath, + [ + path.join(REPO, "src/kc_storage/src/main.ts"), + "restore", + "--data-dir", + path.join(profileB, "data", "library"), + "--from", + tar, + ], + { encoding: "utf8" } ); - await close(ctx); + assert.equal(restored.status, 0, restored.stderr); ctx = await launch(profileB, log); - assert.deepEqual(await readLibrary(ctx.page), library); + await ui.selectProject(ctx.page, PROJECT); + assert.deepEqual( + sortByTitle((await readLibrary(ctx.page))[0].sources), + sortByTitle(library[0].sources) + ); await ui.openTable(ctx.page); - await shot(ctx.page, run, "b-restored-table"); - await close(ctx); + assert.equal((await ui.tableTitles(ctx.page)).length, 3); + await ui.openSource(ctx.page, "recovery-fixture.pdf"); + await ui.openDocumentTab(ctx.page); + assert.match(await documentSource(ctx.page), /^blob:/); + await ctx.page.waitForTimeout(2000); + await shot(ctx.page, run, "b-restored-pdf"); + + // Desktop functions read the managed file, not the deleted original + const thumbnail = await ctx.page.evaluate( + (assetId) => + new Promise((resolve) => { + window.api.receive("E2A:FileSystem:FileThumbnail", (responses) => { + const mine = responses.find( + (r) => r.success?.data?.id === "probe" + ); + if (mine) resolve(mine.success.data.thumbnail); + }); + window.api.send("A2E:FileSystem:FileThumbnail", [ + { + path: "/nonexistent/recovery-fixture.pdf", + assetId, + id: "probe", + }, + ]); + }), + pdfAsset + ); + assert.match(thumbnail, /^data:image\/png;base64,.{100,}/); + + await ctx.page.evaluate(async (assetId) => { + const { chat } = await window.api.invoke("A2E:Backend:Info"); + await fetch(`${chat.url}/sources/summarize`, { + method: "POST", + headers: { + Authorization: `Bearer ${chat.token}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + source: { + id: { value: "probe-source" }, + title: "probe", + ingestType: "file", + accessLink: "/nonexistent/recovery-fixture.pdf", + assetId, + }, + messages: [], + }), + }); + }, pdfAsset); + const extracted = JSON.parse( + fs.readFileSync( + path.join( + profileB, + "data", + "storage", + "sources", + "probe-source.json" + ), + "utf8" + ) + ); + assert.match(extracted[0], /quasar-meridian-77/); } finally { + if (ctx) await close(ctx); site.close(); } } diff --git a/package.json b/package.json index d801993e..7f4c0f46 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,8 @@ "type": "commonjs", "workspaces": [ "src/kc_electron", - "src/kc_angular" + "src/kc_angular", + "src/kc_storage" ], "repository": { "type": "git", @@ -38,6 +39,8 @@ "start-debug": "electron . --inspect=5858", "test": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --test \"tests/*.test.mts\"", "e2e": "node --test --test-concurrency=1 \"e2e/*.e2e.mjs\"", + "storage": "node src/kc_storage/src/main.ts", + "test-storage": "yarn workspace kc_storage test", "watch-main": "cd src/kc_angular && yarn run watch-main", "watch-main-dev": "cd src/kc_angular && yarn run watch-main-dev", "watch-electron": "cd src/kc_electron && yarn run watch-electron", diff --git a/src/kc_angular/src/app/app.module.ts b/src/kc_angular/src/app/app.module.ts index 9f0a5a33..ca154045 100755 --- a/src/kc_angular/src/app/app.module.ts +++ b/src/kc_angular/src/app/app.module.ts @@ -18,8 +18,9 @@ import { HTTP_INTERCEPTORS } from '@angular/common/http'; import { BackendAuthInterceptor, BackendService, - loadBackend, + initializeBackend, } from '@services/ipc-services/backend.service'; +import { StorageService } from '@services/ipc-services/storage.service'; import { A11yModule } from '@angular/cdk/a11y'; import { AccordionModule } from 'primeng/accordion'; import { AppComponent } from '@app/app.component'; @@ -331,8 +332,8 @@ import { CarouselModule } from 'primeng/carousel'; providers: [ { provide: APP_INITIALIZER, - useFactory: loadBackend, - deps: [BackendService], + useFactory: initializeBackend, + deps: [BackendService, StorageService], multi: true, }, { diff --git a/src/kc_angular/src/app/components/settings/storage-settings.component.ts b/src/kc_angular/src/app/components/settings/storage-settings.component.ts index f58417d0..01866f04 100755 --- a/src/kc_angular/src/app/components/settings/storage-settings.component.ts +++ b/src/kc_angular/src/app/components/settings/storage-settings.component.ts @@ -29,14 +29,43 @@ import { SettingsService } from '@services/ipc-services/settings.service';
-
Backup
+
Library Backup
- A backup holds projects, sources, annotations, chat history - and the inbox. It does not hold settings or imported files. + Projects, sources, topics, metadata, and copies of imported + files. Restore it with the storage service command line (see + DEVELOPMENT.md). It does not hold chat history, preferences, + or settings. +
+ +
+ +
+
+
+
+
+ +
+ + +
+
Chat and Preferences Backup
+
+
+ +
+
+ Chat history, the inbox, and UI preferences from this window's + local storage. It does not hold projects, sources, or files.
@@ -79,6 +108,8 @@ export class StorageSettingsComponent { exporting = false; + exportingLibrary = false; + form: FormGroup; constructor( @@ -106,6 +137,26 @@ export class StorageSettingsComponent { } } + async onExportLibrary() { + this.exportingLibrary = true; + try { + const size = await this.storage.exportLibrary(); + this.notifications.success( + 'Backup', + 'Library Exported', + `${(size / 1024).toFixed(0)} KB` + ); + } catch (e: any) { + this.notifications.error( + 'Backup', + 'Library Export Failed', + e?.error?.error?.message ?? e?.message ?? `${e}` + ); + } finally { + this.exportingLibrary = false; + } + } + onImport($event: any) { const input: HTMLInputElement = $event.target; const file = input.files?.[0]; @@ -120,7 +171,7 @@ export class StorageSettingsComponent { this.notifications.success( 'Backup', 'Backup Restored', - `${result.projectsAdded} projects added, ${result.projectsReplaced} replaced. Reloading...` + `${result.keysWritten} entries restored. Reloading...` ); // Reload so every service reads the restored storage diff --git a/src/kc_angular/src/app/components/source-components/ks-preview.component.ts b/src/kc_angular/src/app/components/source-components/ks-preview.component.ts index 5aad7a96..2c29f091 100755 --- a/src/kc_angular/src/app/components/source-components/ks-preview.component.ts +++ b/src/kc_angular/src/app/components/source-components/ks-preview.component.ts @@ -191,7 +191,11 @@ export class KsPreviewComponent implements OnInit, OnDestroy { } if (clickEvent.openClicked) { - this.ipc.openLocalFile(this.fileViewConfig.filePath); + if (this.ks.assetId) { + this.ipc.openAsset(this.ks.assetId); + } else { + this.ipc.openLocalFile(this.fileViewConfig.filePath); + } } } diff --git a/src/kc_angular/src/app/components/source-components/ks-thumbnail.component.ts b/src/kc_angular/src/app/components/source-components/ks-thumbnail.component.ts index 49d05ca8..91d1bd6c 100755 --- a/src/kc_angular/src/app/components/source-components/ks-thumbnail.component.ts +++ b/src/kc_angular/src/app/components/source-components/ks-thumbnail.component.ts @@ -143,6 +143,7 @@ export class KsThumbnailComponent implements OnDestroy, OnChanges { this.ipcService.getFileThumbnail([ { path: link, + assetId: this.ks.assetId, id: this.ks.id.value, }, ]); diff --git a/src/kc_angular/src/app/components/source-components/source.document.component.ts b/src/kc_angular/src/app/components/source-components/source.document.component.ts index 572e5a54..b36afbff 100755 --- a/src/kc_angular/src/app/components/source-components/source.document.component.ts +++ b/src/kc_angular/src/app/components/source-components/source.document.component.ts @@ -14,7 +14,16 @@ * limitations under the License. */ -import { Component, ElementRef, OnInit, ViewChild } from '@angular/core'; +import { + Component, + ElementRef, + OnDestroy, + OnInit, + ViewChild, +} from '@angular/core'; +import { HttpClient } from '@angular/common/http'; +import { firstValueFrom } from 'rxjs'; +import { StorageService } from '@services/ipc-services/storage.service'; import { KnowledgeSource } from '@app/models/knowledge.source.model'; import { DomSanitizer, SafeUrl } from '@angular/platform-browser'; @@ -50,7 +59,7 @@ import { DomSanitizer, SafeUrl } from '@angular/platform-browser'; `, ], }) -export class SourceDocumentComponent implements OnInit { +export class SourceDocumentComponent implements OnInit, OnDestroy { source!: KnowledgeSource; safeUrl: SafeUrl | undefined; @@ -72,17 +81,43 @@ export class SourceDocumentComponent implements OnInit { @ViewChild('documentContainer', { static: true }) documentContainer!: ElementRef; - constructor(private sanitizer: DomSanitizer) {} + private objectUrl?: string; + + constructor( + private sanitizer: DomSanitizer, + private http: HttpClient, + private storage: StorageService + ) {} + + ngOnDestroy() { + if (this.objectUrl) { + URL.revokeObjectURL(this.objectUrl); + } + } + + /** Managed files come from the storage service. Others use the file path. */ + private async documentUrl(): Promise { + if (this.source.assetId) { + const blob = await firstValueFrom( + this.http.get(this.storage.assetContentUrl(this.source.assetId), { + responseType: 'blob', + }) + ); + this.objectUrl = URL.createObjectURL(blob); + return this.objectUrl; + } + return 'file://' + encodeURI(this.source.accessLink as string); + } ngOnInit(): void { - setTimeout(() => { - if (typeof this.source.accessLink === 'string') { + setTimeout(async () => { + if (typeof this.source.accessLink === 'string' || this.source.assetId) { this.safeUrl = this.sanitizer.bypassSecurityTrustResourceUrl( - 'file://' + encodeURI(this.source.accessLink) + await this.documentUrl() ); for (const type of this.styles.unset) { - if (this.source.accessLink?.endsWith(type)) { + if (String(this.source.accessLink ?? '').endsWith(type)) { this.style = { 'max-height': '100%', 'max-width': '100%', diff --git a/src/kc_angular/src/app/models/knowledge.source.model.ts b/src/kc_angular/src/app/models/knowledge.source.model.ts index 3c3f2fde..ba97a720 100755 --- a/src/kc_angular/src/app/models/knowledge.source.model.ts +++ b/src/kc_angular/src/app/models/knowledge.source.model.ts @@ -93,6 +93,9 @@ export class KnowledgeSource { importMethod?: ImportMethod = 'manual'; thumbnail?: string; + /** Managed copy of a file source in the storage service. */ + assetId?: string; + constructor( title: string, id: UUID, diff --git a/src/kc_angular/src/app/services/command-services/ks-command.service.ts b/src/kc_angular/src/app/services/command-services/ks-command.service.ts index e87e7cf1..487710bb 100755 --- a/src/kc_angular/src/app/services/command-services/ks-command.service.ts +++ b/src/kc_angular/src/app/services/command-services/ks-command.service.ts @@ -122,7 +122,24 @@ export class KsCommandService { } open(ks: Partial) { - if (ks.ingestType === 'file' && typeof ks.accessLink === 'string') { + if (ks.ingestType === 'file' && ks.assetId) { + this.ipc + .openAsset(ks.assetId) + .then(() => + this.notifications.success( + 'Source Command', + 'File Opened', + ks.title ?? '' + ) + ) + .catch((e) => + this.notifications.error( + 'Source Command', + 'Unable to Open File', + `${e}` + ) + ); + } else if (ks.ingestType === 'file' && typeof ks.accessLink === 'string') { this.ipc.openLocalFile(ks.accessLink).then((result) => { if (result) { this.notifications.success( diff --git a/src/kc_angular/src/app/services/ipc-services/backend.service.ts b/src/kc_angular/src/app/services/ipc-services/backend.service.ts index 6f7f3d64..02c3ace9 100644 --- a/src/kc_angular/src/app/services/ipc-services/backend.service.ts +++ b/src/kc_angular/src/app/services/ipc-services/backend.service.ts @@ -31,6 +31,7 @@ export interface BackendEndpoint { export interface BackendInfo { chat: BackendEndpoint; + storage: BackendEndpoint; } /** @@ -43,20 +44,28 @@ export interface BackendInfo { export class BackendService { private info: BackendInfo = { chat: { error: 'Backend information not loaded.' }, + storage: { error: 'Backend information not loaded.' }, }; get chat(): BackendEndpoint { return this.info.chat; } + get storage(): BackendEndpoint { + return this.info.storage; + } + async load() { try { this.info = await window.api.invoke('A2E:Backend:Info'); } catch (e) { - this.info = { chat: { error: `Backend information unavailable: ${e}` } }; + const error = `Backend information unavailable: ${e}`; + this.info = { chat: { error }, storage: { error } }; } - if (this.info.chat.error) { - console.error('[Backend]:', this.info.chat.error); + for (const [name, endpoint] of Object.entries(this.info)) { + if (endpoint.error) { + console.error(`[Backend]: ${name}: ${endpoint.error}`); + } } } @@ -93,6 +102,22 @@ export class BackendAuthInterceptor implements HttpInterceptor { } } -export function loadBackend(backend: BackendService) { - return () => backend.load(); +/** + * Runs before the app starts: get this instance's server addresses, then + * load projects and sources from the storage service. + */ +export function initializeBackend( + backend: BackendService, + storage: { load(): Promise } +) { + return async () => { + await backend.load(); + try { + await storage.load(); + } catch (e: any) { + const reason = e?.error?.error?.message ?? e?.message ?? String(e); + console.error('[Storage]: load failed', e); + alert(`Knowledge could not load your library.\n\n${reason}`); + } + }; } diff --git a/src/kc_angular/src/app/services/ipc-services/backup.ts b/src/kc_angular/src/app/services/ipc-services/backup.ts index b36c7f5e..b4c999c9 100644 --- a/src/kc_angular/src/app/services/ipc-services/backup.ts +++ b/src/kc_angular/src/app/services/ipc-services/backup.ts @@ -15,18 +15,21 @@ */ /* - * Backup and restore for renderer storage. + * Backup and restore for renderer storage (localStorage). * - * A backup is a full snapshot of localStorage. It holds projects, their - * sources and annotations, chat history, the inbox, topics and UI state. - * It does not hold settings, imported files or the extracted-text cache. + * The renderer keeps chat history, the inbox, the current project, the + * favicon cache, and UI preferences. Projects, sources, and files are in + * the storage service and its library backup, not here. + * + * A renderer backup is a full snapshot. If old project keys exist, they + * are included, so the file can also be migration input + * (see docs/storage-service.md). * * This module has no Angular or Electron imports so that Node can test it. */ export const BACKUP_FORMAT = 'knowledge-backup'; export const BACKUP_VERSION = 1; -export const PROJECT_LIST_KEY = 'kc-projects'; /** The subset of the Web Storage API that backup and restore use. */ export interface KeyValueStore { @@ -46,8 +49,6 @@ export interface Backup { export interface RestoreResult { keysWritten: number; - projectsAdded: number; - projectsReplaced: number; } export function createBackup( @@ -77,45 +78,19 @@ export function createBackup( } /** - * Restore a backup into the store. - * - * Keys from the backup replace keys with the same name. The project list is - * merged so that projects already in the store remain listed. - * Throws an Error with a user-facing message if the input is not a backup. + * Restore a backup into the store. Keys from the backup replace keys with + * the same name. Throws an Error with a user-facing message if the input + * is not a backup. */ export function restoreBackup( store: KeyValueStore, input: unknown ): RestoreResult { const data = toKeyValues(input); - - const existingIds = readProjectIds(store.getItem(PROJECT_LIST_KEY)); - const importedIds = readProjectIds(data[PROJECT_LIST_KEY] ?? null); - - let keysWritten = 0; for (const [key, value] of Object.entries(data)) { - if (key === PROJECT_LIST_KEY) { - continue; - } store.setItem(key, value); - keysWritten++; - } - - const merged = [...existingIds]; - let projectsAdded = 0; - let projectsReplaced = 0; - for (const id of importedIds) { - if (merged.includes(id)) { - projectsReplaced++; - } else { - merged.push(id); - projectsAdded++; - } } - store.setItem(PROJECT_LIST_KEY, JSON.stringify(merged)); - keysWritten++; - - return { keysWritten, projectsAdded, projectsReplaced }; + return { keysWritten: Object.keys(data).length }; } /** @@ -147,17 +122,3 @@ function toKeyValues(input: unknown): Record { throw new Error('The file is not a Knowledge backup.'); } - -function readProjectIds(raw: string | null): string[] { - if (!raw) { - return []; - } - try { - const parsed = JSON.parse(raw); - return Array.isArray(parsed) - ? parsed.filter((id) => typeof id === 'string') - : []; - } catch { - return []; - } -} diff --git a/src/kc_angular/src/app/services/ipc-services/electron-ipc.service.ts b/src/kc_angular/src/app/services/ipc-services/electron-ipc.service.ts index 44c068c7..0ba35ee5 100755 --- a/src/kc_angular/src/app/services/ipc-services/electron-ipc.service.ts +++ b/src/kc_angular/src/app/services/ipc-services/electron-ipc.service.ts @@ -13,6 +13,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ +import type { AssetRecord } from '@contracts/storage'; import { Injectable, NgZone } from '@angular/core'; import { BehaviorSubject } from 'rxjs'; import { @@ -347,6 +348,19 @@ export class ElectronIpcService { }); } + /** + * Copy a local file into the storage service. Desktop only: the renderer + * cannot read local paths. The original file is not changed. + */ + importFile(path: string, mediaType?: string): Promise { + return window.api.invoke('A2E:Storage:ImportFile', { path, mediaType }); + } + + /** Open a managed file in its default application. Desktop only. */ + openAsset(assetId: string): Promise { + return window.api.invoke('A2E:Storage:OpenAsset', assetId); + } + generateUuid(quantity: number): Promise { return new Promise((resolve, reject) => { if (quantity <= 0) { diff --git a/src/kc_angular/src/app/services/ipc-services/storage.service.ts b/src/kc_angular/src/app/services/ipc-services/storage.service.ts index 7026e5f3..0fddb462 100755 --- a/src/kc_angular/src/app/services/ipc-services/storage.service.ts +++ b/src/kc_angular/src/app/services/ipc-services/storage.service.ts @@ -1,5 +1,5 @@ /* - * Copyright (c) 2023 Rob Royce + * Copyright (c) 2023-2026 Rob Royce * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -13,11 +13,37 @@ * See the License for the specific language governing permissions and * limitations under the License. */ + +/* + * Projects and sources. The storage service owns them. This service keeps + * an in-memory copy for the UI and writes every change to the service in + * order. It does not use localStorage for projects or sources. + * + * localStorage keeps only UI state: the current project, chat history, + * favicon cache, the inbox, and preferences. + */ + import { Injectable } from '@angular/core'; +import { HttpClient, HttpErrorResponse } from '@angular/common/http'; +import { firstValueFrom } from 'rxjs'; import { KcProject } from '@app/models/project.model'; import { KnowledgeSource } from '@app/models/knowledge.source.model'; import { AutoscanService } from '@services/ingest-services/autoscan.service'; import { NotificationsService } from '@services/user-services/notifications.service'; +import { BackendService } from '@services/ipc-services/backend.service'; +import { ElectronIpcService } from '@services/ipc-services/electron-ipc.service'; +import { + projectToRecord, + recordToProject, + recordToSource, + sourceToRecord, +} from '@contracts/mapping'; +import type { + ProjectList, + ProjectRecord, + SourceList, + SourceRecord, +} from '@contracts/storage'; import { createBackup, restoreBackup, RestoreResult } from './backup'; @Injectable({ @@ -25,123 +51,75 @@ import { createBackup, restoreBackup, RestoreResult } from './backup'; }) export class StorageService { readonly KC_CURRENT_PROJECT = 'current-project'; - private KC_ALL_PROJECT_IDS = 'kc-projects'; private db = window.localStorage; - private projectList: KcProject[] | null = null; + private projectList: KcProject[] = []; + + /** JSON of the last record written to or read from the service, by ID. */ + private synced = new Map(); + + /** Writes run one at a time, in call order. */ + private queue: Promise = Promise.resolve(); constructor( + private http: HttpClient, + private backend: BackendService, + private ipc: ElectronIpcService, private autoscan: AutoscanService, private notifications: NotificationsService ) {} - get projects(): KcProject[] { - const projects: KcProject[] = []; - - // Get and parse Project list from local storage - const projectsStr: string | null = this.db.getItem(this.KC_ALL_PROJECT_IDS); - if (!projectsStr) { - this.notifications.warn( - 'Storage Service', - 'Project List Unavailable', - 'Project list does not exist in storage system, creating...' - ); - this.db.setItem(this.KC_ALL_PROJECT_IDS, JSON.stringify([])); - return projects; - } - - const projectIds: string[] = JSON.parse(projectsStr); - if (!projectIds) { - this.notifications.warn( - 'Storage Service', - 'Invalid Project List', - 'Could not deserialize Project list JSON template... ' + private get api() { + const url = this.backend.storage.url; + if (!url) { + throw new Error( + this.backend.storage.error ?? 'Storage service unavailable.' ); - return projects; } + return `${url}/v1`; + } - if (projectIds.length === 0) { - this.notifications.warn('Storage Service', 'No Projects Available', ''); - return projects; + /** Load all projects and sources. Runs once before the app starts. */ + async load() { + const [{ projects }, { sources }] = await Promise.all([ + firstValueFrom(this.http.get(`${this.api}/projects`)), + firstValueFrom(this.http.get(`${this.api}/sources`)), + ]); + + const byProject = new Map(); + for (const record of sources) { + this.synced.set(`s:${record.id}`, JSON.stringify(this.writable(record))); + const ks = this.revive(recordToSource(record) as KnowledgeSource); + const list = byProject.get(record.projectId) ?? []; + list.push(ks); + byProject.set(record.projectId, list); } - let pStr: string | null = null; - let project: KcProject | null = null; - - // Deserialize projects from list of Project IDs - for (const pId of projectIds) { - pStr = this.db.getItem(pId); - if (!pStr) { - this.notifications.warn('Storage Service', 'Invalid Project Id', pId); - pStr = null; - continue; - } - - project = JSON.parse(pStr); - if (!project) { - this.notifications.warn( - 'Storage Service', - 'Invalid Project Object', - 'Could not deserialize Project JSON template...' - ); - project = null; - continue; - } - - // TODO: a fix for refactoring project source list and calendar - - // Pre-populate list of Knowledge Sources for later consumption - if (project.knowledgeSource && project.knowledgeSource.length > 0) { - for (const ks of project.knowledgeSource) { - // TODO : this is a temporary fix after refactoring ks timelines, should be removed eventually... - if (!ks.dateAccessed) { - ks.dateAccessed = []; - } - if (!ks.dateModified) { - ks.dateModified = []; - } - if (!ks.dateCreated) { - ks.dateCreated = new Date(); - } - - ks.dateCreated = new Date(ks.dateCreated); - - const accessed = []; - for (const d of ks.dateAccessed) { - accessed.push(new Date(d)); - } - ks.dateAccessed = accessed; - - const modified = []; - for (const d of ks.dateModified) { - modified.push(new Date(d)); - } - ks.dateModified = modified; - ks.dateDue = ks.dateDue ? new Date(ks.dateDue) : ks.dateDue; - ks.associatedProject = project.id; - ks.icon = undefined; - } - } - projects.push(project); - } + this.projectList = projects.map((record) => { + this.synced.set(`p:${record.id}`, JSON.stringify(this.writable(record))); + return recordToProject( + record, + byProject.get(record.id) ?? [] + ) as KcProject; + }); + } - if (projects.length > 0) { - this.projectList = projects; - } else { - this.projectList = null; - } + /** Dates arrive as strings. The UI expects Date objects. */ + private revive(ks: KnowledgeSource): KnowledgeSource { + ks.dateCreated = new Date(ks.dateCreated ?? Date.now()); + ks.dateAccessed = (ks.dateAccessed ?? []).map((d) => new Date(d)); + ks.dateModified = (ks.dateModified ?? []).map((d) => new Date(d)); + ks.dateDue = ks.dateDue ? new Date(ks.dateDue) : ks.dateDue; + return ks; + } - return projects; + /** The fields a client sends. The service sets position and times. */ + private writable(r: Partial) { + const { position, createdAt, updatedAt, ...rest } = r; + return JSON.parse(JSON.stringify(rest)); } - set projects(projectModels: KcProject[]) { - let pStr: string; - const pids = []; - for (const project of projectModels) { - pStr = JSON.stringify(project); - this.db.setItem(project.id.value, pStr); - pids.push(project.id.value); - } - this.db.setItem(this.KC_ALL_PROJECT_IDS, JSON.stringify(pids)); + get projects(): KcProject[] { + return this.projectList; } get kcCurrentProject(): string | null { @@ -149,163 +127,174 @@ export class StorageService { } set kcCurrentProject(id: string | null) { - if (id === null) { - this.db.setItem(this.KC_CURRENT_PROJECT, ''); - return; - } - this.db.setItem(this.KC_CURRENT_PROJECT, id); + this.db.setItem(this.KC_CURRENT_PROJECT, id ?? ''); } - /** - * Returns a list of all Knowledge Sources stored in the application - */ async ksList() { - const projectIdsStr: string | null = this.db.getItem( - this.KC_ALL_PROJECT_IDS - ); + return this.projectList.flatMap((p) => p.knowledgeSource ?? []); + } - if (projectIdsStr === null) { - this.notifications.warn( - 'Storage Service', - 'Project List Unavailable', - 'Project list does not exist in storage system when retrieving source list...' - ); - return []; - } + async getProjects() { + return this.projectList; + } - const projectIds = JSON.parse(projectIdsStr); - if (!projectIds) { - this.notifications.warn( - 'Storage Service', - 'Invalid Project List', - 'Could not deserialize Project list JSON template when retrieving source list...' - ); - return []; + async saveProject(project: KcProject) { + if (!this.projectList.find((p) => p.id.value === project.id.value)) { + this.projectList.push(project); } + return this.enqueue(() => this.write(project)); + } - const ksList: KnowledgeSource[] = []; - for (const projectId of projectIds) { - const projectStr = this.db.getItem(projectId); - if (!projectStr) break; - const project: KcProject = JSON.parse(projectStr); - if (project.knowledgeSource) - for (const ks of project.knowledgeSource) ksList.push(ks); + async saveProjectList(projects: KcProject[]) { + for (const project of projects) { + this.saveProject(project); } + } - return ksList; + async updateProject(project: KcProject) { + return this.enqueue(() => this.write(project)); } - async getProjects() { - if (this.projectList) return this.projectList; - return this.projects; + deleteProject(id: string) { + this.projectList = this.projectList.filter((p) => p.id.value !== id); + this.db.removeItem(`chat-${id}`); + this.enqueue(() => this.remove(`p:${id}`, `${this.api}/projects/${id}`)); } - async saveProject(project: KcProject) { - // Update project in local cache - const idx = this.projectList?.findIndex( - (p) => p.id.value === project.id.value + deleteKnowledgeSource(ks: KnowledgeSource) { + this.db.removeItem(`icon-${ks.id.value}`); + this.db.removeItem(`chat-${ks.id.value}`); + this.enqueue(() => + this.remove(`s:${ks.id.value}`, `${this.api}/sources/${ks.id.value}`) ); - if (idx === -1) { - // If project does not exist in memory, add it - if (this.projectList) this.projectList.push(project); - else this.projectList = [project]; - } else { - // Otherwise update the project in-place - if (this.projectList && idx && idx) { - this.projectList[idx] = project; - } + if (ks.importMethod === 'autoscan' && typeof ks.accessLink === 'string') { + this.autoscan.delete(ks.accessLink); } + } - // Update the project in database - await this.updateProject(project); + private enqueue(task: () => Promise): Promise { + this.queue = this.queue.then(task).catch((e) => this.report(e)); + return this.queue; + } - // Get list of all project IDs from local storage - let projectList: string[] = []; - let projectListString = this.db.getItem(this.KC_ALL_PROJECT_IDS); - if (projectListString) { - projectList = JSON.parse(projectListString); - } + private report(e: unknown) { + const message = + e instanceof HttpErrorResponse + ? e.error?.error?.message ?? e.message + : e instanceof Error + ? e.message + : String(e); + console.error('[Storage]:', e); + this.notifications.error('Storage', 'Changes Not Saved', message, 'toast'); + } - if (projectList.length > 0) { - const id = projectList.find((item) => item === project.id.value); - // Check if project ID is in the list. If it's not, add it, otherwise continue - if (!id) { - projectList.push(project.id.value); - projectListString = JSON.stringify(projectList); - this.db.setItem(this.KC_ALL_PROJECT_IDS, projectListString); + /** Write the project and every changed source. Copy new files first. */ + private async write(project: KcProject) { + const projectRecord = projectToRecord(project); + await this.put( + `p:${projectRecord.id}`, + `${this.api}/projects/${projectRecord.id}`, + projectRecord + ); + + for (const ks of project.knowledgeSource ?? []) { + if (ks.ingestType === 'file' && !ks.assetId) { + await this.copyFile(ks); } - } else { - // If there is no project list, create one and add project to it - projectList.push(project.id.value); - projectListString = JSON.stringify(projectList); - this.db.setItem(this.KC_ALL_PROJECT_IDS, projectListString); + const record = sourceToRecord(ks, projectRecord.id); + await this.put( + `s:${record.id}`, + `${this.api}/sources/${record.id}`, + record + ); } } - async saveProjectList(projects: KcProject[]) { - for (const project of projects) { - this.saveProject(project); + private async copyFile(ks: KnowledgeSource) { + const file = ks.reference?.source?.file; + const path = + file?.path ?? (typeof ks.accessLink === 'string' ? ks.accessLink : ''); + try { + const asset = await this.ipc.importFile(path, file?.type || undefined); + ks.assetId = asset.id; + } catch (e) { + // The source is saved without a managed copy. Report it. + this.notifications.error( + 'Storage', + 'File Not Copied', + `${ks.title}: ${e instanceof Error ? e.message : e}`, + 'toast' + ); } } - async updateProject(project: KcProject) { - const projectString = JSON.stringify(project); - this.db.setItem(project.id.value, projectString); - } - - deleteProject(id: string) { - if (!this.projectList) { - this.notifications.warn( - 'Storage Service', - 'Invalid Project', - 'Attempting to delete a project that does not exist in memory...' - ); + private async put(key: string, url: string, record: object) { + const json = JSON.stringify(this.writable(record)); + if (this.synced.get(key) === json) { return; } + await firstValueFrom(this.http.put(url, JSON.parse(json))); + this.synced.set(key, json); + } - // Remove project from list - this.projectList = this.projectList.filter((p) => p.id.value !== id); + private async remove(key: string, url: string) { + try { + await firstValueFrom(this.http.delete(url)); + } catch (e) { + if (!(e instanceof HttpErrorResponse && e.status === 404)) { + throw e; + } + } + this.synced.delete(key); + } - // Remove project saved under this string. - this.db.removeItem(id); - this.db.removeItem(`chat-${id}`); + /** URL of a managed file's content. Fetch it with HttpClient (adds the token). */ + assetContentUrl(assetId: string) { + return `${this.api}/assets/${assetId}/content`; + } - // Persist changes - const projectIds: string[] = []; - for (const project of this.projectList) { - projectIds.push(project.id.value); - } - const projectListStr = JSON.stringify(projectIds); - this.db.setItem(this.KC_ALL_PROJECT_IDS, projectListStr); + /** Download the library backup (projects, sources, managed files). */ + async exportLibrary() { + const blob = await firstValueFrom( + this.http.get(`${this.api}/backup`, { responseType: 'blob' }) + ); + const date = new Date().toISOString().slice(0, 10); + this.download(blob, `knowledge-library-${date}.tar`); + return blob.size; } /** - * Download a backup of all renderer storage as a JSON file. + * Download a backup of renderer storage: chat history and preferences. + * Projects and sources are not in it. They are in the library backup. */ export(appVersion?: string) { const backup = createBackup(this.db, appVersion); const date = backup.exportedAt.slice(0, 10); - const blob = new Blob([JSON.stringify(backup)], { - type: 'application/json;charset=utf-8;', - }); + this.download( + new Blob([JSON.stringify(backup)], { + type: 'application/json;charset=utf-8;', + }), + `knowledge-backup-${date}.json` + ); + return Object.keys(backup.data).length; + } + private download(blob: Blob, filename: string) { const link = document.createElement('a'); link.style.display = 'none'; link.href = URL.createObjectURL(blob); - link.download = `knowledge-backup-${date}.json`; + link.download = filename; document.body.appendChild(link); link.click(); document.body.removeChild(link); // Chromium reads the blob after click() returns setTimeout(() => URL.revokeObjectURL(link.href), 10000); - - return Object.keys(backup.data).length; } /** - * Restore a backup file. The caller must reload the app afterwards. + * Restore a renderer backup. The caller must reload the app afterwards. * Throws an Error with a user-facing message if the file is invalid. */ restore(fileText: string): RestoreResult { @@ -315,22 +304,6 @@ export class StorageService { } catch { throw new Error('The file is not valid JSON.'); } - const result = restoreBackup(this.db, parsed); - this.projectList = null; - return result; - } - - deleteKnowledgeSource(ks: KnowledgeSource) { - this.db.removeItem(`icon-${ks.id.value}`); - this.db.removeItem(`chat-${ks.id.value}`); - - if (!ks.importMethod) { - return; - } else { - // TODO: autoscan service is only used in this one place here.. should consider moving elsewhere... - if (ks.importMethod === 'autoscan' && typeof ks.accessLink === 'string') { - this.autoscan.delete(ks.accessLink); - } - } + return restoreBackup(this.db, parsed); } } diff --git a/src/kc_angular/src/app/services/user-services/data.service.ts b/src/kc_angular/src/app/services/user-services/data.service.ts index c6419dc8..dee4ed6e 100755 --- a/src/kc_angular/src/app/services/user-services/data.service.ts +++ b/src/kc_angular/src/app/services/user-services/data.service.ts @@ -55,30 +55,12 @@ export class DataService implements OnDestroy { }, get: async (uuids: UUID[]): Promise => { - const ksList: KnowledgeSource[] = []; - for (const id of uuids) { - const lookup = `ks-${id.value}`; - const kstr = localStorage.getItem(lookup); - if (kstr) { - const ks = JSON.parse(kstr); - if (ks) { - ksList.push(ks); - } - } - } - return ksList; + const ids = new Set(uuids.map((id) => id.value)); + return (await this.storage.ksList()).filter((ks) => ids.has(ks.id.value)); }, + /** Sources live inside their project. The storage service persists both. */ update: async (ksList: KnowledgeSource[]) => { - for (const ks of ksList) { - const lookup = `ks-${ks.id.value}`; - const kstr = JSON.stringify(ks); - if (kstr) { - localStorage.setItem(lookup, kstr); - } - } - - // TODO: Remove this after projects no longer carry entire KS objects... for (const ks of ksList) { await this._projects.updateProjects([ { @@ -89,13 +71,6 @@ export class DataService implements OnDestroy { } }, - delete: async (ksList: KnowledgeSource[]) => { - for (const ks of ksList) { - const lookup = `ks-${ks.id.value}`; - localStorage.removeItem(lookup); - } - }, - count: this.__ksList.asObservable().pipe(map((sources) => sources.length)), typeCount: this.__ksList.asObservable().pipe( diff --git a/src/kc_angular/tsconfig.json b/src/kc_angular/tsconfig.json index 13e1fdb5..e049ac54 100755 --- a/src/kc_angular/tsconfig.json +++ b/src/kc_angular/tsconfig.json @@ -43,6 +43,9 @@ "@shared/*": [ "../kc_shared/*" ], + "@contracts/*": [ + "../kc_contracts/*" + ], "@services/*": [ "src/app/services/*" ] diff --git a/src/kc_contracts/mapping.ts b/src/kc_contracts/mapping.ts new file mode 100644 index 00000000..a38e6e57 --- /dev/null +++ b/src/kc_contracts/mapping.ts @@ -0,0 +1,109 @@ +/* + * Copyright (c) 2026 Rob Royce + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Conversion between the UI object shapes (KcProject, KnowledgeSource) and + * storage records. The UI and the migration use the same rules. + * + * This file must not import from any application. It uses structural + * types so that both sides can call it. + */ + +type Json = { [key: string]: any }; + +export interface MappedProject { + id: string; + parentId: string | null; + name: string; + data: Json; +} + +export interface MappedSource { + id: string; + projectId: string; + title: string; + ingestType: string; + assetId: string | null; + data: Json; +} + +/** Source fields that the UI derives again. Clients do not store them. */ +export const DERIVED_SOURCE_FIELDS = ["icon", "thumbnail"]; + +const PROJECT_COLUMNS = ["id", "name", "parentId", "knowledgeSource"]; +const SOURCE_COLUMNS = [ + "id", + "title", + "ingestType", + "associatedProject", + "assetId", +]; + +function without(obj: Json, keys: string[]): Json { + const out: Json = {}; + for (const [key, value] of Object.entries(obj)) { + if (!keys.includes(key) && value !== undefined) { + out[key] = value; + } + } + return out; +} + +export function projectToRecord(project: Json): MappedProject { + const parent = project["parentId"]?.value; + return { + id: project["id"]?.value, + parentId: typeof parent === "string" && parent ? parent : null, + name: project["name"], + data: without(project, PROJECT_COLUMNS), + }; +} + +export function sourceToRecord(source: Json, projectId: string): MappedSource { + return { + id: source["id"]?.value, + projectId, + title: source["title"] ?? "", + ingestType: source["ingestType"], + assetId: source["assetId"] ?? null, + data: without(source, [...SOURCE_COLUMNS, ...DERIVED_SOURCE_FIELDS]), + }; +} + +/** Rebuild the UI project object. Sources must be in project order. */ +export function recordToProject(project: MappedProject, sources: Json[]): Json { + return { + ...project.data, + id: { value: project.id }, + name: project.name, + parentId: { value: project.parentId ?? "" }, + knowledgeSource: sources, + }; +} + +export function recordToSource(source: MappedSource): Json { + const out: Json = { + ...source.data, + id: { value: source.id }, + title: source.title, + ingestType: source.ingestType, + associatedProject: { value: source.projectId }, + }; + if (source.assetId) { + out["assetId"] = source.assetId; + } + return out; +} diff --git a/src/kc_contracts/package.json b/src/kc_contracts/package.json new file mode 100644 index 00000000..e3f3ca5f --- /dev/null +++ b/src/kc_contracts/package.json @@ -0,0 +1,6 @@ +{ + "name": "kc_contracts", + "private": true, + "description": "Storage API contracts shared by the storage service and the clients. No application imports.", + "type": "module" +} diff --git a/src/kc_contracts/storage.ts b/src/kc_contracts/storage.ts new file mode 100644 index 00000000..97e458c6 --- /dev/null +++ b/src/kc_contracts/storage.ts @@ -0,0 +1,101 @@ +/* + * Copyright (c) 2026 Rob Royce + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Storage service API contract, version 1. + * + * Types only. This file must not import from any application. + */ + +export type JsonObject = { [key: string]: unknown }; + +export interface ProjectRecord { + id: string; + parentId: string | null; + name: string; + /** Every project field that has no column. */ + data: JsonObject; + createdAt?: string; + updatedAt?: string; +} + +export interface SourceRecord { + id: string; + projectId: string; + title: string; + ingestType: string; + /** Managed file of a file source. */ + assetId: string | null; + /** Order inside the project. Set by the service. */ + position?: number; + /** Every source field that has no column. */ + data: JsonObject; + createdAt?: string; + updatedAt?: string; +} + +export interface AssetRecord { + id: string; + sha256: string; + size: number; + mediaType: string; + filename: string; + /** Location of the file when it was imported. Information only. */ + originalPath: string | null; + createdAt: string; +} + +export interface HealthResponse { + status: "ok"; + service: "kc-storage"; + version: string; + schemaVersion: number; +} + +export type StorageErrorCode = + | "bad_request" + | "unauthorized" + | "forbidden" + | "not_found" + | "conflict" + | "payload_too_large" + | "internal"; + +export interface ErrorResponse { + error: { code: StorageErrorCode; message: string }; +} + +export interface ProjectList { + projects: ProjectRecord[]; +} + +export interface SourceList { + sources: SourceRecord[]; +} + +/** Request headers for POST /v1/assets. Values are percent-encoded. */ +export const ASSET_FILENAME_HEADER = "x-knowledge-filename"; +export const ASSET_ORIGINAL_PATH_HEADER = "x-knowledge-original-path"; + +export interface LibraryBackupManifest { + format: "knowledge-library-backup"; + version: 1; + createdAt: string; + service: { version: string; schemaVersion: number }; + projects: ProjectRecord[]; + sources: SourceRecord[]; + assets: (AssetRecord & { path: string })[]; +} diff --git a/src/kc_electron/src/app/backend.ts b/src/kc_electron/src/app/backend.ts index 1d638581..c185c895 100644 --- a/src/kc_electron/src/app/backend.ts +++ b/src/kc_electron/src/app/backend.ts @@ -33,6 +33,7 @@ export interface BackendEndpoint { export interface BackendInfo { chat: BackendEndpoint; + storage: BackendEndpoint; } export function newToken() { @@ -47,6 +48,6 @@ export function registerBackendInfo(endpoints: { [K in keyof BackendInfo]: Promise; }) { ipcMain.handle("A2E:Backend:Info", async (): Promise => { - return { chat: await endpoints.chat }; + return { chat: await endpoints.chat, storage: await endpoints.storage }; }); } diff --git a/src/kc_electron/src/app/ipc/ipc-files.ts b/src/kc_electron/src/app/ipc/ipc-files.ts index 727dffe7..eb70eff1 100755 --- a/src/kc_electron/src/app/ipc/ipc-files.ts +++ b/src/kc_electron/src/app/ipc/ipc-files.ts @@ -19,6 +19,8 @@ import { ThumbnailRequest, } from "../../../../kc_shared/models/electron.ipc.model"; +import { materializeAsset } from "../storage.client"; + const fse = require("fs-extra"); const settingsService = require("../services/settings.service"); @@ -162,11 +164,14 @@ getFileThumbnail = ipcMain.on( for (const request of requests) { const height = request.height ?? 1920; const width = request.width ?? 1080; + // Managed files come from the storage service. Others use the given path. + const file = request.assetId + ? materializeAsset(request.assetId) + : Promise.resolve(path.resolve(request.path)); actions.push( - nativeImage.createThumbnailFromPath(path.resolve(request.path), { - width: width, - height: height, - }) + file.then((p: string) => + nativeImage.createThumbnailFromPath(p, { width, height }) + ) ); } diff --git a/src/kc_electron/src/app/storage.client.ts b/src/kc_electron/src/app/storage.client.ts new file mode 100644 index 00000000..50ae187f --- /dev/null +++ b/src/kc_electron/src/app/storage.client.ts @@ -0,0 +1,172 @@ +/* + * Copyright (c) 2026 Rob Royce + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Desktop functions that need both the local filesystem and the storage + * service. The renderer reaches them only through these IPC channels: + * + * A2E:Storage:ImportFile path -> AssetRecord (copy a local file into storage) + * A2E:Storage:OpenAsset assetId -> open the managed file in its default app + * + * Thumbnails and chat text extraction use materializeAsset(). + */ + +import fs from "fs"; +import http from "http"; +import path from "path"; +import { ipcMain, shell } from "electron"; +import type { AssetRecord } from "../../../kc_contracts/storage"; +import { BackendEndpoint } from "./backend"; + +const ASSET_ID = + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; + +let endpoint: Promise | undefined; +let cacheDir = ""; + +function request( + method: string, + route: string, + headers: http.OutgoingHttpHeaders = {} +): Promise<{ + ep: BackendEndpoint; + req: http.ClientRequest; + res: Promise; +}> { + if (!endpoint) { + throw new Error("Storage client is not configured."); + } + return endpoint.then((ep) => { + if (!ep.url || !ep.token) { + throw new Error(ep.error ?? "Storage service unavailable."); + } + let resolveRes: (r: http.IncomingMessage) => void = () => undefined; + let rejectRes: (e: Error) => void = () => undefined; + const res = new Promise((ok, fail) => { + resolveRes = ok; + rejectRes = fail; + }); + const req = http.request(`${ep.url}${route}`, { + method, + headers: { ...headers, Authorization: `Bearer ${ep.token}` }, + }); + req.on("response", resolveRes); + req.on("error", rejectRes); + return { ep, req, res }; + }); +} + +async function readBody(res: http.IncomingMessage) { + const chunks: Buffer[] = []; + for await (const chunk of res) chunks.push(chunk); + return Buffer.concat(chunks).toString("utf8"); +} + +async function errorFrom(res: http.IncomingMessage) { + const text = await readBody(res); + try { + return new Error(JSON.parse(text).error.message); + } catch { + return new Error(`Storage service returned ${res.statusCode}.`); + } +} + +/** Copy a local file into managed storage. The original is not changed. */ +export async function importFile( + filePath: string, + mediaType?: string +): Promise { + const resolved = path.resolve(filePath); + const stat = fs.statSync(resolved); + if (!stat.isFile()) { + throw new Error(`Not a regular file: ${resolved}`); + } + const { req, res } = await request("POST", "/v1/assets", { + "Content-Type": mediaType || "application/octet-stream", + "Content-Length": stat.size, + "x-knowledge-filename": encodeURIComponent(path.basename(resolved)), + "x-knowledge-original-path": encodeURIComponent(resolved), + }); + fs.createReadStream(resolved).pipe(req); + const response = await res; + if (response.statusCode !== 201) { + throw await errorFrom(response); + } + return JSON.parse(await readBody(response)).asset; +} + +/** + * A local copy of a managed file, for functions that need a path + * (thumbnails, default app, text extraction). Cached by asset ID. + */ +export async function materializeAsset(assetId: string): Promise { + if (!ASSET_ID.test(assetId)) { + throw new Error("Invalid asset ID."); + } + const meta = await request("GET", `/v1/assets/${assetId}`); + meta.req.end(); + const metaRes = await meta.res; + if (metaRes.statusCode !== 200) { + throw await errorFrom(metaRes); + } + const asset: AssetRecord = JSON.parse(await readBody(metaRes)).asset; + + const target = path.join(cacheDir, assetId, path.basename(asset.filename)); + if (fs.existsSync(target) && fs.statSync(target).size === asset.size) { + return target; + } + fs.mkdirSync(path.dirname(target), { recursive: true }); + + const content = await request("GET", `/v1/assets/${assetId}/content`); + content.req.end(); + const contentRes = await content.res; + if (contentRes.statusCode !== 200) { + throw await errorFrom(contentRes); + } + const partial = `${target}.partial`; + await new Promise((resolve, reject) => { + const out = fs.createWriteStream(partial); + contentRes.pipe(out); + out.on("finish", resolve); + out.on("error", reject); + contentRes.on("error", reject); + }); + fs.renameSync(partial, target); + return target; +} + +export function configureStorageClient( + storage: Promise, + assetCacheDir: string +) { + endpoint = storage; + cacheDir = assetCacheDir; + + ipcMain.handle( + "A2E:Storage:ImportFile", + (_event, args: { path: string; mediaType?: string }) => + importFile(args.path, args.mediaType) + ); + + ipcMain.handle("A2E:Storage:OpenAsset", async (_event, assetId: string) => { + const file = await materializeAsset(assetId); + const outcome = await shell.openPath(file); + if (outcome) { + throw new Error(outcome); + } + return true; + }); +} diff --git a/src/kc_electron/src/app/storage.process.ts b/src/kc_electron/src/app/storage.process.ts new file mode 100644 index 00000000..7f22ddaa --- /dev/null +++ b/src/kc_electron/src/app/storage.process.ts @@ -0,0 +1,137 @@ +/* + * Copyright (c) 2026 Rob Royce + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Starts the standalone storage service (src/kc_storage) with Node.js. + * + * Electron's own Node.js is too old for node:sqlite, so the service runs + * with the Node.js on PATH, or with KC_NODE. The service stops when this + * process closes its standard input, including after a crash. + */ + +import { spawn, ChildProcess } from "child_process"; +import fs from "fs"; +import path from "path"; +import { app } from "electron"; +import { BackendEndpoint, newToken } from "./backend"; + +const START_TIMEOUT_MS = 20000; + +let child: ChildProcess | undefined; + +export function startStorageService(dataDir: string): Promise { + const entry = path.join( + app.getAppPath(), + "src", + "kc_storage", + "src", + "main.ts" + ); + const node = process.env.KC_NODE || "node"; + const token = newToken(); + + return new Promise((resolve) => { + if (!fs.existsSync(entry)) { + resolve({ error: `Storage service not found at ${entry}.` }); + return; + } + + let stderr = ""; + let settled = false; + const finish = (endpoint: BackendEndpoint) => { + if (!settled) { + settled = true; + resolve(endpoint); + } + }; + + // Only the variables the service needs. ELECTRON_RUN_AS_NODE and + // test-runner variables must not reach it. + const env: NodeJS.ProcessEnv = { + PATH: process.env.PATH, + HOME: process.env.HOME, + KC_STORAGE_TOKEN: token, + }; + + child = spawn( + node, + [ + entry, + "serve", + "--data-dir", + dataDir, + "--port", + "0", + "--allow-origin", + "null", + "--exit-on-stdin-close", + ], + { env, stdio: ["pipe", "pipe", "pipe"] } + ); + + const timer = setTimeout(() => { + finish({ + error: `Storage service did not start in time. ${stderr}`.trim(), + }); + child?.kill(); + }, START_TIMEOUT_MS); + + child.on("error", (e) => { + clearTimeout(timer); + finish({ + error: `Could not run "${node}": ${e.message}. Install Node.js 24 or set KC_NODE.`, + }); + }); + + child.stderr?.on("data", (d) => { + stderr += d; + process.stderr.write(`[storage] ${d}`); + }); + + let stdout = ""; + child.stdout?.on("data", (d) => { + stdout += d; + const newline = stdout.indexOf("\n"); + if (newline < 0 || settled) { + return; + } + clearTimeout(timer); + try { + const ready = JSON.parse(stdout.slice(0, newline)); + console.log(`[Knowledge]: storage service listening on ${ready.url}`); + finish({ url: ready.url, token }); + } catch { + finish({ error: `Unexpected storage service output: ${stdout}` }); + } + }); + + child.on("exit", (code) => { + clearTimeout(timer); + finish({ + error: `Storage service exited with code ${code}. ${stderr}`.trim(), + }); + if (settled && code !== 0) { + console.error(`[Knowledge]: storage service stopped (code ${code})`); + } + }); + }); +} + +export function stopStorageService() { + if (child && child.exitCode === null) { + child.stdin?.end(); + } +} diff --git a/src/kc_electron/src/local/middleware/SourceParser.ts b/src/kc_electron/src/local/middleware/SourceParser.ts index c37eee73..aaaa1318 100644 --- a/src/kc_electron/src/local/middleware/SourceParser.ts +++ b/src/kc_electron/src/local/middleware/SourceParser.ts @@ -20,6 +20,7 @@ import { getDocument } from "pdfjs-dist"; import { htmlToText, HtmlToTextOptions } from "html-to-text"; import fs from "fs"; import TextUtils from "../utils/text.utils"; +import { materializeAsset } from "../../app/storage.client"; import { codeMarkdownMap } from "../../../../kc_shared/constants/supported.file.types"; export class SourceParser { @@ -44,7 +45,10 @@ export class SourceParser { const accessLink = req.body.accessLink; if (ingestType === "file") { - req.body.text = await SourceParser.fromFile(accessLink); + // Managed files come from the storage service, not the original path + const assetId = req.body.source?.assetId; + const filePath = assetId ? await materializeAsset(assetId) : accessLink; + req.body.text = await SourceParser.fromFile(filePath); } else { req.body.text = await SourceParser.fromWeb(new URL(accessLink)); } diff --git a/src/kc_electron/src/main.ts b/src/kc_electron/src/main.ts index 6d0f2aef..4c8ff08e 100755 --- a/src/kc_electron/src/main.ts +++ b/src/kc_electron/src/main.ts @@ -41,6 +41,8 @@ import * as uuid from "uuid"; import KnowledgeIpc from "./local/utils/ipc.utils"; import ChatServer from "./local/chat.api"; import { newToken, registerBackendInfo } from "./app/backend"; +import { startStorageService, stopStorageService } from "./app/storage.process"; +import { configureStorageClient } from "./app/storage.client"; const settingsService = require("./app/services/settings.service"); @@ -83,11 +85,16 @@ if (!app.requestSingleInstanceLock()) { console.error( "[Knowledge]: this profile is already open in another instance. Exiting." ); - app.exit(0); + process.exit(0); } +const dataPath = settingsService.getSettings().system.appPath; +// /storage belongs to the chat server text cache +const storage = startStorageService(path.join(dataPath, "library")); +configureStorageClient(storage, path.join(dataPath, "tmp", "assets")); + const chatServer = new ChatServer(newToken()); -registerBackendInfo({ chat: chatServer.start() }); +registerBackendInfo({ chat: chatServer.start(), storage }); // Setup knowledge source ingestion require("./app/services/index"); @@ -238,8 +245,23 @@ function checkForUpdates() { }); } -app.on("ready", function () { +app.on("ready", async function () { + // Projects and sources live in the storage service. Without it the app cannot work. + const endpoint = await storage; + if (endpoint.error) { + dialog.showErrorBox( + "Knowledge cannot start", + `The storage service did not start.\n\n${endpoint.error}` + ); + app.exit(1); + return; + } + createMainWindow(); kcMainWindow.loadFile(MAIN_ENTRY); checkForUpdates(); }); + +app.on("will-quit", () => { + stopStorageService(); +}); diff --git a/src/kc_electron/src/preload.js b/src/kc_electron/src/preload.js index 515a0b7d..f5d8ea9a 100755 --- a/src/kc_electron/src/preload.js +++ b/src/kc_electron/src/preload.js @@ -55,7 +55,11 @@ const ipcSendChannels = [ "A2E:Window:ZoomIn", ]; -const ipcInvokeChannels = ["A2E:Backend:Info"]; +const ipcInvokeChannels = [ + "A2E:Backend:Info", + "A2E:Storage:ImportFile", + "A2E:Storage:OpenAsset", +]; const ipcReceiveOnceChannels = [ "E2A:BrowserView:Close", diff --git a/src/kc_electron/tsconfig.json b/src/kc_electron/tsconfig.json index 2ab5ce2d..67c56b9c 100755 --- a/src/kc_electron/tsconfig.json +++ b/src/kc_electron/tsconfig.json @@ -102,7 +102,8 @@ }, "include": [ "src/**/*", - "./../kc_shared/**/*" + "./../kc_shared/**/*", + "./../kc_contracts/**/*.ts" ], "exclude": [ "node_modules", diff --git a/src/kc_shared/models/electron.ipc.model.ts b/src/kc_shared/models/electron.ipc.model.ts index 281cc91a..5f89f5f9 100755 --- a/src/kc_shared/models/electron.ipc.model.ts +++ b/src/kc_shared/models/electron.ipc.model.ts @@ -60,6 +60,8 @@ export interface DialogRequest { export interface ThumbnailRequest { path: string; + /** Managed file in the storage service. Used instead of path when set. */ + assetId?: string; width?: number; height?: number; id?: string; diff --git a/src/kc_storage/package.json b/src/kc_storage/package.json new file mode 100644 index 00000000..f38d7cfe --- /dev/null +++ b/src/kc_storage/package.json @@ -0,0 +1,21 @@ +{ + "name": "kc_storage", + "version": "0.1.0", + "description": "Standalone storage service for Knowledge: projects, sources, and managed files in SQLite.", + "author": "Rob Royce (https://robroyce.dev)", + "license": "Apache-2.0", + "private": true, + "type": "module", + "engines": { + "node": ">=24" + }, + "scripts": { + "start": "node src/main.ts serve", + "test": "node --test --test-concurrency=1 \"test/*.test.ts\"", + "typecheck": "tsc --noEmit -p tsconfig.json" + }, + "devDependencies": { + "@types/node": "^24.0.0", + "typescript": "^5.9.0" + } +} diff --git a/src/kc_storage/src/assets.ts b/src/kc_storage/src/assets.ts new file mode 100644 index 00000000..e83d2d42 --- /dev/null +++ b/src/kc_storage/src/assets.ts @@ -0,0 +1,113 @@ +/* + * Managed file storage. Files live at assets//. + * New content goes to assets/tmp first and moves into place after the + * hash and size are known. + */ + +import crypto from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import { pipeline } from "node:stream/promises"; +import { Transform, type Readable } from "node:stream"; +import { tooLarge } from "./errors.ts"; +import { validAssetId } from "./validate.ts"; + +export interface StagedFile { + tmpPath: string; + sha256: string; + size: number; +} + +export class AssetStore { + readonly root: string; + private tmpDir: string; + + constructor(root: string) { + this.root = root; + this.tmpDir = path.join(root, "tmp"); + fs.mkdirSync(this.tmpDir, { recursive: true }); + } + + /** Path of a managed file. Only a valid asset ID can form a path. */ + pathFor(id: string): string { + validAssetId(id); + return path.join(this.root, id.slice(0, 2), id); + } + + /** Write a stream to a temporary file and hash it. */ + async stage(input: Readable, maxBytes: number): Promise { + const tmpPath = path.join(this.tmpDir, crypto.randomUUID()); + const hash = crypto.createHash("sha256"); + let size = 0; + + const meter = new Transform({ + transform(chunk: Buffer, _enc, done) { + size += chunk.length; + if (size > maxBytes) { + done(tooLarge(`The file is larger than ${maxBytes} bytes.`)); + return; + } + hash.update(chunk); + done(null, chunk); + }, + }); + + try { + await pipeline( + input, + meter, + fs.createWriteStream(tmpPath, { flags: "wx" }) + ); + } catch (e) { + fs.rmSync(tmpPath, { force: true }); + throw e; + } + return { tmpPath, sha256: hash.digest("hex"), size }; + } + + /** Move a staged file to its permanent path. */ + commit(staged: StagedFile, id: string) { + const target = this.pathFor(id); + fs.mkdirSync(path.dirname(target), { recursive: true }); + fs.renameSync(staged.tmpPath, target); + } + + discard(staged: StagedFile) { + fs.rmSync(staged.tmpPath, { force: true }); + } + + remove(id: string) { + fs.rmSync(this.pathFor(id), { force: true }); + } + + exists(id: string): boolean { + return fs.existsSync(this.pathFor(id)); + } + + /** + * Remove temporary files and files with no database record. Call only + * while this process holds the data-directory lock and no request runs. + */ + sweep(known: Set): string[] { + const removed: string[] = []; + for (const entry of fs.readdirSync(this.tmpDir)) { + fs.rmSync(path.join(this.tmpDir, entry), { + force: true, + recursive: true, + }); + removed.push(`tmp/${entry}`); + } + for (const dir of fs.readdirSync(this.root)) { + if (dir === "tmp" || !/^[0-9a-f]{2}$/.test(dir)) { + continue; + } + for (const file of fs.readdirSync(path.join(this.root, dir))) { + if (!known.has(file)) { + fs.rmSync(path.join(this.root, dir, file), { force: true }); + removed.push(`${dir}/${file}`); + } + } + } + return removed; + } +} diff --git a/src/kc_storage/src/backup.ts b/src/kc_storage/src/backup.ts new file mode 100644 index 00000000..95c7be80 --- /dev/null +++ b/src/kc_storage/src/backup.ts @@ -0,0 +1,161 @@ +/* + * Portable library backup: a tar file with manifest.json and assets/. + * It holds projects, sources, and managed files. It does not hold chat + * history, UI preferences, settings, or credentials. + */ + +import fs from "node:fs"; +import path from "node:path"; +import type { Writable } from "node:stream"; +import type { LibraryBackupManifest } from "../../kc_contracts/storage.ts"; +import { DATABASE_FILE, openDataDir, type DataDir } from "./datadir.ts"; +import { SCHEMA_VERSION, transaction } from "./db.ts"; +import type { StagedFile } from "./assets.ts"; +import { + readTarEntry, + readTarIndex, + streamTarEntry, + TarWriter, +} from "./tar.ts"; +import { validAssetId, validProject, validSource } from "./validate.ts"; +import { VERSION } from "./version.ts"; + +export const BACKUP_FORMAT = "knowledge-library-backup"; + +export async function writeBackup(dir: DataDir, out: Writable) { + const { library, assets } = dir; + const manifest: LibraryBackupManifest = { + format: BACKUP_FORMAT, + version: 1, + createdAt: new Date().toISOString(), + service: { version: VERSION, schemaVersion: SCHEMA_VERSION }, + projects: library.listProjects(), + sources: library.listSources(), + assets: library.listAssets().map((a) => ({ ...a, path: `assets/${a.id}` })), + }; + + for (const asset of manifest.assets) { + if (!assets.exists(asset.id)) { + throw new Error(`Managed file for asset ${asset.id} is missing.`); + } + } + + const tar = new TarWriter(out); + await tar.addBuffer( + "manifest.json", + Buffer.from(JSON.stringify(manifest, null, 2)) + ); + for (const asset of manifest.assets) { + await tar.addFile(asset.path, assets.pathFor(asset.id)); + } + await tar.finish(); + return manifest; +} + +export interface RestoreSummary { + projects: number; + sources: number; + assets: number; +} + +/** + * Restore a backup into a new or empty data directory. Validates the + * manifest and every file hash before the database changes. + */ +export async function restoreBackup( + archive: string, + root: string +): Promise { + if (fs.existsSync(path.join(root, DATABASE_FILE))) { + const probe = openDataDir(root); + const empty = probe.library.isEmpty(); + probe.close(); + if (!empty) { + throw new Error( + `The data directory ${root} already has a library. Use an empty directory.` + ); + } + } + + const index = readTarIndex(archive); + const manifestEntry = index.get("manifest.json"); + if (!manifestEntry) { + throw new Error("The backup has no manifest.json."); + } + const manifest = JSON.parse( + readTarEntry(archive, manifestEntry).toString("utf8") + ); + if (manifest?.format !== BACKUP_FORMAT || manifest.version !== 1) { + throw new Error("The file is not a knowledge-library-backup version 1."); + } + + const projects = (manifest.projects ?? []).map((p: any) => + validProject(p.id, p) + ); + const sources = (manifest.sources ?? []).map((s: any) => + validSource(s.id, s) + ); + const assetList = (manifest.assets ?? []).map((a: any) => { + validAssetId(a.id); + const entry = index.get(`assets/${a.id}`); + if (!entry || entry.size !== a.size) { + throw new Error(`The backup has no valid file for asset ${a.id}.`); + } + return { record: a, entry }; + }); + + const dir = openDataDir(root); + const staged: { id: string; file: StagedFile }[] = []; + try { + for (const { record, entry } of assetList) { + const file = await dir.assets.stage( + streamTarEntry(archive, entry), + Number.MAX_SAFE_INTEGER + ); + staged.push({ id: record.id, file }); + if (file.sha256 !== record.sha256) { + throw new Error( + `Hash mismatch for asset ${record.id}. The backup is damaged.` + ); + } + } + + const createdAt = new Map(); + for (const r of [ + ...(manifest.projects ?? []), + ...(manifest.sources ?? []), + ]) { + if (r.createdAt) createdAt.set(r.id, r.createdAt); + } + + transaction(dir.db, () => { + for (const { record } of assetList) { + dir.library.insertAsset({ + id: record.id, + sha256: record.sha256, + size: record.size, + mediaType: record.mediaType, + filename: record.filename, + originalPath: record.originalPath ?? null, + createdAt: record.createdAt, + }); + } + for (const p of projects) dir.library.putProject(p, createdAt.get(p.id)); + for (const s of sources) dir.library.putSource(s, createdAt.get(s.id)); + for (const { id, file } of staged) dir.assets.commit(file, id); + }); + return { + projects: projects.length, + sources: sources.length, + assets: assetList.length, + }; + } catch (e) { + for (const { id, file } of staged) { + dir.assets.discard(file); + if (!dir.library.getAsset(id)) dir.assets.remove(id); + } + throw e; + } finally { + dir.close(); + } +} diff --git a/src/kc_storage/src/cli.ts b/src/kc_storage/src/cli.ts new file mode 100644 index 00000000..1c552102 --- /dev/null +++ b/src/kc_storage/src/cli.ts @@ -0,0 +1,207 @@ +/* + * Command line interface for the storage service. + */ + +import fs from "node:fs"; +import path from "node:path"; +import { parseArgs } from "node:util"; +import type { AddressInfo } from "node:net"; +import { restoreBackup, writeBackup } from "./backup.ts"; +import { openDataDir } from "./datadir.ts"; +import { createServer } from "./http.ts"; +import { migrate } from "./migrate.ts"; +import { VERSION } from "./version.ts"; + +const USAGE = `kc-storage ${VERSION} + +Usage: + node src/main.ts serve --data-dir [--port ] [--allow-origin ]... + [--max-upload-mb ] [--exit-on-stdin-close] + node src/main.ts migrate --data-dir --from [--dry-run] + node src/main.ts backup --data-dir --out + node src/main.ts restore --data-dir --from + +serve needs the bearer token in the KC_STORAGE_TOKEN environment variable. +serve listens on 127.0.0.1 only. Port 0 (default) picks a free port. +When ready, serve prints one JSON line: {"event":"ready","url":...}.`; + +function fail(message: string): never { + console.error(`kc-storage: ${message}`); + process.exit(2); +} + +export async function main(argv: string[]) { + const [command, ...rest] = argv; + let parsed; + try { + parsed = parseArgs({ + args: rest, + options: { + "data-dir": { type: "string" }, + port: { type: "string", default: "0" }, + "allow-origin": { type: "string", multiple: true, default: [] }, + "exit-on-stdin-close": { type: "boolean", default: false }, + "max-upload-mb": { type: "string", default: "2048" }, + from: { type: "string" }, + out: { type: "string" }, + "dry-run": { type: "boolean", default: false }, + help: { type: "boolean", default: false }, + }, + }); + } catch (e: any) { + fail(`${e.message}\n\n${USAGE}`); + } + const values = parsed.values; + if (!command || values.help) { + console.log(USAGE); + return; + } + const dataDir = values["data-dir"]; + if (!dataDir) fail("--data-dir is required."); + + switch (command) { + case "serve": + return serve(dataDir, values); + case "migrate": { + if (!values.from) fail("--from is required."); + const dir = openDataDir(dataDir); + try { + const report = await migrate(dir, values.from, { + dryRun: values["dry-run"], + }); + console.log(JSON.stringify(summary(report), null, 2)); + if (report.status === "failed") process.exitCode = 1; + } finally { + dir.close(); + } + return; + } + case "backup": { + if (!values.out) fail("--out is required."); + const dir = openDataDir(dataDir); + const tmp = `${values.out}.partial`; + try { + const out = fs.createWriteStream(tmp, { flags: "wx" }); + const manifest = await writeBackup(dir, out); + await new Promise((resolve, reject) => { + out.on("error", reject); + out.end(resolve); + }); + fs.renameSync(tmp, values.out); + console.log( + JSON.stringify({ + backup: path.resolve(values.out), + projects: manifest.projects.length, + sources: manifest.sources.length, + assets: manifest.assets.length, + }) + ); + } catch (e) { + fs.rmSync(tmp, { force: true }); + throw e; + } finally { + dir.close(); + } + return; + } + case "restore": { + if (!values.from) fail("--from is required."); + const result = await restoreBackup(values.from, dataDir); + console.log( + JSON.stringify({ restored: result, dataDir: path.resolve(dataDir) }) + ); + return; + } + default: + fail(`Unknown command: ${command}\n\n${USAGE}`); + } +} + +function summary(report: Awaited>) { + const count = (o: Record) => + Object.fromEntries(Object.entries(o).map(([k, v]) => [k, v.length])); + return { + status: report.status, + error: report.error, + runDirectory: report.runDirectory, + projects: count(report.projects), + sources: count(report.sources), + files: count(report.files), + missingFiles: report.files.missing, + superseded: report.superseded.length, + rendererKeys: report.rendererKeys.length, + }; +} + +function serve(dataDir: string, values: Record) { + const token = process.env.KC_STORAGE_TOKEN; + if (!token || token.length < 32) { + fail( + "KC_STORAGE_TOKEN must be set to a random value of at least 32 characters." + ); + } + const port = Number(values.port); + if (!Number.isInteger(port) || port < 0 || port > 65535) { + fail("--port must be an integer from 0 to 65535."); + } + + const maxUploadMb = Number(values["max-upload-mb"]); + if (!Number.isFinite(maxUploadMb) || maxUploadMb <= 0) { + fail("--max-upload-mb must be a positive number."); + } + + const dir = openDataDir(dataDir); + if (dir.swept.length > 0) { + console.error( + `[kc-storage] removed ${dir.swept.length} unreferenced or temporary files` + ); + } + const server = createServer({ + dir, + token, + allowedOrigins: values["allow-origin"], + maxUploadBytes: Math.floor(maxUploadMb * 1024 * 1024), + }); + + let stopping = false; + const shutdown = (reason: string) => { + if (stopping) return; + stopping = true; + console.error(`[kc-storage] stopping (${reason})`); + server.close(() => { + dir.close(); + process.exit(0); + }); + server.closeAllConnections(); + setTimeout(() => { + dir.close(); + process.exit(1); + }, 5000).unref(); + }; + + process.on("SIGTERM", () => shutdown("SIGTERM")); + process.on("SIGINT", () => shutdown("SIGINT")); + if (values["exit-on-stdin-close"]) { + process.stdin.on("end", () => shutdown("stdin closed")); + process.stdin.on("error", () => shutdown("stdin error")); + process.stdin.resume(); + } + + server.on("error", (e) => { + console.error(`[kc-storage] server error: ${e.message}`); + dir.close(); + process.exit(1); + }); + + server.listen(port, "127.0.0.1", () => { + const { port: actual } = server.address() as AddressInfo; + console.log( + JSON.stringify({ + event: "ready", + url: `http://127.0.0.1:${actual}`, + version: VERSION, + dataDir: dir.root, + }) + ); + }); +} diff --git a/src/kc_storage/src/datadir.ts b/src/kc_storage/src/datadir.ts new file mode 100644 index 00000000..c4265ebd --- /dev/null +++ b/src/kc_storage/src/datadir.ts @@ -0,0 +1,103 @@ +/* + * A data directory: library.sqlite, assets/, migrations/, and a lock file. + * Only one process may open a data directory at a time. + */ + +import fs from "node:fs"; +import path from "node:path"; +import type { DatabaseSync } from "node:sqlite"; +import { AssetStore } from "./assets.ts"; +import { openDatabase } from "./db.ts"; +import { Library } from "./library.ts"; + +export interface DataDir { + root: string; + db: DatabaseSync; + library: Library; + assets: AssetStore; + /** Files removed at open because no record referred to them. */ + swept: string[]; + close(): void; +} + +export const DATABASE_FILE = "library.sqlite"; + +function alive(pid: number) { + try { + process.kill(pid, 0); + return true; + } catch (e: any) { + return e.code === "EPERM"; + } +} + +function acquireLock(root: string): () => void { + const lockPath = path.join(root, "storage.lock"); + const content = JSON.stringify({ + pid: process.pid, + startedAt: new Date().toISOString(), + }); + + for (let attempt = 0; attempt < 2; attempt++) { + try { + fs.writeFileSync(lockPath, content, { flag: "wx" }); + return () => { + try { + const held = JSON.parse(fs.readFileSync(lockPath, "utf8")); + if (held.pid === process.pid) { + fs.rmSync(lockPath, { force: true }); + } + } catch { + // The lock file is already gone + } + }; + } catch (e: any) { + if (e.code !== "EEXIST") { + throw e; + } + let holder = 0; + try { + holder = JSON.parse(fs.readFileSync(lockPath, "utf8")).pid; + } catch { + // Unreadable lock file: treat as stale + } + if (holder && holder !== process.pid && alive(holder)) { + throw new Error( + `The data directory ${root} is in use by process ${holder}.` + ); + } + fs.rmSync(lockPath, { force: true }); + } + } + throw new Error(`Could not lock the data directory ${root}.`); +} + +export function openDataDir(root: string): DataDir { + const resolved = path.resolve(root); + fs.mkdirSync(resolved, { recursive: true }); + const release = acquireLock(resolved); + + try { + const db = openDatabase(path.join(resolved, DATABASE_FILE)); + const library = new Library(db); + const assets = new AssetStore(path.join(resolved, "assets")); + + // Recover from an interrupted upload, migration, or restore + const swept = assets.sweep(library.assetIds()); + + return { + root: resolved, + db, + library, + assets, + swept, + close() { + db.close(); + release(); + }, + }; + } catch (e) { + release(); + throw e; + } +} diff --git a/src/kc_storage/src/db.ts b/src/kc_storage/src/db.ts new file mode 100644 index 00000000..364aed6a --- /dev/null +++ b/src/kc_storage/src/db.ts @@ -0,0 +1,126 @@ +/* + * SQLite database and versioned schema migrations. + */ + +import { DatabaseSync } from "node:sqlite"; + +interface Migration { + version: number; + sql: string; +} + +const MIGRATIONS: Migration[] = [ + { + version: 1, + sql: ` + CREATE TABLE projects ( + id TEXT PRIMARY KEY, + parent_id TEXT, + name TEXT NOT NULL, + data TEXT NOT NULL CHECK (json_valid(data)), + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ) STRICT; + + CREATE TABLE assets ( + id TEXT PRIMARY KEY, + sha256 TEXT NOT NULL, + size INTEGER NOT NULL, + media_type TEXT NOT NULL, + filename TEXT NOT NULL, + original_path TEXT, + created_at TEXT NOT NULL + ) STRICT; + + CREATE INDEX assets_sha256 ON assets (sha256); + + CREATE TABLE sources ( + id TEXT PRIMARY KEY, + project_id TEXT NOT NULL REFERENCES projects (id) ON DELETE CASCADE, + title TEXT NOT NULL, + ingest_type TEXT NOT NULL, + asset_id TEXT REFERENCES assets (id) ON DELETE RESTRICT, + position INTEGER NOT NULL, + data TEXT NOT NULL CHECK (json_valid(data)), + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ) STRICT; + + CREATE INDEX sources_project ON sources (project_id, position); + + CREATE TABLE import_runs ( + id TEXT PRIMARY KEY, + input_sha256 TEXT NOT NULL, + started_at TEXT NOT NULL, + finished_at TEXT, + status TEXT NOT NULL, + report TEXT + ) STRICT; + `, + }, +]; + +export const SCHEMA_VERSION = MIGRATIONS[MIGRATIONS.length - 1].version; + +export function now() { + return new Date().toISOString(); +} + +export function openDatabase(file: string): DatabaseSync { + const db = new DatabaseSync(file); + db.exec(` + PRAGMA journal_mode = WAL; + PRAGMA foreign_keys = ON; + PRAGMA busy_timeout = 5000; + `); + applyMigrations(db); + return db; +} + +export function schemaVersion(db: DatabaseSync): number { + const row = db + .prepare("SELECT MAX(version) AS version FROM schema_migrations") + .get() as { version: number | null }; + return row.version ?? 0; +} + +function applyMigrations(db: DatabaseSync) { + db.exec(` + CREATE TABLE IF NOT EXISTS schema_migrations ( + version INTEGER PRIMARY KEY, + applied_at TEXT NOT NULL + ) STRICT + `); + + const current = schemaVersion(db); + if (current > SCHEMA_VERSION) { + throw new Error( + `Database schema version ${current} is newer than this service supports (${SCHEMA_VERSION}).` + ); + } + + for (const migration of MIGRATIONS) { + if (migration.version <= current) { + continue; + } + transaction(db, () => { + db.exec(migration.sql); + db.prepare( + "INSERT INTO schema_migrations (version, applied_at) VALUES (?, ?)" + ).run(migration.version, now()); + }); + } +} + +/** Run fn in one write transaction. Roll back if it throws. */ +export function transaction(db: DatabaseSync, fn: () => T): T { + db.exec("BEGIN IMMEDIATE"); + try { + const result = fn(); + db.exec("COMMIT"); + return result; + } catch (e) { + db.exec("ROLLBACK"); + throw e; + } +} diff --git a/src/kc_storage/src/errors.ts b/src/kc_storage/src/errors.ts new file mode 100644 index 00000000..b4f33d59 --- /dev/null +++ b/src/kc_storage/src/errors.ts @@ -0,0 +1,26 @@ +import type { StorageErrorCode } from "../../kc_contracts/storage.ts"; + +/** An error with an HTTP status and a message that callers may see. */ +export class StorageError extends Error { + readonly status: number; + readonly code: StorageErrorCode; + + constructor(status: number, code: StorageErrorCode, message: string) { + super(message); + this.status = status; + this.code = code; + } +} + +export const badRequest = (message: string) => + new StorageError(400, "bad_request", message); +export const unauthorized = (message: string) => + new StorageError(401, "unauthorized", message); +export const forbidden = (message: string) => + new StorageError(403, "forbidden", message); +export const notFound = (message: string) => + new StorageError(404, "not_found", message); +export const conflict = (message: string) => + new StorageError(409, "conflict", message); +export const tooLarge = (message: string) => + new StorageError(413, "payload_too_large", message); diff --git a/src/kc_storage/src/http.ts b/src/kc_storage/src/http.ts new file mode 100644 index 00000000..f03c8db2 --- /dev/null +++ b/src/kc_storage/src/http.ts @@ -0,0 +1,302 @@ +/* + * HTTP API, version 1. See docs/storage-service.md. + */ + +import crypto from "node:crypto"; +import fs from "node:fs"; +import http, { type IncomingMessage, type ServerResponse } from "node:http"; +import type { AddressInfo } from "node:net"; +import { + ASSET_FILENAME_HEADER, + ASSET_ORIGINAL_PATH_HEADER, + type ErrorResponse, + type HealthResponse, +} from "../../kc_contracts/storage.ts"; +import { writeBackup } from "./backup.ts"; +import type { DataDir } from "./datadir.ts"; +import { now, SCHEMA_VERSION } from "./db.ts"; +import { + badRequest, + forbidden, + notFound, + StorageError, + tooLarge, + unauthorized, +} from "./errors.ts"; +import { + validFilename, + validId, + validMediaType, + validOriginalPath, + validProject, + validSource, +} from "./validate.ts"; +import { VERSION } from "./version.ts"; + +export interface ServerOptions { + dir: DataDir; + token: string; + /** Value of the Origin header that may use the API. "null" for file:// pages. */ + allowedOrigins: string[]; + maxJsonBytes?: number; + maxUploadBytes?: number; +} + +const ALLOWED_HEADERS = [ + "authorization", + "content-type", + ASSET_FILENAME_HEADER, + ASSET_ORIGINAL_PATH_HEADER, +].join(", "); + +function send(res: ServerResponse, status: number, body?: unknown) { + if (body === undefined) { + res.writeHead(status).end(); + return; + } + const json = JSON.stringify(body); + res.writeHead(status, { + "Content-Type": "application/json; charset=utf-8", + "Content-Length": Buffer.byteLength(json), + "Cache-Control": "no-store", + }); + res.end(json); +} + +async function readJson(req: IncomingMessage, limit: number): Promise { + const type = req.headers["content-type"] ?? ""; + if (!type.startsWith("application/json")) { + throw badRequest("Content-Type must be application/json."); + } + const chunks: Buffer[] = []; + let size = 0; + for await (const chunk of req) { + size += chunk.length; + if (size > limit) { + throw tooLarge(`The request body is larger than ${limit} bytes.`); + } + chunks.push(chunk); + } + try { + return JSON.parse(Buffer.concat(chunks).toString("utf8")); + } catch { + throw badRequest("The request body is not valid JSON."); + } +} + +function tokenMatches(header: string | undefined, token: string) { + const expected = Buffer.from(`Bearer ${token}`); + const actual = Buffer.from(header ?? ""); + return ( + actual.length === expected.length && + crypto.timingSafeEqual(actual, expected) + ); +} + +export function createServer(options: ServerOptions): http.Server { + const { dir, token, allowedOrigins } = options; + const { library, assets } = dir; + const maxJson = options.maxJsonBytes ?? 10 * 1024 * 1024; + const maxUpload = options.maxUploadBytes ?? 2 * 1024 ** 3; + + async function route(req: IncomingMessage, res: ServerResponse) { + const url = new URL(req.url ?? "/", "http://localhost"); + let parts: string[]; + try { + parts = url.pathname.split("/").filter(Boolean).map(decodeURIComponent); + } catch { + throw badRequest("The path is not valid percent-encoding."); + } + const method = req.method ?? "GET"; + + if (method === "GET" && url.pathname === "/health") { + const body: HealthResponse = { + status: "ok", + service: "kc-storage", + version: VERSION, + schemaVersion: SCHEMA_VERSION, + }; + return send(res, 200, body); + } + + if (parts[0] !== "v1") { + throw notFound(`No route for ${method} ${url.pathname}.`); + } + if (!tokenMatches(req.headers.authorization, token)) { + throw unauthorized("Missing or invalid bearer token."); + } + + const [, resource, id, sub] = parts; + + if (resource === "projects" && !sub) { + if (method === "GET" && !id) + return send(res, 200, { projects: library.listProjects() }); + if (method === "PUT" && id) { + const record = validProject(id, await readJson(req, maxJson)); + const result = library.putProject(record); + return send(res, result === "created" ? 201 : 200, { + project: library.getProject(record.id), + }); + } + if (method === "DELETE" && id) { + if (!library.deleteProject(validId(id))) + throw notFound(`Project ${id} does not exist.`); + return send(res, 204); + } + } + + if (resource === "sources" && !sub) { + if (method === "GET" && !id) { + const projectId = url.searchParams.get("projectId"); + return send(res, 200, { + sources: library.listSources( + projectId ? validId(projectId, "projectId") : undefined + ), + }); + } + if (method === "PUT" && id) { + const record = validSource(id, await readJson(req, maxJson)); + const result = library.putSource(record); + return send(res, result === "created" ? 201 : 200, { + source: library.getSource(record.id), + }); + } + if (method === "DELETE" && id) { + if (!library.deleteSource(validId(id))) + throw notFound(`Source ${id} does not exist.`); + return send(res, 204); + } + } + + if (resource === "assets") { + if (method === "POST" && !id) { + const filename = validFilename( + req.headers[ASSET_FILENAME_HEADER] as string | undefined + ); + const originalPath = validOriginalPath( + req.headers[ASSET_ORIGINAL_PATH_HEADER] as string | undefined + ); + const mediaType = validMediaType(req.headers["content-type"]); + const declared = Number(req.headers["content-length"] ?? 0); + if (declared > maxUpload) { + throw tooLarge(`The file is larger than ${maxUpload} bytes.`); + } + + const staged = await assets.stage(req, maxUpload); + const assetId = crypto.randomUUID(); + try { + assets.commit(staged, assetId); + library.insertAsset({ + id: assetId, + sha256: staged.sha256, + size: staged.size, + mediaType, + filename, + originalPath, + createdAt: now(), + }); + } catch (e) { + assets.discard(staged); + assets.remove(assetId); + throw e; + } + return send(res, 201, { asset: library.getAsset(assetId) }); + } + + if (method === "GET" && id) { + const asset = library.getAsset(id); + if (!asset) throw notFound(`Asset ${id} does not exist.`); + if (!sub) return send(res, 200, { asset }); + if (sub === "content") { + const file = assets.pathFor(asset.id); + if (!fs.existsSync(file)) { + throw new StorageError( + 500, + "internal", + `Managed file for asset ${asset.id} is missing.` + ); + } + res.writeHead(200, { + "Content-Type": asset.mediaType, + "Content-Length": asset.size, + "Content-Disposition": `inline; filename*=UTF-8''${encodeURIComponent( + asset.filename + )}`, + "X-Content-Type-Options": "nosniff", + "Cache-Control": "no-store", + }); + fs.createReadStream(file).pipe(res); + return; + } + } + } + + if (resource === "backup" && !id && method === "GET") { + const date = new Date().toISOString().slice(0, 10); + res.writeHead(200, { + "Content-Type": "application/x-tar", + "Content-Disposition": `attachment; filename="knowledge-library-${date}.tar"`, + "Cache-Control": "no-store", + }); + await writeBackup(dir, res); + res.end(); + return; + } + + throw notFound(`No route for ${method} ${url.pathname}.`); + } + + const server = http.createServer(async (req, res) => { + try { + // Reject requests that do not address this server by its loopback name + const port = (server.address() as AddressInfo).port; + const host = req.headers.host; + if (host !== `127.0.0.1:${port}` && host !== `localhost:${port}`) { + throw forbidden("Invalid Host header."); + } + + // CORS lets the configured page read responses. It is not authentication. + const origin = req.headers.origin; + if (origin !== undefined) { + if (!allowedOrigins.includes(origin)) { + throw forbidden("Origin not allowed."); + } + res.setHeader("Access-Control-Allow-Origin", origin); + res.setHeader("Vary", "Origin"); + res.setHeader("Access-Control-Expose-Headers", "content-disposition"); + } + if (req.method === "OPTIONS") { + res.setHeader("Access-Control-Allow-Methods", "GET, PUT, POST, DELETE"); + res.setHeader("Access-Control-Allow-Headers", ALLOWED_HEADERS); + res.setHeader("Access-Control-Max-Age", "600"); + return send(res, 204); + } + + await route(req, res); + } catch (e: any) { + const error = + e instanceof StorageError + ? e + : new StorageError( + 500, + "internal", + "Internal error. See the service log." + ); + if (!(e instanceof StorageError)) { + console.error( + `[kc-storage] ${req.method} ${req.url}: ${e?.stack ?? e}` + ); + } + if (res.headersSent) { + res.destroy(); + return; + } + req.resume(); + const body: ErrorResponse = { + error: { code: error.code, message: error.message }, + }; + send(res, error.status, body); + } + }); + return server; +} diff --git a/src/kc_storage/src/library.ts b/src/kc_storage/src/library.ts new file mode 100644 index 00000000..2f7dbbe9 --- /dev/null +++ b/src/kc_storage/src/library.ts @@ -0,0 +1,291 @@ +/* + * Projects, sources, and asset records. All SQL for these tables is here. + */ + +import type { DatabaseSync } from "node:sqlite"; +import type { + AssetRecord, + ProjectRecord, + SourceRecord, +} from "../../kc_contracts/storage.ts"; +import { conflict } from "./errors.ts"; +import { now } from "./db.ts"; + +export type WriteResult = "created" | "updated" | "unchanged"; + +interface ProjectRow { + id: string; + parent_id: string | null; + name: string; + data: string; + created_at: string; + updated_at: string; +} + +interface SourceRow { + id: string; + project_id: string; + title: string; + ingest_type: string; + asset_id: string | null; + position: number; + data: string; + created_at: string; + updated_at: string; +} + +interface AssetRow { + id: string; + sha256: string; + size: number; + media_type: string; + filename: string; + original_path: string | null; + created_at: string; +} + +const toProject = (r: ProjectRow): ProjectRecord => ({ + id: r.id, + parentId: r.parent_id, + name: r.name, + data: JSON.parse(r.data), + createdAt: r.created_at, + updatedAt: r.updated_at, +}); + +const toSource = (r: SourceRow): SourceRecord => ({ + id: r.id, + projectId: r.project_id, + title: r.title, + ingestType: r.ingest_type, + assetId: r.asset_id, + position: r.position, + data: JSON.parse(r.data), + createdAt: r.created_at, + updatedAt: r.updated_at, +}); + +const toAsset = (r: AssetRow): AssetRecord => ({ + id: r.id, + sha256: r.sha256, + size: r.size, + mediaType: r.media_type, + filename: r.filename, + originalPath: r.original_path, + createdAt: r.created_at, +}); + +export class Library { + private db: DatabaseSync; + + constructor(db: DatabaseSync) { + this.db = db; + } + + listProjects(): ProjectRecord[] { + const rows = this.db + .prepare("SELECT * FROM projects ORDER BY created_at, id") + .all() as unknown as ProjectRow[]; + return rows.map(toProject); + } + + getProject(id: string): ProjectRecord | undefined { + const row = this.db.prepare("SELECT * FROM projects WHERE id = ?").get(id); + return row ? toProject(row as unknown as ProjectRow) : undefined; + } + + putProject(p: ProjectRecord, createdAt?: string): WriteResult { + const data = JSON.stringify(p.data); + const old = this.db + .prepare("SELECT * FROM projects WHERE id = ?") + .get(p.id) as unknown as ProjectRow | undefined; + + if (!old) { + const time = now(); + this.db + .prepare( + "INSERT INTO projects (id, parent_id, name, data, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?)" + ) + .run(p.id, p.parentId, p.name, data, createdAt ?? time, time); + return "created"; + } + if ( + old.parent_id === p.parentId && + old.name === p.name && + old.data === data + ) { + return "unchanged"; + } + this.db + .prepare( + "UPDATE projects SET parent_id = ?, name = ?, data = ?, updated_at = ? WHERE id = ?" + ) + .run(p.parentId, p.name, data, now(), p.id); + return "updated"; + } + + deleteProject(id: string): boolean { + return ( + this.db.prepare("DELETE FROM projects WHERE id = ?").run(id).changes > 0 + ); + } + + listSources(projectId?: string): SourceRecord[] { + const rows = (projectId + ? this.db + .prepare( + "SELECT * FROM sources WHERE project_id = ? ORDER BY position, id" + ) + .all(projectId) + : this.db + .prepare("SELECT * FROM sources ORDER BY project_id, position, id") + .all()) as unknown as SourceRow[]; + return rows.map(toSource); + } + + getSource(id: string): SourceRecord | undefined { + const row = this.db.prepare("SELECT * FROM sources WHERE id = ?").get(id); + return row ? toSource(row as unknown as SourceRow) : undefined; + } + + putSource(s: SourceRecord, createdAt?: string): WriteResult { + if (!this.getProject(s.projectId)) { + throw conflict(`Project ${s.projectId} does not exist.`); + } + if (s.assetId && !this.getAsset(s.assetId)) { + throw conflict(`Asset ${s.assetId} does not exist.`); + } + + const data = JSON.stringify(s.data); + const old = this.db + .prepare("SELECT * FROM sources WHERE id = ?") + .get(s.id) as unknown as SourceRow | undefined; + + // New sources and moved sources go to the end of their project + const position = + old && old.project_id === s.projectId + ? old.position + : this.nextPosition(s.projectId); + + if (!old) { + const time = now(); + this.db + .prepare( + `INSERT INTO sources (id, project_id, title, ingest_type, asset_id, position, data, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)` + ) + .run( + s.id, + s.projectId, + s.title, + s.ingestType, + s.assetId, + position, + data, + createdAt ?? time, + time + ); + return "created"; + } + if ( + old.project_id === s.projectId && + old.title === s.title && + old.ingest_type === s.ingestType && + old.asset_id === s.assetId && + old.data === data + ) { + return "unchanged"; + } + this.db + .prepare( + `UPDATE sources SET project_id = ?, title = ?, ingest_type = ?, asset_id = ?, position = ?, data = ?, updated_at = ? + WHERE id = ?` + ) + .run( + s.projectId, + s.title, + s.ingestType, + s.assetId, + position, + data, + now(), + s.id + ); + return "updated"; + } + + deleteSource(id: string): boolean { + return ( + this.db.prepare("DELETE FROM sources WHERE id = ?").run(id).changes > 0 + ); + } + + private nextPosition(projectId: string): number { + const row = this.db + .prepare("SELECT MAX(position) AS p FROM sources WHERE project_id = ?") + .get(projectId) as { p: number | null }; + return (row.p ?? -1) + 1; + } + + listAssets(): AssetRecord[] { + const rows = this.db + .prepare("SELECT * FROM assets ORDER BY created_at, id") + .all() as unknown as AssetRow[]; + return rows.map(toAsset); + } + + getAsset(id: string): AssetRecord | undefined { + const row = this.db.prepare("SELECT * FROM assets WHERE id = ?").get(id); + return row ? toAsset(row as unknown as AssetRow) : undefined; + } + + /** An existing asset with the same content and origin, if any. */ + findAsset( + sha256: string, + originalPath: string | null + ): AssetRecord | undefined { + const row = this.db + .prepare( + "SELECT * FROM assets WHERE sha256 = ? AND original_path IS ? ORDER BY created_at LIMIT 1" + ) + .get(sha256, originalPath); + return row ? toAsset(row as unknown as AssetRow) : undefined; + } + + insertAsset(a: AssetRecord) { + this.db + .prepare( + `INSERT INTO assets (id, sha256, size, media_type, filename, original_path, created_at) + VALUES (?, ?, ?, ?, ?, ?, ?)` + ) + .run( + a.id, + a.sha256, + a.size, + a.mediaType, + a.filename, + a.originalPath, + a.createdAt + ); + } + + deleteAsset(id: string) { + this.db.prepare("DELETE FROM assets WHERE id = ?").run(id); + } + + assetIds(): Set { + const rows = this.db.prepare("SELECT id FROM assets").all() as { + id: string; + }[]; + return new Set(rows.map((r) => r.id)); + } + + isEmpty(): boolean { + const row = this.db + .prepare( + "SELECT (SELECT COUNT(*) FROM projects) + (SELECT COUNT(*) FROM sources) + (SELECT COUNT(*) FROM assets) AS n" + ) + .get() as { n: number }; + return row.n === 0; + } +} diff --git a/src/kc_storage/src/main.ts b/src/kc_storage/src/main.ts new file mode 100644 index 00000000..317e2ccc --- /dev/null +++ b/src/kc_storage/src/main.ts @@ -0,0 +1,19 @@ +/* + * Entry point. Checks the Node.js version before loading node:sqlite. + */ + +const major = Number(process.versions.node.split(".")[0]); +if (major < 24) { + console.error( + `kc-storage needs Node.js 24 or later (found ${process.versions.node}). Set KC_NODE to a newer node binary.` + ); + process.exit(2); +} + +const { main } = await import("./cli.ts"); +try { + await main(process.argv.slice(2)); +} catch (e: any) { + console.error(`kc-storage: ${e?.message ?? e}`); + process.exit(1); +} diff --git a/src/kc_storage/src/migrate.ts b/src/kc_storage/src/migrate.ts new file mode 100644 index 00000000..827ad48b --- /dev/null +++ b/src/kc_storage/src/migrate.ts @@ -0,0 +1,434 @@ +/* + * Controlled import of renderer data (a Settings > Backup file, + * format "knowledge-backup" version 1) into the library. + * + * - The input file is never changed. A copy goes into the run directory. + * - The whole input is parsed and validated before any write. + * - The database is copied (VACUUM INTO) before any write. + * - All records are written in one transaction. Copied files are removed + * if the transaction fails. + * - IDs are kept. A repeated run updates or skips records. + */ + +import crypto from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import { + DERIVED_SOURCE_FIELDS, + projectToRecord, + sourceToRecord, +} from "../../kc_contracts/mapping.ts"; +import type { + ProjectRecord, + SourceRecord, +} from "../../kc_contracts/storage.ts"; +import type { StagedFile } from "./assets.ts"; +import type { DataDir } from "./datadir.ts"; +import { now, transaction } from "./db.ts"; +import { StorageError } from "./errors.ts"; +import { validProject, validSource } from "./validate.ts"; + +const INPUT_FORMAT = "knowledge-backup"; +const MAX_FILE_BYTES = 2 * 1024 ** 3; + +interface Item { + id: string; + title?: string; + reason?: string; +} + +export interface MigrationReport { + runId: string; + status: "completed" | "failed" | "dry-run"; + startedAt: string; + finishedAt?: string; + input: { path: string; sha256: string }; + runDirectory: string | null; + projects: { + imported: Item[]; + updated: Item[]; + unchanged: Item[]; + failed: Item[]; + }; + sources: { + imported: Item[]; + updated: Item[]; + unchanged: Item[]; + skipped: Item[]; + failed: Item[]; + }; + files: { + copied: { sourceId: string; path: string; assetId: string }[]; + reused: { sourceId: string; path: string; assetId: string }[]; + kept: { sourceId: string; assetId: string }[]; + missing: { sourceId: string; title: string; path: string | null }[]; + }; + /** ks- records that differ from the copy inside the project. */ + superseded: { key: string; differentFields: string[] }[]; + /** Derived source fields that were not stored, by field name. */ + droppedDerivedFields: Record; + /** Renderer keys that stay in localStorage (chat history, preferences). */ + rendererKeys: string[]; + rollback: { snapshot: string | null }; + error?: string; +} + +interface PlannedSource { + record: SourceRecord; + filePath: string | null; +} + +function isProjectValue(value: any, key: string) { + return ( + value && + typeof value === "object" && + value.id?.value === key && + typeof value.name === "string" && + Array.isArray(value.knowledgeSource) + ); +} + +function parse(text: string) { + try { + return JSON.parse(text); + } catch { + throw new Error("The input is not valid JSON."); + } +} + +function filePathOf(source: any): string | null { + const candidate = source?.reference?.source?.file?.path ?? source?.accessLink; + return typeof candidate === "string" && path.isAbsolute(candidate) + ? candidate + : null; +} + +function differentFields(a: any, b: any): string[] { + const keys = new Set([...Object.keys(a ?? {}), ...Object.keys(b ?? {})]); + return [...keys].filter( + (k) => JSON.stringify(a?.[k]) !== JSON.stringify(b?.[k]) + ); +} + +export async function migrate( + dir: DataDir, + inputPath: string, + options: { dryRun?: boolean } = {} +): Promise { + const text = fs.readFileSync(inputPath, "utf8"); + const inputSha = crypto.createHash("sha256").update(text).digest("hex"); + const runId = `${now().replace(/[:.]/g, "-")}-${crypto + .randomBytes(3) + .toString("hex")}`; + + const report: MigrationReport = { + runId, + status: options.dryRun ? "dry-run" : "completed", + startedAt: now(), + input: { path: path.resolve(inputPath), sha256: inputSha }, + runDirectory: null, + projects: { imported: [], updated: [], unchanged: [], failed: [] }, + sources: { + imported: [], + updated: [], + unchanged: [], + skipped: [], + failed: [], + }, + files: { copied: [], reused: [], kept: [], missing: [] }, + superseded: [], + droppedDerivedFields: {}, + rendererKeys: [], + rollback: { snapshot: null }, + }; + + // 1. Validate the input as a whole. Nothing is written yet. + const input = parse(text); + if ( + input?.format !== INPUT_FORMAT || + input.version !== 1 || + typeof input.data !== "object" + ) { + throw new Error(`The input is not a ${INPUT_FORMAT} version 1 file.`); + } + const data: Record = input.data; + + const projects: ProjectRecord[] = []; + const projectIds = new Set(); + const planned: PlannedSource[] = []; + const plannedIds = new Set(); + const separate = new Map(); + const consumed = new Set(); + + for (const [key, raw] of Object.entries(data)) { + if (typeof raw !== "string") { + throw new Error(`The value of ${key} is not text.`); + } + let value: any; + try { + value = JSON.parse(raw); + } catch { + continue; + } + // ks- holds a separate source copy. Other ks- keys are UI preferences. + if (key.startsWith("ks-") && value?.id?.value === key.slice(3)) { + separate.set(key, raw); + continue; + } + if (!isProjectValue(value, key)) { + continue; + } + consumed.add(key); + try { + const mapped = projectToRecord(value); + const record = validProject(mapped.id, mapped); + projects.push(record); + projectIds.add(record.id); + for (const ks of value.knowledgeSource) { + planSource(ks, record.id); + } + } catch (e: any) { + report.projects.failed.push({ + id: key, + title: value.name, + reason: e.message, + }); + for (const ks of value.knowledgeSource) { + report.sources.skipped.push({ + id: ks?.id?.value ?? "(no id)", + title: ks?.title, + reason: `Project ${key} failed validation.`, + }); + } + } + } + consumed.add("kc-projects"); + + function planSource(ks: any, projectId: string) { + const id = ks?.id?.value ?? "(no id)"; + try { + for (const field of DERIVED_SOURCE_FIELDS) { + if (ks?.[field] !== undefined) { + report.droppedDerivedFields[field] = + (report.droppedDerivedFields[field] ?? 0) + 1; + } + } + const mapped = sourceToRecord(ks, projectId); + const record = validSource(mapped.id, mapped); + if (plannedIds.has(record.id)) { + report.sources.skipped.push({ + id, + title: ks.title, + reason: "Duplicate source ID in input.", + }); + return; + } + plannedIds.add(record.id); + planned.push({ + record, + filePath: record.ingestType === "file" ? filePathOf(ks) : null, + }); + } catch (e: any) { + report.sources.failed.push({ id, title: ks?.title, reason: e.message }); + } + } + + // Separate ks- records: superseded by the embedded copy, or standalone + for (const [key, raw] of separate) { + consumed.add(key); + let ks: any; + try { + ks = JSON.parse(raw); + } catch { + report.sources.failed.push({ id: key, reason: "Not valid JSON." }); + continue; + } + const id = key.slice(3); + const embedded = planned.find((p) => p.record.id === id); + if (embedded) { + const fields = differentFields( + sourceToRecord(ks, embedded.record.projectId).data, + embedded.record.data + ); + if (fields.length > 0) { + report.superseded.push({ key, differentFields: fields }); + } + continue; + } + const projectId = ks?.associatedProject?.value; + if (typeof projectId === "string" && projectIds.has(projectId)) { + planSource(ks, projectId); + } else { + report.sources.skipped.push({ + id, + title: ks?.title, + reason: `Separate record ${key} has no project in the input.`, + }); + } + } + + report.rendererKeys = Object.keys(data) + .filter((k) => !consumed.has(k)) + .sort(); + + // 2. Find the files. A missing file is reported, and the source is kept. + for (const p of planned) { + if (p.record.ingestType !== "file") continue; + const existing = dir.library.getSource(p.record.id); + const usable = + p.filePath && + fs.existsSync(p.filePath) && + fs.statSync(p.filePath).isFile(); + if (!usable) { + if (existing?.assetId) { + p.record.assetId = existing.assetId; + report.files.kept.push({ + sourceId: p.record.id, + assetId: existing.assetId, + }); + } else { + report.files.missing.push({ + sourceId: p.record.id, + title: p.record.title, + path: p.filePath, + }); + } + p.filePath = null; + } + } + + if (options.dryRun) { + for (const p of projects) + report.projects.imported.push({ id: p.id, title: p.name }); + for (const s of planned) + report.sources.imported.push({ id: s.record.id, title: s.record.title }); + report.finishedAt = now(); + return report; + } + + // 3. Prepare the run directory and a database snapshot for rollback + const runDir = path.join(dir.root, "migrations", runId); + fs.mkdirSync(runDir, { recursive: true }); + report.runDirectory = runDir; + fs.copyFileSync(inputPath, path.join(runDir, "input.json")); + const snapshot = path.join(runDir, "pre-migration.sqlite"); + dir.db.exec(`VACUUM INTO '${snapshot.replace(/'/g, "''")}'`); + report.rollback.snapshot = snapshot; + + // 4. Stage file copies, then write everything in one transaction + const staged: { sourceId: string; path: string; file: StagedFile }[] = []; + const committed: string[] = []; + try { + for (const p of planned) { + if (p.filePath) { + const file = await dir.assets.stage( + fs.createReadStream(p.filePath), + MAX_FILE_BYTES + ); + staged.push({ sourceId: p.record.id, path: p.filePath, file }); + } + } + + transaction(dir.db, () => { + for (const p of projects) { + const created = p.data["dateCreated"]; + const result = dir.library.putProject( + p, + typeof created === "string" ? created : undefined + ); + const item = { id: p.id, title: p.name }; + if (result === "created") report.projects.imported.push(item); + else if (result === "updated") report.projects.updated.push(item); + else report.projects.unchanged.push(item); + } + + for (const p of planned) { + const stage = staged.find((s) => s.sourceId === p.record.id); + if (stage) { + const reuse = dir.library.findAsset(stage.file.sha256, stage.path); + if (reuse) { + p.record.assetId = reuse.id; + report.files.reused.push({ + sourceId: p.record.id, + path: stage.path, + assetId: reuse.id, + }); + } else { + const id = crypto.randomUUID(); + const filename = path.basename(stage.path); + dir.library.insertAsset({ + id, + sha256: stage.file.sha256, + size: stage.file.size, + mediaType: + (p.record.data as any)?.reference?.source?.file?.type || + "application/octet-stream", + filename, + originalPath: stage.path, + createdAt: now(), + }); + dir.assets.commit(stage.file, id); + committed.push(id); + p.record.assetId = id; + report.files.copied.push({ + sourceId: p.record.id, + path: stage.path, + assetId: id, + }); + } + } + + const result = dir.library.putSource(p.record); + const item = { id: p.record.id, title: p.record.title }; + if (result === "created") report.sources.imported.push(item); + else if (result === "updated") report.sources.updated.push(item); + else report.sources.unchanged.push(item); + } + + if (process.env.KC_STORAGE_TEST_FAULT === "migrate-before-commit") { + throw new Error("Injected test fault before commit."); + } + }); + } catch (e: any) { + for (const id of committed) dir.assets.remove(id); + report.status = "failed"; + report.error = + e instanceof StorageError || e instanceof Error ? e.message : String(e); + // Nothing was written. Keep validation results, clear write results. + for (const list of [ + report.projects.imported, + report.projects.updated, + report.projects.unchanged, + ]) + list.length = 0; + for (const list of [ + report.sources.imported, + report.sources.updated, + report.sources.unchanged, + ]) + list.length = 0; + report.files.copied.length = 0; + report.files.reused.length = 0; + } finally { + for (const s of staged) dir.assets.discard(s.file); + } + + report.finishedAt = now(); + dir.db + .prepare( + "INSERT INTO import_runs (id, input_sha256, started_at, finished_at, status, report) VALUES (?, ?, ?, ?, ?, ?)" + ) + .run( + runId, + inputSha, + report.startedAt, + report.finishedAt, + report.status, + JSON.stringify(report) + ); + fs.writeFileSync( + path.join(runDir, "report.json"), + JSON.stringify(report, null, 2) + ); + return report; +} diff --git a/src/kc_storage/src/tar.ts b/src/kc_storage/src/tar.ts new file mode 100644 index 00000000..cba222e8 --- /dev/null +++ b/src/kc_storage/src/tar.ts @@ -0,0 +1,142 @@ +/* + * Minimal POSIX ustar writer and reader for backups. Regular files only. + */ + +import fs from "node:fs"; +import { once } from "node:events"; +import { Readable, type Writable } from "node:stream"; + +const BLOCK = 512; + +function header(name: string, size: number, mtime: Date): Buffer { + if (Buffer.byteLength(name) > 100) { + throw new Error(`Tar entry name is too long: ${name}`); + } + const h = Buffer.alloc(BLOCK); + const octal = (value: number, length: number) => + value.toString(8).padStart(length - 1, "0") + "\0"; + + h.write(name, 0, 100, "utf8"); + h.write(octal(0o644, 8), 100, 8, "ascii"); + h.write(octal(0, 8), 108, 8, "ascii"); + h.write(octal(0, 8), 116, 8, "ascii"); + h.write(octal(size, 12), 124, 12, "ascii"); + h.write(octal(Math.floor(mtime.getTime() / 1000), 12), 136, 12, "ascii"); + h.write(" ", 148, 8, "ascii"); + h.write("0", 156, 1, "ascii"); + h.write("ustar\0", 257, 6, "ascii"); + h.write("00", 263, 2, "ascii"); + + let sum = 0; + for (const byte of h) sum += byte; + h.write(sum.toString(8).padStart(6, "0") + "\0 ", 148, 8, "ascii"); + return h; +} + +function padding(size: number) { + return Buffer.alloc((BLOCK - (size % BLOCK)) % BLOCK); +} + +export class TarWriter { + private out: Writable; + + constructor(out: Writable) { + this.out = out; + } + + private async write(chunk: Buffer) { + if (!this.out.write(chunk)) { + await once(this.out, "drain"); + } + } + + async addBuffer(name: string, data: Buffer) { + await this.write(header(name, data.length, new Date())); + await this.write(data); + await this.write(padding(data.length)); + } + + async addFile(name: string, file: string) { + const { size, mtime } = fs.statSync(file); + await this.write(header(name, size, mtime)); + let written = 0; + for await (const chunk of fs.createReadStream(file)) { + written += chunk.length; + await this.write(chunk); + } + if (written !== size) { + throw new Error(`File changed while it was being archived: ${file}`); + } + await this.write(padding(size)); + } + + async finish() { + await this.write(Buffer.alloc(BLOCK * 2)); + } +} + +export interface TarEntry { + name: string; + offset: number; + size: number; +} + +/** Read the entry index of a tar file. Validates every header checksum. */ +export function readTarIndex(file: string): Map { + const fd = fs.openSync(file, "r"); + const entries = new Map(); + try { + const total = fs.fstatSync(fd).size; + const h = Buffer.alloc(BLOCK); + let offset = 0; + + while (offset + BLOCK <= total) { + fs.readSync(fd, h, 0, BLOCK, offset); + if (h.every((b) => b === 0)) { + return entries; + } + + const stored = parseInt(h.toString("ascii", 148, 156).trim(), 8); + let sum = 0; + for (let i = 0; i < BLOCK; i++) sum += i >= 148 && i < 156 ? 32 : h[i]; + if (stored !== sum) { + throw new Error(`Invalid tar header checksum at offset ${offset}.`); + } + + const name = h.toString("utf8", 0, 100).replace(/\0.*$/s, ""); + const size = parseInt(h.toString("ascii", 124, 136).trim(), 8); + const type = h.toString("ascii", 156, 157); + if (!Number.isFinite(size) || size < 0 || offset + BLOCK + size > total) { + throw new Error(`Invalid tar entry size for ${name}.`); + } + if (type === "0" || type === "\0") { + entries.set(name, { name, offset: offset + BLOCK, size }); + } + offset += BLOCK + Math.ceil(size / BLOCK) * BLOCK; + } + throw new Error("The tar file ends without an end-of-archive marker."); + } finally { + fs.closeSync(fd); + } +} + +export function readTarEntry(file: string, entry: TarEntry): Buffer { + const buf = Buffer.alloc(entry.size); + const fd = fs.openSync(file, "r"); + try { + fs.readSync(fd, buf, 0, entry.size, entry.offset); + } finally { + fs.closeSync(fd); + } + return buf; +} + +export function streamTarEntry(file: string, entry: TarEntry) { + if (entry.size === 0) { + return Readable.from([]); + } + return fs.createReadStream(file, { + start: entry.offset, + end: entry.offset + entry.size - 1, + }); +} diff --git a/src/kc_storage/src/validate.ts b/src/kc_storage/src/validate.ts new file mode 100644 index 00000000..7cd99cc1 --- /dev/null +++ b/src/kc_storage/src/validate.ts @@ -0,0 +1,167 @@ +/* + * Request validation. Every value from a client passes through here. + */ + +import type { + JsonObject, + ProjectRecord, + SourceRecord, +} from "../../kc_contracts/storage.ts"; +import { badRequest } from "./errors.ts"; + +const ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9_-]{0,127}$/; +const ASSET_ID_PATTERN = + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; +const MEDIA_TYPE_PATTERN = + /^[a-z0-9][a-z0-9!#$&^_.+-]*\/[a-z0-9][a-z0-9!#$&^_.+-]*$/; + +export const INGEST_TYPES = [ + "file", + "website", + "generic", + "topic", + "search", + "note", + "message", +]; + +export function validId(value: unknown, field = "id"): string { + if (typeof value !== "string" || !ID_PATTERN.test(value)) { + throw badRequest( + `${field} must be 1-128 letters, digits, "-" or "_", starting with a letter or digit.` + ); + } + return value; +} + +export function validAssetId(value: unknown): string { + if (typeof value !== "string" || !ASSET_ID_PATTERN.test(value)) { + throw badRequest("assetId must be a lowercase UUID v4."); + } + return value; +} + +function text(value: unknown, field: string, max: number, allowEmpty: boolean) { + if (typeof value !== "string") { + throw badRequest(`${field} must be a string.`); + } + if (!allowEmpty && value.trim().length === 0) { + throw badRequest(`${field} must not be empty.`); + } + if (value.length > max || /[\u0000]/.test(value)) { + throw badRequest(`${field} must be at most ${max} characters with no NUL.`); + } + return value; +} + +function object(value: unknown, field: string): JsonObject { + if (!value || typeof value !== "object" || Array.isArray(value)) { + throw badRequest(`${field} must be a JSON object.`); + } + return value as JsonObject; +} + +function onlyKeys(body: JsonObject, allowed: string[]) { + const extra = Object.keys(body).filter((k) => !allowed.includes(k)); + if (extra.length > 0) { + throw badRequest(`Unknown fields: ${extra.join(", ")}. Put them in data.`); + } +} + +export function validProject(pathId: string, input: unknown): ProjectRecord { + const body = object(input, "body"); + onlyKeys(body, ["id", "parentId", "name", "data", "createdAt", "updatedAt"]); + const id = validId(pathId); + if (body["id"] !== undefined && body["id"] !== id) { + throw badRequest("id in the body does not match the path."); + } + const parent = body["parentId"]; + return { + id, + parentId: + parent === null || parent === undefined || parent === "" + ? null + : validId(parent, "parentId"), + name: text(body["name"], "name", 1024, false), + data: object(body["data"], "data"), + }; +} + +export function validSource(pathId: string, input: unknown): SourceRecord { + const body = object(input, "body"); + onlyKeys(body, [ + "id", + "projectId", + "title", + "ingestType", + "assetId", + "position", + "data", + "createdAt", + "updatedAt", + ]); + const id = validId(pathId); + if (body["id"] !== undefined && body["id"] !== id) { + throw badRequest("id in the body does not match the path."); + } + const ingestType = body["ingestType"]; + if (typeof ingestType !== "string" || !INGEST_TYPES.includes(ingestType)) { + throw badRequest(`ingestType must be one of: ${INGEST_TYPES.join(", ")}.`); + } + const asset = body["assetId"]; + return { + id, + projectId: validId(body["projectId"], "projectId"), + title: text(body["title"] ?? "", "title", 4096, true), + ingestType, + assetId: asset === null || asset === undefined ? null : validAssetId(asset), + data: object(body["data"], "data"), + }; +} + +/** A display filename. Path separators and control characters are removed. */ +export function validFilename(raw: string | undefined): string { + if (!raw) { + throw badRequest("The filename header is required."); + } + let name: string; + try { + name = decodeURIComponent(raw); + } catch { + throw badRequest("The filename header is not valid percent-encoding."); + } + name = name.replace(/[\u0000-\u001f\u007f]/g, ""); + name = name.split(/[\\/]/).pop() ?? ""; + if (!name || name === "." || name === "..") { + throw badRequest("The filename is empty after removing path parts."); + } + return text(name, "filename", 255, false); +} + +/** Original location of an upload. Stored as text. The service never opens it. */ +export function validOriginalPath(raw: string | undefined): string | null { + if (!raw) { + return null; + } + try { + return text(decodeURIComponent(raw), "originalPath", 4096, false); + } catch (e) { + if (e instanceof URIError) { + throw badRequest( + "The original path header is not valid percent-encoding." + ); + } + throw e; + } +} + +export function validMediaType(raw: string | undefined): string { + const value = (raw ?? "").split(";")[0].trim().toLowerCase(); + if (!value) { + return "application/octet-stream"; + } + if (!MEDIA_TYPE_PATTERN.test(value) || value.length > 255) { + throw badRequest("Content-Type is not a valid media type."); + } + return value; +} diff --git a/src/kc_storage/src/version.ts b/src/kc_storage/src/version.ts new file mode 100644 index 00000000..52c905c1 --- /dev/null +++ b/src/kc_storage/src/version.ts @@ -0,0 +1 @@ +export const VERSION = "0.1.0"; diff --git a/src/kc_storage/test/backup.test.ts b/src/kc_storage/test/backup.test.ts new file mode 100644 index 00000000..02f053ce --- /dev/null +++ b/src/kc_storage/test/backup.test.ts @@ -0,0 +1,185 @@ +import { afterEach, test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; +import { + FIXTURES, + cli, + client, + startService, + stopAll, + tempDir, +} from "./helpers.ts"; + +afterEach(stopAll); + +async function populate(dataDir: string, work: string) { + const pdf = path.join(work, "paper.pdf"); + const txt = path.join(work, "note.txt"); + fs.copyFileSync(path.join(FIXTURES, "recovery-fixture.pdf"), pdf); + fs.copyFileSync(path.join(FIXTURES, "recovery-note.txt"), txt); + + const svc = await startService(dataDir); + const api = client(svc); + await api.put("/v1/projects/p1", { + name: "Backup Project", + parentId: null, + data: { description: "d" }, + }); + await api.put("/v1/projects/p2", { name: "Child", parentId: "p1", data: {} }); + const a1 = (await api.upload(pdf, "paper.pdf", "application/pdf", pdf)).body + .asset; + const a2 = (await api.upload(txt, "note.txt", "text/plain", txt)).body.asset; + const file = (id: string, title: string, assetId: string) => ({ + projectId: "p1", + title, + ingestType: "file", + assetId, + data: { topics: ["x"], meta: [{ key: "annotation", value: title }] }, + }); + await api.put("/v1/sources/s-pdf", file("s-pdf", "paper.pdf", a1.id)); + await api.put("/v1/sources/s-txt", file("s-txt", "note.txt", a2.id)); + await api.put("/v1/sources/s-web", { + projectId: "p2", + title: "Site", + ingestType: "website", + assetId: null, + data: { accessLink: "https://example.com/" }, + }); + return { svc, api, pdf, txt, assets: [a1, a2] }; +} + +async function snapshot(api: ReturnType) { + const strip = (r: any) => { + const { updatedAt, ...rest } = r; + return rest; + }; + return { + projects: (await api.get("/v1/projects")).body.projects.map(strip), + sources: (await api.get("/v1/sources")).body.sources.map(strip), + }; +} + +test("backup restores into a different directory without the original files", async () => { + const work = tempDir("bk-work"); + const source = tempDir("bk-source"); + const { svc, api, pdf, txt, assets } = await populate(source, work); + const expected = await snapshot(api); + const bytes = { + [assets[0].id]: fs.readFileSync(pdf), + [assets[1].id]: fs.readFileSync(txt), + }; + + // Backup over HTTP and through the command line + const res = await fetch(`${svc.url}/v1/backup`, { + headers: { Authorization: `Bearer ${svc.token}` }, + }); + assert.equal(res.status, 200); + assert.match( + res.headers.get("content-disposition")!, + /knowledge-library-.*\.tar/ + ); + const httpTar = path.join(work, "http.tar"); + fs.writeFileSync(httpTar, Buffer.from(await res.arrayBuffer())); + await svc.stop(); + + const cliTar = path.join(work, "cli.tar"); + const made = cli(["backup", "--data-dir", source, "--out", cliTar]); + assert.equal(made.status, 0, made.stderr); + assert.deepEqual(JSON.parse(made.stdout).assets, 2); + + // The originals are no longer available + fs.rmSync(pdf); + fs.rmSync(txt); + + for (const tar of [httpTar, cliTar]) { + const target = tempDir("bk-target"); + const restored = cli(["restore", "--data-dir", target, "--from", tar]); + assert.equal(restored.status, 0, restored.stderr); + assert.deepEqual(JSON.parse(restored.stdout).restored, { + projects: 2, + sources: 3, + assets: 2, + }); + + const svc2 = await startService(target); + const api2 = client(svc2); + assert.deepEqual(await snapshot(api2), expected); + for (const asset of assets) { + const meta = (await api2.get(`/v1/assets/${asset.id}`)).body.asset; + assert.deepEqual(meta, asset); + assert.ok((await api2.content(asset.id)).bytes.equals(bytes[asset.id])); + } + await svc2.stop(); + } +}); + +test("restore refuses a directory that already has a library", async () => { + const work = tempDir("bk2-work"); + const source = tempDir("bk2-source"); + const { svc } = await populate(source, work); + await svc.stop(); + const tar = path.join(work, "b.tar"); + assert.equal(cli(["backup", "--data-dir", source, "--out", tar]).status, 0); + + const result = cli(["restore", "--data-dir", source, "--from", tar]); + assert.equal(result.status, 1); + assert.match(result.stderr, /already has a library/); +}); + +test("a damaged backup is rejected and leaves the target empty", async () => { + const work = tempDir("bk3-work"); + const source = tempDir("bk3-source"); + const { svc } = await populate(source, work); + await svc.stop(); + const tar = path.join(work, "b.tar"); + assert.equal(cli(["backup", "--data-dir", source, "--out", tar]).status, 0); + + // Change one byte inside the last asset's content (before the end blocks) + const buf = fs.readFileSync(tar); + const marker = buf.indexOf("Knowledge recovery test note"); + assert.ok(marker > 0); + buf[marker] ^= 0xff; + fs.writeFileSync(tar, buf); + + const target = tempDir("bk3-target"); + const result = cli(["restore", "--data-dir", target, "--from", tar]); + assert.equal(result.status, 1); + assert.match(result.stderr, /Hash mismatch/); + + const svc2 = await startService(target); + const api2 = client(svc2); + assert.deepEqual((await api2.get("/v1/projects")).body.projects, []); + const assetDirs = fs + .readdirSync(path.join(target, "assets")) + .filter((d) => d !== "tmp"); + assert.deepEqual( + assetDirs.flatMap((d) => fs.readdirSync(path.join(target, "assets", d))), + [] + ); + await svc2.stop(); +}); + +test("the manifest holds no credentials and states its format", async () => { + const work = tempDir("bk4-work"); + const source = tempDir("bk4-source"); + const { svc } = await populate(source, work); + const res = await fetch(`${svc.url}/v1/backup`, { + headers: { Authorization: `Bearer ${svc.token}` }, + }); + const tar = Buffer.from(await res.arrayBuffer()); + assert.ok(!tar.includes(Buffer.from(svc.token))); + const manifest = JSON.parse( + tar + .subarray(512, 512 + parseInt(tar.toString("ascii", 124, 135), 8)) + .toString() + ); + assert.equal(manifest.format, "knowledge-library-backup"); + assert.equal(manifest.version, 1); + assert.equal(manifest.service.schemaVersion, 1); + assert.deepEqual( + manifest.assets.map((a: any) => a.path).sort(), + manifest.assets.map((a: any) => `assets/${a.id}`).sort() + ); + await svc.stop(); +}); diff --git a/src/kc_storage/test/helpers.ts b/src/kc_storage/test/helpers.ts new file mode 100644 index 00000000..84757a96 --- /dev/null +++ b/src/kc_storage/test/helpers.ts @@ -0,0 +1,193 @@ +/* + * Test helpers. Each test starts the service as a separate Node.js process + * with a temporary data directory. Electron is not involved. + */ + +import { spawn, spawnSync, type ChildProcess } from "node:child_process"; +import crypto from "node:crypto"; +import fs from "node:fs"; +import http from "node:http"; +import os from "node:os"; +import path from "node:path"; +import { once } from "node:events"; + +export const MAIN = path.resolve(import.meta.dirname, "..", "src", "main.ts"); +export const FIXTURES = path.resolve( + import.meta.dirname, + "..", + "..", + "..", + "e2e", + "fixtures" +); + +export function tempDir(name: string) { + return fs.mkdtempSync(path.join(os.tmpdir(), `kc-storage-${name}-`)); +} + +export function sha256(file: string) { + return crypto + .createHash("sha256") + .update(fs.readFileSync(file)) + .digest("hex"); +} + +const running = new Set(); + +/** Stop every service that a test started. Use in afterEach. */ +export async function stopAll() { + await Promise.all([...running].map((s) => s.stop())); +} + +export interface Service { + url: string; + port: number; + token: string; + child: ChildProcess; + stderr: () => string; + stop(): Promise; +} + +export async function startService( + dataDir: string, + args: string[] = [], + env: Record = {} +): Promise { + const token = crypto.randomBytes(32).toString("hex"); + const child = spawn( + process.execPath, + [MAIN, "serve", "--data-dir", dataDir, ...args], + { + env: { ...process.env, KC_STORAGE_TOKEN: token, ...env }, + stdio: ["pipe", "pipe", "pipe"], + } + ); + let err = ""; + child.stderr!.on("data", (d) => (err += d)); + + const line = await new Promise((resolve, reject) => { + let out = ""; + child.stdout!.on("data", (d) => { + out += d; + if (out.includes("\n")) resolve(out.split("\n")[0]); + }); + child.on("exit", (code) => + reject(new Error(`service exited (${code}): ${err}`)) + ); + }); + const ready = JSON.parse(line); + const service: Service = { + url: ready.url, + port: Number(new URL(ready.url).port), + token, + child, + stderr: () => err, + async stop() { + if (child.exitCode !== null) return child.exitCode; + child.kill("SIGTERM"); + const [code] = await once(child, "exit"); + return code as number | null; + }, + }; + running.add(service); + child.on("exit", () => running.delete(service)); + return service; +} + +export function cli(args: string[], env: Record = {}) { + const result = spawnSync(process.execPath, [MAIN, ...args], { + env: { ...process.env, ...env }, + encoding: "utf8", + }); + return { + status: result.status, + stdout: result.stdout, + stderr: result.stderr, + }; +} + +export function client(service: Service) { + const headers = (extra: Record = {}) => ({ + Authorization: `Bearer ${service.token}`, + ...extra, + }); + const json = async (method: string, route: string, body?: unknown) => { + const res = await fetch(service.url + route, { + method, + headers: headers( + body === undefined ? {} : { "Content-Type": "application/json" } + ), + body: body === undefined ? undefined : JSON.stringify(body), + }); + const text = await res.text(); + return { status: res.status, body: text ? JSON.parse(text) : undefined }; + }; + return { + get: (route: string) => json("GET", route), + put: (route: string, body: unknown) => json("PUT", route, body), + del: (route: string) => json("DELETE", route), + async upload( + file: string, + filename: string, + type: string, + originalPath?: string + ) { + const extra: Record = { + "Content-Type": type, + "x-knowledge-filename": encodeURIComponent(filename), + }; + if (originalPath) + extra["x-knowledge-original-path"] = encodeURIComponent(originalPath); + const res = await fetch(service.url + "/v1/assets", { + method: "POST", + headers: headers(extra), + body: fs.readFileSync(file), + }); + return { status: res.status, body: (await res.json()) as any }; + }, + async content(assetId: string) { + const res = await fetch(`${service.url}/v1/assets/${assetId}/content`, { + headers: headers(), + }); + return { + status: res.status, + headers: res.headers, + bytes: Buffer.from(await res.arrayBuffer()), + }; + }, + }; +} + +/** Raw request with full control of headers (Host, Origin). */ +export function rawRequest( + service: Service, + options: { method?: string; path: string; headers?: Record } +): Promise<{ + status: number; + headers: http.IncomingHttpHeaders; + body: string; +}> { + return new Promise((resolve, reject) => { + const req = http.request( + { + host: "127.0.0.1", + port: service.port, + method: options.method ?? "GET", + path: options.path, + headers: options.headers, + }, + (res) => { + let body = ""; + res.on("data", (d) => (body += d)); + res.on("end", () => + resolve({ status: res.statusCode!, headers: res.headers, body }) + ); + } + ); + req.on("error", reject); + req.setTimeout(5000, () => + req.destroy(new Error(`Timed out: ${options.path}`)) + ); + req.end(); + }); +} diff --git a/src/kc_storage/test/migration.test.ts b/src/kc_storage/test/migration.test.ts new file mode 100644 index 00000000..58a321c9 --- /dev/null +++ b/src/kc_storage/test/migration.test.ts @@ -0,0 +1,366 @@ +import { afterEach, test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; +import { DatabaseSync } from "node:sqlite"; +import { + FIXTURES, + cli, + client, + sha256, + startService, + stopAll, + tempDir, +} from "./helpers.ts"; + +afterEach(stopAll); + +const P1 = "11111111-1111-4111-8111-111111111111"; + +/** + * A renderer backup with the record shapes found in real profiles: + * embedded sources, a differing ks- copy, a standalone ks- record, + * an orphan ks- record, a missing file, derived fields, and other keys. + */ +function legacyBackup(work: string) { + const pdf = path.join(work, "paper.pdf"); + const txt = path.join(work, "note.txt"); + fs.copyFileSync(path.join(FIXTURES, "recovery-fixture.pdf"), pdf); + fs.copyFileSync(path.join(FIXTURES, "recovery-note.txt"), txt); + const gone = path.join(work, "deleted.pdf"); + + const fileSource = ( + id: string, + title: string, + file: string, + type: string, + extra = {} + ) => ({ + id: { value: id }, + title, + ingestType: "file", + associatedProject: { value: P1 }, + accessLink: file, + reference: { + ingestType: "file", + source: { file: { filename: title, path: file, size: 1, type } }, + link: file, + }, + topics: [], + meta: [{ key: "knowledge:type", value: "file" }], + notes: [], + icon: { changingThisBreaksApplicationSecurity: "blob:file:///x" }, + thumbnail: "data:image/png;base64,AAAA", + ...extra, + }); + + const sources = [ + { + id: { value: "web-1" }, + title: "Fixture Site", + ingestType: "website", + associatedProject: { value: P1 }, + accessLink: "http://127.0.0.1/index.html", + topics: ["fixture"], + meta: [{ key: "annotation", value: "web note" }], + notes: [{ title: "kept note", body: "notes field is preserved" }], + futureField: { unknown: true }, + }, + fileSource("pdf-1", "paper.pdf", pdf, "application/pdf", { + topics: ["physics"], + meta: [{ key: "annotation", value: "important" }], + }), + fileSource("txt-1", "note.txt", txt, "text/plain"), + fileSource("gone-1", "deleted.pdf", gone, "application/pdf"), + ]; + + const project = { + id: { value: P1 }, + name: "Legacy Project", + parentId: { value: "" }, + subprojects: [], + description: "from renderer", + topics: ["t"], + calendar: { events: [] }, + icon: "pi pi-folder", + sources: [], + knowledgeSource: sources, + }; + + const data: Record = { + [P1]: JSON.stringify(project), + "kc-projects": JSON.stringify([P1]), + // ks- copy that differs from the embedded copy: the embedded copy wins + "ks-pdf-1": JSON.stringify({ ...sources[1], meta: [] }), + // standalone ks- record whose project exists + "ks-extra-1": JSON.stringify({ + id: { value: "extra-1" }, + title: "Standalone", + ingestType: "website", + associatedProject: { value: P1 }, + accessLink: "https://example.com/", + }), + // ks- record without a project + "ks-orphan-1": JSON.stringify({ + id: { value: "orphan-1" }, + title: "Orphan", + ingestType: "website", + associatedProject: { value: "no-such-project" }, + }), + "chat-web-1": JSON.stringify([{ text: "hello" }]), + // UI preference whose key looks like a source key + "ks-table-rows": "10", + theme: JSON.stringify({ name: "dark" }), + "ingest-queue": "[]", + "current-project": P1, + }; + + const file = path.join(work, "knowledge-backup.json"); + fs.writeFileSync( + file, + JSON.stringify({ + format: "knowledge-backup", + version: 1, + exportedAt: "2026-10-06T00:00:00Z", + data, + }) + ); + return { file, pdf, txt }; +} + +function migrate(dataDir: string, input: string, env = {}) { + const result = cli(["migrate", "--data-dir", dataDir, "--from", input], env); + return { + ...result, + summary: result.stdout ? JSON.parse(result.stdout) : undefined, + }; +} + +function report(summary: any) { + return JSON.parse( + fs.readFileSync(path.join(summary.runDirectory, "report.json"), "utf8") + ); +} + +function count(dataDir: string, table: string) { + const db = new DatabaseSync(path.join(dataDir, "library.sqlite"), { + readOnly: true, + }); + try { + return (db.prepare(`SELECT COUNT(*) AS n FROM ${table}`).get() as any).n; + } finally { + db.close(); + } +} + +function assetFiles(dataDir: string) { + const root = path.join(dataDir, "assets"); + return fs + .readdirSync(root) + .filter((d) => d !== "tmp") + .flatMap((d) => fs.readdirSync(path.join(root, d))); +} + +test("migration imports, reports, and preserves data", async () => { + const work = tempDir("mig-work"); + const dataDir = tempDir("mig-data"); + const { file, pdf } = legacyBackup(work); + const inputSha = sha256(file); + + const first = migrate(dataDir, file); + assert.equal(first.status, 0, first.stderr); + const r = report(first.summary); + + assert.equal(r.status, "completed"); + assert.deepEqual( + r.projects.imported.map((i: any) => i.id), + [P1] + ); + assert.deepEqual(r.sources.imported.map((i: any) => i.id).sort(), [ + "extra-1", + "gone-1", + "pdf-1", + "txt-1", + "web-1", + ]); + assert.deepEqual( + r.sources.skipped.map((i: any) => i.id), + ["orphan-1"] + ); + assert.equal(r.files.copied.length, 2); + assert.deepEqual( + r.files.missing.map((m: any) => m.sourceId), + ["gone-1"] + ); + assert.deepEqual(r.superseded, [ + { key: "ks-pdf-1", differentFields: ["meta"] }, + ]); + assert.deepEqual(r.droppedDerivedFields, { icon: 3, thumbnail: 3 }); + assert.deepEqual(r.rendererKeys, [ + "chat-web-1", + "current-project", + "ingest-queue", + "ks-table-rows", + "theme", + ]); + + // Input unchanged; copy, report, and rollback snapshot exist + assert.equal(sha256(file), inputSha); + assert.equal(sha256(path.join(r.runDirectory, "input.json")), inputSha); + assert.ok(fs.existsSync(r.rollback.snapshot)); + + // Data through the API + const svc = await startService(dataDir); + const api = client(svc); + const sources = (await api.get(`/v1/sources?projectId=${P1}`)).body.sources; + const byId = Object.fromEntries(sources.map((s: any) => [s.id, s])); + assert.deepEqual(byId["web-1"].data.notes, [ + { title: "kept note", body: "notes field is preserved" }, + ]); + assert.deepEqual(byId["web-1"].data.futureField, { unknown: true }); + assert.deepEqual(byId["pdf-1"].data.meta, [ + { key: "annotation", value: "important" }, + ]); + assert.equal(byId["pdf-1"].data.icon, undefined); + assert.equal(byId["gone-1"].assetId, null); + assert.equal(byId["gone-1"].data.accessLink, path.join(work, "deleted.pdf")); + + const asset = (await api.get(`/v1/assets/${byId["pdf-1"].assetId}`)).body + .asset; + assert.equal(asset.originalPath, pdf); + assert.equal(asset.filename, "paper.pdf"); + assert.equal(asset.mediaType, "application/pdf"); + assert.ok((await api.content(asset.id)).bytes.equals(fs.readFileSync(pdf))); + + const projects = (await api.get("/v1/projects")).body.projects; + assert.equal(projects[0].data.icon, "pi pi-folder"); + assert.equal(projects[0].data.description, "from renderer"); + await svc.stop(); +}); + +test("a repeated migration creates no duplicates and keeps assets of deleted originals", () => { + const work = tempDir("rep-work"); + const dataDir = tempDir("rep-data"); + const { file, pdf, txt } = legacyBackup(work); + + assert.equal(migrate(dataDir, file).status, 0); + const counts = () => + ["projects", "sources", "assets"].map((t) => count(dataDir, t)); + const before = counts(); + const files = assetFiles(dataDir).sort(); + + const again = report(migrate(dataDir, file).summary); + assert.equal(again.status, "completed"); + assert.equal(again.projects.unchanged.length, 1); + assert.equal(again.sources.unchanged.length, 5); + assert.equal(again.sources.imported.length + again.sources.updated.length, 0); + assert.equal(again.files.reused.length, 2); + assert.deepEqual(counts(), before); + assert.deepEqual(assetFiles(dataDir).sort(), files); + + // Originals deleted: the sources keep their managed copies + fs.rmSync(pdf); + fs.rmSync(txt); + const third = report(migrate(dataDir, file).summary); + assert.equal(third.files.kept.length, 2); + assert.deepEqual( + third.files.missing.map((m: any) => m.sourceId), + ["gone-1"] + ); + assert.equal(third.sources.unchanged.length, 5); + assert.deepEqual(counts(), before); +}); + +test("a failed migration changes nothing and a later run succeeds", () => { + const work = tempDir("fail-work"); + const dataDir = tempDir("fail-data"); + const { file } = legacyBackup(work); + + const failed = migrate(dataDir, file, { + KC_STORAGE_TEST_FAULT: "migrate-before-commit", + }); + assert.equal(failed.status, 1); + const r = report(failed.summary); + assert.equal(r.status, "failed"); + assert.match(r.error, /Injected test fault/); + assert.equal(count(dataDir, "projects"), 0); + assert.equal(count(dataDir, "sources"), 0); + assert.equal(count(dataDir, "assets"), 0); + assert.deepEqual(assetFiles(dataDir), []); + assert.deepEqual(fs.readdirSync(path.join(dataDir, "assets", "tmp")), []); + + const ok = migrate(dataDir, file); + assert.equal(ok.status, 0); + assert.equal(count(dataDir, "sources"), 5); +}); + +test("invalid input is rejected before any write", () => { + const work = tempDir("bad-work"); + const dataDir = tempDir("bad-data"); + const inputs = { + "not-json.json": "{oops", + "wrong-format.json": JSON.stringify({ projects: [] }), + "bad-value.json": JSON.stringify({ + format: "knowledge-backup", + version: 1, + data: { a: 1 }, + }), + }; + for (const [name, content] of Object.entries(inputs)) { + const file = path.join(work, name); + fs.writeFileSync(file, content); + const result = migrate(dataDir, file); + assert.equal(result.status, 1, name); + assert.match( + result.stderr, + /not valid JSON|not a knowledge-backup|is not text/, + name + ); + } + assert.ok(!fs.existsSync(path.join(dataDir, "migrations"))); + assert.equal(count(dataDir, "projects"), 0); +}); + +test("dry run reports without writing", () => { + const work = tempDir("dry-work"); + const dataDir = tempDir("dry-data"); + const { file } = legacyBackup(work); + const result = cli([ + "migrate", + "--data-dir", + dataDir, + "--from", + file, + "--dry-run", + ]); + assert.equal(result.status, 0); + const summary = JSON.parse(result.stdout); + assert.equal(summary.status, "dry-run"); + assert.equal(summary.sources.imported, 5); + assert.equal(count(dataDir, "sources"), 0); + assert.ok(!fs.existsSync(path.join(dataDir, "migrations"))); +}); + +test("the documented rollback restores the pre-migration state", async () => { + const work = tempDir("rb-work"); + const dataDir = tempDir("rb-data"); + const { file } = legacyBackup(work); + + const r = report(migrate(dataDir, file).summary); + assert.equal(count(dataDir, "sources"), 5); + assert.equal(assetFiles(dataDir).length, 2); + + // Rollback: with the service stopped, replace the database with the snapshot + for (const suffix of ["", "-wal", "-shm"]) { + fs.rmSync(path.join(dataDir, `library.sqlite${suffix}`), { force: true }); + } + fs.copyFileSync(r.rollback.snapshot, path.join(dataDir, "library.sqlite")); + + // The next start removes the copied files that no record refers to + const svc = await startService(dataDir); + const api = client(svc); + assert.deepEqual((await api.get("/v1/projects")).body.projects, []); + assert.deepEqual(assetFiles(dataDir), []); + assert.match(svc.stderr(), /removed 2 unreferenced or temporary files/); + await svc.stop(); +}); diff --git a/src/kc_storage/test/service.test.ts b/src/kc_storage/test/service.test.ts new file mode 100644 index 00000000..089245bd --- /dev/null +++ b/src/kc_storage/test/service.test.ts @@ -0,0 +1,324 @@ +import { afterEach, test } from "node:test"; +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; +import { once } from "node:events"; +import { + FIXTURES, + cli, + client, + rawRequest, + sha256, + startService, + stopAll, + tempDir, +} from "./helpers.ts"; + +afterEach(stopAll); + +const PDF = path.join(FIXTURES, "recovery-fixture.pdf"); +const TXT = path.join(FIXTURES, "recovery-note.txt"); + +const project = (name: string, data: object = {}) => ({ + name, + parentId: null, + data, +}); +const source = (projectId: string, title: string, extra: object = {}) => ({ + projectId, + title, + ingestType: "website", + assetId: null, + data: { accessLink: "https://example.com/", topics: ["t"], ...extra }, +}); + +test("projects and sources persist across a restart, in order, with all fields", async () => { + const dataDir = tempDir("persist"); + let svc = await startService(dataDir); + let api = client(svc); + + assert.equal( + ( + await api.put( + "/v1/projects/p1", + project("One", { anything: { nested: [1, 2] } }) + ) + ).status, + 201 + ); + assert.equal( + (await api.put("/v1/projects/p2", { ...project("Two"), parentId: "p1" })) + .status, + 201 + ); + for (const id of ["s1", "s2", "s3"]) { + assert.equal( + ( + await api.put( + `/v1/sources/${id}`, + source("p1", `Source ${id}`, { meta: [{ key: "k", value: id }] }) + ) + ).status, + 201 + ); + } + // Update keeps position; move goes to the end of the new project + assert.equal( + (await api.put("/v1/sources/s1", source("p1", "Source s1 renamed"))).status, + 200 + ); + assert.equal( + (await api.put("/v1/sources/s2", source("p2", "Source s2"))).status, + 200 + ); + assert.equal(await svc.stop(), 0); + + svc = await startService(dataDir); + api = client(svc); + const projects = (await api.get("/v1/projects")).body.projects; + assert.deepEqual( + projects.map((p: any) => [p.id, p.name, p.parentId]), + [ + ["p1", "One", null], + ["p2", "Two", "p1"], + ] + ); + assert.deepEqual(projects[0].data, { anything: { nested: [1, 2] } }); + + const p1 = (await api.get("/v1/sources?projectId=p1")).body.sources; + assert.deepEqual( + p1.map((s: any) => s.title), + ["Source s1 renamed", "Source s3"] + ); + assert.deepEqual(p1[1].data.meta, [{ key: "k", value: "s3" }]); + const p2 = (await api.get("/v1/sources?projectId=p2")).body.sources; + assert.deepEqual( + p2.map((s: any) => s.id), + ["s2"] + ); + + // Deleting a project deletes its sources + assert.equal((await api.del("/v1/projects/p2")).status, 204); + assert.equal( + (await api.get("/v1/sources?projectId=p2")).body.sources.length, + 0 + ); + assert.equal(await svc.stop(), 0); +}); + +test("uploaded files are copied, hashed, and served; originals stay unchanged", async () => { + const dataDir = tempDir("assets"); + const work = tempDir("assets-originals"); + const original = path.join(work, "paper.pdf"); + fs.copyFileSync(PDF, original); + const before = { + sha: sha256(original), + mtime: fs.statSync(original).mtimeMs, + }; + + const svc = await startService(dataDir); + const api = client(svc); + const up = await api.upload( + original, + "../../evil/paper.pdf", + "application/pdf", + original + ); + assert.equal(up.status, 201); + const asset = up.body.asset; + assert.equal(asset.filename, "paper.pdf"); + assert.equal(asset.originalPath, original); + assert.equal(asset.sha256, before.sha); + assert.equal(asset.size, fs.statSync(original).size); + + // The original is not moved or changed + assert.equal(sha256(original), before.sha); + assert.equal(fs.statSync(original).mtimeMs, before.mtime); + + // The managed copy is independent of the original + fs.rmSync(original); + const got = await api.content(asset.id); + assert.equal(got.status, 200); + assert.equal(got.headers.get("content-type"), "application/pdf"); + assert.match( + got.headers.get("content-disposition")!, + /filename\*=UTF-8''paper\.pdf/ + ); + assert.ok(got.bytes.equals(fs.readFileSync(PDF))); + + // A source refers to the asset by ID + await api.put("/v1/projects/p1", project("P")); + const put = await api.put("/v1/sources/f1", { + ...source("p1", "paper.pdf"), + ingestType: "file", + assetId: asset.id, + }); + assert.equal(put.status, 201); + assert.equal(put.body.source.assetId, asset.id); + await svc.stop(); +}); + +test("requests are validated and cannot reach arbitrary files", async () => { + const dataDir = tempDir("validate"); + const svc = await startService(dataDir, [ + "--allow-origin", + "null", + "--max-upload-mb", + "0.001", + ]); + const api = client(svc); + const bearer = { Authorization: `Bearer ${svc.token}` }; + + // Token + assert.equal((await rawRequest(svc, { path: "/v1/projects" })).status, 401); + assert.equal( + ( + await rawRequest(svc, { + path: "/v1/projects", + headers: { Authorization: "Bearer wrong" }, + }) + ).status, + 401 + ); + assert.equal((await rawRequest(svc, { path: "/health" })).status, 200); + + // Host and Origin + assert.equal( + ( + await rawRequest(svc, { + path: "/v1/projects", + headers: { ...bearer, Host: `evil.example:${svc.port}` }, + }) + ).status, + 403 + ); + assert.equal( + ( + await rawRequest(svc, { + path: "/v1/projects", + headers: { ...bearer, Origin: "https://evil.example" }, + }) + ).status, + 403 + ); + const allowed = await rawRequest(svc, { + path: "/v1/projects", + headers: { ...bearer, Origin: "null" }, + }); + assert.equal(allowed.status, 200); + assert.equal(allowed.headers["access-control-allow-origin"], "null"); + + // Paths and IDs + for (const p of [ + "/v1/assets/..%2F..%2Flibrary.sqlite/content", + "/v1/assets/%2E%2E/content", + "/v1/assets/00000000-0000-4000-8000-000000000000/content", + ]) { + const res = await rawRequest(svc, { path: p, headers: bearer }); + assert.ok([400, 404].includes(res.status), `${p} -> ${res.status}`); + assert.ok( + !res.body.includes("SQLite format"), + `${p} returned database bytes` + ); + } + assert.equal((await api.put("/v1/projects/..", project("x"))).status, 404); + assert.equal( + (await api.put("/v1/projects/bad%20id", project("x"))).status, + 400 + ); + + // Bodies + assert.equal( + (await api.put("/v1/projects/p1", { name: "", parentId: null, data: {} })) + .status, + 400 + ); + assert.equal( + (await api.put("/v1/projects/p1", { ...project("x"), extra: 1 })).status, + 400 + ); + assert.equal( + (await api.put("/v1/projects/p1", { ...project("x"), id: "other" })).status, + 400 + ); + assert.equal( + (await api.put("/v1/sources/s1", source("missing", "x"))).status, + 409 + ); + await api.put("/v1/projects/p1", project("x")); + assert.equal( + ( + await api.put("/v1/sources/s1", { + ...source("p1", "x"), + assetId: "00000000-0000-4000-8000-000000000000", + }) + ).status, + 409 + ); + assert.equal( + ( + await api.put("/v1/sources/s1", { + ...source("p1", "x"), + ingestType: "exe", + }) + ).status, + 400 + ); + + // Upload limits and headers + const big = path.join(tempDir("big"), "big.bin"); + fs.writeFileSync(big, Buffer.alloc(4096, 1)); + assert.equal( + (await api.upload(big, "big.bin", "application/octet-stream")).status, + 413 + ); + const noName = await fetch(`${svc.url}/v1/assets`, { + method: "POST", + headers: bearer, + body: "x", + }); + assert.equal(noName.status, 400); + + // Nothing partial remains after rejected uploads + assert.deepEqual(fs.readdirSync(path.join(dataDir, "assets", "tmp")), []); + await svc.stop(); +}); + +test("one process per data directory, clean shutdown on SIGTERM and stdin close", async () => { + const dataDir = tempDir("lock"); + const svc = await startService(dataDir); + + const second = cli(["serve", "--data-dir", dataDir], { + KC_STORAGE_TOKEN: "x".repeat(64), + }); + assert.equal(second.status, 1); + assert.match(second.stderr, /in use by process/); + + assert.equal(await svc.stop(), 0); + assert.ok(!fs.existsSync(path.join(dataDir, "storage.lock"))); + + const viaStdin = await startService(dataDir, ["--exit-on-stdin-close"]); + viaStdin.child.stdin!.end(); + const [code] = await once(viaStdin.child, "exit"); + assert.equal(code, 0); + assert.match(viaStdin.stderr(), /stdin closed/); +}); + +test("serve fails clearly without a token", () => { + const result = cli(["serve", "--data-dir", tempDir("notoken")], { + KC_STORAGE_TOKEN: "", + }); + assert.equal(result.status, 2); + assert.match(result.stderr, /KC_STORAGE_TOKEN/); +}); + +test("text uploads keep their content", async () => { + const svc = await startService(tempDir("text")); + const api = client(svc); + const up = await api.upload(TXT, "recovery-note.txt", "text/plain"); + assert.equal(up.status, 201); + assert.ok( + (await api.content(up.body.asset.id)).bytes.equals(fs.readFileSync(TXT)) + ); + await svc.stop(); +}); diff --git a/src/kc_storage/tsconfig.json b/src/kc_storage/tsconfig.json new file mode 100644 index 00000000..13aefd7a --- /dev/null +++ b/src/kc_storage/tsconfig.json @@ -0,0 +1,18 @@ +{ + "compilerOptions": { + "target": "es2023", + "module": "nodenext", + "moduleResolution": "nodenext", + "lib": ["es2023"], + "types": ["node"], + "strict": true, + "noEmit": true, + "allowImportingTsExtensions": true, + "erasableSyntaxOnly": true, + "verbatimModuleSyntax": true, + "skipLibCheck": true, + "noUnusedLocals": true, + "noFallthroughCasesInSwitch": true + }, + "include": ["src/**/*.ts", "test/**/*.ts", "../kc_contracts/**/*.ts"] +} diff --git a/tests/backup.test.mts b/tests/backup.test.mts index 535d86c5..7644ae75 100644 --- a/tests/backup.test.mts +++ b/tests/backup.test.mts @@ -7,7 +7,6 @@ import assert from "node:assert/strict"; import { createBackup, restoreBackup, - PROJECT_LIST_KEY, } from "../src/kc_angular/src/app/services/ipc-services/backup.ts"; import type { KeyValueStore } from "../src/kc_angular/src/app/services/ipc-services/backup.ts"; @@ -48,7 +47,7 @@ function populated() { meta: [{ key: "annotation", value: "mentions chlorophyll" }], }; store.setItem("project-a", JSON.stringify(project("project-a", [source]))); - store.setItem(PROJECT_LIST_KEY, JSON.stringify(["project-a"])); + store.setItem("kc-projects", JSON.stringify(["project-a"])); store.setItem("chat-source-1", JSON.stringify([{ text: "hello" }])); store.setItem("ingest-queue", JSON.stringify([{ title: "pending" }])); store.setItem("current-project", "project-a"); @@ -63,8 +62,7 @@ test("backup round trip restores every key into an empty profile", () => { const result = restoreBackup(fresh, backup); assert.deepEqual(fresh.snapshot(), original.snapshot()); - assert.equal(result.projectsAdded, 1); - assert.equal(result.projectsReplaced, 0); + assert.equal(result.keysWritten, original.length); }); test("backup keeps chat history, inbox and source annotations", () => { @@ -78,22 +76,22 @@ test("backup keeps chat history, inbox and source annotations", () => { ]); }); -test("restore merges the project list and does not duplicate IDs", () => { +test("restore replaces keys with the same name and keeps other keys", () => { const backup = createBackup(populated()); const target = new MemoryStore(); - target.setItem("project-b", JSON.stringify(project("project-b"))); - target.setItem(PROJECT_LIST_KEY, JSON.stringify(["project-b", "project-a"])); + target.setItem("current-project", "old"); + target.setItem("chat-other", "kept"); const result = restoreBackup(target, backup); - assert.deepEqual(JSON.parse(target.getItem(PROJECT_LIST_KEY)!), [ - "project-b", - "project-a", - ]); - assert.ok(target.getItem("project-b")); - assert.equal(result.projectsAdded, 0); - assert.equal(result.projectsReplaced, 1); + assert.equal(target.getItem("current-project"), "project-a"); + assert.equal(target.getItem("chat-other"), "kept"); + assert.equal( + target.getItem("chat-source-1"), + JSON.stringify([{ text: "hello" }]) + ); + assert.equal(result.keysWritten, Object.keys(backup.data).length); }); test("restore rejects invalid files and leaves the store unchanged", () => { diff --git a/yarn.lock b/yarn.lock index ed647eb5..4f4c9344 100644 --- a/yarn.lock +++ b/yarn.lock @@ -3070,6 +3070,15 @@ __metadata: languageName: node linkType: hard +"@types/node@npm:^24.0.0": + version: 24.19.1 + resolution: "@types/node@npm:24.19.1" + dependencies: + undici-types: ">=7.24.0 <7.24.7" + checksum: 2d3a555fe260fa741213a4e2450336337bdebf476091a6a5996dc7edab2e3b44252c8e64ff1026f020499123d86a42120881f70417723feb6d91ed0ae94bcea5 + languageName: node + linkType: hard + "@types/normalize-package-data@npm:^2.4.0": version: 2.4.1 resolution: "@types/normalize-package-data@npm:2.4.1" @@ -9219,6 +9228,15 @@ __metadata: languageName: unknown linkType: soft +"kc_storage@workspace:src/kc_storage": + version: 0.0.0-use.local + resolution: "kc_storage@workspace:src/kc_storage" + dependencies: + "@types/node": ^24.0.0 + typescript: ^5.9.0 + languageName: unknown + linkType: soft + "keyv@npm:^4.5.2": version: 4.5.2 resolution: "keyv@npm:4.5.2" @@ -13579,6 +13597,16 @@ __metadata: languageName: node linkType: hard +"typescript@npm:^5.9.0": + version: 5.9.3 + resolution: "typescript@npm:5.9.3" + bin: + tsc: bin/tsc + tsserver: bin/tsserver + checksum: 0d0ffb84f2cd072c3e164c79a2e5a1a1f4f168e84cb2882ff8967b92afe1def6c2a91f6838fb58b168428f9458c57a2ba06a6737711fdd87a256bbe83e9a217f + languageName: node + linkType: hard + "typescript@patch:typescript@<4.9.0#~builtin": version: 4.8.4 resolution: "typescript@patch:typescript@npm%3A4.8.4#~builtin::version=4.8.4&hash=701156" @@ -13599,6 +13627,16 @@ __metadata: languageName: node linkType: hard +"typescript@patch:typescript@^5.9.0#~builtin": + version: 5.9.3 + resolution: "typescript@patch:typescript@npm%3A5.9.3#~builtin::version=5.9.3&hash=701156" + bin: + tsc: bin/tsc + tsserver: bin/tsserver + checksum: 8bb8d86819ac86a498eada254cad7fb69c5f74778506c700c2a712daeaff21d3a6f51fd0d534fe16903cb010d1b74f89437a3d02d4d0ff5ca2ba9a4660de8497 + languageName: node + linkType: hard + "uglify-js@npm:^3.1.4": version: 3.17.4 resolution: "uglify-js@npm:3.17.4" @@ -13608,6 +13646,13 @@ __metadata: languageName: node linkType: hard +"undici-types@npm:>=7.24.0 <7.24.7": + version: 7.24.6 + resolution: "undici-types@npm:7.24.6" + checksum: e79802e37dc472c6324850d938b5df6b9e796f6a1bb5f8ccd24f3a6d17eae94a9cc125506c8b339a3293ffbad81daeba30b843e564c1d237b3c8deafd98b6267 + languageName: node + linkType: hard + "unicode-canonical-property-names-ecmascript@npm:^2.0.0": version: 2.0.0 resolution: "unicode-canonical-property-names-ecmascript@npm:2.0.0"