fix(search): discard obsolete embedding work and sweep orphan vectors

ARCHITECTURE.md

@@ -117,7 +117,9 @@ graph TB
 
 **Hybrid query:** MCP client → `vault_search` → FTS5 BM25 ranks (notes + file content) + sqlite-vec KNN ranks (notes + file content) → RRF fusion → cross-encoder reranking → response.
 
-**Invariant — vault is source of truth:** The vault `.md` files are canonical. SQLite FTS5 is derived — rebuildable from scratch. Never write to the index directly. The sqlite-vec embeddings are equally derived — they persist across rebuilds as an optimization but can always be regenerated from the vault.
+**Invariant — vault is source of truth:** Vault files are canonical. MCP content edits must write to those files, never directly to the index. The watcher and startup rebuild derive the SQLite search index from the files.
+
+Embeddings persist across rebuilds to reuse unchanged content and can be regenerated from the vault.
 
 ## MCP Tools
 
@@ -468,15 +470,23 @@ When no embedder is configured (`EMBEDDING_ENABLED=false`), no vectors are index
 2. **Pass 2** — extract links (with the complete path list for resolution), then index file content (canvas, PDF, text → FTS5)
 3. **Pass 3 (background)** — embed notes, then file content. Search works with FTS-only until vectors are ready
 
+**Startup cleanup:** before resetting the source tables, the rebuild removes vectors whose parent chunk or memory-entry row is missing and retains vectors with surviving parents.
+
 Vector tables persist across restarts and rebuilds (only FTS, notes, links, tasks, non-md, and file content tables are cleared). Pass 3 cleans up vectors for deleted notes and files, then embeds only new or modified chunks via content-hash gating.
 
 **Incremental updates:** the file watcher calls `embedNote` after `upsertNote` and `embedFileContent` after `upsertFileContent`; deletion cleans up both vectors and chunks.
 
+**Embedding freshness:**
+
+- Each successful source upsert returns a unique `sourceVersion`. Queued watcher jobs and background rebuild snapshots retain that version, so deletion or replacement invalidates earlier work even when content or modification time repeats.
+- Note, file-content and memory-entry writers check the captured version before model work and after each model await. Obsolete jobs skip derived writes, note/file tail pruning and later memory batches.
+- The watcher assigns an event token before reading each file and checks it before indexing. Unlink or a newer event invalidates earlier reads; embedding stays serialized per path to limit model concurrency.
+
 **Embedding pipeline:** Controlled by `EMBEDDING_ENABLED` (default: `true`). Markdown syntax is stripped before embedding (`plaintext.ts`). Short notes (under 500 body tokens) stay a single title-prefixed chunk. Longer notes split into per-heading sections via `chunker.ts`:
 
 - **Two views per note:** each top-level heading spans its full subtree (the aggregate view, so child text embeds twice); deeper headings own only the lines above the next heading of any level (the disjoint leaf view)
 - **Chunk prefixes:** every fragment starts with the note title; aggregate and leaf fragments add a `Section:` line naming the heading's ancestor path (capped at the remaining token budget — leading ancestors are dropped when deep nesting with long names would floor the body budget, keeping the deepest segments; the Section line is suppressed entirely when the title and metadata exhaust the budget), while preamble fragments, a singleton wrapper's aggregate, and the TOC chunk keep the bare title. A heading whose slice is empty emits no section chunk — its name still rides the TOC chunk, and descendant chunks' Section lines carry it when it has children. A top-level heading with children but no body of its own still emits its aggregate (the slice spans the subtree)
-- **Table-of-contents chunk:** each split note with named headings emits one short chunk (folder segments + title on one line, then heading names in document order, truncated at the chunk budget). Generic intent-phrased queries are structurally won by short chunks under the embedding model, so every split note gets one deliberately short chunk, made unique by its folder path
+- **Table-of-contents chunk:** each split note with named headings emits one short chunk with its folder path and title on the first line, followed by heading names in document order within the chunk budget. This gives a broad query a compact view of the note's topics without requiring one section to represent the whole note
 - **Sub-splitting:** oversized sections split at paragraph boundaries (MAX_CHUNK_TOKENS = 450, minus each chunk's prefix cost), with a sub-minimum trailing fragment merged backward
 
 Content-hash gating (SHA-256 per chunk) skips re-embedding unchanged content on both incremental file-watcher updates and full rebuilds.
@@ -887,10 +897,11 @@ graph LR
    `/home/obsidian/.config` (persists across restarts for incremental sync —
    critical for embedding ingestion).
 3. **`svc-vault-mcp`** — MCP server. Drops to the same `obsidian` user, so
-   both processes read/write the shared `/vault` volume. On startup: builds
-   the FTS5 search index, bootstraps memory templates if the memory folder
+   both processes read/write the shared `/vault` volume. On startup: bootstraps
+   memory templates if the memory folder
    doesn't exist, `MEMORY_ENABLED` is not `false`, and the server is not in
-   `READONLY_MODE`, then starts the file watcher.
+   `READONLY_MODE`, builds the FTS5 search index including those templates,
+   then starts the file watcher.
 
 `svc-vault-mcp` declares `svc-obsidian-sync` in its `dependencies.d`, so the
 MCP server starts only after the full init chain has finished and the sync

deploy/local/README.md

@@ -188,11 +188,15 @@ docker compose pull && docker compose up -d
 
 ## Restart
 
-The server runs startup tasks on every boot: it rebuilds the search index,
-creates memory template files if the memory folder doesn't exist (skipped
-when `MEMORY_ENABLED=false` or `READONLY_MODE=true`), and starts
-the file watcher. Restarting the container re-runs this flow (useful when
-testing bootstrap behavior). The command is the same for both setup methods,
+The server runs these startup tasks on every boot:
+
+1. Create memory template files if the memory folder doesn't exist (skipped
+   when `MEMORY_ENABLED=false` or `READONLY_MODE=true`).
+2. Rebuild the search index, including any new memory template files.
+3. Start the file watcher.
+
+Restarting the container re-runs this flow (useful when testing bootstrap
+behavior). The command is the same for both setup methods,
 since both name the container `vault-cortex`:
 
 ```bash

deploy/remote/README.md

@@ -432,12 +432,16 @@ and unchanged notes are not re-embedded.
 
 ## Restart
 
-The container runs startup tasks on every boot: a catch-up sync runs before
-the server starts to bring the vault current, then the server rebuilds the
-search index, creates memory template files if the memory folder doesn't
-exist (skipped when `MEMORY_ENABLED=false` or `READONLY_MODE=true`), and
-starts the file watcher. Restarting the container re-runs this flow (useful
-when testing bootstrap behavior). The command is the same for every setup
+On every boot, a catch-up sync brings the vault current before the server
+starts. The server then runs these startup tasks:
+
+1. Create memory template files if the memory folder doesn't exist (skipped
+   when `MEMORY_ENABLED=false` or `READONLY_MODE=true`).
+2. Rebuild the search index, including any new memory template files.
+3. Start the file watcher.
+
+Restarting the container re-runs this flow (useful when testing bootstrap
+behavior). The command is the same for every setup
 method:
 
 ```bash

src/__tests__/integration/server-integration.test.ts

@@ -12,7 +12,7 @@
 
 import { describe, it, expect, beforeAll, afterAll, onTestFinished, vi } from "vitest"
 import { DateTime } from "luxon"
-import { readFile, stat, writeFile } from "node:fs/promises"
+import { readFile, readdir, stat, writeFile } from "node:fs/promises"
 import { join } from "node:path"
 import Database from "better-sqlite3"
 import { fileExists } from "../../utils/fs.js"
@@ -32,6 +32,51 @@ import type { ToolResult } from "./test-harness.js"
 
 vi.setConfig({ testTimeout: 15_000 })
 
+it("indexes newly bootstrapped memory templates before accepting requests", async () => {
+  const server = await startServer(await freePort(), { MEMORY_DIR: "Fresh Memory" })
+  onTestFinished(server.cleanup)
+  const client = await createTestClient(server.port)
+  onTestFinished(() => client.close())
+  const database = new Database(join(server.dataDir, "search.db"), { readonly: true })
+  onTestFinished(() => {
+    database.close()
+  })
+
+  expect((await readdir(join(server.vaultPath, "Fresh Memory"))).toSorted()).toEqual([
+    "Agents.md",
+    "Me.md",
+    "Opinions.md",
+    "Principles.md",
+    "Routines.md",
+  ])
+  const result = await callTool({
+    client,
+    name: "vault_search",
+    args: { query: '"subject of every entry"', filters: { folder: "Fresh Memory" } },
+  })
+  const parsedResult = JSON.parse(textContent(result))
+
+  expect(parsedResult.results.map((entry: { path: string }) => entry.path)).toEqual([
+    "Fresh Memory/Agents.md",
+  ])
+  expect(
+    database.prepare("SELECT path FROM notes WHERE path LIKE 'Fresh Memory/%' ORDER BY path").all(),
+  ).toEqual([
+    { path: "Fresh Memory/Agents.md" },
+    { path: "Fresh Memory/Me.md" },
+    { path: "Fresh Memory/Opinions.md" },
+    { path: "Fresh Memory/Principles.md" },
+    { path: "Fresh Memory/Routines.md" },
+  ])
+  const expectedPreferences = await readFile(
+    join(import.meta.dirname, "fixtures/vault/About Me/Preferences.md"),
+    "utf8",
+  )
+  expect(await readFile(join(server.vaultPath, "About Me/Preferences.md"), "utf8")).toBe(
+    expectedPreferences,
+  )
+})
+
 /** Extract joined text from a prompt result's messages. */
 const promptText = (result: Awaited<ReturnType<Client["getPrompt"]>>): string => {
   return result.messages