From ccee3a79d2c7621c9744f75c3fd5f030e2377999 Mon Sep 17 00:00:00 2001 From: Colby McHenry Date: Tue, 27 Oct 2026 12:48:26 -0500 Subject: [PATCH] fix(cli): read commands and the prompt hook open the index read-only Running query, files, node, callers, callees, impact, context, explore or affected while `codegraph index` rebuilt a project could make the rebuild fail with "database is locked", and so could the hidden prompt-hook, which Claude Code runs on its own for structural prompts. Each opened the index read-write, so its open ran the writer's repairs: during the rebuild's bulk load they recreated the secondary indexes the indexer had dropped, and the indexer's next write, whose transaction had already read, failed with SQLITE_BUSY_SNAPSHOT (517). PR #2930 fixed the same thing for status. These commands now open the index read-only, through one helper, and leave migrations, index recovery and WAL maintenance to the writer: - query, files, node, callers, callees, impact, context, explore, affected and prompt-hook are read-only. explore and node run the MCP tool handler in process without a lifecycle, so they never sync or open another project. - While another live process holds rebuild.pid they say so in one line on stderr (a --json answer stays parseable), as status does. - The prompt hook stays out of a running rebuild altogether: the index is half-built until it ends, the MCP tools answer "rebuild in progress" meanwhile, and not opening it keeps the hook from holding the file the rebuild replaces. Its gate records the new outcome noop-rebuild. - The hook's one write, filling an empty segment vocabulary (issue #1142), goes through a writer connection of its own, opened only once it holds the index lock, which every index and sync holds while it writes. - status (read-only since PR #2930), sync, index, init and uninit are unchanged; the last four write. A read-only open migrates nothing. QueryBuilder.hasSchema lets each read that needs a later table or column answer as that migration's defaults would: files.generated (v9), unresolved_refs.status and name_tail (v8), and name_segment_vocab (v7). The status pending count from PR #2930 uses it too. Validation (Windows 11, Node 22.22): - The issue's interleaving, each command in another process inside the writer's edge transaction: on main all 14 command forms make the insert fail with errcode 517 and bring the 5 dropped edge indexes back; with this change all 14 inserts succeed and the indexes stay dropped. - Real rebuilds of excalidraw 95307fa1 (736 files) with query and the prompt hook looping beside them, default settings and with the resolution window forced: main failed 6 of 14 ("store worker: database is locked" with an empty index, or "database is locked" partway through resolution), and 12 of its 109 query runs failed at open with SQLITE_BUSY. This change failed 0 of 14, with 0 failures in 299 query and 284 hook runs and the same graph every time (14,236 nodes, 61,210 edges). - Old indexes: 24 command forms on indexes at schemas 4 to 16 (11 schemas rebuilt from a current index, plus indexes built by 0.9.9, 1.4.0, 1.5.0, 1.6.0 and 1.6.2): output identical to main's, which migrates first, 384 of 384. The schema stays as it was, except where the hook fills the vocabulary of a pre-v7 index; that writer's open migrates it, as every read-write open did. - Timing on excalidraw at rest, 22 interleaved runs per command: medians within -5.8% to +3.3%, -0.9% in total. - read-commands-during-rebuild.test.ts: 38/38 pass. With main's sources 35 fail; with only the CLI opens reverted 33, only the schema fallbacks 19, only the read-only vocabulary fill 2. Follow-up to PR #2930. Fixes #2913 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + .../read-commands-during-rebuild.test.ts | 385 ++++++++++++++++++ docs/design/telemetry.md | 4 +- src/bin/codegraph.ts | 66 ++- src/db/queries.ts | 78 +++- src/index.ts | 25 +- 6 files changed, 517 insertions(+), 42 deletions(-) create mode 100644 __tests__/read-commands-during-rebuild.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 92cf17b48c..9ac1377ddb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,6 +77,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph upgrade` now upgrades a global npm install when run from your home folder, which holds npm's global folder on Windows and with tools like nvm: before, it ran `npm install` without `-g` there, which left the global copy on the old version and put a second copy in your home folder. A copy installed in a project is now upgraded in that project wherever you run the command, with a warning if npm left it on the old version, and a copy installed with pnpm, yarn or bun gets the command that upgrades it instead of npm. Thanks @timlincool for the report and @maxmilian for the first fix. (#2477) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Running `codegraph status` while `codegraph index` rebuilds a project no longer makes the rebuild fail with "database is locked": status now only reads the index, and it says when a rebuild is running in another process, in its output and as `rebuildInProgress` in `status --json`. Thanks @sealad886 for the report, and @danusha2345 and @KissFeng for the fixes. (#2913) +- Running `codegraph query`, `explore`, `node`, `callers`, `callees`, `impact`, `context`, `files` or `affected` while `codegraph index` rebuilds a project no longer makes the rebuild fail with "database is locked", and neither does the context hook Claude Code runs on its own when you ask about your code: like `codegraph status`, they now only read the index and say when a rebuild is running, and the hook stays quiet until the rebuild is done. They still answer from an index an older CodeGraph version built. Thanks @sealad886 for the report. (#2913) - `codegraph init` and `codegraph index` no longer report a file you deleted as one that "could not be read" while git still lists it, which it does until the deletion is staged: such a file is now skipped, and so is a symlink that points at nothing. A file that is there but can't be read is still reported. Thanks @xianzhi-xzw for the report and the first fix. (#2904) - In CFML, a `` or `` block the parser can't read no longer marks its whole file as contributing nothing to the graph: `codegraph init` no longer warns about the file, and `codegraph status` no longer lists it among files whose symbols are missing, since its component, functions and calls are all indexed. The file's recorded errors now name the block and its line instead. Thanks @HarryMuc for the report. (#2654) - In Erlang, a comment inside an argument or parameter list, like the one in `add_to_includes( % give access to .hrl in app's src/` before its two arguments, no longer counts as an argument. The call now links to the function of its real arity, and a function whose parameter list holds a comment keeps its own arity. Before, the comment raised the arity by one, so the call linked to nothing. Re-index Erlang projects after upgrading. Thanks @Dshuishui. (#1619) diff --git a/__tests__/read-commands-during-rebuild.test.ts b/__tests__/read-commands-during-rebuild.test.ts new file mode 100644 index 0000000000..29eba2cb87 --- /dev/null +++ b/__tests__/read-commands-during-rebuild.test.ts @@ -0,0 +1,385 @@ +/** + * The CLI's read commands and the prompt hook while another process writes + * the index (#2913). #2930 fixed `codegraph status`; these are the rest. + * + * Opened read-write, a command ran the writer's open-time repairs: during a + * bulk load they recreated the secondary edge indexes the writer had dropped, + * and the writer's next edge write failed with SQLITE_BUSY_SNAPSHOT + * ("database is locked"). Every command that only reads now opens the index + * read-only. A read-only open migrates nothing, so the same commands must also + * answer from an index an older release built. + * + * The CLI runs against the built binary, as the issue's reproduction did. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { spawnSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { DatabaseConnection } from '../src/db'; +import { QueryBuilder } from '../src/db/queries'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +function runCli(cwd: string, args: string[], stdin?: string): { status: number | null; stdout: string; stderr: string } { + const result = spawnSync(process.execPath, [BIN, ...args], { + cwd, + input: stdin, + encoding: 'utf-8', + timeout: 60_000, + env: { + ...process.env, + CODEGRAPH_NO_DAEMON: '1', CODEGRAPH_TELEMETRY: '0', DO_NOT_TRACK: '1', + CODEGRAPH_NO_RELAUNCH: '1', CODEGRAPH_WASM_RELAUNCHED: '1', + CODEGRAPH_NO_PROMPT_HOOK: '0', CODEGRAPH_PROMPT_HOOK: '1', + NO_COLOR: '1', + }, + }); + return { status: result.status, stdout: result.stdout, stderr: result.stderr }; +} + +interface ReadCommand { + name: string; + args: string[]; + /** The prompt the hook is handed, for `prompt-hook`. */ + prompt?: string; + answer: RegExp; +} + +const READ_COMMANDS: ReadCommand[] = [ + { name: 'query', args: ['query', 'loadOrderState'], answer: /function\s+loadOrderState/ }, + { name: 'files', args: ['files', '--json'], answer: /"path": "src\/order\.ts"/ }, + { name: 'node', args: ['node', 'loadOrderState'], answer: /\*\*loadOrderState\*\* \(function\)/ }, + { name: 'callers', args: ['callers', 'loadOrderState'], answer: /getUserProfile/ }, + { name: 'callees', args: ['callees', 'getUserProfile'], answer: /loadOrderState/ }, + { name: 'impact', args: ['impact', 'loadOrderState'], answer: /getUserProfile/ }, + { name: 'context', args: ['context', 'getUserProfile', 'loadOrderState'], answer: /loadOrderState/ }, + { name: 'explore', args: ['explore', 'getUserProfile', 'loadOrderState'], answer: /src\/order\.ts/ }, + { name: 'affected', args: ['affected', 'src/order.ts'], answer: /src\/user\.test\.ts/ }, + { name: 'prompt-hook', args: ['prompt-hook'], prompt: 'how does getUserProfile reach loadOrderState?', answer: / path.join(dir, '.codegraph', 'codegraph.db'); + +/** The secondary indexes on `edges`, the ones a bulk load drops. */ +function edgeIndexes(conn: DatabaseConnection): string[] { + return (conn.getDb().prepare( + "SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = 'edges' AND sql IS NOT NULL" + ).all() as Array<{ name: string }>).map((r) => r.name).sort(); +} + +function schemaVersion(dir: string): number | undefined { + const conn = DatabaseConnection.open(dbOf(dir), { readOnly: true }); + try { + return conn.getSchemaVersion()?.version; + } finally { + conn.close(); + } +} + +/** + * Take a current index back to the schema an older release wrote, undoing + * what the later migrations added (src/db/migrations.ts). Schema 7 is 1.3.0 to + * 1.4.x; schema 4 is 0.7.6 to 0.9.9, the oldest any release tag wrote. + */ +function downgradeTo(dir: string, version: 4 | 7): void { + const conn = DatabaseConnection.open(dbOf(dir)); + try { + const db = conn.getDb(); + const dropIndexesOn = (table: string, columns: RegExp): void => { + const indexes = db.prepare( + "SELECT name, sql FROM sqlite_master WHERE type = 'index' AND tbl_name = ? AND sql IS NOT NULL" + ).all(table) as Array<{ name: string; sql: string }>; + for (const { name, sql } of indexes) if (columns.test(sql)) db.exec(`DROP INDEX ${name}`); + }; + db.exec('DROP TABLE synthesis_inputs; DROP INDEX idx_edges_synthesis_site'); // v10 + dropIndexesOn('files', /generated/); + db.exec('ALTER TABLE files DROP COLUMN generated'); // v9 + dropIndexesOn('unresolved_refs', /status|name_tail/); + // Before v8 a finished resolution pass deleted what it could not resolve. + db.exec("DELETE FROM unresolved_refs WHERE status = 'failed'"); + db.exec('ALTER TABLE unresolved_refs DROP COLUMN status'); + db.exec('ALTER TABLE unresolved_refs DROP COLUMN name_tail'); // v8 + if (version < 7) { + db.exec('DROP TABLE name_segment_vocab'); // v7 + db.exec('DROP INDEX idx_edges_identity'); // v6 + db.exec('ALTER TABLE nodes DROP COLUMN return_type'); // v5 + } + db.exec(`DELETE FROM schema_versions; INSERT INTO schema_versions (version, applied_at, description) VALUES (${version}, 0, 'older release')`); + } finally { + conn.close(); + } +} + +describe('read commands while another process writes the index (#2913)', () => { + let dir: string; + + beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-read-rebuild-')); + writeProject(dir); + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + cg.close(); + }, 60_000); + + afterAll(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + /** The locks the writer holds: a rebuild's (`codegraph index`), or a sync's index lock. */ + function holdLocks(kind: 'rebuild' | 'sync'): () => void { + const dataDir = path.join(dir, '.codegraph'); + const files = kind === 'rebuild' ? ['rebuild.pid', 'writer.pid'] : ['codegraph.lock']; + const content = kind === 'rebuild' + ? JSON.stringify({ pid: process.pid, mode: 'rebuild', startedAt: Date.now(), ready: false }) + : String(process.pid); + for (const file of files) fs.writeFileSync(path.join(dataDir, file), content); + return () => { for (const file of files) fs.rmSync(path.join(dataDir, file), { force: true }); }; + } + + /** + * The issue's interleaving: the writer drops the edge indexes for a bulk + * load, reads its endpoints inside the edge transaction, the command runs in + * another process, then the writer inserts. + */ + async function insideWritersEdgeTransaction( + locks: 'rebuild' | 'sync', + edgeLine: number, + observe: () => void, + ): Promise<{ dropped: string[]; afterInsert: string[] }> { + const writer = DatabaseConnection.open(dbOf(dir)); + const release = holdLocks(locks); + try { + const complete = edgeIndexes(writer); + writer.beginBulkEdgeLoad(); + const dropped = edgeIndexes(writer); + expect(dropped.length).toBeLessThan(complete.length); + + const db = writer.getDb(); + const ids = (db.prepare("SELECT id FROM nodes WHERE kind = 'function' ORDER BY name").all() as Array<{ id: string }>).map((r) => r.id); + const edgeCount = (): number => (db.prepare('SELECT COUNT(*) AS n FROM edges').get() as { n: number }).n; + const before = edgeCount(); + const queries = new QueryBuilder(db); + // The endpoint read is private; hook it as the issue's reproduction did. + const hooked = queries as unknown as { getExistingNodeIds: (ids: readonly string[]) => Set }; + const readEndpoints = hooked.getExistingNodeIds.bind(queries); + hooked.getExistingNodeIds = (endpointIds) => { + const existing = readEndpoints(endpointIds); + observe(); + return existing; + }; + // Read-write, the command recreated the dropped indexes in between, and + // this insert failed with SQLITE_BUSY_SNAPSHOT (517). + expect(() => queries.insertEdges([{ source: ids[0]!, target: ids[1]!, kind: 'references', line: edgeLine }])).not.toThrow(); + expect(edgeCount()).toBe(before + 1); + const afterInsert = edgeIndexes(writer); + await writer.endBulkEdgeLoad(); + expect(edgeIndexes(writer)).toEqual(complete); + return { dropped, afterInsert }; + } finally { + release(); + writer.close(); + } + } + + it.each(READ_COMMANDS.map((command, i) => [command.name, command, i] as const))( + '%s answers inside the writer\'s bulk load without failing its next write', + async (_name, command, i) => { + // The hook keeps out of a rebuild altogether (below), so it meets a + // sync's bulk load here; the commands meet the issue's rebuild. + const locks = command.args[0] === 'prompt-hook' ? 'sync' : 'rebuild'; + let result: ReturnType | undefined; + const { dropped, afterInsert } = await insideWritersEdgeTransaction(locks, 1000 + i, () => { + result = run(dir, command); + }); + expect(result?.status, result?.stderr).toBe(0); + expect(result?.stdout).toMatch(command.answer); + // Mid-rebuild, the answer comes from a graph still being built. + if (locks === 'rebuild') expect(result?.stderr).toContain('An index rebuild ("codegraph index") is running in another process'); + // The command repaired nothing: the bulk load's indexes are still down. + expect(afterInsert).toEqual(dropped); + }, + 60_000, + ); + + it('the prompt hook leaves an index alone while `codegraph index` rebuilds it', async () => { + let result: ReturnType | undefined; + const hook = READ_COMMANDS.find((c) => c.name === 'prompt-hook')!; + await insideWritersEdgeTransaction('rebuild', 2000, () => { + result = run(dir, hook); + }); + // Silent, as on every path it cannot serve: the index is half-built until + // the rebuild ends. Once it has, the hook answers again. + expect(result?.status, result?.stderr).toBe(0); + expect(result?.stdout).toBe(''); + expect(run(dir, hook).stdout).toMatch(hook.answer); + }, 60_000); +}); + +describe('read commands on an index an older release built (#2913)', () => { + let base: string; + const copies: string[] = []; + + beforeAll(async () => { + base = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-read-legacy-')); + writeProject(base); + const cg = CodeGraph.initSync(base); + await cg.indexAll(); + cg.close(); + }, 60_000); + + afterAll(() => { + for (const copy of [base, ...copies]) fs.rmSync(copy, { recursive: true, force: true }); + }); + + function legacyCopy(version: 4 | 7, prepare?: (db: DatabaseConnection) => void): string { + const copy = fs.mkdtempSync(path.join(os.tmpdir(), `codegraph-read-v${version}-`)); + copies.push(copy); + fs.cpSync(base, copy, { recursive: true }); + if (prepare) { + const conn = DatabaseConnection.open(dbOf(copy)); + try { + prepare(conn); + } finally { + conn.close(); + } + } + downgradeTo(copy, version); + return copy; + } + + // Schema 7 has no reference status (v8), generated-file flag (v9) or + // synthesis inputs (v10); schema 4 has no segment vocabulary (v7) and no + // return types (v5) either. The commands read each index as it is. + it.each(READ_COMMANDS.flatMap((command) => ([7, 4] as const).map((version) => [command.name, version, command] as const)))( + '%s answers from a schema-%i index', + (_name, version, command) => { + const copy = legacyCopy(version); + const result = run(copy, command); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toMatch(command.answer); + // The hook's segment tier needs the vocabulary that schema 4 predates. + // It fills it under the index lock through a writer of its own, whose + // open upgrades the index, as every read-write open used to. + const upgraded = version === 4 && command.name === 'prompt-hook, segment tier'; + expect(schemaVersion(copy)).toBe(upgraded ? schemaVersion(base) : version); + }, + 60_000, + ); + + it.each([7, 4] as const)('every read of a later table or column answers from a schema-%i index read-only', (version) => { + const copy = legacyCopy(version, (writer) => { + // A reference an interrupted run left behind, pending as the v8 migration marks it. + const from = (writer.getDb().prepare("SELECT id FROM nodes WHERE name = 'getUserProfile'").get() as { id: string }).id; + writer.getDb().prepare( + "INSERT INTO unresolved_refs (from_node_id, reference_name, reference_kind, line, col, file_path, language) VALUES (?, 'missingSink', 'calls', 2, 0, 'src/user.ts', 'typescript')" + ).run(from); + }); + const conn = DatabaseConnection.open(dbOf(copy), { readOnly: true }); + try { + const queries = new QueryBuilder(conn.getDb()); + expect(queries.getGeneratedPathsAmong(['src/order.ts', 'src/user.ts']).size).toBe(0); + expect(queries.countGeneratedFiles()).toBe(0); + const unreferenced = queries.getUnreferencedNodes(['function'], 50); + expect(unreferenced.map((r) => r.node.name)).toContain('profileTest'); + expect(unreferenced.every((r) => !r.generated)).toBe(true); + expect([...queries.getUnresolvedNamesAmong(['missingSink', 'other'])]).toEqual(['missingSink']); + expect(queries.getUnresolvedReferencesCount()).toBe(1); + const vocab = version >= 7; + expect(queries.isNameSegmentVocabEmpty()).toBe(!vocab); + expect(queries.getSegmentCoOccurrence([{ segment: 'order', word: 'order' }, { segment: 'state', word: 'state' }], 2, 5).map((r) => r.name)) + .toEqual(vocab ? expect.arrayContaining(['OrderStateMachine']) : []); + expect((queries.getSegmentNameCounts(['order']).get('order') ?? 0) > 0).toBe(vocab); + expect(queries.getNamesForSegment('machine', 5)).toEqual(vocab ? expect.arrayContaining(['OrderStateMachine']) : []); + expect(queries.getExploreMissDiagnostics('order nothinglikethis').matched).toEqual(['order']); + } finally { + conn.close(); + } + expect(schemaVersion(copy)).toBe(version); + }, 60_000); +}); + +describe('the prompt hook fills an empty segment vocabulary under the index lock (#1142, #2913)', () => { + let dir: string; + const segmentTier = READ_COMMANDS.find((c) => c.name === 'prompt-hook, segment tier')!; + + beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-read-vocab-')); + writeProject(dir); + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + cg.close(); + const conn = DatabaseConnection.open(dbOf(dir)); + try { + conn.getDb().exec('DELETE FROM name_segment_vocab'); + } finally { + conn.close(); + } + }, 60_000); + + afterAll(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + function vocabRows(): number { + const conn = DatabaseConnection.open(dbOf(dir), { readOnly: true }); + try { + return (conn.getDb().prepare('SELECT COUNT(*) AS n FROM name_segment_vocab').get() as { n: number }).n; + } finally { + conn.close(); + } + } + + it('writes nothing while an index or sync holds the lock', () => { + const lock = path.join(dir, '.codegraph', 'codegraph.lock'); + fs.writeFileSync(lock, String(process.pid)); + try { + const result = run(dir, segmentTier); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toBe(''); + expect(vocabRows()).toBe(0); + } finally { + fs.rmSync(lock, { force: true }); + } + }, 60_000); + + it('fills it once the lock is free, and answers from it', () => { + const result = run(dir, segmentTier); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toMatch(segmentTier.answer); + expect(vocabRows()).toBeGreaterThan(0); + expect(fs.existsSync(path.join(dir, '.codegraph', 'codegraph.lock'))).toBe(false); + }, 60_000); +}); diff --git a/docs/design/telemetry.md b/docs/design/telemetry.md index bf5deed9cc..72bf2f57cc 100644 --- a/docs/design/telemetry.md +++ b/docs/design/telemetry.md @@ -80,7 +80,7 @@ Event types: counters named `prompt-hook-gate-`, outcome ∈ `high-keyword` / `high-token` / `medium-segment` / `nudge-projects` / `noop-shape` / `noop-no-index` / `noop-unverified` / `noop-explore-keyword` / - `noop-explore-token` / `noop-vocab-empty` — decision names only, never + `noop-explore-token` / `noop-vocab-empty` / `noop-rebuild` — decision names only, never prompt content. This is the gate's measured recall/precision funnel: a rising `noop-*` share against the `high`/`medium` tiers is the signal that the gate (keyword table or segment matching) is missing real questions. @@ -89,6 +89,8 @@ Event types: `noop-explore-` instead (#1143), and a MEDIUM-eligible prompt hitting a not-yet-backfilled segment vocabulary records `noop-vocab-empty` rather than polluting `noop-unverified` (#1142). + A prompt that arrives while `codegraph index` rebuilds the project records + `noop-rebuild`: the hook leaves a rebuilding index alone (#2913). - **`uninstall`** — one per `uninstall`/`uninit` run (churn signal). Props: `targets`. One legacy field is still *accepted* and belongs in the mirror even though nothing sends diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 7719497eff..c4bbc4eb45 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -378,6 +378,32 @@ function warn(message: string): void { console.log(chalk.yellow(getGlyphs().warn) + ' ' + message); } +/** + * Open a project's index for a command that only reads it: `query`, + * `explore`, `node`, the prompt hook and the rest (#2913). + * + * Opened read-write, the index runs the writer's repairs: migrations, the + * rebuild of the indexes and search triggers an interrupted bulk load left + * out, and WAL maintenance. While `codegraph index` or a sync writes in + * another process, those repairs recreate the indexes its bulk load dropped, + * and its next write fails with SQLITE_BUSY_SNAPSHOT, "database is locked". + * Read-only, a command leaves all of that to the writer, as `status` does + * since #2930; an index an older release built is read as it is (see + * QueryBuilder.hasSchema). + * + * During a rebuild the answer comes from a graph still being built, so the + * command says so, on stderr to keep a `--json` answer parseable. + */ +async function openIndexToRead(projectPath: string): Promise { + const { default: CodeGraph } = await loadCodeGraph(); + const cg = await CodeGraph.open(projectPath, { readOnly: true }); + const { isRebuildInProgress } = await import('../mcp/writer-lock'); + if (isRebuildInProgress(projectPath)) { + console.error(`${chalk.yellow(getGlyphs().warn)} An index rebuild ("codegraph index") is running in another process — this answer comes from the part of the index it has rebuilt so far.`); + } + return cg; +} + /** "not found" (+ optional did-you-mean) when no exact symbol matches. */ function formatSymbolNotFound(symbol: string, fuzzyNames: string[]): string { const suggestions = [...new Set(fuzzyNames.filter((n) => n !== symbol))].slice(0, 3); @@ -1344,8 +1370,7 @@ program process.exit(1); } - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(projectPath); + const cg = await openIndexToRead(projectPath); const limit = parseInt(options.limit || '10', 10); const rawResults = cg.searchNodes(search, { @@ -1430,8 +1455,7 @@ program process.exit(1); } - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(projectPath); + const cg = await openIndexToRead(projectPath); const { ToolHandler } = await import('../mcp/tools'); const handler = new ToolHandler(cg); @@ -1487,8 +1511,7 @@ program process.exit(1); } - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(projectPath); + const cg = await openIndexToRead(projectPath); const result = await cg.buildContext(taskParts.join(' '), { format, @@ -1592,8 +1615,15 @@ program `${lead}\n${projects.map((p) => ` - projectPath: "${p}"`).join('\n')}\n`; if (plan.exploreRoot) { - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(plan.exploreRoot); + // A `codegraph index` in another process is rebuilding this index: it + // holds part of the graph until the rebuild ends, and the rebuild + // replaces the file, which an open handle blocks on Windows. Inject + // nothing, as the MCP tools answer "rebuild in progress" (#2913). + const { isRebuildInProgress } = await import('../mcp/writer-lock'); + if (isRebuildInProgress(plan.exploreRoot)) { gate('noop-rebuild'); return; } + // Claude Code runs this hook on its own, on every prompt: read-only, so + // it never repairs what an index or sync is in the middle of writing. + const cg = await openIndexToRead(plan.exploreRoot); try { const others = plan.nudgeProjects.length ? `\n${nudge(plan.nudgeProjects, 'Other indexed projects in this workspace — pass projectPath to query them:')}` @@ -1644,8 +1674,9 @@ program // (#1142). Heal it here: on a populated vocab this is one SELECT; // the actual backfill is a one-time batched pass whose cost the MCP // server's own catch-up sync usually pays first (it runs at every - // session start). A distinct noop outcome keeps a dormant vocab - // from polluting the noop-unverified recall signal. + // session start). It writes through a connection of its own, under + // the index lock (#2913). A distinct noop outcome keeps a dormant + // vocab from polluting the noop-unverified recall signal. const vocabReady = await cg.healSegmentVocabIfEmpty().catch(() => false); if (!vocabReady) { gate('noop-vocab-empty'); return; } const related = cg.getSegmentMatches(proseWords); @@ -1716,8 +1747,7 @@ program process.exit(1); } - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(projectPath); + const cg = await openIndexToRead(projectPath); const { ToolHandler } = await import('../mcp/tools'); const handler = new ToolHandler(cg); @@ -1788,8 +1818,7 @@ program process.exit(1); } - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(projectPath); + const cg = await openIndexToRead(projectPath); let files = cg.getFiles(); if (files.length === 0) { @@ -2361,8 +2390,7 @@ for (const direction of ['callers', 'callees'] as const) { process.exit(1); } - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(projectPath); + const cg = await openIndexToRead(projectPath); try { const limit = parseInt(options.limit || '20', 10); const { nodes: targets } = lookupSymbolNodes(cg, symbol, goPackageReaderFor(cg)); @@ -2485,8 +2513,7 @@ program process.exit(1); } - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(projectPath); + const cg = await openIndexToRead(projectPath); try { const depth = Math.min(Math.max(parseInt(options.depth || '2', 10), 1), 10); const { nodes: targets } = lookupSymbolNodes(cg, symbol, goPackageReaderFor(cg)); @@ -2626,8 +2653,7 @@ program process.exit(0); } - const { default: CodeGraph } = await loadCodeGraph(); - const cg = await CodeGraph.open(projectPath); + const cg = await openIndexToRead(projectPath); const maxDepth = parseInt(options.depth || '5', 10); // Custom filter pattern diff --git a/src/db/queries.ts b/src/db/queries.ts index 3377e288ed..4675e3b65e 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -466,6 +466,33 @@ export class QueryBuilder { // connection, or a worker following a rebuilt index keeps its answer (#1864). this.dominantFileMemo = undefined; this.readMemos.clear(); + this.presentSchema.clear(); + } + + /** + * Tables (`name`) and columns (`table.column`) that later migrations added, + * found in this connection's database. See {@link hasSchema}. + */ + private presentSchema = new Set(); + + /** + * Whether the database holds `table`, or its `column`. An index an older + * release built lacks what later migrations add until a writer's open + * migrates it, and a read-only open migrates nothing: the CLI's read + * commands and the prompt hook open read-only so that a running rebuild is + * left alone (#2913). A read that needs a newer table or column checks for + * it here and, without it, answers as the migration's defaults would. + * Found once and remembered, since a schema only grows; a missing one is + * looked for again, so a writer's later migration is seen. + */ + private hasSchema(table: string, column?: string): boolean { + const key = column ? `${table}.${column}` : table; + if (this.presentSchema.has(key)) return true; + const found = column + ? this.db.prepare('SELECT 1 FROM pragma_table_info(?) WHERE name = ?').get(table, column) !== undefined + : this.db.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?").get(table) !== undefined; + if (found) this.presentSchema.add(key); + return found; } /** @@ -885,6 +912,8 @@ export class QueryBuilder { /** True when the vocab has no rows — an index built before the table existed. * `sync` uses this to heal such databases (see rebuildNameSegmentVocabFrom). */ isNameSegmentVocabEmpty(): boolean { + // Before schema v7 there is no table, which its migration creates empty (#2913). + if (!this.hasSchema('name_segment_vocab')) return true; const row = this.db.prepare('SELECT 1 FROM name_segment_vocab LIMIT 1').get(); return row === undefined; } @@ -931,7 +960,7 @@ export class QueryBuilder { minWords: number, limit: number, ): Array<{ name: string; matches: number }> { - if (variants.length === 0) return []; + if (variants.length === 0 || !this.hasSchema('name_segment_vocab')) return []; const placeholders = variants.map(() => '?').join(', '); const whens = variants.map(() => 'WHEN ? THEN ?').join(' '); const rows = this.db @@ -956,7 +985,7 @@ export class QueryBuilder { /** How many distinct names each segment appears in — the rarity signal that * separates a discriminative word ("checkout") from a ubiquitous one ("state"). */ getSegmentNameCounts(segments: string[]): Map { - if (segments.length === 0) return new Map(); + if (segments.length === 0 || !this.hasSchema('name_segment_vocab')) return new Map(); const placeholders = segments.map(() => '?').join(', '); const rows = this.db .prepare( @@ -969,6 +998,7 @@ export class QueryBuilder { /** Names containing the given segment (rare-single-word tier). */ getNamesForSegment(segment: string, limit: number): string[] { + if (!this.hasSchema('name_segment_vocab')) return []; const rows = this.db .prepare('SELECT name FROM name_segment_vocab WHERE segment = ? ORDER BY length(name) ASC LIMIT ?') .all(segment, limit) as Array<{ name: string }>; @@ -1916,32 +1946,34 @@ export class QueryBuilder { // EXISTS uses FTS postings and the segment primary key, never source scans. // Vocab rows can outlive deleted definitions, so verify them against nodes. + // An index from before schema v7 has no vocab: FTS alone answers (#2913). + const vocab = this.hasSchema('name_segment_vocab'); const rows = this.db.prepare(` WITH words(word, pattern) AS (VALUES ${checked.map(() => '(?, ?)').join(', ')}) SELECT word, ( - EXISTS (SELECT 1 FROM nodes_fts WHERE nodes_fts MATCH pattern) + EXISTS (SELECT 1 FROM nodes_fts WHERE nodes_fts MATCH pattern)${vocab ? ` OR EXISTS ( SELECT 1 FROM name_segment_vocab v WHERE v.segment = word AND EXISTS (SELECT 1 FROM nodes n WHERE n.name = v.name AND n.kind NOT IN ('file', 'import')) - ) + )` : ''} ) AS matched FROM words `).all(...checked.flatMap(w => [w, `{name qualified_name signature docstring} : "${w}"*`])) as Array<{ word: string; matched: number }>; - const names = this.db.prepare(` + const names = this.db.prepare(`${vocab ? ` SELECT name FROM ( SELECT v.name FROM name_segment_vocab v WHERE v.segment IN (${checked.map(() => '?').join(', ')}) AND EXISTS (SELECT 1 FROM nodes n WHERE n.name = v.name AND n.kind NOT IN ('file', 'import')) LIMIT 12 ) - UNION ALL + UNION ALL` : ''} SELECT name FROM ( SELECT n.name FROM nodes_fts JOIN nodes n ON n.rowid = nodes_fts.rowid WHERE nodes_fts MATCH ? AND n.kind NOT IN ('file', 'import') LIMIT 12 ) - `).all(...checked, `name : (${checked.map(w => `"${w}"*`).join(' OR ')})`) as Array<{ name: string }>; + `).all(...(vocab ? checked : []), `name : (${checked.map(w => `"${w}"*`).join(' OR ')})`) as Array<{ name: string }>; return { matched: rows.filter(r => r.matched).map(r => r.word), unmatched: rows.filter(r => !r.matched).map(r => r.word), @@ -2647,9 +2679,12 @@ export class QueryBuilder { ): Array<{ node: Node; generated: boolean }> { if (kinds.length === 0 || limit <= 0) return []; const placeholders = kinds.map(() => '?').join(','); + // Before schema v9 no file carries the flag, as its migration's default + // leaves them (#2913). + const generated = this.hasSchema('files', 'generated') ? 'COALESCE(f.generated, 0)' : '0'; const rows = this.db .prepare( - `SELECT n.*, COALESCE(f.generated, 0) AS file_generated + `SELECT n.*, ${generated} AS file_generated FROM nodes n LEFT JOIN files f ON f.path = n.file_path WHERE n.kind IN (${placeholders}) @@ -2687,22 +2722,29 @@ export class QueryBuilder { const unique = [...new Set(names)].filter((name) => name.length > 0); const found = new Set(); if (unique.length === 0) return found; + // Before schema v8 a reference had no tail and was never parked, and the + // migration leaves every row pending with an empty tail: only the name + // can match (#2913). + const parked = this.hasSchema('unresolved_refs', 'status'); for (let i = 0; i < unique.length; i += SQLITE_PARAM_CHUNK_SIZE) { const chunk = unique.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); const placeholders = chunk.map(() => '?').join(','); - const rows = this.db - .prepare( - `SELECT DISTINCT reference_name AS name FROM unresolved_refs - WHERE reference_name IN (${placeholders}) + const byName = `SELECT DISTINCT reference_name AS name FROM unresolved_refs + WHERE reference_name IN (${placeholders})`; + const rows = (parked + ? this.db + .prepare( + `${byName} UNION SELECT DISTINCT name_tail AS name FROM unresolved_refs WHERE name_tail IN (${placeholders}) UNION SELECT DISTINCT ${MEMBER_TAIL_SQL} AS name FROM unresolved_refs WHERE ${MODULE_PARKED_SQL} AND ${MEMBER_TAIL_SQL} IN (${placeholders})` - ) - .all(...chunk, ...chunk, ...chunk) as Array<{ name: string }>; + ) + .all(...chunk, ...chunk, ...chunk) + : this.db.prepare(byName).all(...chunk)) as Array<{ name: string }>; for (const row of rows) found.add(row.name); } return found; @@ -3745,6 +3787,9 @@ export class QueryBuilder { const unique = [...new Set(filePaths)]; const found = new Set(); if (unique.length === 0) return found; + // An index from before schema v9 has no flag, which its migration adds + // as 0 everywhere: the path convention alone decides (#2913). + if (!this.hasSchema('files', 'generated')) return found; for (let i = 0; i < unique.length; i += SQLITE_PARAM_CHUNK_SIZE) { const chunk = unique.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); @@ -3893,6 +3938,8 @@ export class QueryBuilder { /** How many indexed files carry the generated flag. Surfaced by `status`. */ countGeneratedFiles(): number { + // Before schema v9 no file carries the flag (#2913). + if (!this.hasSchema('files', 'generated')) return 0; const row = this.db .prepare('SELECT COUNT(*) AS n FROM files WHERE generated = 1') .get() as { n: number } | undefined; @@ -4138,8 +4185,7 @@ export class QueryBuilder { // and the v8 migration marks every one of its rows pending. A read-only // open migrates nothing (`codegraph status`, #2913), so count them all // until a writer's open has migrated it. - const columns = this.db.prepare('PRAGMA table_info(unresolved_refs)').all() as Array<{ name: string }>; - if (!columns.some((c) => c.name === 'status')) { + if (!this.hasSchema('unresolved_refs', 'status')) { return (this.db.prepare('SELECT COUNT(*) as count FROM unresolved_refs').get() as { count: number }).count; } this.stmts.getUnresolvedCount = this.db.prepare( diff --git a/src/index.ts b/src/index.ts index 2610c7ad84..5d9c25ce96 100644 --- a/src/index.ts +++ b/src/index.ts @@ -2360,6 +2360,12 @@ export class CodeGraph { * usable (already populated — the overwhelmingly common one-SELECT case — * or healed here); false when it isn't (empty graph, or another process * holds the index lock — that process's own sync heals it). + * + * A read-only instance, as the hook opens since #2913, writes through a + * connection of its own, opened only once it holds the index lock. Every + * index and sync holds that lock while it writes, so the repairs a writer's + * open runs, and the migration an index from before the table needs, cannot + * disturb a bulk load in progress. */ async healSegmentVocabIfEmpty(): Promise { const empty = (() => { @@ -2375,7 +2381,16 @@ export class CodeGraph { } try { if (!this.queries.isNameSegmentVocabEmpty()) return true; // raced: healed meanwhile - await this.rebuildNameSegmentVocab(); + if (!this.db.readOnly) { + await this.rebuildNameSegmentVocab(); + return true; + } + const writer = DatabaseConnection.open(this.db.getPath()); + try { + await this.rebuildNameSegmentVocab(new QueryBuilder(writer.getDb())); + } finally { + writer.close(); + } return true; } finally { this.fileLock.release(); @@ -2387,15 +2402,15 @@ export class CodeGraph { * Rebuild the segment vocabulary from the current graph, batched and * yielding — the upgrade-heal path for indexes built before the vocab table * existed. Runs inside the index mutex/lock (sync and - * healSegmentVocabIfEmpty hold them). + * healSegmentVocabIfEmpty hold them), through `queries`. */ - private async rebuildNameSegmentVocab(): Promise { + private async rebuildNameSegmentVocab(queries: QueryBuilder = this.queries): Promise { const maybeYield = createYielder(); const BATCH = 2000; for (let offset = 0; ; offset += BATCH) { - const names = this.queries.getDistinctNodeNames(BATCH, offset); + const names = queries.getDistinctNodeNames(BATCH, offset); if (names.length === 0) break; - this.queries.insertNameSegmentsBatch(names); + queries.insertNameSegmentsBatch(names); await maybeYield(); } }