From b74619ce43d0da61234baaef8416a797834122d5 Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Tue, 13 Oct 2026 18:11:57 -0600 Subject: [PATCH 1/2] perf(extraction): read the next file batch while the current one is dispatched Parse-loop time falls 3-4% on supabase and n8n (4 runs per build, ranges not overlapping). Whole-index wall time is within the run-to-run variation on all five measured projects. Graph fingerprints are identical and peak memory is within 1%. --- CHANGELOG.md | 1 + src/extraction/index.ts | 75 ++++++++++++++++++++++------------------- 2 files changed, 42 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7fb0db2095..7562165faf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -101,6 +101,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - **Syncing a changed file in a large Python project is faster.** When a Python class has no method of the name being linked (a test class that passes `Handler.__init__` as a value, say), the resolver checks every class with that method for one that derives from it. For each class whose base is a package import (`import unittest`, then `unittest.TestCase`) it parsed the whole file again, and its cache holds only a few files, so two such references took about a second each. A plain one-line `import a.b` is now read from its line; other forms still use the parse. In CPython a one-file sync takes a median 5.5 s instead of 7.7 s on Node and 5.2 s instead of 8.1 s on Bun (4 alternating runs per build, ranges not overlapping; the phase that retries unresolved references drops from 4.6 s to 2.3 s). A full index is within the run-to-run variation, and the graph is the same on CPython and Django (no edge differs). - **Questions to the MCP server are faster in a project with no `codegraph.json`.** While ranking results, CodeGraph checked for the project's `codegraph.json` once per result path, and with no such file each check raised and caught an error. It now asks the file system without raising, so `codegraph_explore` and the other query tools spend less CPU on most projects. Editing, adding or deleting `codegraph.json` is still picked up without a restart (#491). +- **Indexing reads the next batch of files while the current batch is being parsed.** The parse loop used to wait for each batch of 10 files to be read before handing any of it to the parser. The next batch is now read in the background, so the time spent parsing falls by a median 3–4% on large projects (supabase 5.85 s to 5.67 s, n8n 15.48 s to 14.85 s; 4 runs per build, ranges not overlapping on those two). Whole-index time is within the run-to-run variation on all five projects measured (Alamofire, gin, pretix, supabase, n8n), the graph is identical, and peak memory is within 1%. At most one extra batch of file contents is held in memory. - **Indexing a project with a bundled minified JavaScript file is faster.** For each reference in a one-line file such as `jquery.min.js`, the resolver counted characters from the start of the line to find the reference's column, many times per reference. It now skips straight to the column when the text before it is ASCII. Alamofire, whose docs folder carries jQuery, resolves its references in a median 0.93 s instead of 1.25 s (profiling runs on a busy host, 5,614 nodes and 13,673 edges either way). Separately, the resolver now looks nodes up by query only when there are fewer than 15,000 pending references and fewer than 0.3 per node; at either limit it loads the node table. A full index of a small project such as gin (about 4.4 references per node) therefore loads the table, saving about 60 ms of its one-second index, while a one-file sync still queries. In the 2026-10-12 benchmark on Node, a full index of Alamofire takes 1.81 s against upstream's 1.73 s and gin 1.15 s against 1.13 s ([results](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md)). - **Indexing a project with long minified lines that contain a non-ASCII character is faster.** The resolver turns a reference's column into a byte offset many times per reference, and when any character before the column was not ASCII it walked every character on the line each time. It now counts a 64-byte block at a time and walks characters only inside the block that holds the column. Indexing JetBrains Exposed, whose docs folder carries minified scripts, takes a median 25.7 s instead of 37.0 s (5 alternating runs per build, baseline against baseline within 0.3%; 28,689 nodes and 82,842 edges either way). nocodb, with no long lines, is unchanged within the run-to-run band (6.53 s against 6.74 s). - **Indexing a Kotlin project is faster.** When a call could reach several same-named functions, the resolver read the call site, checked the receiver and typed it again for each one, although all of that depends only on the call. It now does that once per call. Indexing JetBrains Exposed takes a median 31.2 s instead of 37.3 s (5 alternating runs per build, baseline against baseline within 0.1%; 28,689 nodes, 82,842 edges and the same 36,202 call edges either way). diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 7892b1c8bd..5db61bf7cd 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -2444,44 +2444,51 @@ export class ExtractionOrchestrator { } }; + // Read one batch of files in parallel (with path validation before any I/O). + // Never rejects: every per-file failure is returned as that file's `error`. + const readBatch = (batch: string[]) => Promise.all( + batch.map(async (fp) => { + try { + // Indexing read: follow in-root symlinks the directory walk already + // descended into (the `../` guard still applies) so files reached + // via an in-root symlink-to-outside still index (#935). + const fullPath = validatePathWithinRoot(this.rootDir, fp, { allowSymlinkEscape: true }); + if (!fullPath) { + logWarn('Path traversal blocked in batch reader', { filePath: fp }); + return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: new Error('Path traversal blocked') }; + } + // Stat first: a file over the size limit is stored as skipped + // without ever being read or decoded (#1910), so ten oversize + // fixtures in one I/O batch no longer cost their size in RSS. + const { stats, bytes } = await readBoundedSource(fullPath); + if (bytes === null) { + return { filePath: fp, content: oversizeStamp(stats.size), stats, error: null as Error | null }; + } + // Read bytes, not text: a `.ts` that is really an MPEG transport + // stream (#1910) is recognised from its head here, at no extra I/O, + // and never decoded or parsed. + if (isMpegTsBytes(fp, bytes)) { + return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: null as Error | null, skipped: true }; + } + const content = bytes.toString('utf-8'); + return { filePath: fp, content, stats, error: null as Error | null }; + } catch (err) { + return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: err as Error }; + } + }) + ); + const tParseLoop = Date.now(); + let nextRead = files.length > 0 ? readBatch(files.slice(0, FILE_IO_BATCH_SIZE)) : null; for (let i = 0; i < files.length; i += FILE_IO_BATCH_SIZE) { if (signal?.aborted) { aborted = true; break; } - const batch = files.slice(i, i + FILE_IO_BATCH_SIZE); - - // Read files in parallel (with path validation before any I/O) - const fileContents = await Promise.all( - batch.map(async (fp) => { - try { - // Indexing read: follow in-root symlinks the directory walk already - // descended into (the `../` guard still applies) so files reached - // via an in-root symlink-to-outside still index (#935). - const fullPath = validatePathWithinRoot(this.rootDir, fp, { allowSymlinkEscape: true }); - if (!fullPath) { - logWarn('Path traversal blocked in batch reader', { filePath: fp }); - return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: new Error('Path traversal blocked') }; - } - // Stat first: a file over the size limit is stored as skipped - // without ever being read or decoded (#1910), so ten oversize - // fixtures in one I/O batch no longer cost their size in RSS. - const { stats, bytes } = await readBoundedSource(fullPath); - if (bytes === null) { - return { filePath: fp, content: oversizeStamp(stats.size), stats, error: null as Error | null }; - } - // Read bytes, not text: a `.ts` that is really an MPEG transport - // stream (#1910) is recognised from its head here, at no extra I/O, - // and never decoded or parsed. - if (isMpegTsBytes(fp, bytes)) { - return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: null as Error | null, skipped: true }; - } - const content = bytes.toString('utf-8'); - return { filePath: fp, content, stats, error: null as Error | null }; - } catch (err) { - return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: err as Error }; - } - }) - ); + const fileContents = await nextRead!; + // Start reading the next batch before dispatching this one, so file I/O + // overlaps the parse window instead of idling the pool during each read. + // Holds at most one extra batch of file contents in memory. + const nextStart = i + FILE_IO_BATCH_SIZE; + nextRead = nextStart < files.length ? readBatch(files.slice(nextStart, nextStart + FILE_IO_BATCH_SIZE)) : null; // Dispatch each readable file into the bounded parse window; the window // stores results on the main thread as they arrive. From 44e0b1266772619ecbd71d3e955908ea5b0c2f40 Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Wed, 14 Oct 2026 03:38:07 -0600 Subject: [PATCH 2/2] fix(extraction): do not start reading the next file batch once indexing is cancelled --- src/extraction/index.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 5db61bf7cd..6512a728c5 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -2479,7 +2479,7 @@ export class ExtractionOrchestrator { ); const tParseLoop = Date.now(); - let nextRead = files.length > 0 ? readBatch(files.slice(0, FILE_IO_BATCH_SIZE)) : null; + let nextRead = files.length > 0 && !signal?.aborted ? readBatch(files.slice(0, FILE_IO_BATCH_SIZE)) : null; for (let i = 0; i < files.length; i += FILE_IO_BATCH_SIZE) { if (signal?.aborted) { aborted = true; break; } @@ -2488,7 +2488,7 @@ export class ExtractionOrchestrator { // overlaps the parse window instead of idling the pool during each read. // Holds at most one extra batch of file contents in memory. const nextStart = i + FILE_IO_BATCH_SIZE; - nextRead = nextStart < files.length ? readBatch(files.slice(nextStart, nextStart + FILE_IO_BATCH_SIZE)) : null; + nextRead = nextStart < files.length && !signal?.aborted ? readBatch(files.slice(nextStart, nextStart + FILE_IO_BATCH_SIZE)) : null; // Dispatch each readable file into the bounded parse window; the window // stores results on the main thread as they arrive.