diff --git a/CHANGELOG.md b/CHANGELOG.md index 474595495b..ab3c4e089f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - **Syncing a changed file in a large Python project is faster.** When a Python class has no method of the name being linked (a test class that passes `Handler.__init__` as a value, say), the resolver checks every class with that method for one that derives from it. For each class whose base is a package import (`import unittest`, then `unittest.TestCase`) it parsed the whole file again, and its cache holds only a few files, so two such references took about a second each. A plain one-line `import a.b` is now read from its line; other forms still use the parse. In CPython a one-file sync takes a median 5.5 s instead of 7.7 s on Node and 5.2 s instead of 8.1 s on Bun (4 alternating runs per build, ranges not overlapping; the phase that retries unresolved references drops from 4.6 s to 2.3 s). A full index is within the run-to-run variation, and the graph is the same on CPython and Django (no edge differs). - **The same sync resolves each Python class's bases once.** The scan above resolved the base classes of every candidate again for each reference that needed it. The bases of a class depend only on its own header and file, so they are now kept for the rest of the run. In CPython the phase that retries unresolved references drops from a median 2.29 s to 2.08 s on Node (-9%, 4 alternating runs per build, ranges not overlapping) and a one-file sync by about 3% (5.33 s to 5.15 s on the second sync); on Bun the sync time shows no clear change. A full index is within the run-to-run variation, and no edge, node or unresolved reference differs on CPython or Django. - **Questions to the MCP server are faster in a project with no `codegraph.json`.** While ranking results, CodeGraph checked for the project's `codegraph.json` once per result path, and with no such file each check raised and caught an error. It now asks the file system without raising, so `codegraph_explore` and the other query tools spend less CPU on most projects. Editing, adding or deleting `codegraph.json` is still picked up without a restart (#491). +- **Indexing reads the next batch of files while the current batch is being parsed.** The parse loop used to wait for each batch of 10 files to be read before handing any of it to the parser. The next batch is now read in the background, so the time spent parsing falls by a median 3–4% on large projects (supabase 5.85 s to 5.67 s, n8n 15.48 s to 14.85 s; 4 runs per build, ranges not overlapping on those two). Whole-index time is within the run-to-run variation on all five projects measured (Alamofire, gin, pretix, supabase, n8n), the graph is identical, and peak memory is within 1%. At most one extra batch of file contents is held in memory. - **Ranking looks at `codegraph.json` once per query, not once per result path.** The check above still ran for every path a query ranked; it now runs once per ranking pass and the answer is reused for each path. A `codegraph_explore` call is a median 4% faster on a Go project (gin), 7% on a NestJS project and 0.6% on a large TypeScript project (n8n), the first two inside the run-to-run variation of the measurement (5 rounds, answers identical in every round). Editing `codegraph.json` is still picked up without a restart. - **Indexing a project with a bundled minified JavaScript file is faster.** For each reference in a one-line file such as `jquery.min.js`, the resolver counted characters from the start of the line to find the reference's column, many times per reference. It now skips straight to the column when the text before it is ASCII. Alamofire, whose docs folder carries jQuery, resolves its references in a median 0.93 s instead of 1.25 s (profiling runs on a busy host, 5,614 nodes and 13,673 edges either way). Separately, the resolver now looks nodes up by query only when there are fewer than 15,000 pending references and fewer than 0.3 per node; at either limit it loads the node table. A full index of a small project such as gin (about 4.4 references per node) therefore loads the table, saving about 60 ms of its one-second index, while a one-file sync still queries. In the 2026-10-12 benchmark on Node, a full index of Alamofire takes 1.81 s against upstream's 1.73 s and gin 1.15 s against 1.13 s ([results](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md)). - **Indexing a project with long minified lines that contain a non-ASCII character is faster.** The resolver turns a reference's column into a byte offset many times per reference, and when any character before the column was not ASCII it walked every character on the line each time. It now counts a 64-byte block at a time and walks characters only inside the block that holds the column. Indexing JetBrains Exposed, whose docs folder carries minified scripts, takes a median 25.7 s instead of 37.0 s (5 alternating runs per build, baseline against baseline within 0.3%; 28,689 nodes and 82,842 edges either way). nocodb, with no long lines, is unchanged within the run-to-run band (6.53 s against 6.74 s). diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 7892b1c8bd..6512a728c5 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -2444,44 +2444,51 @@ export class ExtractionOrchestrator { } }; + // Read one batch of files in parallel (with path validation before any I/O). + // Never rejects: every per-file failure is returned as that file's `error`. + const readBatch = (batch: string[]) => Promise.all( + batch.map(async (fp) => { + try { + // Indexing read: follow in-root symlinks the directory walk already + // descended into (the `../` guard still applies) so files reached + // via an in-root symlink-to-outside still index (#935). + const fullPath = validatePathWithinRoot(this.rootDir, fp, { allowSymlinkEscape: true }); + if (!fullPath) { + logWarn('Path traversal blocked in batch reader', { filePath: fp }); + return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: new Error('Path traversal blocked') }; + } + // Stat first: a file over the size limit is stored as skipped + // without ever being read or decoded (#1910), so ten oversize + // fixtures in one I/O batch no longer cost their size in RSS. + const { stats, bytes } = await readBoundedSource(fullPath); + if (bytes === null) { + return { filePath: fp, content: oversizeStamp(stats.size), stats, error: null as Error | null }; + } + // Read bytes, not text: a `.ts` that is really an MPEG transport + // stream (#1910) is recognised from its head here, at no extra I/O, + // and never decoded or parsed. + if (isMpegTsBytes(fp, bytes)) { + return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: null as Error | null, skipped: true }; + } + const content = bytes.toString('utf-8'); + return { filePath: fp, content, stats, error: null as Error | null }; + } catch (err) { + return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: err as Error }; + } + }) + ); + const tParseLoop = Date.now(); + let nextRead = files.length > 0 && !signal?.aborted ? readBatch(files.slice(0, FILE_IO_BATCH_SIZE)) : null; for (let i = 0; i < files.length; i += FILE_IO_BATCH_SIZE) { if (signal?.aborted) { aborted = true; break; } - const batch = files.slice(i, i + FILE_IO_BATCH_SIZE); - - // Read files in parallel (with path validation before any I/O) - const fileContents = await Promise.all( - batch.map(async (fp) => { - try { - // Indexing read: follow in-root symlinks the directory walk already - // descended into (the `../` guard still applies) so files reached - // via an in-root symlink-to-outside still index (#935). - const fullPath = validatePathWithinRoot(this.rootDir, fp, { allowSymlinkEscape: true }); - if (!fullPath) { - logWarn('Path traversal blocked in batch reader', { filePath: fp }); - return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: new Error('Path traversal blocked') }; - } - // Stat first: a file over the size limit is stored as skipped - // without ever being read or decoded (#1910), so ten oversize - // fixtures in one I/O batch no longer cost their size in RSS. - const { stats, bytes } = await readBoundedSource(fullPath); - if (bytes === null) { - return { filePath: fp, content: oversizeStamp(stats.size), stats, error: null as Error | null }; - } - // Read bytes, not text: a `.ts` that is really an MPEG transport - // stream (#1910) is recognised from its head here, at no extra I/O, - // and never decoded or parsed. - if (isMpegTsBytes(fp, bytes)) { - return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: null as Error | null, skipped: true }; - } - const content = bytes.toString('utf-8'); - return { filePath: fp, content, stats, error: null as Error | null }; - } catch (err) { - return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: err as Error }; - } - }) - ); + const fileContents = await nextRead!; + // Start reading the next batch before dispatching this one, so file I/O + // overlaps the parse window instead of idling the pool during each read. + // Holds at most one extra batch of file contents in memory. + const nextStart = i + FILE_IO_BATCH_SIZE; + nextRead = nextStart < files.length && !signal?.aborted ? readBatch(files.slice(nextStart, nextStart + FILE_IO_BATCH_SIZE)) : null; // Dispatch each readable file into the bounded parse window; the window // stores results on the main thread as they arrive.