perf(js): indexing a JavaScript or TypeScript project is faster

CHANGELOG.md

@@ -118,6 +118,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - **Indexing a project with long minified lines that contain a non-ASCII character is faster.** The resolver turns a reference's column into a byte offset many times per reference, and when any character before the column was not ASCII it walked every character on the line each time. It now counts a 64-byte block at a time and walks characters only inside the block that holds the column. Indexing JetBrains Exposed, whose docs folder carries minified scripts, takes a median 25.7 s instead of 37.0 s (5 alternating runs per build, baseline against baseline within 0.3%; 28,689 nodes and 82,842 edges either way). nocodb, with no long lines, is unchanged within the run-to-run band (6.53 s against 6.74 s).
 - **Indexing a Kotlin project is faster.** When a call could reach several same-named functions, the resolver read the call site, checked the receiver and typed it again for each one, although all of that depends only on the call. It now does that once per call. Indexing JetBrains Exposed takes a median 31.2 s instead of 37.3 s (5 alternating runs per build, baseline against baseline within 0.1%; 28,689 nodes, 82,842 edges and the same 36,202 call edges either way).
 - **Indexing a Kotlin project uses less memory.** Each of the resolver's worker threads compiled and kept its own copy of up to 512 name patterns, and a pattern with a capture group or a word boundary holds up to about 800 KB once compiled. The threads now share one cache of compiled patterns and each keeps only its 16 to 32 most recent. Peak memory while indexing JetBrains Exposed falls from a median 2,409 MiB to 1,939 MiB (-19.5%) and ktor from 2,492 MiB to 1,992 MiB (-20.1%), and Exposed also indexes 5.9% faster (9.63 s to 9.06 s). Alamofire peaks 7.6% lower; pretix and CPython show no clear change. 3 alternating runs per build on Node 24, each change outside the spread of two identical builds of the change, with identical graphs.
+- **Indexing a JavaScript or TypeScript project is faster.** Every JavaScript or TypeScript file that contains the word `require` was parsed a second time to look for `require('./x')` calls, although most such files only say `typeof require`, `required:` or `require('lodash')`. Now the text is read first, and the second parse runs only when it could hold a `require` call whose quoted argument is a relative path, a `#` import or a path with a slash, or when the native parser found a syntax error in the file. A full index of pretix takes a median 9.67 s instead of 10.17 s (its parse phase 2.07 s instead of 2.48 s), and supabase uses 6.7% less CPU (69.9 s instead of 75.0 s; parse phase 5.70 s instead of 6.19 s) with its wall time inside the run-to-run variation. Node 24, 3 alternating runs per build, the same graph either way.
 - **Code written without a space before a name resolves like spaced code.** Where one token ends exactly where the next begins, the resolver read the syntax at that point as the earlier token: in `x=1;local function helper()` it saw the `;`, so the Lua local read as global and calls from other files linked to it, and in `if ($x instanceof Foo)$x->bar();` it saw the `)`, so the PHP guard did not type `$x`. It now reads the token that starts there, the same way the TypeScript side already did. On json, jq and CPython the graph is unchanged.
 - **A Kotlin infix call on a parenthesized receiver now reaches the right class.** `(b) put 1`, `(b)put(1)` and `((b)) put (1)` linked to whichever class had a `put`, so with `b: A` and another class `B` that also declared `put`, the call landed on `B::put`. They now follow what `b` holds, as `b put 1` already did. An extension on a project class called through a typed receiver is still not linked, as before; the parenthesized form no longer guesses it by name.
 - **A Vite plugin that spreads another plugin no longer turns off a React Router app's file routes.** Any `...` in a root `vite.config`, `react-router.config` or `remix.config` was taken for custom route configuration, so React Router's own RSC playground, which merges `...mdx({ … })` into a plugin entry, got no file routes. A spread of a call inside a `plugins` list is now ignored, since it only builds a plugin; a spread of anything else (`...shared`, `...makeConfig()` at the top level, `[...plugins]`, or inside a plugin call's arguments) still turns file routes off. The playground goes from 0 to 12 file routes.

__tests__/commonjs-require-prefilter.test.ts

@@ -0,0 +1,89 @@
+/**
+ * The CommonJS require pass re-parses a file only when its text could hold a
+ * literal `require('./x')` call. The text check is sound only for a tree
+ * without syntax errors (recovery can make a call of `require@('./x')`), so it
+ * applies only when the kernel reports that its parse of the same text was
+ * clean.
+ */
+import { describe, it, expect, vi, beforeEach } from 'vitest';
+import { commonJsRequireRefs, mayRequireModulePath } from '../src/extraction/commonjs-requires';
+import { tryKernelExtractRaw } from '../src/extraction/kernel';
+import * as parseTree from '../src/extraction/parse-tree';
+import type { Language } from '../src/types';
+
+// Absolute, so Bun resolves it the same way (oven-sh/bun#43685).
+vi.mock(`${__dirname}/../src/extraction/parse-tree`, async (importOriginal) => {
+  const actual = await importOriginal<typeof import('../src/extraction/parse-tree')>();
+  return { ...actual, parseSourceTreeSync: vi.fn(actual.parseSourceTreeSync) };
+});
+const parse = vi.mocked(parseTree.parseSourceTreeSync);
+beforeEach(() => { parse.mockClear(); });
+
+const names = (source: string, language: Language, clean: boolean) =>
+  commonJsRequireRefs('entry.ts', source, language, clean).map((ref) => ref.referenceName);
+
+describe('CommonJS require pre-filter', () => {
+  it('skips the re-parse of a clean file that only mentions require', () => {
+    const source = `// images that do not require color spaces
+if (typeof require === 'function') module.exports = factory(require('jquery'));
+const f = __webpack_require__(12), g = jest.requireActual('./x'), required = true;`;
+    expect(names(source, 'javascript', true)).toEqual([]);
+    expect(parse).not.toHaveBeenCalled();
+  });
+
+  it('still parses when a clean file has a literal module-path call, through comments, optional calls and empty slots', () => {
+    expect(names("const a = require /* loader */ ?.( // path\n './a');", 'javascript', true)).toEqual(['./a']);
+    expect(names("import x = require('./b');", 'typescript', true)).toEqual(['./b']);
+    expect(names('const c = require<Mod>(`./c`);', 'typescript', true)).toEqual(['./c']);
+    // The grammar allows an empty argument slot, so this file parses cleanly.
+    expect(names("require(, './d');", 'javascript', true)).toEqual(['./d']);
+    expect(parse).toHaveBeenCalledTimes(4);
+  });
+
+  it('keeps the tree walk for a file that did not parse cleanly', () => {
+    // Error recovery reads this as a call; the text check would not.
+    expect(mayRequireModulePath("require@('#x')", 'javascript')).toBe(false);
+    expect(names("require@('#x')", 'javascript', false)).toEqual(['#x']);
+  });
+
+  it('never rules out a clean file the walk reads a require from', () => {
+    const pieces = ['require', 'require', ',', ', ', '...', ' ', '\n', '\t', '/* c */', '// c\n', '?.', '(', '(', ')', "'./a'", '"b/c"', '`./t`', "'pkg'", "'#x'",
+      "'.'", '<T>', '<', '>', '<!-- x\n', '-->', '\u2028', '\u00a0', '\u200b', ';', 'x', '=', 'typeof ', '.', ',', '`', '${', '}', '\\',
+      "'\\x2e/a'", 'import r = ', 'function(require){', '{', '[', ']', '!', '@', 'new ', 'const ', '+', '-', '#', ':', '"', "'", '*', '/'];
+    const literals = ["'./a'", '"b/c"', '`./t`', "'#x'", "'..'", "'pkg'", "'\\x2e/a'", '`./${x}`'];
+    let seed = 7;
+    const pick = <T>(list: T[]) => {
+      seed = (seed + 0x6d2b79f5) >>> 0; // mulberry32
+      let z = Math.imul(seed ^ (seed >>> 15), seed | 1);
+      z ^= z + Math.imul(z ^ (z >>> 7), z | 61);
+      return list[((z ^ (z >>> 14)) >>> 0) % list.length]!;
+    };
+    const junk = (max: number) => Array.from({ length: pick([...Array(max + 1).keys()]) }, () => pick(pieces)).join('');
+    const languages: Language[] = ['javascript', 'jsx', 'typescript', 'tsx'];
+    let checked = 0;
+    let read = 0;
+    for (let n = 0; n < 6000; n++) {
+      const source = junk(2) + 'require' + junk(1) + '(' + junk(1) + pick(literals) + junk(1) + ')' + junk(2);
+      const language = languages[n % 4]!;
+      const tree = parseTree.parseSourceTreeSync(source, language);
+      const clean = !!tree && !tree.rootNode.hasError;
+      tree?.delete();
+      if (!clean) continue;
+      checked++;
+      const walked = names(source, language, false);
+      if (walked.length) read++;
+      expect(names(source, language, true), `${language}: ${JSON.stringify(source)}`).toEqual(walked);
+    }
+    expect(checked).toBeGreaterThan(400);
+    expect(read).toBeGreaterThan(150);
+  });
+});
+
+describe('kernel parse flag', () => {
+  it('is set for a clean parse of the file itself and clear otherwise', () => {
+    expect(tryKernelExtractRaw('a.js', "const a = require('./a');\n", 'javascript')?.sourceParsedClean).toBe(true);
+    expect(tryKernelExtractRaw('b.js', "const b = require@('./b');\n", 'javascript')?.sourceParsedClean).toBe(false);
+    // A Flow file is parsed from a blanked copy, not its own text.
+    expect(tryKernelExtractRaw('c.js', "// @flow\ntype T = {| a: number |};\nconst c = require('./c');\n", 'tsx')?.sourceParsedClean).toBe(false);
+  });
+});

codegraph-kernel/src/buffers.rs

@@ -12,7 +12,10 @@
 //!
 //! meta (40 bytes):
 //!   0   u8   KERNEL_ABI_VERSION
-//!   1   [3]  pad
+//!   1   u8   flags: bit 0 = the walker's tree parsed without errors. An older
+//!            v3 kernel leaves the byte 0, which reads as "unknown", so the
+//!            layout stays v3.
+//!   2   [2]  pad
 //!   4   u32  node count
 //!   8   u32  edge count
 //!   12  u32  ref count
@@ -528,7 +531,7 @@ impl EmitOut {
             reexport_imports: 0,
         };
         EmitOut {
-            meta: build_meta(&tables, self.arena.len() as u32, NONE_STR, duration_ms),
+            meta: build_meta(&tables, self.arena.len() as u32, NONE_STR, duration_ms, 0),
             nodes: Vec::new(),
             edges: Vec::new(),
             refs: Vec::new(),
@@ -577,10 +580,14 @@ pub fn parse_collapse_warning(
     arena.put(&json)
 }
 
-pub fn build_meta(t: &Tables, arena_len: u32, errors_json: StrRef, duration_ms: f64) -> Vec<u8> {
+/// Meta byte 1, bit 0: the tree the walker read has no ERROR or MISSING node.
+pub const META_PARSED_CLEAN: u8 = 1;
+
+pub fn build_meta(t: &Tables, arena_len: u32, errors_json: StrRef, duration_ms: f64, flags: u8) -> Vec<u8> {
     let mut m = Vec::with_capacity(META_SIZE);
     m.push(KERNEL_ABI_VERSION);
-    m.extend_from_slice(&[0u8; 3]);
+    m.push(flags);
+    m.extend_from_slice(&[0u8; 2]);
     m.extend_from_slice(&t.node_count.to_le_bytes());
     m.extend_from_slice(&t.edge_count.to_le_bytes());
     m.extend_from_slice(&t.ref_count.to_le_bytes());
@@ -643,7 +650,7 @@ mod tests {
             from_id_str: NONE_STR,
         });
         assert_eq!(t.refs.len(), REF_ROW_SIZE);
-        let meta = build_meta(&t, a.len(), NONE_STR, 0.0);
+        let meta = build_meta(&t, a.len(), NONE_STR, 0.0, 0);
         assert_eq!(meta.len(), META_SIZE);
     }
 }
@@ -698,7 +705,8 @@ pub fn push_file_node_with_docstring<'p>(
 pub fn finish(mut arena: Arena, tables: Tables, has_error: bool, file_path: &str, t0: std::time::Instant) -> EmitOut {
     let duration_ms = t0.elapsed().as_secs_f64() * 1000.0;
     let errors_json = parse_collapse_warning(&mut arena, &tables, has_error, file_path);
-    let meta = build_meta(&tables, arena.len(), errors_json, duration_ms);
+    let flags = if has_error { 0 } else { META_PARSED_CLEAN };
+    let meta = build_meta(&tables, arena.len(), errors_json, duration_ms, flags);
     EmitOut {
         meta,
         nodes: tables.nodes,

src/extraction/commonjs-requires.ts

@@ -40,8 +40,65 @@ function scopeFor(node: TreeNode, functionScoped = false): TreeNode {
   return node;
 }
 
-export function commonJsRequireRefs(filePath: string, source: string, language: Language): UnresolvedReference[] {
+/**
+ * Index past the whitespace and comments the grammar skips between tokens, or -1
+ * where the text could be something this reader does not follow (non-ASCII
+ * whitespace, an HTML-like comment, an unclosed block comment).
+ */
+function skipExtras(source: string, i: number): number {
+  for (;;) {
+    const c = source.charCodeAt(i);
+    if (c === 0x20 || (c >= 0x09 && c <= 0x0d)) { i++; continue; }
+    if (c === 0x2f && source.charCodeAt(i + 1) === 0x2f) {
+      for (i += 2; i < source.length && !'\n\r\u2028\u2029'.includes(source[i]!); i++);
+      continue;
+    }
+    if (c === 0x2f && source.charCodeAt(i + 1) === 0x2a) {
+      const close = source.indexOf('*/', i + 2);
+      if (close < 0) return -1;
+      i = close + 2;
+      continue;
+    }
+    if (c > 0x7e || c < 0x20 || source.startsWith('<!--', i) || source.startsWith('-->', i)) return -1;
+    return i;
+  }
+}
+
+/**
+ * Whether an error-free parse of the text could hold a call the tree walk
+ * reads: `require`, an optional `?.`, `(`, then a quoted module path. It only
+ * rules files out: anything it cannot read (type arguments, escapes, odd
+ * spacing) counts as a call and the walk decides. Most files that say
+ * "require" have no such call (`typeof require`, prose, `required`), and their
+ * re-parse was the cost. A tagged template (require`…`) is no call here: the
+ * walk never reads one. A tree with errors is out of its reach: recovery can
+ * make a call of `require@('./x')`.
+ */
+export function mayRequireModulePath(source: string, language: Language): boolean {
+  const typed = language === 'typescript' || language === 'tsx';
+  for (let at = source.indexOf('require'); at !== -1; at = source.indexOf('require', at + 7)) {
+    let i = skipExtras(source, at + 7);
+    if (i >= 0 && source.startsWith('?.', i)) i = skipExtras(source, i + 2);
+    if (i < 0 || (typed && source[i] === '<')) return true;
+    if (source[i] !== '(') continue;
+    // The grammar allows empty argument slots: `require(, './x')` is clean.
+    do i = skipExtras(source, i + 1); while (i >= 0 && source[i] === ',');
+    if (i < 0) return true;
+    const quote = source[i];
+    if (quote !== "'" && quote !== '"' && quote !== '`') continue;
+    let end = i + 1;
+    for (; end < source.length && source[end] !== quote; end++) {
+      if (source[end] === '\\' || (quote === '`' ? source[end] === '$' : '\n\r\u2028\u2029'.includes(source[end]!))) return true;
+    }
+    if (end >= source.length || MODULE_SPECIFIER.test(source.slice(i + 1, end))) return true;
+  }
+  return false;
+}
+
+/** `parsedClean`: the caller's parser read `source` without a syntax error (the kernel's flag), so the text check may skip the re-parse. */
+export function commonJsRequireRefs(filePath: string, source: string, language: Language, parsedClean = false): UnresolvedReference[] {
   if (!COMMONJS_LANGUAGES.has(language) || !source.includes('require')) return [];
+  if (parsedClean && !mayRequireModulePath(source, language)) return [];
   const tree = parseSourceTreeSync(source, language);
   if (!tree) return [];
   const scopes: TreeNode[] = [];

src/extraction/kernel/index.ts

@@ -21,6 +21,7 @@ import { captureLiterals } from '../literal-capture';
 import {
   KERNEL_ABI_VERSION as LAYOUT_ABI,
   META as LAYOUT_META,
+  META_PARSED_CLEAN,
   NONE as LAYOUT_NONE,
 } from './layout';
 
@@ -68,6 +69,13 @@ export interface KernelRawResult {
   buffers: NonNullable<ExtractionResult['kernelBuffers']>;
   counts: { nodes: number; edges: number; refs: number };
   errors: ExtractionResult['errors'];
+  /** The kernel parsed `source` itself, not a pre-parsed rewrite, without a syntax error. */
+  sourceParsedClean: boolean;
+}
+
+/** Whether the walker's tree is `source`'s own and has no ERROR or MISSING node. */
+function sourceParsedClean(meta: Buffer, pre: string, source: string): boolean {
+  return (meta.readUInt8(LAYOUT_META.flags) & META_PARSED_CLEAN) !== 0 && pre === source;
 }
 
 /**
@@ -106,7 +114,7 @@ export function tryKernelExtractRaw(
         buffers.arena.toString('utf8', errorsOff, errorsOff + errorsLen)
       ) as ExtractionResult['errors'];
     }
-    return { buffers: { ...buffers, literalSource: pre }, counts, errors };
+    return { buffers: { ...buffers, literalSource: pre }, counts, errors, sourceParsedClean: sourceParsedClean(meta, pre, source) };
   } catch (err) {
     const message = err instanceof Error ? err.message : String(err);
     // `defer:` is the stack guard's signal; the generic extractor takes the
@@ -179,7 +187,7 @@ export function tryKernelExtract(
     // finding holders.
     captureLiterals(pre, result.nodes);
     // Read beside the tables, as TreeSitterExtractor does (a CommonJS `require`).
-    result.unresolvedReferences.push(...commonJsRequireRefs(filePath, source, language));
+    result.unresolvedReferences.push(...commonJsRequireRefs(filePath, source, language, sourceParsedClean(buffers.meta, pre, source)));
     result.durationMs = Date.now() - t0;
     return result;
   } catch (err) {

src/extraction/kernel/layout.ts

@@ -30,6 +30,7 @@ export const BINDING_ROW_SIZE = 64;
 /** meta byte offsets */
 export const META = {
   version: 0, // u8
+  flags: 1, // u8 (bit 0 META_PARSED_CLEAN; 0 from an older v3 kernel)
   nodeCount: 4, // u32
   edgeCount: 8, // u32
   refCount: 12, // u32
@@ -40,6 +41,9 @@ export const META = {
   bindingCount: 36, // u32 (v3)
 } as const;
 
+/** META.flags bit: the walker's tree has no ERROR or MISSING node. */
+export const META_PARSED_CLEAN = 1;
+
 /** binding row byte offsets (v3) — see BindingRow in buffers.rs */
 export const BINDING = {
   kind: 0, // u8 — BINDING_KINDS index

src/extraction/parse-worker.ts

@@ -63,7 +63,7 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st
             nodes: [],
             edges: [],
             // Read beside the kernel's tables (extractFromSource's own post-step).
-            unresolvedReferences: commonJsRequireRefs(filePath!, content!, language),
+            unresolvedReferences: commonJsRequireRefs(filePath!, content!, language, raw.sourceParsedClean),
             errors: raw.errors,
             durationMs: 0,
             kernelBuffers: raw.buffers,