fix(ts): a type-only re-export like export type * from './types' now parses

CHANGELOG.md

@@ -282,6 +282,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - Python imports now resolve the way Python finds modules: from your project's package roots — the repository root, `src/` layouts, each service with its own `pyproject.toml`, `setup.cfg` or `setup.py`, and the package directories those declare — so `from shop.cart import total` reaches `src/shop/cart.py`, `from pkg import name` reaches what the package's `__init__.py` defines or re-exports (respecting `__all__`), and `import json` or `import requests` no longer links to a same-named file nested in a test fixture, an example, a vendored copy or another package. After a module, package or `__init__.py` is added or removed, a package's `__init__.py` is edited, or a build config is edited, the next sync brings Python imports back in line with a fresh index; the file watcher doesn't react to a build-config edit by itself, so run `codegraph sync` after one. Re-index Python projects after upgrading. (#1704)
 - An Express route keeps its named handler when a middleware before it contains an arrow function. `rateLimit({ keyGenerator: (req) => req.ip })` or `(req, res, next) => next()` no longer hides the handler at the end of the route (re-index to update an existing project).
 - A TypeScript type re-exported through a barrel (`export type { Foo } from`, `export { type Foo } from`, `export type * from`) now links to its real declaration instead of a same-named type elsewhere in the project, and import lists with comments, string names, or a JSDoc `@import` are read correctly. Re-index to update an existing project.
+- In TypeScript, a file that re-exports a whole module's types, like `export type * from './types'` or `export type * as Types from './types'`, now parses cleanly. Before, the parser couldn't read that line, so a barrel made of re-exports was reported as producing no symbols and `codegraph status` listed it among the files it could not parse. Re-index a project that uses it after upgrading. Thanks @gavinwilliams for the report and @danusha2345 for the pull request. (#2474)
 - Git sync hooks now refresh the project they were installed for, so a project inside a larger repository stays current after commit, pull and checkout; installing them no longer breaks an existing hook written in another language, and runs even when your own hook ends with `exit`.
 - In TypeScript and JavaScript, a call through a field the calling class inherits, like `this._store.add(…)` in a class whose base class declares `_store`, now links to the method of the field's class, with the base class found the way the subclass's file imports it, under its own name, an alias or a re-export. Before, such a call linked only when some other class with the calling class's name happened to declare the field, which could be another app's class: that last resort now needs a type the calling file imports or declares. A call on a field or variable whose type comes from a package outside your project, like `TelemetryReporter` from `@vscode/extension-telemetry`, no longer links to a project class that only shares the type's name, unless a file in your project declares that package (`declare module '…'`). Re-index TypeScript and JavaScript projects after upgrading.
 - In TypeScript and JavaScript, a call on a variable, parameter or field whose class the file imports, like `store.add(…)` with `store: DisposableStore`, now links to the method of the class that import names when other classes in the project share its name, such as a vendored copy, a test mock or another app's class. Before, it linked to whichever of them was indexed first, or to the one in the nearest folder: in vscode, calls on classes imported from vscode's own `vs/base` and `vs/platform` reached the copies the Copilot extension vendors or its test fixtures, in bitwarden the desktop app's `initService.init()` reached the browser extension's `InitService`, and in react-native `Keyboard` and `WebSocket` reached the Jest mock of `NativeEventEmitter`. A call whose first match was a test file's class, which then linked to nothing, now reaches the imported class too. The import is followed through re-exports, `import type`, namespace imports like `new async.Delayer()` and `require(…)`. Re-index TypeScript and JavaScript projects after upgrading.

__tests__/status-index-health.test.ts

@@ -5,9 +5,9 @@
  * "Up to date" used to mean only "every file's content hash matches the
  * index". Two kinds of file pass that check with nothing usable in the graph:
  *
- *   - a file the parser could not read — `export type * from` is valid
- *     TypeScript 5.0 the bundled grammar has no rule for, so the file is stored
- *     with a parse error and no symbols (#2336);
+ *   - a file the parser could not read, like one left with merge-conflict
+ *     markers, which is stored with a parse error and no symbols (#2336; the
+ *     report's file, TypeScript 5.0's `export type * from`, parses since #2474);
  *   - a row stored without its symbols — by an engine whose grammar failed to
  *     load (#2335), or the #1541 wipe — which `codegraph sync` re-indexes.
  *
@@ -96,7 +96,12 @@ describe('status reports files indexed without their symbols (#2336, #2335)', ()
   it('a file the parser could not read keeps status from reporting "up to date"', async () => {
     fs.writeFileSync(
       path.join(dir, 'src', 'index.ts'),
-      'export * from "./constants/index.js";\nexport type * from "./types/database.js";\n'
+      'export * from "./constants/index.js";\n' +
+        '<<<<<<< HEAD\n' +
+        'export type { Row } from "./types/database.js";\n' +
+        '=======\n' +
+        'export type { RowId } from "./types/database.js";\n' +
+        '>>>>>>> feature\n'
     );
     await index();
 

__tests__/ts-export-type-star.test.ts

@@ -0,0 +1,240 @@
+/**
+ * TypeScript 5.0's type-only re-export of a whole module parses (#2474):
+ *
+ *   export { FooClient, type FooClientOptions } from "./client.js";
+ *   export type * from "./types.js";
+ *   export type * as Shapes from "./shapes.js";
+ *
+ * The grammar has no rule for `export type *` (upstream's neither), so `type`
+ * parsed as an error. A barrel of re-exports was then reported as producing no
+ * symbols, `codegraph status` listed it among the files it could not parse,
+ * and the native kernel handed every file holding such a line to the wasm
+ * parser. The TypeScript extractor's preParse now blanks that `type` to spaces
+ * where a statement starts, offsets kept, so the line reads as the `export *`
+ * it is to the graph, and both engines parse the same bytes. The import
+ * resolver reads the statement as written from the file, so a name imported
+ * through the barrel still reaches the file that declares it.
+ */
+import { describe, it, expect, beforeAll, beforeEach, afterEach, afterAll } from 'vitest';
+import * as fs from 'fs';
+import * as os from 'os';
+import * as path from 'path';
+import { CodeGraph } from '../src';
+import { extractFromSource } from '../src/extraction';
+import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars';
+import { tryKernelExtract, resetKernelForTests } from '../src/extraction/kernel';
+import { blankExportTypeStar } from '../src/extraction/languages/typescript';
+import type { ExtractionResult, Language } from '../src/types';
+
+const KERNEL_PATH = path.join(
+  __dirname,
+  '..',
+  'codegraph-kernel',
+  'prebuilds',
+  `${process.platform}-${process.arch}`,
+  'codegraph-kernel.node'
+);
+const kernelAvailable = fs.existsSync(KERNEL_PATH) || process.env.CODEGRAPH_KERNEL_EXPECT === '1';
+
+/** The issue's barrel, plus the namespace form. */
+const BARREL = [
+  'export { FooClient, FooApiError, type FooClientOptions } from "./client.js";',
+  'export { DAYS_OF_WEEK } from "./types.js";',
+  'export type * from "./types.js";',
+  'export type * as Shapes from "./shapes.js";',
+  '',
+].join('\n');
+
+describe('blankExportTypeStar', () => {
+  it('blanks the `type` of a type-only star re-export, keeping every offset', () => {
+    const cases: Array<[string, string]> = [
+      ['export type * from "./types";', 'export      * from "./types";'],
+      ['export type * as Types from "./types";', 'export      * as Types from "./types";'],
+      ["export type * as 'my types' from './types';", "export      * as 'my types' from './types';"],
+      ['export type *from"./types";', 'export      *from"./types";'],
+      ['export\ttype\t*\tfrom "./types";', 'export\t    \t*\tfrom "./types";'],
+      // `from` on the next line.
+      ['export type *\n  from "./types";', 'export      *\n  from "./types";'],
+      // Indented in an ambient module, and after `{`, `;` or a block comment on the same line.
+      ['declare module "m" {\n  export type * from "./y";\n}', 'declare module "m" {\n  export      * from "./y";\n}'],
+      ['declare module "m" { export type * from "./y"; }', 'declare module "m" { export      * from "./y"; }'],
+      ['export * from "./a"; export type * from "./b";', 'export * from "./a"; export      * from "./b";'],
+      ['/* types */ export type * from "./b";', '/* types */ export      * from "./b";'],
+      ['export * from "./a";\r\nexport type * from "./b";\r\n', 'export * from "./a";\r\nexport      * from "./b";\r\n'],
+    ];
+    for (const [source, expected] of cases) {
+      const blanked = blankExportTypeStar(source);
+      expect(blanked).toBe(expected);
+      expect(blanked).toHaveLength(source.length);
+    }
+  });
+
+  it('leaves everything else as written', () => {
+    for (const source of [
+      'export type Star = "*";',
+      'export type { A } from "./a";',
+      'export { type A } from "./a";',
+      'export * from "./a";',
+      'export * as ns from "./a";',
+      // Not where a statement starts: a mention in a comment, a doc or a string.
+      '// Re-export them with export type * from "./types".',
+      '/**\n * export type * from "./types";\n */',
+      'const hint = "export type * from \'./types\'";',
+    ]) {
+      expect(blankExportTypeStar(source)).toBe(source);
+    }
+  });
+});
+
+const ENV_KEYS = ['CODEGRAPH_KERNEL', 'CODEGRAPH_KERNEL_LANGS'] as const;
+
+describe('a type-only star re-export parses in both engines', () => {
+  let savedEnv: Record<string, string | undefined> = {};
+
+  beforeAll(async () => {
+    await initGrammars();
+    await loadGrammarsForLanguages(['typescript', 'tsx']);
+  });
+
+  beforeEach(() => {
+    savedEnv = Object.fromEntries(ENV_KEYS.map((k) => [k, process.env[k]]));
+    resetKernelForTests();
+  });
+
+  afterEach(() => {
+    for (const k of ENV_KEYS) {
+      if (savedEnv[k] === undefined) delete process.env[k];
+      else process.env[k] = savedEnv[k];
+    }
+    resetKernelForTests();
+  });
+
+  function extract(backend: 'kernel' | 'wasm', file: string, source: string, language: Language): ExtractionResult {
+    if (backend === 'wasm') {
+      process.env.CODEGRAPH_KERNEL = '0';
+      return extractFromSource(file, source, language);
+    }
+    delete process.env.CODEGRAPH_KERNEL;
+    process.env.CODEGRAPH_KERNEL_LANGS = 'all';
+    // Null when the kernel defers the file to the wasm parser, as it does every file with a parse error.
+    const result = tryKernelExtract(file, source, language);
+    expect(result, `kernel extraction of ${file}`).not.toBeNull();
+    return result!;
+  }
+
+  const backends = kernelAvailable ? (['kernel', 'wasm'] as const) : (['wasm'] as const);
+
+  for (const crlf of [false, true]) {
+    const eol = (s: string) => (crlf ? s.replace(/\n/g, '\r\n') : s);
+    for (const [file, language] of [
+      ['src/index.ts', 'typescript'],
+      ['src/index.tsx', 'tsx'],
+    ] as const) {
+      it.each(backends)(`a barrel in ${file}${crlf ? ' (CRLF)' : ''}: %s`, (backend) => {
+        const result = extract(backend, file, eol(BARREL), language);
+        expect(result.errors).toEqual([]);
+        // Each statement imports its module, as `export * from` does.
+        expect(result.nodes.filter((n) => n.kind === 'import').map((n) => `${n.name}@${n.startLine}`)).toEqual([
+          './client.js@1',
+          './types.js@2',
+          './types.js@3',
+          './shapes.js@4',
+        ]);
+        expect(result.unresolvedReferences.map((r) => `${r.referenceName}@${r.line}`).sort()).toEqual([
+          './client.js@1',
+          './shapes.js@4',
+          './types.js@2',
+          './types.js@3',
+          'DAYS_OF_WEEK@2',
+          'FooApiError@1',
+          'FooClient@1',
+          'FooClientOptions@1',
+        ]);
+      });
+    }
+  }
+
+  it.each(backends)('in a declaration file\'s declare module body: %s', (backend) => {
+    const source = 'declare module "sdk" {\n  export type * from "./types";\n  export function connect(): void;\n}\n';
+    const result = extract(backend, 'typings/sdk.d.ts', source, 'typescript');
+    expect(result.errors).toEqual([]);
+    expect(result.nodes.filter((n) => n.kind !== 'file').map((n) => `${n.kind} ${n.name}`)).toEqual([
+      'import ./types',
+      'function connect',
+    ]);
+  });
+
+  it.runIf(kernelAvailable)('the kernel and the wasm extractor agree', () => {
+    const canon = (r: ExtractionResult) => [
+      ...r.nodes.map(({ updatedAt: _u, ...n }) => JSON.stringify(n, Object.keys(n).sort())).sort(),
+      ...r.unresolvedReferences.map((ref) => JSON.stringify(ref, Object.keys(ref).sort())).sort(),
+    ];
+    for (const [file, language] of [
+      ['src/index.ts', 'typescript'],
+      ['src/index.tsx', 'tsx'],
+    ] as const) {
+      expect(canon(extract('kernel', file, BARREL, language))).toEqual(canon(extract('wasm', file, BARREL, language)));
+    }
+  });
+});
+
+describe('indexing a barrel with `export type *` (#2474)', () => {
+  let root = '';
+  let cg: CodeGraph;
+
+  beforeAll(async () => {
+    root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-export-type-star-'));
+    const files: Record<string, string> = {
+      'packages/sdk/src/client.ts':
+        'export interface FooClientOptions { baseUrl: string }\n' +
+        'export class FooApiError extends Error {}\n' +
+        'export class FooClient {\n' +
+        '  constructor(private opts: FooClientOptions) {}\n' +
+        '  get(path: string) { return this.opts.baseUrl + path; }\n' +
+        '}\n',
+      'packages/sdk/src/types.ts': "export const DAYS_OF_WEEK = ['mon', 'tue'];\nexport type Schedule = { day: string };\n",
+      'packages/sdk/src/shapes.ts': 'export interface Circle { radius: number }\n',
+      'packages/sdk/src/index.ts': BARREL,
+      // Same names, declared where nothing re-exports them: what a name match could pick.
+      'app/shapes.ts': 'export interface FooClientOptions { decoy: true }\nexport type Schedule = { decoy: true };\n',
+      'app/main.ts':
+        'import { FooClient, type FooClientOptions } from "../packages/sdk/src/index.js";\n' +
+        'import type { Schedule } from "../packages/sdk/src/index.js";\n' +
+        'export function run(o: FooClientOptions, s: Schedule) {\n' +
+        '  return new FooClient(o).get(s.day);\n' +
+        '}\n',
+    };
+    for (const [rel, content] of Object.entries(files)) {
+      fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true });
+      fs.writeFileSync(path.join(root, rel), content);
+    }
+    cg = await CodeGraph.init(root, { index: true });
+  });
+
+  afterAll(() => {
+    cg?.close();
+    if (root) fs.rmSync(root, { recursive: true, force: true });
+  });
+
+  it('stores the barrel without a parse error', () => {
+    const barrel = cg.getFile('packages/sdk/src/index.ts');
+    expect(barrel).not.toBeNull();
+    expect(barrel!.errors ?? []).toEqual([]);
+  });
+
+  it('links each name imported through the barrel to the file that declares it', () => {
+    const ids = cg.getNodesInFile('app/main.ts').map((n) => n.id);
+    const imported = cg
+      .getOutgoingEdgesFrom(ids, ['imports'])
+      .map((e) => cg.getNode(e.target)!)
+      .filter((t) => t.kind !== 'file')
+      .map((t) => `${t.name} -> ${t.filePath}`)
+      .sort();
+    expect(imported).toEqual([
+      'FooClient -> packages/sdk/src/client.ts',
+      'FooClientOptions -> packages/sdk/src/client.ts',
+      // Reached only through `export type * from "./types.js"`.
+      'Schedule -> packages/sdk/src/types.ts',
+    ]);
+  });
+});

src/extraction/languages/typescript.ts

@@ -69,6 +69,32 @@ export function blankFlowSyntax(source: string, filePath?: string): string {
     .replace(/\.\.\.(?=\s*[},])/g, '   ');
 }
 
+/** Whether the file writes `export type *` anywhere: most files don't, and skip the rest. */
+const EXPORT_TYPE_STAR_HINT = /export\s+type\s*\*/;
+/**
+ * `export type *` where a statement starts (a line's first word, or after `;`,
+ * `{`, `}` or the end of a block comment), followed by `from` or `as`.
+ */
+const EXPORT_TYPE_STAR = /(^|[;{}]|\*\/)([ \t]*export\s+)type(?=\s*\*\s*(?:from|as)\b)/gm;
+
+/**
+ * TypeScript 5.0's type-only re-export of a whole module, `export type * from
+ * './types'` and `export type * as ns from './types'`, with its `type` blanked
+ * to spaces (offsets survive), so it reads as the `export *` it is to the
+ * graph. The grammar has no rule for it (nor does upstream's): `type` parsed
+ * as an error, so a barrel of re-exports was reported as producing no symbols
+ * and counted by `codegraph status` as a file it could not parse, and the
+ * kernel handed every such file to the wasm parser (#2474). Only a statement
+ * start is blanked, so a mention in a comment or a doc keeps its text. The
+ * import resolver reads the statement as written, `type` included, from the
+ * file itself (`extractReExports`). Applied before both engines parse (the
+ * kernel's preParse hoist), so they agree.
+ */
+export function blankExportTypeStar(source: string): string {
+  if (!EXPORT_TYPE_STAR_HINT.test(source)) return source;
+  return source.replace(EXPORT_TYPE_STAR, '$1$2    ');
+}
+
 /**
  * Whether `node` is declared in a `declare module 'x' { … }` or `declare
  * global { … }` body, which exports it without an `export` keyword.
@@ -147,7 +173,7 @@ export function isAmbientFunctionSignature(node: SyntaxNode, filePath: string):
 }
 
 export const typescriptExtractor: LanguageExtractor = {
-  preParse: blankFlowSyntax,
+  preParse: (source, filePath) => blankExportTypeStar(blankFlowSyntax(source, filePath)),
   functionTypes: ['function_declaration', 'generator_function_declaration', 'arrow_function', 'function_expression', 'generator_function'],
   isAmbientFunction: isAmbientFunctionSignature,
   // `class` is a class expression, `register(class Widget extends Base {…})`: