Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

- **Syncing a changed file in a large Python project is faster.** When a Python class has no method of the name being linked (a test class that passes `Handler.__init__` as a value, say), the resolver checks every class with that method for one that derives from it. For each class whose base is a package import (`import unittest`, then `unittest.TestCase`) it parsed the whole file again, and its cache holds only a few files, so two such references took about a second each. A plain one-line `import a.b` is now read from its line; other forms still use the parse. In CPython a one-file sync takes a median 5.5 s instead of 7.7 s on Node and 5.2 s instead of 8.1 s on Bun (4 alternating runs per build, ranges not overlapping; the phase that retries unresolved references drops from 4.6 s to 2.3 s). A full index is within the run-to-run variation, and the graph is the same on CPython and Django (no edge differs).
- **Questions to the MCP server are faster in a project with no `codegraph.json`.** While ranking results, CodeGraph checked for the project's `codegraph.json` once per result path, and with no such file each check raised and caught an error. It now asks the file system without raising, so `codegraph_explore` and the other query tools spend less CPU on most projects. Editing, adding or deleting `codegraph.json` is still picked up without a restart (#491).
- **Ranking looks at `codegraph.json` once per query, not once per result path.** The check above still ran for every path a query ranked; it now runs once per ranking pass and the answer is reused for each path. A `codegraph_explore` call is a median 4% faster on a Go project (gin), 7% on a NestJS project and 0.6% on a large TypeScript project (n8n), the first two inside the run-to-run variation of the measurement (5 rounds, answers identical in every round). Editing `codegraph.json` is still picked up without a restart.
- **Indexing a project with a bundled minified JavaScript file is faster.** For each reference in a one-line file such as `jquery.min.js`, the resolver counted characters from the start of the line to find the reference's column, many times per reference. It now skips straight to the column when the text before it is ASCII. Alamofire, whose docs folder carries jQuery, resolves its references in a median 0.93 s instead of 1.25 s (profiling runs on a busy host, 5,614 nodes and 13,673 edges either way). Separately, the resolver now looks nodes up by query only when there are fewer than 15,000 pending references and fewer than 0.3 per node; at either limit it loads the node table. A full index of a small project such as gin (about 4.4 references per node) therefore loads the table, saving about 60 ms of its one-second index, while a one-file sync still queries. In the 2026-10-12 benchmark on Node, a full index of Alamofire takes 1.81 s against upstream's 1.73 s and gin 1.15 s against 1.13 s ([results](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md)).
- **Indexing a project with long minified lines that contain a non-ASCII character is faster.** The resolver turns a reference's column into a byte offset many times per reference, and when any character before the column was not ASCII it walked every character on the line each time. It now counts a 64-byte block at a time and walks characters only inside the block that holds the column. Indexing JetBrains Exposed, whose docs folder carries minified scripts, takes a median 25.7 s instead of 37.0 s (5 alternating runs per build, baseline against baseline within 0.3%; 28,689 nodes and 82,842 edges either way). nocodb, with no long lines, is unchanged within the run-to-run band (6.53 s against 6.74 s).
- **Indexing a Kotlin project is faster.** When a call could reach several same-named functions, the resolver read the call site, checked the receiver and typed it again for each one, although all of that depends only on the call. It now does that once per call. Indexing JetBrains Exposed takes a median 31.2 s instead of 37.3 s (5 alternating runs per build, baseline against baseline within 0.1%; 28,689 nodes, 82,842 edges and the same 36,202 call edges either way).
Expand Down
38 changes: 37 additions & 1 deletion __tests__/deprioritize-config.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,13 @@
* paths stay indexed and findable; they just stop winning. Locked below.
*/

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';
import * as fs from 'fs';
import * as path from 'path';
import * as os from 'os';
import { CodeGraph } from '../src';
import { initGrammars, loadAllGrammars } from '../src/extraction/grammars';
import * as projectConfig from '../src/project-config';
import { loadDeprioritizePatterns } from '../src/project-config';
import { nameMatchBonus, scorePathRelevance } from '../src/search/query-utils';
import { DEPRIORITIZED_NAME_BONUS_SCALE } from '../src/db/queries';
Expand Down Expand Up @@ -183,6 +184,20 @@ describe('#982 minimal repro — ranking with and without deprioritize', () => {
expect(results.slice(0, 2).every(isHelper)).toBe(true);
});

it('reads the config once per search, not once per result path', () => {
const configReads = (cg: CodeGraph): number => {
const spy = vi.spyOn(projectConfig, 'loadDeprioritizePatterns');
try {
expect(cg.searchNodes(QUERY, { limit: 20 }).length).toBeGreaterThanOrEqual(4);
return spy.mock.calls.length;
} finally {
spy.mockRestore();
}
};
expect(configReads(cfgCg)).toBe(1);
expect(configReads(baseCg)).toBe(1);
});

it('with deprioritize, product code outranks the peripheral helpers', () => {
const results = cfgCg.searchNodes(QUERY, { limit: 20 });
const firstHelper = results.findIndex(isHelper);
Expand Down Expand Up @@ -227,6 +242,27 @@ describe('#982 minimal repro — ranking with and without deprioritize', () => {
expect(matcher!('apps/desktop/statusbar/StatusBar.ts')).toBe(false);
});

it('keeps the exported QueryBuilder setter that takes a plain path predicate', () => {
// SDK callers pass a (filePath) => boolean to setDeprioritizedPathMatcher;
// the provider form is a separate setter.
type Q = {
setDeprioritizedPathMatcher(m: ((p: string) => boolean) | undefined): void;
getDeprioritizedPathMatcher(): ((p: string) => boolean) | undefined;
};
const queries = (baseCg as unknown as { queries: Q & { deprioritizedPathProvider: unknown } }).queries;
const provider = queries.deprioritizedPathProvider;
try {
queries.setDeprioritizedPathMatcher((p) => p.startsWith('gateway/'));
const matcher = queries.getDeprioritizedPathMatcher();
expect(matcher!('gateway/x.ts')).toBe(true);
expect(matcher!('packages/x.ts')).toBe(false);
queries.setDeprioritizedPathMatcher(undefined);
expect(queries.getDeprioritizedPathMatcher()).toBeUndefined();
} finally {
queries.deprioritizedPathProvider = provider;
}
});

it('picks up a config written after the project was opened', async () => {
// wireLayers runs once per open, so a matcher captured there would freeze
// at open time — and the MCP server keeps one CodeGraph per root alive for
Expand Down
33 changes: 21 additions & 12 deletions src/codegraph.ts
Original file line number Diff line number Diff line change
Expand Up @@ -227,27 +227,36 @@ export class CodeGraph {
// the config said when the project opened. The MCP server caches one
// CodeGraph per root for its whole lifetime, so editing `codegraph.json`
// would appear to do nothing until the process restarted — `exclude` and
// `include` do not behave that way. `loadDeprioritizePatterns` is
// mtime-cached, so this costs one `stat`; the compiled matcher is memoized
// on the pattern array's identity, which the cache keeps stable.
// `include` do not behave that way. The provider runs once per ranking
// pass (callers reuse the predicate for every path in it), and
// `loadDeprioritizePatterns` is mtime-cached, so a pass costs one `stat`;
// the compiled matcher is memoized on the pattern array's identity, which
// the cache keeps stable. No patterns yields no predicate at all.
let cachedPatterns: string[] | undefined;
let cachedMatcher: ReturnType<typeof ignore> | undefined;
this.queries.setDeprioritizedPathMatcher((filePath: string): boolean => {
this.queries.setDeprioritizedPathProvider(() => {
try {
const patterns = loadDeprioritizePatterns(this.projectRoot);
if (patterns.length === 0) return false;
if (patterns.length === 0) return undefined;
if (patterns !== cachedPatterns) {
cachedPatterns = patterns;
cachedMatcher = ignore().add(patterns);
}
const rel = path.isAbsolute(filePath)
? path.relative(this.projectRoot, filePath)
: filePath;
if (!rel || rel.startsWith('..')) return false;
return cachedMatcher!.ignores(rel.split(path.sep).join('/'));
const matcher = cachedMatcher!;
return (filePath: string): boolean => {
try {
const rel = path.isAbsolute(filePath)
? path.relative(this.projectRoot, filePath)
: filePath;
if (!rel || rel.startsWith('..')) return false;
return matcher.ignores(rel.split(path.sep).join('/'));
} catch {
// Ranking must never take the search down with it.
return false;
}
};
} catch {
// Ranking must never take the search down with it.
return false;
return undefined;
}
});

Expand Down
11 changes: 7 additions & 4 deletions src/context/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -229,8 +229,9 @@ export class ContextBuilder {
* content, and `deprioritize` is a ranking lever by definition — `exclude` is
* the lever for taking things out of reach.
*/
private isDeprioritized(filePath: string): boolean {
return this.queries.getDeprioritizedPathMatcher()?.(filePath) ?? false;
private deprioritizedPathMatcher(): (filePath: string) => boolean {
// Reads the config, so take it once per call and reuse it for every path.
return this.queries.getDeprioritizedPathMatcher() ?? (() => false);
}

/**
Expand Down Expand Up @@ -478,6 +479,8 @@ export class ContextBuilder {
return { nodes, edges, roots };
}

const isDeprioritized = this.deprioritizedPathMatcher();

// === HYBRID SEARCH ===

// Step 1: Extract potential symbol names from query
Expand Down Expand Up @@ -943,7 +946,7 @@ export class ContextBuilder {
r.node.filePath,
query,
undefined,
this.isDeprioritized(r.node.filePath),
isDeprioritized(r.node.filePath),
);
const brevityBonus = Math.max(0, 6 - (name.length - titleCased.length) / 4);
termCandidates.push({ node: r.node, score: 8 + brevityBonus + pathScore });
Expand Down Expand Up @@ -1035,7 +1038,7 @@ export class ContextBuilder {
entry.node.filePath,
query,
undefined,
this.isDeprioritized(entry.node.filePath),
isDeprioritized(entry.node.filePath),
);
const brevityBonus = Math.max(0, 6 - entry.node.name.length / 8);
compoundResults.push({
Expand Down
31 changes: 24 additions & 7 deletions src/db/queries.ts
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,7 @@ export class QueryBuilder {
// whole project, not a symbol, so it carries no discriminative signal (#720).
// Set once by the CodeGraph instance; empty by default (no down-weighting).
private projectNameTokens: Set<string> = new Set();
private isDeprioritizedPath: ((filePath: string) => boolean) | undefined;
private deprioritizedPathProvider: (() => ((filePath: string) => boolean) | undefined) | undefined;

// FTS5 availability flag — detected once at construction time (#1532)
private _fts5Available: boolean | undefined;
Expand Down Expand Up @@ -492,15 +492,32 @@ export class QueryBuilder {
* Set the predicate that marks a path as de-prioritized by the project's
* `codegraph.json` `deprioritize` patterns (#982). Ranking-only: those paths
* stay indexed and findable, they just stop outranking first-party code.
* Called once when the project opens; undefined disables the lever.
* undefined disables the lever. A predicate that reads live state per call
* should use `setDeprioritizedPathProvider` instead.
*/
setDeprioritizedPathMatcher(matcher: ((filePath: string) => boolean) | undefined): void {
this.isDeprioritizedPath = matcher;
this.deprioritizedPathProvider = matcher && (() => matcher);
}

/** The `deprioritize` predicate (#982), so other rankers apply the same lever. */
/**
* Like `setDeprioritizedPathMatcher`, but takes a provider of the predicate.
* Called once when the project opens. The provider runs once per ranking pass
* (see `getDeprioritizedPathMatcher`), so the config is read once per pass
* and never once per path.
*/
setDeprioritizedPathProvider(
provider: (() => ((filePath: string) => boolean) | undefined) | undefined
): void {
this.deprioritizedPathProvider = provider;
}

/**
* The `deprioritize` predicate (#982), so other rankers apply the same lever.
* Reads the config now, so call it once per ranking pass and reuse the result
* for every path in it; undefined when nothing is de-prioritized.
*/
getDeprioritizedPathMatcher(): ((filePath: string) => boolean) | undefined {
return this.isDeprioritizedPath;
return this.deprioritizedPathProvider?.();
}

// ===========================================================================
Expand Down Expand Up @@ -1842,15 +1859,15 @@ export class QueryBuilder {
// Apply multi-signal scoring
if (results.length > 0 && (text || query)) {
const scoringQuery = text || query;
const isDeprioritizedPath = this.getDeprioritizedPathMatcher();
results = results.map(r => {
// A path the project de-prioritized is saying its symbol NAMES are not
// the answer, so the exact-name bonus has to be damped too. The -15 path
// penalty alone cannot do it: the bonus is additive and larger (measured
// on #982's repro, a `usage()` helper sat at 74.8 vs 51.2 for the top
// product symbol — -15 lands at 59.8, still ahead). Damped, not zeroed,
// so the tree stays findable when it genuinely is what you asked for.
// Evaluated once and reused: the predicate stats the config file.
const deprioritized = this.isDeprioritizedPath?.(r.node.filePath) ?? false;
const deprioritized = isDeprioritizedPath?.(r.node.filePath) ?? false;
const nameBonus = nameMatchBonus(r.node.name, scoringQuery);
return {
...r,
Expand Down
Loading