Skip to content

fix(sync): detect frameworks introduced by incremental updates - #2516

Closed
nk33-dev wants to merge 1 commit into
colbymchenry:mainfrom
nk33-dev:codex/sync-framework-detection
Closed

nk33-dev wants to merge 1 commit into
colbymchenry:mainfrom
nk33-dev:codex/sync-framework-detection

Conversation

@nk33-dev

Copy link
Copy Markdown

Problem

A long-lived graph keeps the framework list detected by its initial index. When an incremental sync adds a framework manifest together with its first source file, the new file is parsed without that framework's extractor. Adding Express and a route file to an indexed TypeScript project leaves the route absent until a full re-index.

Changes

  • Refresh extraction framework detection after file additions/removals or changes to source paths previously read by detection.
  • Use a fresh file scan for scoped updates, so unchanged files still participate in detection.
  • Re-initialize the reference resolver after file additions/removals.
  • Keep ordinary body-only updates on the cached detection path.

No schema, extraction format, package version, installer, or default MCP tool changes.

Validation

Windows, upstream baseline 7ba8eee.

  • TypeScript: npx tsc --noEmit passed.
  • Two regression cases, full and scoped sync: both fail on the unchanged upstream source and pass with this patch.
  • git diff --check passed.
  • Full suite and other platforms have not been run locally; left to CI.

Scope

This covers source-file updates that change framework detection. A manifest-only edit with no indexed source change is not covered by this patch.

colbymchenry added a commit that referenced this pull request Oct 27, 2026
… runs

The MCP server's watcher holds one CodeGraph for the whole session, and
its framework detection was cached from the first time it ran: the
orchestrator reset it only in indexAll, and CodeGraph.sync never
re-initialized the resolver. A framework that arrived mid-session, like
Express in a new package.json, was missing from every file synced after
it, so a new route file got no route nodes until `codegraph index`. The
resolver's declared-dependencies cache was also keyed by the indexed file
COUNT (PR #2519): swapping one app for another of the same size, or
editing a manifest, kept the old dependency set, even through
reinitializeResolver().

Detection now records what it read: indexed source files, which a sync's
change lists cover, and every other path it read or probed, with a size
and mtime signature, including the per-app manifests the extractor gating
reads. Before a sync extracts files, it detects again only when files
were added or removed since, a source file detection read changed, or one
of those paths changed on disk. A scoped sync takes the file list from
the index plus the files it adds, so it never walks the project (the
first sync in a process used to run a full scan for its first detection).
What detection read is kept between detections and dropped by the same
change lists and signatures, so detecting again reads only what changed.
When the detected set differs from the resolver's, the sync
re-initializes the resolver. Its initialize() now clears its caches
before detecting, and clearCaches() forgets the declared-dependencies
cache with the other per-context memos.

Files indexed before a framework change are not re-extracted. A framework
like Express applies to every file of its language, and re-reading them
all inside a watcher's sync would cost what a full index does. Instead a
full index records the frameworks it used, a sync records how detection
moved since, and `codegraph status` (text, and --json with
reindexRecommended and frameworksChangedSinceIndex) recommends
`codegraph index` while they differ. Status only reads that record.

Validation:
- sync-framework-detection.test.ts (new, 8 cases): Express arriving with a
  new route file, by full and by scoped sync; a .svelte file added with no
  manifest change; an edited source file that detection read; a
  manifest-only edit picked up by the next sync, with the status record,
  cleared on revert and by a full re-index; a fresh process's scoped
  syncs detecting once, from the index, with no scan; the equal-size app
  swap; and the dependency cache forgetting on request. 7 of 8 fail on
  main's source (the 8th needs the new export). Ten mutations, one per
  mechanism, each fail a test.
- The triage probes: main detects [] after Express arrives and routes.js
  gets no route; the fix detects express and extracts GET /hello, by full
  and by scoped sync. For the swap, main's reinitializeResolver() gives
  [react] and the fix [express].
- Cost on excalidraw (737 files, scoped syncs in one long-lived
  instance): the detection step takes 8 ms on a sync where nothing it
  read changed and 8-13 ms when it detects again, against syncs of about
  260 ms (add) to 2 s (modify). The first sync in a process detects once
  (about 300 ms) and no longer scans the project.

Reported in PRs #2516 and #2519 by @nk33-dev. This reimplements both
fixes independently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
colbymchenry added a commit that referenced this pull request Oct 27, 2026
… runs

The MCP server's watcher holds one CodeGraph for the whole session, and
its framework detection was cached from the first time it ran: the
orchestrator reset it only in indexAll, and CodeGraph.sync never
re-initialized the resolver. A framework that arrived mid-session, like
Express in a new package.json, was missing from every file synced after
it, so a new route file got no route nodes until `codegraph index`. The
resolver's declared-dependencies cache was also keyed by the indexed file
COUNT (PR #2519): swapping one app for another of the same size, or
editing a manifest, kept the old dependency set, even through
reinitializeResolver().

Detection now records what it read: indexed source files, which a sync's
change lists cover, and every other path it read or probed, with a size
and mtime signature, including the per-app manifests the extractor gating
reads. Before a sync extracts files, it detects again only when files
were added or removed since, a source file detection read changed, or one
of those paths changed on disk. A scoped sync takes the file list from
the index plus the files it adds, so it never walks the project (the
first sync in a process used to run a full scan for its first detection).
What detection read is kept between detections and dropped by the same
change lists and signatures, so detecting again reads only what changed.
When the detected set differs from the resolver's, the sync
re-initializes the resolver. Its initialize() now clears its caches
before detecting, and clearCaches() forgets the declared-dependencies
cache with the other per-context memos.

Files indexed before a framework change are not re-extracted. A framework
like Express applies to every file of its language, and re-reading them
all inside a watcher's sync would cost what a full index does. Instead a
full index records the frameworks it used, a sync records how detection
moved since, and `codegraph status` (text, and --json with
reindexRecommended and frameworksChangedSinceIndex) recommends
`codegraph index` while they differ. Status only reads that record.

Validation:
- sync-framework-detection.test.ts (new, 8 cases): Express arriving with a
  new route file, by full and by scoped sync; a .svelte file added with no
  manifest change; an edited source file that detection read; a
  manifest-only edit picked up by the next sync, with the status record,
  cleared on revert and by a full re-index; a fresh process's scoped
  syncs detecting once, from the index, with no scan; the equal-size app
  swap; and the dependency cache forgetting on request. 7 of 8 fail on
  main's source (the 8th needs the new export). Ten mutations, one per
  mechanism, each fail a test.
- The triage probes: main detects [] after Express arrives and routes.js
  gets no route; the fix detects express and extracts GET /hello, by full
  and by scoped sync. For the swap, main's reinitializeResolver() gives
  [react] and the fix [express].
- Cost on excalidraw (737 files, scoped syncs in one long-lived
  instance): the detection step takes 8 ms on a sync where nothing it
  read changed and 8-13 ms when it detects again, against syncs of about
  260 ms (add) to 2 s (modify). The first sync in a process detects once
  (about 300 ms) and no longer scans the project.

Reported in PRs #2516 and #2519 by @nk33-dev. This reimplements both
fixes independently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
colbymchenry added a commit that referenced this pull request Oct 27, 2026
… runs (#2941)

The MCP server's watcher holds one CodeGraph for the whole session, and
its framework detection was cached from the first time it ran: the
orchestrator reset it only in indexAll, and CodeGraph.sync never
re-initialized the resolver. A framework that arrived mid-session, like
Express in a new package.json, was missing from every file synced after
it, so a new route file got no route nodes until `codegraph index`. The
resolver's declared-dependencies cache was also keyed by the indexed file
COUNT (PR #2519): swapping one app for another of the same size, or
editing a manifest, kept the old dependency set, even through
reinitializeResolver().

Detection now records what it read: indexed source files, which a sync's
change lists cover, and every other path it read or probed, with a size
and mtime signature, including the per-app manifests the extractor gating
reads. Before a sync extracts files, it detects again only when files
were added or removed since, a source file detection read changed, or one
of those paths changed on disk. A scoped sync takes the file list from
the index plus the files it adds, so it never walks the project (the
first sync in a process used to run a full scan for its first detection).
What detection read is kept between detections and dropped by the same
change lists and signatures, so detecting again reads only what changed.
When the detected set differs from the resolver's, the sync
re-initializes the resolver. Its initialize() now clears its caches
before detecting, and clearCaches() forgets the declared-dependencies
cache with the other per-context memos.

Files indexed before a framework change are not re-extracted. A framework
like Express applies to every file of its language, and re-reading them
all inside a watcher's sync would cost what a full index does. Instead a
full index records the frameworks it used, a sync records how detection
moved since, and `codegraph status` (text, and --json with
reindexRecommended and frameworksChangedSinceIndex) recommends
`codegraph index` while they differ. Status only reads that record.

Validation:
- sync-framework-detection.test.ts (new, 8 cases): Express arriving with a
  new route file, by full and by scoped sync; a .svelte file added with no
  manifest change; an edited source file that detection read; a
  manifest-only edit picked up by the next sync, with the status record,
  cleared on revert and by a full re-index; a fresh process's scoped
  syncs detecting once, from the index, with no scan; the equal-size app
  swap; and the dependency cache forgetting on request. 7 of 8 fail on
  main's source (the 8th needs the new export). Ten mutations, one per
  mechanism, each fail a test.
- The triage probes: main detects [] after Express arrives and routes.js
  gets no route; the fix detects express and extracts GET /hello, by full
  and by scoped sync. For the swap, main's reinitializeResolver() gives
  [react] and the fix [express].
- Cost on excalidraw (737 files, scoped syncs in one long-lived
  instance): the detection step takes 8 ms on a sync where nothing it
  read changed and 8-13 ms when it detects again, against syncs of about
  260 ms (add) to 2 s (modify). The first sync in a process detects once
  (about 300 ms) and no longer scans the project.

Reported in PRs #2516 and #2519 by @nk33-dev. This reimplements both
fixes independently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@colbymchenry

Copy link
Copy Markdown
Owner

Thank you, @nk33-dev: both PRs found real problems, the framework set frozen for a long-running session and the dependency cache keyed by file count. We've landed the fix in #2941, reimplemented with credit to you in the changelog. Detection now re-runs only when what it read changes, without a full scan on the watcher's path, and files indexed before a framework change are flagged by codegraph status with a re-index recommendation. Closing this (and #2519) in favor of it; it ships in the next release.

hospitaler17 added a commit to Union-Work-Place/codegraph that referenced this pull request Oct 29, 2026
…g for Qt work

Main gained hooks the Qt support was written before: a framework that
answers a reference alone declares it through ownsReference, and
framework detection is cached and re-run by a sync that touches a file
it read (colbymchenry#2516). The Qt code had three Qt-specific blocks in
ReferenceResolver and a detect() that read every .cpp/.h/.hpp of the
project, so in a non-Qt C++ tree each of those files became a detection
input and every sync that changed one re-ran the whole scan.

- QML imports, Q_PROPERTY accessors and owner-qualified QML members stay
  unresolved when Qt cannot prove them, now through
  qtResolver.ownsReference instead of three branches in the generic
  resolver. The links are unchanged.
- detect() checks .qml files and the CMake/qmake manifests first, then
  reads at most 200 C++ files, as React Native's does.
- blankQtMacros skips a file that merely contains the words emit,
  signals or slots: only the shapes it rewrites (a signals:/slots:
  header, emit name() ) count. A 1.5 MB non-Qt file: 48 ms to 3 ms.
- loadGrammarsForLanguages(['qml']) loads the JavaScript grammar through
  expandGrammarLanguages, like vue, svelte and astro, instead of a
  separate preload-only helper.
- A .qml file counts as markup in the synthesis input gate, so an edit
  to one with no handler still refreshes the Qt passes on sync.
- Restores a comment the Qt commit deleted from SYNTH_PASSES.
- agent-eval corpus: a QML block (cool-retro-term, caelestia-shell,
  plasma-desktop) and a Qt block (sddm, kdeconnect-kde, keepassxc,
  qgroundcontrol). The Qt commit's second "C++" key made JSON.parse
  drop kdeconnect-kde.
- CHANGELOG: the QML entries are rewritten for users and cite colbymchenry#759.

Validation:
- tsc and npm run build clean; qml, qml-resolution,
  qt-signal-synthesizer, sync-framework-detection, preload-languages
  and synthesis-progress tests green. Linux (WSL, Node 22): 10 files,
  322 passed, 3 skipped.
- cool-retro-term indexed with the build: 39 files, 939 nodes, 1,135
  edges, identical after a re-index; 30 qt-signal-channel edges, 5
  QML to C++.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants