Automatically rebuild stale indexes in the background after engine upgrades #1798

Activity

1. bompus

   changed the title [-]Automatically rebuild stale indexes before serving queries after engine upgrades[/-] [+]Automatically rebuild stale indexes in the background after engine upgrades[/+] on Sep 8, 2026
2. inth3shadows commented on Sep 10, 2026

   

   inth3shadows

   on Sep 10, 2026

   Last edited by inth3shadows

   Contributor

   More actions

   This reproduces as a live bug today, not only as a missing feature — and the cheap half of it is one call. Posting here rather than opening a duplicate, since your reading of the foundation is already exactly right.

   sync says "Already up to date" about an index the same tool says needs a re-index

   Reproduced just now, twice: once on published 1.6.0 (the bundled runtime), once on a build of main @ 3ed73bc under Node 22. Same result both times.

   Set an index's stamp behind the running engine (any real upgrade does this; here it is done directly so the repro needs one binary, not two):

   node -e 'const {DatabaseSync}=require("node:sqlite");
   const db=new DatabaseSync(".codegraph/codegraph.db");
   db.prepare("update project_metadata set value=? where key=?").run("25","indexed_with_extraction_version");'

   status is honest about it:

   "index": { "builtWithExtractionVersion": 25, "currentExtractionVersion": 26,
              "reindexRecommended": true, "state": "complete" }

   sync is not:

   ●  Already up to date
   

   And with a real edit in the tree, sync does its incremental work, reports success, and still leaves the bulk on extractor 25:

   ●  Added: 1 — 2 nodes in 118ms
   "index": { "builtWithExtractionVersion": 25, "currentExtractionVersion": 26, "reindexRecommended": true }
   

   So the two signals a user reaches for disagree, and the one that runs the work is the one that is wrong. Nothing in the sync path ever prints the recommendation status already computed.

   What it cost me

   I hit this before finding your issue. A fixed extractor became the default on PATH; the index was synced after the fix and still carried every fabricated call edge that fix removes, because none of the files carrying them had been edited in between — the index was newer than the fix and contained none of its output. A downstream report went out saying the bug was "still present after the fix." It wasn't.

   The trap worth stating explicitly: the release version cannot be the signal. A synced index and a freshly built one of the same commit both report indexed_with_version: 1.6.0 while their extraction versions are 25 and 26. A fork, or a patch release, can change extraction without moving the release string. builtWithExtractionVersion is the only field that discriminates — an argument for the design you propose, not against it.

   Enforcement is the gap, not knowledge

   Re-verified against main @ 3ed73bc. You cite 374b3b4; the three functions involved are byte-identical between the two revisions (getChangedFiles, isIndexStale, collectGitStatus — hashed at both revs), so nothing here has drifted since you filed.

   isIndexStale() (src/index.ts:1238) has exactly two callers in the tree:

   src/bin/codegraph.ts:1041       the `status` command
   src/ui-server/api/stats.ts:113  the viewer
   

   Neither is sync. And the invariant sync violates is already written down, at src/index.ts:686:

   // Stamp the index with the engine that built it, so `codegraph status`
   // and `codegraph upgrade` can recommend a re-index when the running
   // engine produces richer extraction than the one on disk. Only on a
   // real full index — a sync touches a subset, so it must NOT advance the
   // extraction stamp (the bulk would still be stale). See extraction-version.ts.

   The codebase already knows a sync leaves the bulk stale, deliberately withholds the stamp for that reason, computes isIndexStale() from it, and surfaces reindexRecommended — then sync runs without ever asking. extraction-version.ts even names this precise case as a bump reason: "a resolver fix that materially changes which edges exist."

   Suggested staging

   The background rebuild you describe is the right end state, but it needs a serving-while-rebuilding story and a concurrency story. There is a subset that closes the silent-wrong-answer window immediately and is independently useful:

   Have sync consult isIndexStale() and either escalate to a full re-extract or refuse with a pointer to index. The signal, the threshold and the stated intent are all already in the tree — wiring, not new machinery — and it makes the failure loud instead of silent while the background design is worked out.

   Happy to send that as a PR if you want it split that way.

   Environment for both runs: Linux x86_64 (WSL2); published 1.6.0 via its bundled runtime, and main @ 3ed73bc built from source on Node 22.23.2 (the source build's Node gate rejects Node 26).

   ---

   Filed separately as #1829, because it is a different mechanism in a different file: status.pendingChanges reports zero for a change that is committed but unindexed — so the other signal a user would reach for to check this one is also unreliable.
3. bompus commented on Sep 11, 2026

   

   bompus

   on Sep 11, 2026

   ContributorAuthor

   More actions

   Thanks for the reproduction and for splitting the silent-sync case from the background rebuild.

   The cheap half you described is already up as #1842: sync consults the existing extraction-staleness signal and refuses with a codegraph index recovery path instead of printing “Already up to date.” It does not escalate sync into a full re-extract, and it does not implement the background rebuild this issue asks for.

   The committed-but-unindexed status signal you filed as #1829 is #1843.
4. inth3shadows commented on Sep 12, 2026

   

   inth3shadows

   on Sep 12, 2026

   Contributor

   More actions

   Appreciated — and thank you for pointing me at both PRs rather than leaving me to find them. #1842 is the right scope for this half: refusing is more conservative than the escalate-to-full-reindex shape I had, and it keeps the decision with the person running the command.

   One question on it rather than an objection: --quiet exits 1 as well. That is the git-hook path (codegraph sync -q), so on a repo whose hook is wired to fail the commit on a non-zero exit, the first commit after an upgrade would be blocked by a signal the user cannot see — quiet suppresses the recovery instruction that explains it. Exiting 0 in quiet mode, or writing the one line to stderr before exiting 1, would keep the signal without that edge. Your call; you know the hook wiring better than I do.

   I have left the background-rebuild question here alone — that is what this issue is actually about, and neither PR touches it.
5. bompus

   added a commit that references this issue on Sep 12, 2026

   fix(cli): explain a refused quiet sync on stderr

   ...

   cb8dbdc
6. bompus commented on Sep 12, 2026

   

   bompus

   on Sep 12, 2026

   ContributorAuthor

   More actions

   Good catch, and taken: #1842 now writes the reason as one line on stderr under --quiet and still exits 1, so a hook that fails on a non-zero exit shows the user why instead of blocking silently (cb8dbdc3, quiet-mode tests updated to assert exactly one stderr line and empty stdout).

   For what it is worth, the hooks codegraph installs itself never hit this: they run codegraph sync in the background with output discarded, so they cannot fail a commit. The edge is a user-authored hook, which is exactly the one that needs the line.

   Agreed the background rebuild is a separate question; leaving that to this issue.
7. bompus

   added 3 commits that reference this issue on Sep 12, 2026

   fix(cli): explain a refused quiet sync on stderr

   ...

   d5f42a5

   fix(cli): explain a refused quiet sync on stderr

   ...

   d3a8cc3

   fix(cli): explain a refused quiet sync on stderr

   ...

   8ac04d4
8. deniskern commented on Oct 6, 2026

   

   deniskern

   on Oct 6, 2026

   More actions

   One more data point for the background rebuild, from the MCP side rather than sync.

   On 1.6.2, an index built with 1.6.0 (status --json: builtWithExtractionVersion: 25, currentExtractionVersion: 27, reindexRecommended: true) answers codegraph_explore with no hint at all. The per-file staleness banner and the connect-time catch-up both treat it as fresh, because no file changed. The agent, which is the main reader of these answers, never sees the signal that status already computes.

   This matters most for releases like the one carrying #2357. A Vue/Nuxt user who upgrades keeps getting the old, incomplete callers answers from MCP until someone runs codegraph index by hand. Nothing in the agent's path says so.

   Until the background rebuild lands, a one-line notice in the MCP answer when isIndexStale() is true would close most of the gap, for example: "Index built with an older extractor; results may miss edges. Run codegraph index."
9. bompus commented on Oct 19, 2026

   

   bompus

   on Oct 19, 2026

   ContributorAuthor

   More actions

   Thanks, that's a gap #1842 doesn't cover. #1842 only changes sync: it refuses to report "Already up to date" on an index built by an older extractor and points to codegraph index. Nothing on the MCP side reads that signal yet, so codegraph_explore still answers from an outdated index without saying so, as you saw.

   A one-line notice in the MCP answer when isIndexStale() is true seems like the right stopgap until the background rebuild exists. I'm happy to open a small PR for it if that's wanted.