Optimizations#74
Merged
Merged
Conversation
* perf: batch ReplaceFileGraph writes during indexing * perf: reduce indexing DB roundtrips (batch file IDs, set-based deletes)
#21) * perf(index): add phase profiling + reduce ReplaceFileGraphsBatch overhead (RETURNING id, prepared deletes) + add delete-path index * perf: batch file-graph deletes and add supporting file_id indexes
#22) * store: batch ResolveEdgesForPaths updates and add unresolved edge-name index * store/schema: use partial unresolved-edge index; fix rows.Err handling
#23) * perf: speed up ResolveEdges suffix strategy and path-scoped updates; add optional sqlite_cgo driver
#24) * perf(store): add Phase 1 pragmas, batch MarkFilesSeen updates, and index active-file language stats * fix(store): avoid partial ResolveEdges counts on rollback; apply auto_vacuum only for new DBs
#25) * perf(index/store): batch token + embedding writes; make index benches scale via CODEGRAPH_BENCH_FILES * Update internal/store/store.go * refactor(store): replace sqlite batch-size literals with named constants
#26) * perf(store): batch multi-row inserts for edges, references, and imports during indexing * refactor(store): Extract common logic from batch insert helpers
…chmark repeatability (#28) * perf(indexing): add phase timings + write_stats counters; improve benchmark repeatability * fix(indexer): use duration-based timing internally for observability
…essure (#29) * store: batch symbols + symbol_fts inserts; reduce delete statement pressure * perf(store): tighten batch arg allocation in hot paths
…ts (#30) * indexing: reduce tokenization allocations and add tokenize timing stats * indexing: reduce tokenization allocations and add tokenize timing stats
…cy-safe (#31) * watch: coalesce dirty queue writes and make DrainDirtyFiles concurrency-safe * store: make DrainDirtyFiles atomic (DELETE RETURNING) with safe BEGIN IMMEDIATE fallback
…esolve) (#32) * indexer: resolve edges path-scoped for update runs (avoid repo-wide resolve) * refactor(indexer): deduplicate path-scoped edge resolution flow
…-scoped win) (#33) * indexer: targeted cross-file edge resolution on update (preserve path-scoped win) * indexer: generalize cross-file resolve for partial runs; simplify ResolveEdgesForNames
…act-match prefilter" This reverts commit 91d1d4d.
…h prefilter (#34) * store: narrow ResolveEdgesForNames candidates with indexed exact-match prefilter * store: drop redundant unresolved dst_name index; reuse sqlite batch constants
…stats (#36) * watch: ignore chmod-only + directory-create events; add repeated-work stats * fix(watcher): count only successful update runs in watcher stats
…cs (#37) * maintenance: add FTS optimize to clean and expand doctor DB diagnostics * doctor/clean: reuse store pragma inspection and dedupe repo-root parsing
…m; add doctor --deep integrity checks (#39) * clean/doctor: add explicit ANALYZE, WAL checkpoint, incremental vacuum; add doctor --deep integrity checks * clean: dedupe maintenance action execution; clarify incremental_vacuum auto_vacuum mode errors
…rse_ms) (#40) * cli/index: stabilize --jsonl scan payloads (add scan_kind, include parse_ms) * indexer: fix parse_ms semantics and drop redundant adapter_parse_ms from scan summaries
…elation fields (#41) * cli/index: stabilize --jsonl scan event envelopes with repo/scan correlation fields * cli/index: dedupe JSONL scan event envelopes and handle scan_phases write errors
…elopes, doctor arrays always present) (#43)
#44) * benchmark: add --sqlite-profile and capture sqlite_profile/host context in JSON output * benchmark: persist num_cpu/cwd in baselines and list benchmark flags in help
…indexing perf diffs (#45) * cli: add index_smoke runner with compact jsonl + median baseline for indexing perf diffs * cli: surface index_smoke baseline/db/config errors; add .codegraphignore
…ymbol references (#46) * store/indexer: purge deleted-file graph rows and nullify cross-file symbol references * store: reuse temp id tables in deleted-file purge
…wth; changelog v1.0.10 (#52) * perf(store): scale ResolveEdges dot-suffix + cap slash-suffix map growth; changelog v1.0.10 * fix(store): restrict dot-tail2 candidate set to single-dot dst_name
…d ResolveEdgesForNames scale bench (#53)
…erf: speed up FindDeadCode with indexes (#54) * fix(store): scope RelatedTests(file) + deterministic symbol lookup; perf: speed up FindDeadCode with indexes * Update internal/query/service.go
…write path (#55) * perf(store): cache batch-insert SQL + speed up edge src selection in write path * fix(store): restore nested-function correctness in srcSymbolChooser
…ess (#56) * fix(store/doctor): harden DB pragmas and migration lifecycle correctness * store: build SQLite file DSNs via net/url; reuse conn across migrations
…agmas on SQLITE_BUSY (#57) * bench(store): add chooser + callers/callees benchmarks; retry applyPragmas on SQLITE_BUSY * store(sqlite): unify busy-retry policy; make SQLITE_BUSY detection driver-aware
… allocs (#58) * store: cache symbol insert SQL + IN placeholders to reduce write-path allocs * store: use LoadOrStore in SQL caches and rename symbolInsertSQL arg
…suffix) (#59) * add per-strategy resolver microbenches (slash-suffix, dot-tail2, dot-suffix) * simplify upsertBenchRepo return type to int64
…ix duplicate token stats (#60)
…#62) * perf(export): page no-focus JSONPaged via ExportSymbolsPage/EdgesPage * fix(export): stably order symbols and edges in JSONPaged fallback
…unresolved edges (#64) * perf(store/resolver): early-return resolveEdgesBySlashSuffix when no unresolved edges * Update internal/store/store.go
…cans (#65) * perf(indexer): overlap ExistingFiles load with FS walk on repo-wide scans * Update internal/indexer/indexer.go
* perf(export): writer-based JSONStream for unbounded JSON export * fix(export): make JSONStream's encodeIndented robust to first-line prefixing
* fix(store): delete test_links pointing at purged target files * fix(store): populate test_links.target_file_id during indexing
…edges (#69) * perf(export): stream unbounded DOT export via paged DISTINCT nodes + edges * refactor(export): dedupe DOT edge formatting into a single helper
…70) * perf(indexer): slim existing-files map value type to drop floor cost * refactor(store): scan existing-files metadata directly into caller's map
#73) * perf(store/indexer): defer ContentHash to lazy lookup; trim ExistingFiles map row size * fix(indexer): propagate hashLookup errors through processFileTask
Owner
Author
|
@codex review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements extensive performance optimizations and architectural hardening across the codebase, including parallelized repository indexing, schema-backed edge resolution, and memory-efficient streaming for graph exports. It also introduces a new performance smoke-testing utility and enhances the diagnostic capabilities of the "doctor" command. Review feedback identifies a bug in CLI argument parsing for repository roots and a potential race condition when the file watcher reloads its configuration. Further improvements were suggested to replace inefficient JSON round-trips in the CLI, remove redundant prefix-stripping logic in the export service, and optimize slice capacities in the indexer to minimize reallocations.
isink17
added a commit
that referenced
this pull request
Apr 27, 2026
* Optimizations (#74) * perf: batch ReplaceFileGraph writes during indexing (#20) * perf: batch ReplaceFileGraph writes during indexing * perf: reduce indexing DB roundtrips (batch file IDs, set-based deletes) * perf(index): add phase profiling + reduce ReplaceFileGraphsBatch over… (#21) * perf(index): add phase profiling + reduce ReplaceFileGraphsBatch overhead (RETURNING id, prepared deletes) + add delete-path index * perf: batch file-graph deletes and add supporting file_id indexes * store: batch ResolveEdgesForPaths updates and add unresolved edge-nam… (#22) * store: batch ResolveEdgesForPaths updates and add unresolved edge-name index * store/schema: use partial unresolved-edge index; fix rows.Err handling * perf: speed up ResolveEdges suffix strategy and path-scoped updates; … (#23) * perf: speed up ResolveEdges suffix strategy and path-scoped updates; add optional sqlite_cgo driver * perf(store): add Phase 1 pragmas, batch MarkFilesSeen updates, and in… (#24) * perf(store): add Phase 1 pragmas, batch MarkFilesSeen updates, and index active-file language stats * fix(store): avoid partial ResolveEdges counts on rollback; apply auto_vacuum only for new DBs * perf(index/store): batch token + embedding writes; make index benches… (#25) * perf(index/store): batch token + embedding writes; make index benches scale via CODEGRAPH_BENCH_FILES * Update internal/store/store.go * refactor(store): replace sqlite batch-size literals with named constants * perf(store): batch multi-row inserts for edges, references, and impor… (#26) * perf(store): batch multi-row inserts for edges, references, and imports during indexing * refactor(store): Extract common logic from batch insert helpers * refactor(store): batch test_links inserts and cleanup prepared statements * perf(indexing): add phase timings + write_stats counters; improve benchmark repeatability (#28) * perf(indexing): add phase timings + write_stats counters; improve benchmark repeatability * fix(indexer): use duration-based timing internally for observability * store: batch symbols + symbol_fts inserts; reduce delete statement pressure (#29) * store: batch symbols + symbol_fts inserts; reduce delete statement pressure * perf(store): tighten batch arg allocation in hot paths * indexing: reduce tokenization allocations and add tokenize timing stats (#30) * indexing: reduce tokenization allocations and add tokenize timing stats * indexing: reduce tokenization allocations and add tokenize timing stats * watch: coalesce dirty queue writes and make DrainDirtyFiles concurrency-safe (#31) * watch: coalesce dirty queue writes and make DrainDirtyFiles concurrency-safe * store: make DrainDirtyFiles atomic (DELETE RETURNING) with safe BEGIN IMMEDIATE fallback * indexer: skip mark_seen DB writes for path-scoped updates * indexer: resolve edges path-scoped for update runs (avoid repo-wide resolve) (#32) * indexer: resolve edges path-scoped for update runs (avoid repo-wide resolve) * refactor(indexer): deduplicate path-scoped edge resolution flow * indexer: targeted cross-file edge resolution on update (preserve path-scoped win) (#33) * indexer: targeted cross-file edge resolution on update (preserve path-scoped win) * indexer: generalize cross-file resolve for partial runs; simplify ResolveEdgesForNames * store: narrow ResolveEdgesForNames candidates with indexed exact-match prefilter * Revert "store: narrow ResolveEdgesForNames candidates with indexed exact-match prefilter" This reverts commit 91d1d4d. * store: narrow ResolveEdgesForNames candidates with indexed exact-match prefilter (#34) * store: narrow ResolveEdgesForNames candidates with indexed exact-match prefilter * store: drop redundant unresolved dst_name index; reuse sqlite batch constants * resolve: add cross-file candidate stats + tests for ResolveEdgesForNames * watch: ignore chmod-only + directory-create events; add repeated-workstats (#36) * watch: ignore chmod-only + directory-create events; add repeated-work stats * fix(watcher): count only successful update runs in watcher stats * maintenance: add FTS optimize to clean and expand doctor DB diagnostics (#37) * maintenance: add FTS optimize to clean and expand doctor DB diagnostics * doctor/clean: reuse store pragma inspection and dedupe repo-root parsing * cli: disable HTML escaping in JSON/JSONL output for schema stability (#38) * clean/doctor: add explicit ANALYZE, WAL checkpoint, incremental vacuum; add doctor --deep integrity checks (#39) * clean/doctor: add explicit ANALYZE, WAL checkpoint, incremental vacuum; add doctor --deep integrity checks * clean: dedupe maintenance action execution; clarify incremental_vacuum auto_vacuum mode errors * cli/index: stabilize --jsonl scan payloads (add scan_kind, include parse_ms) (#40) * cli/index: stabilize --jsonl scan payloads (add scan_kind, include parse_ms) * indexer: fix parse_ms semantics and drop redundant adapter_parse_ms from scan summaries * cli/index: stabilize --jsonl scan event envelopes with repo/scan correlation fields (#41) * cli/index: stabilize --jsonl scan event envelopes with repo/scan correlation fields * cli/index: dedupe JSONL scan event envelopes and handle scan_phases write errors * Q1 (#42) * cli/index: stabilize --jsonl scan event envelopes with repo/scan correlation fields * cli/index: dedupe JSONL scan event envelopes and handle scan_phases write errors * doctor: always emit applied_fixes and recommendations arrays in JSON output * json: stabilize watch/doctor machine-readable output (watch event envelopes, doctor arrays always present) (#43) * benchmark: add --sqlite-profile and capture sqlite_profile/host conte… (#44) * benchmark: add --sqlite-profile and capture sqlite_profile/host context in JSON output * benchmark: persist num_cpu/cwd in baselines and list benchmark flags in help * cli: add index_smoke runner with compact jsonl + median baseline for indexing perf diffs (#45) * cli: add index_smoke runner with compact jsonl + median baseline for indexing perf diffs * cli: surface index_smoke baseline/db/config errors; add .codegraphignore * store/indexer: purge deleted-file graph rows and nullify cross-file symbol references (#46) * store/indexer: purge deleted-file graph rows and nullify cross-file symbol references * store: reuse temp id tables in deleted-file purge * watch: apply repo include/exclude and harden dirty_files draining (#47) * watch: apply repo include/exclude and harden dirty_files draining * watch: batch re-queue dirty_files on update failure * store: speed up edge resolver (dot-tail2 map, dotted-edge index, reso… (#48) * store: speed up edge resolver (dot-tail2 map, dotted-edge index, resolver symbol indexes) * store: make resolver temp-table cleanup safe + avoid eager suffix clones * cli/config: default repo artifacts under .codegraph (db, bench gocach… (#49) * cli/config: default repo artifacts under .codegraph (db, bench gocache) w/ legacy db fallback * cli/config: harden repo DB path stat handling + centralize .codegraph dir constant * watch/store: watch config parity + crash-safe dirty queue (claim/delete) (#50) * watch/store: watch config parity + crash-safe dirty queue (claim/delete) * store: make DeleteClaimedDirtyFiles atomic + reuse sqliteInClauseBatchSize * perf(store): scale ResolveEdges dot-suffix + cap slash-suffix map growth; changelog v1.0.10 (#52) * perf(store): scale ResolveEdges dot-suffix + cap slash-suffix map growth; changelog v1.0.10 * fix(store): restrict dot-tail2 candidate set to single-dot dst_name * docs(changelog): normalize release summaries across all tagged versions * perf(store): de-correlate ResolveEdges 1/2/4 by distinct dst_name; add ResolveEdgesForNames scale bench (#53) * fix(store): scope RelatedTests(file) + deterministic symbol lookup; perf: speed up FindDeadCode with indexes (#54) * fix(store): scope RelatedTests(file) + deterministic symbol lookup; perf: speed up FindDeadCode with indexes * Update internal/query/service.go * perf(store): cache batch-insert SQL + speed up edge src selection in write path (#55) * perf(store): cache batch-insert SQL + speed up edge src selection in write path * fix(store): restore nested-function correctness in srcSymbolChooser * fix(store/doctor): harden DB pragmas and migration lifecycle correctness (#56) * fix(store/doctor): harden DB pragmas and migration lifecycle correctness * store: build SQLite file DSNs via net/url; reuse conn across migrations * bench(store): add chooser + callers/callees benchmarks; retry applyPragmas on SQLITE_BUSY (#57) * bench(store): add chooser + callers/callees benchmarks; retry applyPragmas on SQLITE_BUSY * store(sqlite): unify busy-retry policy; make SQLITE_BUSY detection driver-aware * store: cache symbol insert SQL + IN placeholders to reduce write-path allocs (#58) * store: cache symbol insert SQL + IN placeholders to reduce write-path allocs * store: use LoadOrStore in SQL caches and rename symbolInsertSQL arg * add per-strategy resolver microbenches (slash-suffix, dot-tail2, dot-suffix) (#59) * add per-strategy resolver microbenches (slash-suffix, dot-tail2, dot-suffix) * simplify upsertBenchRepo return type to int64 * trim residual write-path allocations + add multi-file batch bench + fix duplicate token stats (#60) * perf(store/indexer): trim repo-wide change-detection floor cost (#61) * perf(export): page no-focus JSONPaged via ExportSymbolsPage/EdgesPage (#62) * perf(export): page no-focus JSONPaged via ExportSymbolsPage/EdgesPage * fix(export): stably order symbols and edges in JSONPaged fallback * perf(store/resolver): early-return resolveEdgesBySlashSuffix when no unresolved edges (#64) * perf(store/resolver): early-return resolveEdgesBySlashSuffix when no unresolved edges * Update internal/store/store.go * docs(changelog): catch up Unreleased with #52–#64 * perf(indexer): overlap ExistingFiles load with FS walk on repo-wide scans (#65) * perf(indexer): overlap ExistingFiles load with FS walk on repo-wide scans * Update internal/indexer/indexer.go * perf(store/resolver): schema-backed slash-suffix path via qualified_suffix (#66) * perf(export): writer-based JSONStream for unbounded JSON export (#67) * perf(export): writer-based JSONStream for unbounded JSON export * fix(export): make JSONStream's encodeIndented robust to first-line prefixing * fix(store): delete test_links pointing at purged target files (#68) * fix(store): delete test_links pointing at purged target files * fix(store): populate test_links.target_file_id during indexing * perf(export): stream unbounded DOT export via paged DISTINCT nodes + edges (#69) * perf(export): stream unbounded DOT export via paged DISTINCT nodes + edges * refactor(export): dedupe DOT edge formatting into a single helper * perf(indexer): slim existing-files map value type to drop floor cost (#70) * perf(indexer): slim existing-files map value type to drop floor cost * refactor(store): scan existing-files metadata directly into caller's map * perf(store/resolver): schema-back dot-tail2 path via persisted column (#71) * perf(store/resolver): schema-back 2-dot dot-suffix path via dot_tail3 (#72) * perf(store/indexer): defer ContentHash to lazy lookup; trim ExistingF… (#73) * perf(store/indexer): defer ContentHash to lazy lookup; trim ExistingFiles map row size * fix(indexer): propagate hashLookup errors through processFileTask * docs(changelog): clarify hashLookup callback signature + error propagation * fix(cli,watcher,indexer,export): MR review fixups before merge * Update changelog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.