[bompus]

Indexing a large project spends a median 3–4% less time in its parse loop (supabase 5.85 s to 5.67 s, n8n 15.48 s to 14.85 s) because the next batch of files is read while the current one is being parsed. The gain does not reach the whole index time on any project I measured, so this is a small change in one phase, not a faster init.

codegraph init, median of 4 runs per build	Parse loop before	Parse loop after	Parse-loop benefit	Whole index before	Whole index after	Whole-index change
Alamofire (small)	0.37 s	0.36 s	2.1% (ranges overlap)	1.69 s	1.73 s	no clear change (2.1% slower)
gin (Go, small)	0.25 s	0.23 s	7.1% of a 0.25 s phase (ranges overlap)	1.06 s	1.04 s	no clear change (1.9% faster)
pretix (mid)	2.57 s	2.48 s	3.4% (ranges overlap)	10.26 s	9.93 s	no clear change (3.1% faster, ranges overlap)
supabase (large)	5.85 s	5.67 s	3.1% (ranges do not overlap)	19.27 s	19.25 s	no clear change
n8n (largest)	15.48 s	14.85 s	4.0% (ranges do not overlap)	69.89 s	68.95 s	no clear change (1.3% faster)

Peak memory differs by +0.2% to +0.9% (no clear change), and the node and edge counts plus an md5 of the ordered node and edge rows are identical in every run on every project. Builds f045e5f7 and the same with this change; 40 runs, 0 rejected, order B C C B B C C B per project, declared before the runs. Under the rule I declared there ("land only when whole-index time is clear") this was rejected; I am landing it because the parse-loop gain is real on the two largest projects, nothing got worse beyond noise, and the change is small.

What changes

The parse loop read 10 files, waited, then handed them to the parser pool, so the pool sat idle during every read. readBatch is now a function; the loop starts reading batch n+1 before it dispatches batch n. At most one extra batch of file contents is held in memory (peak memory stayed within 1% in the runs above). The read function is unchanged and never rejects, so an abort in the middle leaves no unhandled error.

Checks

• Full vitest suite on this head: 702 files, 8,952 passed, 39 skipped, 0 failed; no Native stack trace or Worker exited unexpectedly in the raw output.
• README: checked "About this fork"; no row mentions the parse loop's read step. CHANGELOG entry added under Fixes.

Not covered: why overlapping the read recovers only part of the 6–9% the stage timers allowed (the UTF-8 decode and path validation still run on the main thread); Bun; Windows and macOS.

---

Summary by cubic

Speeds up indexing by reading the next batch of files while the current batch is being parsed, so the parser pool no longer sits idle during reads. Parse-loop time falls a median 3–4% on large projects (supabase 5.85 s to 5.67 s, n8n 15.48 s to 14.85 s); whole-index time stays within run-to-run variation, graph output is identical, and peak memory stays within 1%. At most one extra batch of file contents is held in memory, and the read function still never rejects.

• Reading the next batch is cancelled once indexing is aborted, so no new reads start after an abort.

^{Written for commit 1cdfdf4. Summary will update on new commits.}

Summary by CodeRabbit

• Performance
  • File reading now overlaps with parsing, reducing median parsing time by 3–4% on two large projects. Overall indexing time remained within normal run-to-run variation across five projects.
  • Index results are unchanged, and peak memory use varied by less than 1% from previous behavior. At most one additional batch of file contents is held in memory during indexing.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository YAML (base), Organization UI (inherited)
• Review profile: CHILL
• Plan: Advanced
• Run ID: 48b41132-fb5a-4995-92b1-8c71bbd18e2e

📥 Commits

Reviewing files that changed from the base of the PR and between b74619c and 1cdfdf4.

📒 Files selected for processing: 2

• CHANGELOG.md
• src/extraction/index.ts

🚧 Files skipped from review as they are similar to previous changes: 1

• CHANGELOG.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

📝 Walkthrough

indexAll now overlaps reading the next file batch with parsing the current batch. The change extracts file reading and validation into readBatch. The changelog reports timing, graph, and memory measurements.

Changes: Batch Reading and Parsing

Layer / File(s)	Summary
Read batches alongside parsing src/extraction/index.ts, CHANGELOG.md	readBatch retains per-file validation, oversize handling, MPEG transport-stream detection, and individual read-error results. indexAll starts the next read before dispatching the current batch for parsing. The changelog records the reported timing, graph, and memory measurements.

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to 1cdfd

No specific issue requiring a fix before merge is established; proceed with normal checks.

Security architecture risk: ⚪ Minimal · up to b7461

The change overlaps existing file reads with parsing without adding permissions or expanding access to source content. File-access checks and ordered graph updates remain intact, and no material security risk introduced by this change was identified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The incremental exposure is earlier acquisition of one batch from the existing discovered file set under the same process filesystem authority. The scheduling change does not expand the permitted file set, grant privileges, or add a content-serving route.

Trust Boundaries and Controls

• observed — CodeGraph.getCode delegates to ContextBuilder.getCode, which validates the node's file path without the indexing-only symlink-escape option before reading content. The traced MCP symbol renderer uses that sink, and its separate file-read path also validates strictly. These unchanged controls remain independent of prefetched indexing bytes.

Resilience and Maintainability Implications

• inferred — Abort or a thrown progress callback can leave an additional acquisition batch running because readBatch has no cancellation token. That producer only returns read results and closes file handles in finally; it cannot independently parse or store them. Abort checks still stop subsequent dispatch, and the new promise and ordering state are local to each invocation.

Pre-merge checks: 6

Passed checks: 6 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Suppressions Explained		The pull request changes only CHANGELOG.md and src/extraction/index.ts. The added lines contain no lint, type-check, compiler-suppression directive, or ignore-entry change. The comments describe n…
User-Visible Changes Documented		PASS. The PR changes only src/extraction/index.ts and CHANGELOG.md. The source diff adds an internal readBatch helper and read-ahead scheduling; it does not add, remove, or rename a CLI command …
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: prefetching the next file batch while dispatching the current batch for parsing.

✨ Finishing Touches: 💡 1

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

🛠️ Fix failing CI checks: 💡

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[cubic-dev-ai]

All reported issues were addressed across 2 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>