Skip to content

feat(explore): markdown index (#361) with a section-first doc tier - #1699

Open
bompus wants to merge 2 commits into
colbymchenry:mainfrom
bompus:feat/markdown-section-first
Open

feat(explore): markdown index (#361) with a section-first doc tier#1699
bompus wants to merge 2 commits into
colbymchenry:mainfrom
bompus:feat/markdown-section-first

Conversation

@bompus

@bompus bompus commented Sep 5, 2026

Copy link
Copy Markdown

Follow-up to #361 and #1439, with the measurement from this comment on #361.

What this carries. #361's markdown extractor as it stands (headings, sections, tables and links as nodes; name-matcher and resolution hooks; its five test files), rebased onto main, plus a doc tier in codegraph_explore and three fixes the index needs to leave code answers alone.

The doc tier. A doc-shaped query that names a markdown file renders that file's best sections first and whole: the top three by idf-weighted line hits, where a term that also appears in the file's own path weighs zero (it located the file, not the line) and a heading the query covers word for word counts as a named section; 8k characters per file, spent in score order and rendered in file order. The blast-radius, relationships and "additional files" blocks stay off unless a code file rendered too. Code queries take the same path they did before.

Three fixes for a shared index.

  • detectGeneratedFile skips the header check for markdown: a README that quotes "generated by" was dropped from the index.
  • Markdown nodes leave the subgraph unless the doc tier seeded them: section bodies share the FTS table with code, so an English word in a README matched into code queries.
  • The explore budget tiers count code files only. Markdown took a 466-file repo to 575 indexed files, across the 500-file breakpoint, and every code answer grew a Relationships block.

The server instructions now say markdown is indexed; #361's text still listed docs under what codegraph does not index, and measured as merged the model never called explore for a doc question (24 cells, 0 calls).

Measured. Headless Claude Code, Opus, a repo with 109 markdown files, six doc-question tasks, three rounds of 12 fresh cells against the shipped build with the same repo rules:

Round 1 Round 2 Round 3
Correct 12 / 12 12 / 12 12 / 12
Tool calls, median (shipped: 4) 1 1 1
Explore adopted 10 / 12 11 / 12 10 / 12
No Grep or Read after explore 8 / 12 9 / 12 9 / 12

Every explore call chose the right file and section; the misses are the model grepping a file the prompt already names, and one prompt it reads as being about its own scheduling tools. Cost per cell $0.30 against $0.54.

Code answers on the same repo, one fresh session per query, 25 code queries against main: 11 byte-identical, 3 the same lines reordered, 10 swap a fourth- or fifth-ranked padding file (markdown documents shift FTS tie ranks), and one prose prompt with no identifier answers from two markdown files instead of five unrelated code files.

Tests. tsc clean. #361's five test files plus the two explore tests that the raw patch broke on main (explore-allocation-1500, explore-session-state): 797 pass; the failures on my Windows host are all EBUSY temp-dir teardown in afterAll, none an assertion.

Re-index after upgrading: the markdown nodes are written while indexing.

… doc tier

Ports QingNagi/codegraph#361 (markdown extractor, heading nodes, name-matcher and
resolution hooks) onto experimental and adds the section-first doc tier from
feature/md-section-first: a doc-shaped query renders the best headed sections of
the markdown file it names, ranked by idf-weighted line hits with path tokens
weighted zero, capped at DOC_FILE_CAP per file. Markdown reaches an answer only
through that tier, generated-file detection ignores markdown bodies, and the
budget tiers count code files only so a README-heavy repo keeps its code answers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant