feat(extract): ingest AsciiDoc documentation (#2938) - #3089
feat(extract): ingest AsciiDoc documentation (#2938)#3089abhay-codes07 wants to merge 2 commits into
Conversation
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds an AsciiDoc extractor that mirrors the markdown one: extract_asciidoc turns a .adoc/.asciidoc file into a page node with nested heading nodes per section title, and mints references edges from include::, xref:, link:, and <<...>> targets whose ids match the linked document's own node so they merge rather than spawn ghosts. External URLs, in-page anchors, unresolved-attribute paths, and delimited block bodies (listing/literal/comment/passthrough/example/sidebar/quote) are skipped, and extension-less xrefs resolve to a sibling .adoc. Registers the new extensions across DOC_EXTENSIONS, the extract dispatch, hook source exts, and the HTML viewer's document-type detection, and documents the new format support in the README.
Worth a look
- .asciidoc files are not included in hook source extensions —
graphify/cli.py:74· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Delimited block closes on any same-character delimiter length —
graphify/extractors/asciidoc.py:49· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- _block_delim matches any line of same char length >=4 including heading underlines / horizontal rules —
graphify/extractors/asciidoc.py:52· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2520 functions depend on the 620 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 500 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
detect()— 110 callers, 15 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
extract_js()— 83 callers, 3 callees - …and 49 more — each is listed as a finding
Verification — 2520 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2473 function(s) in the blast radius were not formally verified this run
Formal verification
No difference found (not proven): No behavior difference found in normalize\_node (not a proof).
The verifier ran both versions of normalize\_node on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
· 1 grounded finding(s) anchored inline below; 56 more finding(s) on lines outside this diff (see the check run).
| return Path(os.path.normpath(str(candidate))) | ||
|
|
||
|
|
||
| def extract_asciidoc(path: Path) -> dict: |
There was a problem hiding this comment.
extract_asciidoc()
fans out to 8 callees (efferent coupling); 11 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
|
Follow-up commit on the review's advisory points: a delimited block now closes only on a delimiter of the same character and length (so a nested |
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 3 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds a first-class AsciiDoc extractor (extract_asciidoc) that treats .adoc/.asciidoc files as document page nodes, mints heading nodes from section titles nested by level, and emits references edges for local include::, xref:, link:, and <<...>> document targets while skipping external URLs, unresolved attribute paths, and delimited block bodies; edge target ids are minted from the resolved absolute path so they merge into each referenced document's own node. Wires the new extension into _DISPATCH and into the doc-extension sets across detect, cli hooks, and callflow_html node typing so AsciiDoc projects are scanned and rendered as documents rather than ignored. Documents the support in the README.
Worth a look
- AsciiDoc link target can crash extraction via NUL byte —
graphify/extractors/asciidoc.py:133· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- AsciiDoc links can probe and disclose files outside the source tree —
graphify/extractors/asciidoc.py:133· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- AsciiDoc heading IDs collide across same-named files —
graphify/extractors/asciidoc.py:183· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2523 functions depend on the 623 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 500 callers, 42 callees - new:
_rebuild_code()— 98 callers, 50 callees - new:
detect()— 110 callers, 15 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
extract_corpus_parallel()— 26 callers, 11 callees - new:
extract_js()— 83 callers, 3 callees - …and 49 more — each is listed as a finding
Verification — 2523 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2476 function(s) in the blast radius were not formally verified this run
Formal verification
No difference found (not proven): No behavior difference found in normalize\_node (not a proof).
The verifier ran both versions of normalize\_node on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
· 1 grounded finding(s) anchored inline below; 56 more finding(s) on lines outside this diff (see the check run).
| return Path(os.path.normpath(str(candidate))) | ||
|
|
||
|
|
||
| def extract_asciidoc(path: Path) -> dict: |
There was a problem hiding this comment.
extract_asciidoc()
fans out to 8 callees (efferent coupling); 12 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
`.adoc` was in no extension set, so a project documented in AsciiDoc had
none of it in the graph — only the PNGs beside the docs were ingested, and
the detector said so in passing.
`.adoc`/`.asciidoc` join every place a document is decided: DOC_EXTENSIONS
(classification and the semantic pass), the splittable-text set (an
oversized manual is sliced, not truncated), the callflow and hook-guard
extension lists, and markdown's linkable set (a `[text](./guide.adoc)`
link now resolves).
A structural pass, extractors/asciidoc.py, mirrors the markdown extractor
so the deterministic layer is on par: the file is a `page` node carrying
the document title and header attributes (as `frontmatter`), section
titles are `heading` nodes nested by level, and `include::`, `xref:`,
`link:<local path>` and the cross-document `<<file.adoc#anchor>>` form
become `references` edges minted from the resolved path, so they merge
into the linked document's own node (stamped `target_file` for the
incremental remap, like markdown). External URLs, in-page anchors,
non-document targets and paths carrying an unresolved attribute
(`{docdir}/x.adoc`) are skipped rather than guessed; delimited blocks and
comment lines are not parsed for headings or links.
…doc in the hook list A nested example block (==== ... ====== ... ====== ... ====) must not close the outer one early, and .asciidoc belongs beside .adoc in the hook-guard source extension list.
da41729 to
ef280d2
Compare
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.
Formal verification. 1 change(s) tested, no difference found (not proven).
Graphify review — findings
Adds an AsciiDoc extractor so .adoc/.asciidoc files enter the graph as page nodes with nested heading nodes per section title, and turns include::, xref:, link:, and <<...>> targets into references edges whose ids are minted from resolved paths so they merge into each referenced document's own node. Registers the new extensions across detection (DOC_EXTENSIONS), dispatch (get_extractor), the HTML viewer's document classification, and the hook source list, and skips external URLs, unresolved {attr} paths, in-page anchors, and delimited block bodies. Documents the support in the README extension table.
Worth a look
- AsciiDoc links can probe and disclose arbitrary absolute local files —
graphify/extractors/asciidoc.py:128· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Header attribute parsing never terminates when document has no title —
graphify/extractors/asciidoc.py:174· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 2825 functions depend on the 656 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 587 callers, 44 callees - new:
_rebuild_code()— 115 callers, 51 callees - new:
detect()— 112 callers, 15 callees - new:
_extract_generic()— 18 callers, 26 callees - new:
save_manifest()— 40 callers, 11 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
extract_js()— 85 callers, 4 callees - new:
extract_xaml()— 19 callers, 17 callees - …and 64 more — each is listed as a finding
Verification — 2825 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 2763 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
274 of 274 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/test_affected_cli.py— impact, full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_agents_platform.py— impact, full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_asciidoc.py— impact, changed-test, full-run-safetytests/test_astro_extraction.py— impact, full-run-safetytests/test_astro_import_ids.py— impact, full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— impact, full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_build.py— impact, full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— impact, full-run-safetytests/test_cache.py— full-run-safetytests/test_callflow_html.py— impact, full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— impact, full-run-safetytests/test_charmap_encoding.py— impact, full-run-safetytests/test_chunking.py— impact, full-run-safetytests/test_cjs_module_extension.py— impact, full-run-safetytests/test_claude_cli_backend.py— impact, full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cluster.py— full-run-safetytests/test_codebuddy.py— impact, full-run-safetytests/test_community_hub_labels.py— full-run-safetytests/test_community_labels_skill.py— full-run-safetytests/test_confidence.py— full-run-safetytests/test_corrupt_graph_json.py— full-run-safetytests/test_cpp_nested_and_cli.py— impact, full-run-safetytests/test_cpp_objc_cross_file_calls.py— impact, full-run-safetytests/test_cpp_preprocess.py— full-run-safetytests/test_cross_extension_reexport_self_cycle.py— impact, full-run-safetytests/test_cross_language_call_resolution.py— impact, full-run-safetytests/test_cross_repo_member_calls.py— impact, full-run-safetytests/test_cross_repo_shared_types.py— full-run-safetytests/test_csharp_call_site_generic_args.py— impact, full-run-safety- … and 224 more
non-code file(s) changed (
README.md) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
README.md) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Formal verification
No difference found (not proven): No behavior difference found in normalize\_node (not a proof).
The verifier ran both versions of normalize\_node on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
· 1 grounded finding(s) anchored inline below; 71 more finding(s) on lines outside this diff (see the check run).
| return Path(os.path.normpath(str(candidate))) | ||
|
|
||
|
|
||
| def extract_asciidoc(path: Path) -> dict: |
There was a problem hiding this comment.
extract_asciidoc()
fans out to 8 callees (efferent coupling); 12 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Closes #2938.
The problem
.adocwas in no extension set, so a project whose documentation is AsciiDoc had none of it in the graph — the detector reported that only the PNGs beside the docs were ingested._PROSE_EXTSalready knew.adocfor word counting, which is how far it got.The change
.adoc/.asciidocjoin every place a document is decided:DOC_EXTENSIONS— classification, and so the semantic pass digests them like markdown;file_slice._SPLITTABLE_TEXT_SUFFIXES— an oversized manual is sliced, not truncated (Oversized .qmd/.html/.yaml/.skill documents are silently truncated at 20k chars before the semantic pass — only 5 of 10 document types are sliceable #2900's contract);_MD_LINKABLE_EXTS— a[text](./guide.adoc)link now resolves.And a structural pass,
extractors/asciidoc.py, so the deterministic layer is on par with markdown's:pagenode carrying the document title (= Title) and the header attributes (:author:…) asfrontmatter, sanitized like markdown's;==,===, …) areheadingnodes nested by level,file → contains → heading,heading → contains → subheading;include::other.adoc[],xref:other.adoc[...]/xref:other#anchor[](an extension-less xref names a sibling.adoc),link:<local path>[...]and the cross-document<<other.adoc#anchor,text>>form becomereferencesedges. The target id is minted from the resolved path — the same recipe as the target's own file node — so it merges into that node rather than spawning a ghost, and an existing target is stampedtarget_filefor the incremental remap, exactly as markdown does (Incremental rebuild drops cross-file references edges between Markdown docs; GRAPHIFY_FORCE does not help in the hook (follow-up to #1018) #2211).Skipped rather than guessed: external URLs, in-page anchors, non-document targets (
diagram.png), and paths carrying an unresolved AsciiDoc attribute (include::{snippets}/gen.adoc[]). Delimited blocks (----,....,////,++++,====,****,____) and//comment lines are not parsed for headings or links, mirroring markdown's fenced-block handling.What it looks like
Tests
tests/test_asciidoc.py— 21 tests: both extensions present in every set that decides a document (classification, dispatch, slicing, markdown linkability) and listed bydetect(); the page node's title and attributes; section nesting; block and comment skipping; duplicate section titles; a title-less fragment; unreadable files; each link form producing one edge per target; the skip rules; thetarget_filestamp; and a corpus-level check that every reference — in both directions between markdown and AsciiDoc — merges into the linked document's real node with nothing dangling and no stamp leaking.With the registrations reverted and only the extractor module kept, the 4 integration tests fail (the extractor's own unit tests rightly keep passing).
test_detect,test_oversized_document_slicing,test_extractors_registryand the markdown/wikilink suites are unchanged (290 passed); the full suite matches thev8baseline.README's file-types table is updated.