perf(resolution): parse each Python file once across both resolution passes - #3502
perf(resolution): parse each Python file once across both resolution passes#3502abhay-codes07 wants to merge 1 commit into
Conversation
…passes The symbol-resolution facts pass and the cross-file import pass each parsed the whole .py corpus, back to back, in the main process — every file read and tree-sitter-parsed twice for nothing. _parse_python_tree now memoizes on (path, mtime, size), so the second pass reuses the first pass's tree while a file changed between watch runs still re-parses. On a 364-file self-corpus a sequential extract drops from ~27s to ~24s, graph byte-identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JJbLfztSxm2tH5cJwBbe9q
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Memoizes _parse_python_tree on (path, mtime_ns, size) via _parse_python_tree_cached so the Python facts pass and the cross-file import pass share a single tree-sitter parse per file instead of reading and parsing the whole .py corpus twice; a file that changes between runs re-parses, and errors still return None (skip the file). _resolve_cross_file_imports now goes through the shared memo and keeps tree_sitter_python only as an availability check. Adds tests covering the source/root return, missing-file handling, within-run reuse, and re-parse on edit.
No blocking issues surfaced. 4 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1848 functions depend on the 144 functions this change touches.
Health — this change adds coupling hotspots:
- new:
extract()— 585 callers, 44 callees - new:
_rebuild_code()— 115 callers, 51 callees - new:
_extract_generic()— 18 callers, 26 callees - new:
extract_js()— 85 callers, 4 callees - new:
extract_xaml()— 19 callers, 17 callees - new:
_resolve_js_module_path()— 34 callers, 9 callees - new:
dispatch_command()— 2 callers, 124 callees - new:
extract_objc()— 27 callers, 9 callees - …and 35 more — each is listed as a finding
Verification — 1848 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 800 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
30 of 269 test file(s) selected (11%) via static blast radius.
tests/test_astro_extraction.py— impacttests/test_build.py— impacttests/test_cjs_module_extension.py— impacttests/test_cpp_nested_and_cli.py— impacttests/test_dotnet.py— impacttests/test_extract.py— impacttests/test_forwarding_review_findings.py— impacttests/test_import_extension_resolution.py— impacttests/test_indirect_dispatch.py— impacttests/test_indirect_dispatch_assign_return.py— impacttests/test_indirect_dispatch_getattr.py— impacttests/test_js_exported_scalar_bindings.py— impacttests/test_languages.py— impacttests/test_multilang.py— impacttests/test_package_json_subpath_imports.py— impacttests/test_pascal.py— impacttests/test_pascal_resolution.py— impacttests/test_phantom_external_import.py— impacttests/test_python_import_resolution.py— impacttests/test_python_parse_memoization.py— impact, changed-testtests/test_python_underscore_resolution.py— impacttests/test_rationale.py— impacttests/test_ruby_resolution.py— impacttests/test_scala_self_type.py— impacttests/test_src_layout_import_resolution.py— impacttests/test_swift_computed_properties.py— impacttests/test_ts_new_expression_calls.py— impacttests/test_typescript_module_extensions.py— impacttests/test_unmapped_at_alias_resolution.py— impacttests/test_vue_extraction.py— impact
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Formal verification
Could not verify: Could not verify \_parse\_python\_tree.
The verifier did not have enough to check \_parse\_python\_tree, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `path` is annotated `Path` — outside the synthesizable primitive/collection set
Could not verify: Could not verify \_resolve\_cross\_file\_imports.
The verifier did not have enough to check \_resolve\_cross\_file\_imports, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: non-vacuity: domain too small (only 2 distinct inputs exercised, need 3) — 'no divergence' would be near-vacuous
· 43 more finding(s) on lines outside this diff (see the check run).
What
The Python symbol-resolution facts pass (
_collect_python_symbol_resolution_facts) and the cross-file import pass (_resolve_cross_file_imports) each parse the entire.pycorpus, back to back, in the main process after the workers return. Every file was read from disk and tree-sitter-parsed twice for no reason — cProfile showedread_bytesfiring ~5× per file andparse~4× across the run._parse_python_treenow memoizes on(path, mtime_ns, size), and the cross-file pass — which previously built its ownParserand re-read each file — routes through it. The second pass reuses the first pass's tree.Why it's safe
_parse_python_treestill returnsNoneon any error exactly as before — callers already treat that as "skip this file".Treeand read.root_node; the shared helper returnsroot_nodedirectly, and the one downstreamtree.root_nodeaccess was updated accordingly.cmpequal).Measured
Sequential extract of graphify's own 364-file Python corpus (best of 3, fresh cache dir each run):
v8A ~11% reduction from eliminating one full parse (and disk read) of every Python file. Composes with the other resolution-pass perf work.
Tests
tests/test_python_parse_memoization.py: parse returns(source, root), missing file →None, one parse reused within a run, and an edited file re-parses rather than replaying. Full suite matches a freshv8(0.9.58) baseline.🤖 Generated with Claude Code
https://claude.ai/code/session_01JJbLfztSxm2tH5cJwBbe9q