Skip to content

gh-153568: Factor shared prefixes in expression rules - #157437

Open
pablogsal wants to merge 1 commit into
python:mainfrom
pablogsal:gh-153568-factor-expression-grammar
Open

gh-153568: Factor shared prefixes in expression rules#157437
pablogsal wants to merge 1 commit into
python:mainfrom
pablogsal:gh-153568-factor-expression-grammar

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 13, 2026

Copy link
Copy Markdown
Member

Several expression rules start by parsing the same thing for each choice. For a - b, the parser tries the + alternative first, then goes back and gets a again to try -. The cache saves us from parsing a from scratch, but we still pay for the extra rule call and cache lookup.

This puts the shared part before the choice: get the left operand once, then check which operator follows. It does the same for tuples, boolean expressions, comparisons and powers. Parse the first expression once, then look for the optional part that follows it. When there is nothing more to match, return the expression we already have.

On my machine, this makes parsing the full standard library and ten repositories about 4.7% faster overall.

Codebase Python files Parsing speed
Standard library 2,060 4.8% faster
Django 2,932 5.5% faster
pandas 1,527 4.8% faster
NumPy 495 4.4% faster
SciPy 999 4.6% faster
scikit-learn 1,036 4.7% faster
SymPy 1,607 5.6% faster
Transformers 4,884 3.9% faster
PyTorch 4,890 4.4% faster
Home Assistant 18,731 4.7% faster
mypy 442 5.5% faster

The table compares this change against main at a60343ed17, using release builds without PGO or LTO. The timings include tokenization, building the AST and cleanup, without compiling bytecode or running the files.

Part of #153568.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant