👌 Fix quadratic inline tokenization on runs of special characters - #411
Open
hdimer wants to merge 1 commit into
Open
👌 Fix quadratic inline tokenization on runs of special characters#411hdimer wants to merge 1 commit into
hdimer wants to merge 1 commit into
Conversation
Long runs of characters that begin an inline rule but do not form a construct (bare `&`/`<`, incomplete entities, ...) were tokenised in O(n^2) time, from two causes: - the inline tokenizer's fallback appended to `state.pending` one character at a time; `str += ch` on an attribute cannot reuse the buffer in place, so each append is O(len). - the `entity` and `html_inline` rules matched their `^`-anchored regexes against `src[pos:]`, slicing the remaining source on every character. Accumulate `pending` through a list buffer materialised lazily, and anchor the entity/html regexes with `.match(src, pos)` instead of slicing. Rendering is now linear (e.g. `'&' * 500_000` drops from seconds to well under one); output is unchanged and the full suite passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Long runs of characters that begin an inline rule but do not form a construct — bare
&/<, incomplete entities like&am&am…, etc. — were tokenised in O(n²) time. A single such character is fine; it is the count that blows up. For example,md.render("&" * 128_000)took over a second and grew quadratically.This is the same class as the previously fixed #367 and #389.
Root causes
Two independent quadratic factors, both hit once per character in these runs:
state.pendingaccumulation. The inline tokenizer's fallback path doesstate.pending += state.src[state.pos]one character at a time. Becausependingis an attribute,str += chcannot use CPython's in-place concatenation optimisation (the attribute keeps a second reference), so each append rebuilds the whole string — O(len) per character.src[pos:]slicing inentity/html_inline. These rules matched^-anchored regexes againststate.src[pos:], copying the remaining source on every&/<.In JavaScript
markdown-itneither is quadratic (rope-backed string concat, and the equivalent regexes are applied with a sticky/lastIndex match), so this is Python-port-specific.Fix
StateInline.pendingnow accumulates through a lazily-materialised list buffer (append_pending), giving amortised O(1) appends. It still presents as a plainstrto every reader (the getter joins the buffer on demand and caches).entityandhtml_inlineanchor their regexes with.match(state.src, pos)instead of slicing (the leading^is dropped, since.matchalready anchors atpos).HTML_TAG_REis only used byhtml_inline, so this is contained.Rendering is now linear in all these cases (measured slope ≈ 1.0, including the
html=Truehtml_inlinepath);"&" * 500_000drops from seconds to well under one. Output is unchanged — the full test suite passes.Tests
Added
test_long_special_char_runs_stay_linearintests/test_api/test_main.py: it asserts correct rendering for the affected constructs and renders a large input whose previous O(n²) behaviour would exceed the global 10s test timeout. Fullpytest tests/passes (982).ruff check/ruff format(0.15.12) andmypy(1.20.2, strict) are clean.Disclosure: this fix was developed with AI assistance; I reviewed it and stand behind it.