Repository navigation
Cut parse cost by about 31% in frames with no reading moved (#1436) - #1459
Merged
Merged
Conversation
_is_emoji ran any() over a generator, a frame per range tried, for every non-ASCII character under strip_emoji (the default). Frames per parse on py3.11: reference row 285.72 -> 285.72, 'Smith, John' 136 -> 136, the long row 473.86 -> 473.86, a Devanagari name 336.04 -> 271.04. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_tokenize_region asked _ignorable of every character of the name, a frame each. The ignorable set is now a regex class per setting of the two strip switches, built from the same parts (whitespace, the name dots, the invisible separators, _BIDI's class, the emoji ranges), and the region is scanned with finditer for the characters that may separate: those, the commas and the transcription dots, whose context tests are unchanged. _ignorable and _stripped match the same classes, so extract and the flank guard read the one definition; a new test holds each class to its parts over every code point, and _is_emoji goes, its ranges now read only through the class. Frames per parse on py3.11: reference row 285.72 -> 258.72, 'Smith, John' 136 -> 126, the long row 473.86 -> 421.86, a Devanagari name 271.04 -> 226.04. Tokenize output for every policy over a grid of the separator candidates in context, and _ignorable/_stripped over every code point, compared byte-identical with master in two trees. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lambda key was a frame per token. Frames per parse on py3.11: reference row 258.72 -> 252.72, 'Smith, John' 126 -> 124, the long row 421.86 -> 411.86. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tuple() over a generator resumed it once per token, a frame each. Frames per parse on py3.11: reference row 252.72 -> 246.72, 'Smith, John' 124 -> 122, the long row 411.86 -> 401.86. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each segment's pieces and piece tags were frozen through a generator, a frame per piece twice over; map over the type does it in C. Frames per parse on py3.11: reference row 246.72 -> 236.72, 'Smith, John' 122 -> 114, the long row 401.86 -> 383.86. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dict was built on every parse, and Enum.__hash__ is a Python method: three frames a parse. Frames per parse on py3.11: reference row 236.72 -> 233.72, 'Smith, John' 114 -> 111, the long row 383.86 -> 380.86. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Token.__post_init__ tested the two coordinates with all() over a generator, three frames a token, and rebuilt the span as a Span even when it was one, a frame for the NamedTuple's __new__. The checks and their messages are unchanged; a span that is not exactly a Span is still rebuilt as one. Frames per parse on py3.11: reference row 233.72 -> 209.72, 'Smith, John' 111 -> 103, the long row 380.86 -> 340.86. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A frozenset is neither, and isinstance against Mapping, an ABC, enters a Python frame; every token the parse returns carries a frozenset. Frames per parse on py3.11: reference row 209.72 -> 203.72, 'Smith, John' 103 -> 101, the long row 340.86 -> 330.86. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_coerce_enum hands a Role back unchanged, a frame a token for nothing on every token the parse builds; a string role is still coerced, and refused, by it. Frames per parse on py3.11: reference row 203.72 -> 197.72, 'Smith, John' 101 -> 99, the long row 330.86 -> 320.86. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the first nine commits: the _stripped and _ignorable docstrings and the module docstring still described the per-character call, and none named U+202F as the one ignorable character context decides. The every-code-point test now matches each class whole, in C (0.16s, was about 3.5s across four parameters), and says what it cannot guard. New rows hold what no test reached: the dot's flanks and U+202F's letters are read inside the region (each fails with the bound dropped), and a Token's span is checked coordinate by coordinate when it is already a Span (each fails with the check moved behind the Span test). Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of 8392358: its reader rows ran under the all-on and all-off policies only, where a reader taking the two switches in the wrong order reads right; under every setting that mutation now fails. The module docstring's claim that extract and tokenize cannot disagree is narrowed to the characters that separate in every context, the two context decides being tokenize's alone. And a span with a bad start and a good end gets its rows. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on the branch rebased onto 688991e (#1458's merge), each interpreter in an environment of its own: parse 285.72 -> 197.72 on 3.11 and 277.72 -> 188.72 on 3.12 to 3.15, so _CALL_BASELINE moves 285 -> 197 and 277 -> 188; the 64-link run 2,404 -> 1,716 and 2,395 -> 1,706, so _LINK_BASELINE moves 2,418 -> 1,716 and 2,398 -> 1,706. decisions/parse-cost records the move with per-commit and per-name figures, the clock, and the checks that no reading moved. Closes #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of f13950e: the comment on why the pin reads the long run mixed the 2026-09-22 clause-run figures, given a wrong date, with the name-link run's current one, and kept the old band in the present tense beside the new; it now states the old arithmetic as measured then and points at the pin for the current band. The decisions entry's recipe compared against origin/master, which reads 0 once this merges, and now names 688991e; and it names both pairs of trees the 114,647-reading grid compared, re-run after the rebase, identical. Refs #1436 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73
force-pushed
the
fix/issue-1436-parse-cost
branch
from
November 19, 2026 00:02
ecfa86e to
1574d60
Compare
derek73
marked this pull request as ready for review
November 19, 2026 00:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1436
Fewer Python frames per parse with no reading moved. Measured on the branch rebased onto 688991e (#1458's merge), each interpreter in its own environment:
_CALL_BASELINEmoves 285 → 197 and 277 → 188;_LINK_BASELINEmoves 2,418 → 1,716 and 2,398 → 1,706, measured at 2,404 → 1,716 and 2,395 → 1,706.The dated entry is in
docs/design/decisions/parse-cost.md. The issue's scratch patches reached about 234. Each change was picked from a profile of the reference row on master, not from the issue's list alone; theTokenvalidation fast paths were not on that list and were the second-largest piece.Per commit (frames, py3.11, measured commit by commit after the rebase)
Smith, John688991e792ebd891_is_emojiloops over its ranges instead ofany()over a generator (a Devanagari name 336 → 271)d7cba425_ignorableon every characteraf9fb9f1attrgetterkey instead of a lambda87a7a4cdtuple([...])instead oftuple(genexpr)cb4e1f27tuple(map(tuple, …))andmap(frozenset, …)e4101c34tail_startby identity instead of from a dict keyed by the enum (Enum.__hash__is a Python method)9a293c19Tokenchecks its span coordinates without a generator and keeps an exactSpanas givenb3b552ecTokenskips the str andMappingrefusals when its tags are a frozenset (asking an ABC enters a frame)22b26aa0Tokencalls_coerce_enumonly when its role is not already aRole18024105,adcaa494f13950ee,1574d607Per name, one warm parse in a fresh process, master then branch:
John SmithSmith, JohnJohn Smith, M.D.John Smith Jr., PhDराम बहादुर के.सी.Each change commit stands on its own, so a reviewer can drop any one of them (the baselines would then move by that commit's figure).
On the "one predicate" rule (mechanisms.md#ONE-PREDICATE-PER-QUESTION):
_ignorable. The separator set is now regex classes, one per setting of the two strip switches, built from the same parts the function used to test._ignorable(which extract also asks) and_stripped(which the flank guard asks) match those same classes, and tokenize scans the boundary class, which is the ignorable class plus the commas and the transcription dots. A new test checks every class over every code point against its parts as the parts' own predicates state them:str.isspace, the two sets, the emoji ranges, and_BIDI.Tokenchanges skip work only where the answer is already fixed. A frozenset can be neither a str nor aMapping,_coerce_enumhands aRoleback unchanged, and an exactSpanwould be rebuilt into an equalSpan. The role test is the one inlined half: it is the first line of_coerce_enum, anisinstancethat returns its argument, so it cannot drift.Wall clock
Frames cannot see C-level work, so here is CPU time for 1000 parses of the reference row (the thousand-names test's loop), the minimum of 15 runs, with the two trees run alternately three times on py3.11 (master
b78d5af2against the branch before the rebase; #1458 touched only the comma path):I checked separately that no change traded frames for a slower C path. Per 1000 parses, minimum of 9 runs:
Smith, John高橋一郎The one place a list replaced a generator, classify, builds every element either way, so no short-circuit was lost. The one
next()over a generator that does short-circuit, in group's maiden take, was left alone (converting it saved one frame).No reading moved
tools/differential/snapshot.jsonlis byte-identical, andtest_snapshot.pypasses after every commit.uv run python tools/differential/measure.py --base <master b78d5af2 checkout>, run before the rebase: 29 steps, no step failed, no moves in any of them. That covers the committed snapshot, CLDR/GND agreement and its locale views, 17 local snapshots (ctgov ×2, cz, dof, fec, govuk, jrc, nepal, opensanctions-parliaments, orcid, patentsview, psc, tse, tw, uit-vinames, ukec, vn-assembly), 6 agreement scores and 4 "alike" checks.Encoding and script grid, compared across two trees (master in its own worktree, each side asserting
nameparser.__file__). It holds 114,647 readings: repr, ambiguities,initials(), bothcapitalized()views and every token's text, span, role and tags. The words in it:José,inž.,oğlu)क्षत्र,श्रीमान)❤️,John️)محمد,John,Ali)The shapes were 12 comma and clause shapes (no comma, full and one-word heads, an empty head, a nickname clause, a maiden marker, the fullwidth and ideographic commas), each under all three orders × both strip switches, plus
locales.ZH. Every reading was byte-identical, both before the rebase (b78d5af2against the branch) and after it (688991e7against the rebased branch). To show this harness can report a difference, dropping U+200B from the invisible separators moves 4,303 of its lines.Tokenize alone was compared across the same two trees:
_ignorableand_strippedover every code point × every strip setting, plusextractandtokenizeoutput over every separator candidate in 13 contexts. Both were identical, and the same one-character mutation moved them. The code review then fuzzed the old_tokenize_regionagainst the new one over 300,000 random texts with random region bounds, empty regions and extracted regions included, and found 0 differences.Behavior-changing candidates
None found. Every change preserves readings by construction.
Checks
After the rebase and re-take:
uv run pytest: 49,969 passed.uv run pytest -n0 -m serial: 38 passed.uv run mypy,uv run ruff check: clean.test_a_run_of_initials_behind_a_full_name_takes_no_count, passes with its exact counts unchanged.Review aspects run
pr-review-toolkit code-reviewer on the nine change commits. It differentially fuzzed tokenize against the old module and found no correctness bug, plus one stale
_strippeddocstring, now fixed.pr-review-toolkit pr-test-analyzer (tests and comment accuracy), using mutations in a scratch worktree. It found:
Spanholding non-ints, and the end coordinate checked alone, untested;All of these are fixed in
83923582, and each new row was checked to fail with its guard removed.The fix commit
83923582reviewed on its own (code-reviewer). It found two problems, both fixed inecfa86e6:_IGNORABLE[strip_emoji, strip_bidi]) still passes, so this was a widened check that had quietly narrowed. The mutation now fails.The rebase-and-retake commit
f13950eereviewed on its own (design-docs-reviewer, which reproduces the figures). It reproduced every figure on 3.11 to 3.15: the reference row, each commit's step on both the reference row and the link run, every per-name count, and the link pin. It found three problems, all fixed in1574d607:origin/master, which will read 0 once this merges, and now names688991e7;688991e7against the rebased branch: identical.🤖 Generated with Claude Code