diff --git a/docs/design/decisions/parse-cost.md b/docs/design/decisions/parse-cost.md index 5367d46ff..60359116b 100644 --- a/docs/design/decisions/parse-cost.md +++ b/docs/design/decisions/parse-cost.md @@ -6,8 +6,8 @@ The 2026-08-31 entry's numbers were measured that date, on py3.11 wherever no ot - The benchmark bounds the function calls a parse makes, not its seconds, as a band of ±2% around a row per interpreter (`_CALL_BASELINE` and `_BAND` in `tests/v2/test_benchmark.py`), so a drop trips it as a rise does, and an interpreter with no row fails with its own number; a 5s backstop over 1000 parses through `parse` stays for what frame counts cannot see, a backtracking regex and C-level work — 2026-08-31 #475; its seconds, and every scaling row's, are the parsing thread's CPU time, not wall time, and only the timed tests run in the `-n0 -m serial` pass — 2026-11-11 #1352. - Raising or lowering a row is a decision, appended here with the interpreter and the harness invocation, every number coming from `tools/perf/call_count.py` and quoted with its interpreter — 2026-08-31 #475. -- The rows stand at `parse` 285 on 3.11 and 277 on 3.12 to 3.15, as the last move left them — 2026-11-09 #1170; the `facade` rows and the facade's thousand-names timing row went with `HumanName`, and `call_count.py` measures `parse()` alone — 2026-10-28 #357. -- The 64-link pin, `_LINK_BASELINE`, stands at 2,418 on 3.11 and 2,398 on 3.12 to 3.15 under the same band — 2026-11-03 #1036. +- The rows stand at `parse` 197 on 3.11 and 188 on 3.12 to 3.15, as the last move left them — 2026-11-18 #1436; the `facade` rows and the facade's thousand-names timing row went with `HumanName`, and `call_count.py` measures `parse()` alone — 2026-10-28 #357. +- The 64-link pin, `_LINK_BASELINE`, stands at 1,716 on 3.11 and 1,706 on 3.12 to 3.15 under the same band — 2026-11-18 #1436. - No loop merges pieces through `merge_pieces` once per member of a run whose length grows with the input: each builds its lists in one pass, `join_connectives` held by the shift count of `_ShiftCountingList` and group's split-pair merge by `merge_pieces`' frame count, one row per arm — 2026-11-03 #1036. - `post_rules` reads the TITLE, GIVEN, MIDDLE and FAMILY positions in one walk of the tokens (`_positions`): once at its head, once before O3's fold and R2's marks, which share it, again after a rule that retags fires, and once in each of P10, W5's pair, O6 and O7 where the parse reaches it, where it had scanned once per role per reader — 2026-11-05 #1089, 2026-11-07 #1157. - The stages copy their state with `_state.copy_with`, a field copy, rather than `dataclasses.replace`, for `WorkToken`, `PendingAmbiguity` and `ParseState` only, and mypy sees it as `replace` imported under that name, keeping the dataclass plugin's checks at every call site — 2026-09-26 #546. @@ -41,11 +41,12 @@ Open: none in this file. - 2026-10-07 #614 (its rows superseded by 2026-10-08 #620 on 3.12-3.15 and by 2026-10-10 #624 on 3.11) -- every row moves DOWN again, on top of the closure inlining above: `parse` 317 → 308 on 3.11 and 296 → 290 on 3.12-3.15, `facade` 354 → 345 and 333 → 327, each measured on its own interpreter with `PYTHONSAFEPATH=1 PYTHONPATH= pythonX.Y /tools/perf/call_count.py` against adf6da88 (#619's merge), the import asserted. `--modules` on 3.11: `_pieces.py` 59 → 51, `_group.py` 26 → 25, `_assign.py` 9 → 10, every other module unchanged (`_vocab.py` read 37 until #614's /simplify inlined the one roman test the counting view asks per word, 2026-10-07). S2's trailing run is read once between P3's joins and the particle chain, so the chain's stop and its re-ask, the reserve's two views and assign's own reading collapse into one, and the counting view it reads through is one call on every segment that reaches the read, and its unit walk (`_counted_as_one`) runs only on a segment holding a particle past its first piece; the link pin reads 2,478 → 2,482, inside its band, and stays. A segment of fewer than three pieces never reaches the read and costs what it did (`John Smith` 152, `Smith, John` 161, `John Smith, PhD` 206 on 3.11, warm). decisions/S2, this date. - 2026-10-28 #357 (its restated `parse` rows superseded by 2026-11-05 #1089) — THE `facade` ROWS GO WITH `HumanName` (decisions/3-0-reevaluations' 2026-10-28 #357, REMOVAL entry): `_CALL_BASELINE` keeps one entry per interpreter, `parse` 307 on 3.11 and 288 on 3.12 to 3.15, unchanged, and loses `facade` 344 and 325, with `test_facade_cost_stays_within_its_band` and the facade's row of the thousand-names timing test; `tools/perf/call_count.py` measures `parse()` alone. No `parse` row moves: `uv run python tools/perf/call_count.py` reads 305 on 3.11 on the removal branch (2026-10-28), inside the ±2% band around 307. SUPERSEDES: the 2026-10-10 #625 entry's `facade` rows and the 2026-08-31 #475 entry's facade entry point. -- 2026-11-03 #1036 — A MERGE LOOP MAY NOT SHIFT THE PIECE LIST ONCE PER MEMBER, and the link pin moves DOWN by 64. `merge_pieces` extends its first piece in place but replaces the slice behind it, which moves every later piece, C-level work no frame guard counts; `join_connectives`' two loops and group's split-pair merge each called it once per member of a run, so a run of n members moved O(n²) elements. The clock rows `title_chain` and `link_run` had read it on CI's runners since 2026-10-31, in 14 failed runs from 2026-10-31T19:51Z to 2026-11-03T20:12Z (counted 2026-11-03T21:05Z: each run `gh run list --workflow python-package.yml --status failure --created '>=2026-10-25'` lists whose `gh run view --log-failed` names `test_benchmark.py::test_(parse|prefixed)_cost_grows_no_worse_than_linearly[title_chain|link_run]`), at 7.5-8.3x and 6.0-6.5x for 4x the input against `_MAX_RATIO` 6.0, ratios tight across 3.11-3.15 and across runs, while a dev machine read 4.1-4.4x on the same tree (py3.11, Apple silicon): the superlinear term is a memory move, cheap where memory is fast, so the clean column the bound was calibrated on included it. Measured on this machine, py3.11, best of five, `title_chain`'s ratio for 4x rose with size on the tree before the change, 4.39x at 1600 → 6400, 5.32x at 6400 → 25600 and 7.88x at 25600 → 102400 (5.2 s), and after it reads 4.07x and 3.97x (1.5 s). The two shapes' terms have different origins, measured the same way at 6400 → 25600 for `title_chain` and 3200 → 12800 for `link_run`: `title_chain`'s arrived with #617, which folded the comma part's title chain into group's shared join loop (83f4e914, its parent, 4.01x; ef98637f, its first commit, 5.75x), the trap root AGENTS.md's frame-budget bullet names for #617 in another form; `link_run`'s is older than #617 (83f4e914 already 4.90x), group's join having merged through `merge_pieces` per join since the in-place merge (92ec3eeb, 2026-07-21). Both stayed that size at 0c54662b (2026-10-07), 2bfa9533, 12384fae, 2cb763a3, 0a6fc3d9, b577e9a8, db7e8fc4 and 962301f7 (2026-11-03): 5.3-5.8x and 4.67-4.96x, while the tested bases read 3.6-4.6x on this machine throughout. The same run image read both a failure and a pass that day, so which runner a job lands on decides it, which the logs do not name. The three loops build fresh lists in one pass and assign them back once; readings are unchanged (the snapshot test, and two differential fuzzes of the old function against the new, 300,000 and 240,000 cases each with a negative control that a planted change reports, and 80,000 parses of split-pair shapes). The guard is deterministic now: `_ShiftCountingList` in `tests/v2/test_benchmark.py` counts the elements an edit shifts, and `join_connectives` shifts none, against a recorded control on 962301f7 of 4,802 → 79,202 (`Mr. and ` at 50 and 200 units), 9,702 → 158,802 (`i Und `) and 2,352 → 39,402 (`and `); group's split-pair merge, which builds its own pieces, is held by `merge_pieces`' frame count in each of its three arms (`Ph. D.`, a trailing `Pharm. D.` and one behind the given name after a family comma), 0 against 50 and 200 calls at 50 and 200 pairs. The pin: `_name_link_run(64)` reads 2,418 on 3.11 and 2,398 on 3.12-3.15, against 2,482 and 2,462 on 962301f7, each measured on its own interpreter, the recorded 2,478 and 2,459 having drifted inside the band; the 64 are `merge_pieces`' calls on that name, 63 from the run loop and 1 from the join. `_CALL_BASELINE` does not move on any interpreter. SUPERSEDES: the 2026-10-07 #619 entry's link pin. +- 2026-11-03 #1036 (its link pin superseded by 2026-11-18 #1436) — A MERGE LOOP MAY NOT SHIFT THE PIECE LIST ONCE PER MEMBER, and the link pin moves DOWN by 64. `merge_pieces` extends its first piece in place but replaces the slice behind it, which moves every later piece, C-level work no frame guard counts; `join_connectives`' two loops and group's split-pair merge each called it once per member of a run, so a run of n members moved O(n²) elements. The clock rows `title_chain` and `link_run` had read it on CI's runners since 2026-10-31, in 14 failed runs from 2026-10-31T19:51Z to 2026-11-03T20:12Z (counted 2026-11-03T21:05Z: each run `gh run list --workflow python-package.yml --status failure --created '>=2026-10-25'` lists whose `gh run view --log-failed` names `test_benchmark.py::test_(parse|prefixed)_cost_grows_no_worse_than_linearly[title_chain|link_run]`), at 7.5-8.3x and 6.0-6.5x for 4x the input against `_MAX_RATIO` 6.0, ratios tight across 3.11-3.15 and across runs, while a dev machine read 4.1-4.4x on the same tree (py3.11, Apple silicon): the superlinear term is a memory move, cheap where memory is fast, so the clean column the bound was calibrated on included it. Measured on this machine, py3.11, best of five, `title_chain`'s ratio for 4x rose with size on the tree before the change, 4.39x at 1600 → 6400, 5.32x at 6400 → 25600 and 7.88x at 25600 → 102400 (5.2 s), and after it reads 4.07x and 3.97x (1.5 s). The two shapes' terms have different origins, measured the same way at 6400 → 25600 for `title_chain` and 3200 → 12800 for `link_run`: `title_chain`'s arrived with #617, which folded the comma part's title chain into group's shared join loop (83f4e914, its parent, 4.01x; ef98637f, its first commit, 5.75x), the trap root AGENTS.md's frame-budget bullet names for #617 in another form; `link_run`'s is older than #617 (83f4e914 already 4.90x), group's join having merged through `merge_pieces` per join since the in-place merge (92ec3eeb, 2026-07-21). Both stayed that size at 0c54662b (2026-10-07), 2bfa9533, 12384fae, 2cb763a3, 0a6fc3d9, b577e9a8, db7e8fc4 and 962301f7 (2026-11-03): 5.3-5.8x and 4.67-4.96x, while the tested bases read 3.6-4.6x on this machine throughout. The same run image read both a failure and a pass that day, so which runner a job lands on decides it, which the logs do not name. The three loops build fresh lists in one pass and assign them back once; readings are unchanged (the snapshot test, and two differential fuzzes of the old function against the new, 300,000 and 240,000 cases each with a negative control that a planted change reports, and 80,000 parses of split-pair shapes). The guard is deterministic now: `_ShiftCountingList` in `tests/v2/test_benchmark.py` counts the elements an edit shifts, and `join_connectives` shifts none, against a recorded control on 962301f7 of 4,802 → 79,202 (`Mr. and ` at 50 and 200 units), 9,702 → 158,802 (`i Und `) and 2,352 → 39,402 (`and `); group's split-pair merge, which builds its own pieces, is held by `merge_pieces`' frame count in each of its three arms (`Ph. D.`, a trailing `Pharm. D.` and one behind the given name after a family comma), 0 against 50 and 200 calls at 50 and 200 pairs. The pin: `_name_link_run(64)` reads 2,418 on 3.11 and 2,398 on 3.12-3.15, against 2,482 and 2,462 on 962301f7, each measured on its own interpreter, the recorded 2,478 and 2,459 having drifted inside the band; the 64 are `merge_pieces`' calls on that name, 63 from the run loop and 1 from the join. `_CALL_BASELINE` does not move on any interpreter. SUPERSEDES: the 2026-10-07 #619 entry's link pin. - 2026-11-05 #1085 — NO ROW MOVES; the three simplify passes of the 3.0 cycle (#1084, #1083, #1085, merged in that order) take the reference row DOWN by 5.92 on every interpreter. `uv run python tools/perf/call_count.py --against 5138879b`, the tree before them, reads 312.72 → 306.80 on 3.11, and with `uv run --python 3.1x` 293.72 → 287.80 on each of 3.12 to 3.15; `--against 1b2c3d4c`, #1085's parent, reads 311.72 → 306.80 on 3.11, so 4.92 of it is #1085's and 1.00 the other two's (measured 2026-11-05 on 401bf62e). Both readings sit inside the ±2% band around `parse` 307 and 288, so `_CALL_BASELINE` does not move, but 293.72 had stood 0.04 below the 3.12-3.15 band's upper edge, 293.76, which the passes moved away from. - 2026-11-05 #1089 (its rows superseded by 2026-11-09 #1170) — EVERY ROW MOVES DOWN, by 18 on 3.11 and 8 on 3.12 to 3.15: `parse` 307 → 289 and 288 → 280. `post_rules` scanned the tokens once per role it read, through `_idx`, a call around a list comprehension, two frames each on 3.11 and one on 3.12 and later (PEP 709 inlined the comprehension); every parse made ten such scans, four at the head, three at P6's site and three at R2's marks. `_positions` builds the TITLE, GIVEN, MIDDLE and FAMILY lists in one plain loop, one frame, and the stage asks it twice a parse, at its head and at R2's marks; a rule that retags walks again where it fires, O1's and O2's rotations now among them, which is what lets P6's site read the lists it is handed instead of scanning again for the rotations' sake (the stale-list shape #359 fixed: O1 and O2 were the one retagging pair ahead of P6's site that had not recomputed; P10, O6 and O7 retag after it without updating the lists, and the readers after them, O3's fold and R2's marks, walk again). The rare readers (O6, O7, P10 and O3's fold) take the same walk, so `_idx` is gone and one function answers where a role's tokens are. Separately, rules/C3's numeral clause in `_comma._whole_name` asks `ROMAN` of the word ending the part's name words at C level ahead of `ends_in_a_generation`, the match being the first test of `reads_as_generation`, which that predicate calls, as group's count and `name_word_count` already ask it, so a suffix-comma name of two or more words before the comma whose last name word is not numeral-shaped pays 2 frames less on every interpreter (`ends_in_a_generation` and the `reads_as_generation` it calls). Measured 2026-11-05 on 21a4fe15 (#1100's merge) and on the change, each interpreter on its own: `uv run python tools/perf/call_count.py --against 21a4fe15` reads 306.80 → 288.80 on 3.11, and with `UV_PROJECT_ENVIRONMENT= uv run --python 3.1x` 287.80 → 279.80 on each of 3.12 to 3.15; `--modules` on 3.11 puts all 18 in `_pipeline/_post_rules.py` (29 → 11), every other module unchanged on the reference row, which holds no comma. Per name, one warm parse counted with a `sys.setprofile` hook (`_frames_for` in `tests/v2/test_benchmark.py`), 3.11 then 3.12 to 3.15: `Smith, John` 159 → 141 and 137 → 129, `John Smith Jr., PhD` 264 → 244 and 239 → 229, `John Smith, PhD` 206 → 186 and 183 → 173, `John Smith` 149 → 131 and 132 → 124. BEHAVIOR IDENTICAL: `snapshot.py --record` and `cldr_agreement.py --record` (with the `ja` extra too) rewrite no committed record, and the two trees read alike under a policy sweep. Its population is every distinct text of the committed `tools/differential/corpus*.jsonl` files (the `name` key, or the line's string; 6,757 on 21a4fe15) plus eight shapes the rotations, P6 and the folds reach that no corpus holds (`Ivan Petrovich Sidorov`, `Aliyev Ilham Heydar oglu`, `Ilham Heydar oglu Aliyev`, `van Berg Jan de`, `Dr. John`, `Dean of Chemistry née Jones`, `Mary Jane Smith Jones`, `Ivan Ivanovich Petrovich`), each parsed by `Parser(policy=Policy(name_order=..., middle_as_family=..., two_family_names=..., two_given_names=..., patronymic_rules=...))` under the three orders, each switch off and on, and the rules empty or both of East Slavic and Turkic (48 policies), the comparator a line per parse holding the `repr` of the text, the policy's values (`patronymic_rules` as a sorted list, a frozenset's order varying with the process's string hash), `tokens` and `ambiguities`, the two trees' files paired line by line: 0 of 324,720 lines differ. The negative control, which inverts the `ROMAN` gate and narrows R2's walk to the GIVEN and MIDDLE parts (`_positions(tokens)[1:3]`), reads 1,754. `test_a_part_ending_in_no_numeral_skips_the_generation_test` holds the gate, with its recorded control (246 frames against 244 on `John Smith Jr., PhD`, py3.11), `test_ends_in_a_generation_needs_the_roman_shape` holds that the match is a necessary half of the predicate, and `test_post_rules_walks_the_roles_once_per_reader` counts the walks and the calls `post_rules` makes directly, and `test_o1_and_o2_read_the_lists_after_their_own_rotation` witnesses the moved recompute (with it dropped, `Sidorov Ivan Petrovich` under RU, `petrovich` listed as a particle, reads middle 'Petrovich' where both trees read family 'Petrovich Sidorov'; the stale-list fix #467 made at P6's site had stood unwitnessed, decisions/P6). By the four frame questions: the scans ran on every parse (question 2) and moved the band (question 3), and the walk replaces the scans rather than standing beside them, so no second answer to where a role's tokens are is left to drift (question 4). No scaling row is added: the walk is linear in the tokens, as each scan was, and runs a fixed number of times a parse. SUPERSEDES: the 2026-10-10 #625 entry's rows, and the 2026-10-28 #357 entry's restated `parse` rows. - 2026-11-07 #1157 — NO ROW MOVES; release checklist step 0's pass over post_rules and assign. O3's fold and R2's marks now share one `_positions` walk, which the fold updates, so a `middle_as_family` parse walks once fewer; P10's and W5's tag gates are one plain walk; a lone name word's field is decided where it is placed, so placing one piece makes no `_name_positions` call; S1's settling credential is asked only of a clause holding an unsettled member; N3's early nickname scan sits behind `state.extracted`. Measured 2026-11-07, py3.11, one warm parse counted with a `sys.setprofile` hook, before and after (dbc2fc22 and the change, and again on b43ad933, #1154's merge, and on c489c9a7, #1162's merge, after #1158's, each against the change rebased onto it, every count the same): the reference name 287 either way; `Smith` 128 → 125, `John (Ret.) Smith` 216 → 213, `John Smith (PhD, MA)` 253 → 250, `王李小明` 155 → 152; `John Smith` 130 and `Smith, John` 140 unchanged; no name rose under the default, `FAMILY_FIRST`, `middle_as_family`, `lone_name_word=FAMILY` or `two_family_names` policies. BEHAVIOR IDENTICAL: `snapshot.jsonl` byte-identical, and, for the switches the snapshot's default policy cannot see, dbc2fc22 and the change (2aeb6b95, before the rebase) read alike under a policy sweep, each tree loaded by path with `PYTHONSAFEPATH=1` and its `__file__` checked: every distinct text of the committed `tools/differential/corpus*.jsonl` files (the `name` key, or the line's string) plus 14 shapes these paths reach (`Juan Smith`, `Smith`, `de`, `María del Carmen García`, `王李小明`, `Smith, John`, `John (Ret.) Smith`, `John Smith (PhD, MA)`, `John Smith (MA)`, `John Smith (MA, Jr.)`, `(Bob)`, `Smith (Bob)`, `Ivan Petrovich Sidorov`, `García López, Juan`), 6,906 texts, each parsed by `Parser(policy=Policy(name_order=..., middle_as_family=..., lone_name_word=..., two_family_names=...))` under the three orders, each switch off and on and the lone field given or family (24 policies, every other field at its default), the comparator a line per parse holding the `repr` of the text, the policy's four values, `tokens` and `ambiguities`, the two files compared with `cmp`: 0 of 165,744 lines differ. The negative control, dbc2fc22 with `_assign._name_positions`' one-piece branch returning `[first]` where it returned `[lone]`, reads 3,157 (measured 2026-11-07 in #1157's review). `test_post_rules_walks_the_roles_once_per_reader` pins the `middle_as_family` row at 2 walks, against 3 before. -- 2026-11-09 #1170 — EVERY ROW MOVES DOWN, by 4 on 3.11 and 3 on 3.12 to 3.15: `parse` 289 → 285 and 280 → 277, the rows re-taken where the tree measures rather than by the change alone, master having drifted to 291.80 and 282.80 inside the band since #1089. Three `post_rules` helpers that every parse calls stop making calls they need not make, the three cleanups #1157's pass declined only because the code was unchanged since its last pass: `_leading_name_piece` tests a piece's tokens in a plain inner loop where an `any()` generator cost a frame per resume; `post_rules` asks `_rotations_apply` only where `patronymic_rules` is set, the rotations' only readers testing a rule's membership; and `_mark_suffix_entries` collects the suffix positions in a plain loop and returns before `delimiter_cores` on a name holding fewer than two, where `zip` over them never ran. Measured 2026-11-09 on 9f6264e4 (#1295's merge) and on the change, each interpreter on its own (`uv run python tools/perf/call_count.py`, and with `UV_PROJECT_ENVIRONMENT= uv run --python 3.1x` for 3.12 to 3.15): 291.80 → 284.80 on 3.11 and 282.80 → 276.80 on each of 3.12 to 3.15. Per name, one warm parse counted with `_frames_for` in `tests/v2/test_benchmark.py`, py3.11, each name in a process of its own: `John Smith` 131 → 126, `Smith, John` 141 → 136, `John Smith Jr., PhD` 264 → 260, `Dr. John Smith Jr.` 241 → 234. A name's absolute count depends on what the process parsed before it, `_frames_for`'s one warm-up parse leaving some first-use work to the name, so the same tree read `Dr. John Smith Jr.` 224, 214 and 234 by the order its names ran in (two reviews of #1170); compare counts taken the same way, and read the difference, which held at 7 for that name in every order. BEHAVIOR IDENTICAL by construction, each change returning where the old code's result was already decided (a rotation needs its rule in `patronymic_rules`; `zip` over fewer than two suffixes is empty; the inner loop is the `any()` it replaces), and `snapshot.jsonl` byte-identical. `test_post_rules_helpers_return_before_their_frames` holds the two calls the band cannot see put back, with its recorded control (9f6264e4: 2 generator frames on `John Smith`, 1 `delimiter_cores` call), and `test_post_rules_walks_the_roles_once_per_reader`'s pin on the calls `post_rules` makes directly moves 7 → 6. By the four frame questions: every parse pays them (question 2) and they move the band (question 3); none is a copy of a predicate (question 4). Declined again, as #1157 declined them: a `stale` flag on the position lists and dropping the recompute after P6, each bringing back the stale-list shape #359 fixed. SUPERSEDES: the 2026-11-05 #1089 entry's rows. +- 2026-11-09 #1170 (its rows superseded by 2026-11-18 #1436) — EVERY ROW MOVES DOWN, by 4 on 3.11 and 3 on 3.12 to 3.15: `parse` 289 → 285 and 280 → 277, the rows re-taken where the tree measures rather than by the change alone, master having drifted to 291.80 and 282.80 inside the band since #1089. Three `post_rules` helpers that every parse calls stop making calls they need not make, the three cleanups #1157's pass declined only because the code was unchanged since its last pass: `_leading_name_piece` tests a piece's tokens in a plain inner loop where an `any()` generator cost a frame per resume; `post_rules` asks `_rotations_apply` only where `patronymic_rules` is set, the rotations' only readers testing a rule's membership; and `_mark_suffix_entries` collects the suffix positions in a plain loop and returns before `delimiter_cores` on a name holding fewer than two, where `zip` over them never ran. Measured 2026-11-09 on 9f6264e4 (#1295's merge) and on the change, each interpreter on its own (`uv run python tools/perf/call_count.py`, and with `UV_PROJECT_ENVIRONMENT= uv run --python 3.1x` for 3.12 to 3.15): 291.80 → 284.80 on 3.11 and 282.80 → 276.80 on each of 3.12 to 3.15. Per name, one warm parse counted with `_frames_for` in `tests/v2/test_benchmark.py`, py3.11, each name in a process of its own: `John Smith` 131 → 126, `Smith, John` 141 → 136, `John Smith Jr., PhD` 264 → 260, `Dr. John Smith Jr.` 241 → 234. A name's absolute count depends on what the process parsed before it, `_frames_for`'s one warm-up parse leaving some first-use work to the name, so the same tree read `Dr. John Smith Jr.` 224, 214 and 234 by the order its names ran in (two reviews of #1170); compare counts taken the same way, and read the difference, which held at 7 for that name in every order. BEHAVIOR IDENTICAL by construction, each change returning where the old code's result was already decided (a rotation needs its rule in `patronymic_rules`; `zip` over fewer than two suffixes is empty; the inner loop is the `any()` it replaces), and `snapshot.jsonl` byte-identical. `test_post_rules_helpers_return_before_their_frames` holds the two calls the band cannot see put back, with its recorded control (9f6264e4: 2 generator frames on `John Smith`, 1 `delimiter_cores` call), and `test_post_rules_walks_the_roles_once_per_reader`'s pin on the calls `post_rules` makes directly moves 7 → 6. By the four frame questions: every parse pays them (question 2) and they move the band (question 3); none is a copy of a predicate (question 4). Declined again, as #1157 declined them: a `stale` flag on the position lists and dropping the recompute after P6, each bringing back the stale-list shape #359 fixed. SUPERSEDES: the 2026-11-05 #1089 entry's rows. - 2026-11-11 #1352 — THE TIMED ROWS READ CPU TIME, AND THE COUNTS RUN IN PARALLEL; NO ROW, BOUND OR BASELINE MOVES. On a laptop running several agent sessions the wall-clock rows failed at random and passed alone. Measured on py3.11, 12 cores (6 performance, 6 efficiency), five `uv run pytest -n0 -m serial tests/v2/test_benchmark.py` runs beside 24 shell busy loops (load average 37-48): 20, 23, 21, 19 and 23 of the module's 62 tests failed on 970b54e5, at up to 10.0x against `_MAX_RATIO` 6.0; on the change, 0 of 34 in each of five runs at 24 busy loops (load 50-62) and at none, and in five at 12, which barely loaded the machine (the before tree failed none there either). Per row, three runs at 24 busy loops (load 18-75): `time.perf_counter()` read 27 of the 33 ratio rows at 6.0 or over at least once, up to 14.0x; `time.thread_time()`, measured back to back, none over 5.76x (as high as 5.76x with no busy loops too). CPU time is not load-immune -- interleaved best-of-7 read a clean row up to 5.6x, the thread charged for a slower core or a colder cache, and a user-interactive QoS hint did not help -- and noise can DEFLATE a ratio as well as inflate it: with more rounds taken only at or over the bound, a planted quadratic reading 6.7x quiet (`given_part_titles`) read 5.93x once at 24 busy loops and passed, and a loop stopping at its first reading under the bound could stop on a dip (both found by #1352's review). So `_ratio` interleaves the two inputs and, where its first 7 rounds read at or over 80% of the bound (`_NEAR`; `bound * _NEAR` is 4.800000000000001 in floating point), takes all 35 and judges the last minimums, each round only moving the two minimums toward the true costs; `test_ratio_takes_more_rounds_only_near_the_bound` holds that on a scripted clock, and fails against both earlier loops. Loaded, the serial pass takes 78-115s against 47s, the escalation's price; quiet it takes 18s, as before. The defects it guards are seen as before. Quiet, the old wall-clock `_best` and `_ratio` fail the same rows in every run under a planted C-level quadratic in the word count, every row within 0.35x. The recorded controls read on `_ratio`, quiet then at 24 busy loops: `link_run` 13.9-14.1 and 14.9-16.4 on b9ed1429; `given_part_run` 8.0 and 10.0-10.3 and `given_part_titles` 10.1 and 11.1-12.5 on 003b4962; `title_chain` 12.0 and 13.5-14.8, `pair_run` 10.0 and 10.2-12.8, `caps_pairs` 10.1-10.2 and 12.2-12.8 and `delimiter_cores` 10.4 and 11.8-13.1 on 9dd271b2; `connective_between_particles` 8.5 and 8.9-10.2 on 5752df69 (#939 before its review); and `slash_chain` 8.8-8.9 and 8.9-9.2 with #1103's per-step copy planted. That is against 3.97-4.19 quiet and up to 5.06 at 24 busy loops on 970b54e5. The weakest planted quadratic, 6.3-7.0x quiet on eleven rows, read 7.18x or more on every row in five runs at 24 busy loops. A planted backtracking regex reads 7.31 s of CPU over the thousand names, against 7.36 s wall. The 5s bound is CPU seconds now and no looser for work on the parse's own thread, CPU time being at most the wall time of a single-threaded parse; it no longer sees time spent blocked or on another thread, which a parse never spends. 962301f7's memory move reads 4.07-4.43 on the CPU clock and 4.07-4.50 on the wall clock, quiet, as the 2026-11-03 #1036 entry found for the wall clock; `_ShiftCountingList` holds it. The 28 untimed tests (the frame, call and shift counts and the table checks) lose the `serial` mark. Each passes alone in a fresh process, after the 21,065 tests of `test_cases.py` and `test_match.py` in one, in three full parallel runs, and under `--cov` on 3.11 and 3.14. The order dependence of an absolute count (the 2026-11-09 #1170 entry) stays inside a ratio, a band, or an exact count of one named function's calls (`only=`/`caller=`, `== 0` up to `== 6`), none of which counts an `lru_cache`'d function. - 2026-11-18 #1435 — NO ROW MOVES; A RUN OF INITIALS BEHIND A FULL NAME ASKS NO COUNT. #1238's C16 block in `_comma.decide` put a dotted run spelling a listed credential, every letter a capital or the name in one case, into `maybe`, so `John Smith, M.D.` paid C4's `name_word_count` and the C11 books, which only a one-word head reads the run by. Where the run is the part's only entry in `maybe` (no word of the class and no caps-shape candidate beside it), `decide` now asks `_lone_word` first (once, its answer kept for the reading), and behind a head that is not one word skips the count and the books: the run stays in `maybe`, so the part is still read as the count's part and never takes the listing form's early return, and with nothing else in `maybe` the count licenses nothing and `head_named` meets no member. Frames, `test_benchmark._frames_for` for one parse in a fresh process, master (b78d5af2) then branch: `John Smith, M.D.` 221 → 199 on 3.11 and 203 → 185 on 3.15; `John Smith, MD` 190 and 178 and `Smith, John` 136 and 125 unchanged; `Smith, J.P.`, which still counts, 233 and 218 unchanged. The issue's table (209 for `John Smith, M.D.` on master) was taken in a process that had parsed other names first, which moves the absolute count and not the difference (the 2026-11-09 #1170 entry). The issue's proposed gate (append the run only beside another class word or behind a one-word head) was not taken: it sends such a part down the no-`maybe` path, equal only where the listing early return and C1's fallback agree, which nothing holds by construction. Checked against master: a grid of 215,280 parses (65 heads, from empty, nickname, title, particle and title-particle heads alone and in pairs to full and one-case names, × 46 tails of dotted runs, listed and ambiguous credentials, paired initials and their company × three comma spellings and a later part × the three orders × six policies) byte-identical, the issue's gate also identical there, and the harness shown to see a move (on the same grid, a `_lone_word` always true moves 34,848 rows, and skipping the count behind a one-word head too moves 4,536, `Smith van, M.D.` reading given 'M.', middle 'D.', family 'Smith van' where master reads given 'Smith', family 'van', suffix 'M.D.', `_lone_word` reading that head as one word while the count finds two); `snapshot.jsonl` byte-identical; and `snapshot.py --local ctgov-officials` and `ctgov-contacts` against master, no move. `test_a_run_of_initials_behind_a_full_name_takes_no_count` holds the saving, its control recorded in its docstring, and `test_the_count_c16_skips_decides_no_reading` (tests/v2/pipeline/test_comma.py) the readings a wrong gate moves. Under each of three mutants (the gate removed; the count skipped behind a one-word head too; `run_alone` forced true, so a run beside another entry of `maybe` such as `John Smith, M.D. M.A` skips the count as well) the frame test fails, and under the last two the reading test does too. A run beside `MA` (`John Smith, M.D. MA`) never joins `maybe`, `MA` being the run's company (C16), so it reaches none of this. +- 2026-11-18 #1436 — EVERY ROW AND THE LINK PIN MOVE DOWN: `parse` 285 → 197 on 3.11 and 277 → 188 on 3.12 to 3.15, `_LINK_BASELINE` 2,418 → 1,716 and 2,398 → 1,706. Measured 2026-11-18 on 688991e7 (#1458's merge, the tree before the change) and on the change rebased onto it, each interpreter on its own (`uv run python tools/perf/call_count.py`, `--against 688991e7` on 3.11, and with `UV_PROJECT_ENVIRONMENT= uv run --python 3.1x` for 3.12 to 3.15): 285.72 → 197.72 on 3.11 and 277.72 → 188.72 on each of 3.12 to 3.15, a fall of 31% and 32%; the link run `_name_link_run(64)`, `_frames_for` in `tests/v2/test_benchmark.py`, 2,404 → 1,716 on 3.11 and 2,395 → 1,706 on 3.12 to 3.15. #1434's simplify pass had measured scratch patches at about 234 on 3.11 and left them for a deliberate move. The changes, each a commit of its own, py3.11 reference row: tokenize scans a region for its separators with a compiled class per setting of the two strip switches rather than calling `_ignorable` on every character (−27), the ignorable set now defined by those classes, which `_ignorable` and `_stripped` match, and held to its parts over every code point by `test_the_separator_classes_hold_their_parts_over_every_code_point`; tokenize's sort key an `attrgetter` (−6); classify's tokens and group's frozen pieces built without a generator (−6, −10); group's tail start read by identity rather than from a dict keyed by the enum, whose `__hash__` is Python (−3); and `Token.__post_init__` paying nothing on the parse's own tokens, its span checked without a generator and an exact `Span` kept, its frozenset tags spared the str and `Mapping` refusals, its role coerced only when it is not a `Role` (−24, −6, −6), the checks and their messages unchanged and each refusal held by a row of `test_token_rejects_a_span_of_anything_but_two_ints` and the tests beside it. On the 64-link run, which holds 67 tokens, the same commits in order read −143, −67, −67, −6, −3 and −268, −67, −67 (`git log 688991e7..` checked out commit by commit): a cost per token or per character is a cost per link there. `_is_emoji` looped over its ranges in an earlier commit (no move on the reference row, 336 → 271 on a Devanagari name) and is gone, its ranges read through the class. Per name, `_frames_for` for one warm parse, each in a fresh process, 688991e7 then the change, 3.11 and 3.15: `John Smith` 126 → 91 and 120 → 84, `Smith, John` 136 → 99 and 125 → 87, `John Smith, M.D.` 199 → 149 and 185 → 134, `Dr. John Smith Jr.` 233 → 170 and 225 → 161, `John Smith Jr., PhD` 258 → 195 and 244 → 180, `राम बहादुर के.सी.` 336 → 193 and 326 → 182. Time, which the frames cannot see: 1000 parses of the reference name, minimum of 15, the two trees run in turn three times (py3.11, b78d5af2 and the change before the rebase), 116.9-117.4 ms → 109.2-109.9 ms of CPU, and per 1000 names, minimum of 9, `Smith, John` 64.7 → 61.0 ms, the long name `Sir John Ronald Reuel Tolkien00 van der Berg Jr., PhD` 176.2 → 165.1 ms, the Devanagari name 105.9 → 92.9 ms, `高橋一郎` 61.8 → 57.9 ms and forty double-spaced words 358.4 → 320.3 ms: no frame was traded for slower C work. BEHAVIOR IDENTICAL: `snapshot.jsonl` byte-identical; `tools/differential/measure.py --base` against b78d5af2 before the rebase, 29 steps with no move; and, across two trees each asserting its `nameparser.__file__`, b78d5af2 against the change before the rebase and again 688991e7 against the change rebased onto it, a grid of 114,647 readings (NFC and NFD spellings, ZWJ and ZWNJ inside Devanagari words, an emoji ZWJ sequence, a variation selector, bidi marks and isolates, ZWSP, the interpunct, the fullwidth stop and the nakaguro between Han names, U+202F in Mongolian, in twelve comma and clause shapes under the three orders and both strip switches and `locales.ZH`, the comparator a line holding the repr, the ambiguities, `initials()`, both case views and every token's text, span, role and tags) byte-identical, the harness shown to see a move (U+200B dropped from the invisible separators moves 4,303 lines), and a differential fuzz of the old `_tokenize_region` against the new over 300,000 random texts with random region bounds, 0 differences (#1459's code review). By the four frame questions: tokenize's call and `Token`'s checks multiply with the input (question 1), every parse pays all of them (question 2), and they move the band (question 3); none inlines a predicate save the role test, `_coerce_enum`'s own first line, which returns its argument and cannot drift (question 4). The long link run keeps its margin: a per-link regression is +64 against a 2% band of 34.3. SUPERSEDES: the 2026-11-09 #1170 entry's rows, and the 2026-11-03 #1036 entry's link pin. diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index c705c2f91..ba4f556f1 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -517,7 +517,9 @@ def classify(state: ParseState) -> ParseState: and (bare := t.text.strip(COMMA_STRIP)) and bare != t.text): readers[i] = copy_with(t, text=bare) folded[i] = _normalize(bare) - tokens = tuple( + # a list, not a generator: tuple() over a generator cost a frame + # per token for each resume + tokens = tuple([ copy_with( t, tags=_tags_for(readers[i], folded[i], state, marker_tags.get(i), one_case_own=one_case and i < clause_at @@ -525,7 +527,7 @@ def classify(state: ParseState) -> ParseState: letter_one_case=whole_one_case or ( one_case and i < clause_at and t.role is None))) - for i, t in enumerate(state.tokens)) + for i, t in enumerate(state.tokens)]) # A word with a delimiter glued to it joins no neighbour across the # delimiter, so the stem's joining tags stay off it: 'y;' is no # connective to the word after it (rules/C10: "no join, maiden diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index be7ffd09a..b7cbbba58 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -1539,9 +1539,12 @@ def group(state: ParseState) -> ParseState: # tokens, which land in `dropped` as the maiden marker does; the # built-in cores, which no v1 release read, cut them too (#889) cores = delimiter_cores(state.policy.extra_suffix_delimiters) - tail_start = {Structure.SUFFIX_COMMA: 1, - Structure.FAMILY_COMMA: 2}.get(state.structure) + # identity tests, not a dict keyed by the enum: Enum.__hash__ is a + # Python method, a frame per key on every parse family_comma = state.structure is Structure.FAMILY_COMMA + tail_start = (2 if family_comma + else 1 if state.structure is Structure.SUFFIX_COMMA + else None) for seg_idx, seg in enumerate(state.segments): if family_comma: bound_join = (BoundJoin.LENIENT if seg_idx == 1 @@ -1663,8 +1666,10 @@ def group(state: ParseState) -> ParseState: for i in piece[1:]: tokens[i] = copy_with( tokens[i], tags=tokens[i].tags | {"joined"}) - all_pieces.append(tuple(tuple(p) for p in pieces)) - all_ptags.append(tuple(frozenset(t) for t in ptags)) + # map over the type, not a generator: the conversion runs in C, + # where a generator cost a frame per piece + all_pieces.append(tuple(map(tuple, pieces))) + all_ptags.append(tuple(map(frozenset, ptags))) all_reads.append(seg_reads[0] if len(seg_reads) == 1 else None) # rules/M1: "a leading recognized marker is dropped where the # clause holds a word past it" and "a clause of nothing but its diff --git a/nameparser/_pipeline/_tokenize.py b/nameparser/_pipeline/_tokenize.py index 83cddc6b5..96316742b 100644 --- a/nameparser/_pipeline/_tokenize.py +++ b/nameparser/_pipeline/_tokenize.py @@ -19,12 +19,16 @@ original exactly as given (v1 contrast: decisions/T1). Beside the stage function, `_ignorable` is shared: extract asks it -where a listed particle's word ends (rules/N2, #1241), so the two -stages cannot disagree about which characters separate a word. +where a listed particle's word ends (rules/N2, #1241). It matches the +ignorable class tokenize's own scan is built from, so the two stages +agree on every character that separates in every context; the ones +context decides (a transcription dot between classified characters, +U+202F between Mongolian letters) are tokenize's alone. """ from __future__ import annotations import bisect +import operator import re from nameparser._pipeline._state import ( @@ -35,12 +39,16 @@ # Ported from v1's "emoji" and "bidi" patterns (the 1.x and 2.x # nameparser.config.regexes, removed in 3.0, #1008); this is now the -# only copy. Integer ranges, not a regex character class: -# the per-char test needs no regex, and CodeQL's py/overly-large-range -# false-positives on literal astral ranges (surrogate decomposition). +# only copy. Written as integer ranges, never as a literal regex class: +# CodeQL's py/overly-large-range false-positives on literal astral +# ranges (surrogate decomposition), so the class below is built from +# the integers. _EMOJI_RANGES = ((0x1F300, 0x1F64F), (0x1F680, 0x1F6FF), (0x2600, 0x26FF), (0x2700, 0x27BF)) -_BIDI = re.compile('[\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069]+') +_BIDI_CHARS = '\u061C\u200E\u200F\u202A-\u202E\u2066-\u2069' +_BIDI = re.compile(f'[{_BIDI_CHARS}]+') +# The same ranges as a character class, for the classes below +_EMOJI_CHARS = ''.join(f'{chr(lo)}-{chr(hi)}' for lo, hi in _EMOJI_RANGES) # rules/T2: "the katakana middle dot, its halfwidth twin and the # Ethiopic wordspace divide a name like whitespace, always." They @@ -84,6 +92,42 @@ # _ignorable. +def _chars(chars: frozenset[str]) -> str: + """A set of single characters as the body of a regex class.""" + return ''.join(re.escape(ch) for ch in sorted(chars)) + + +# rules/T1: "an ignorable character separates its neighbors and +# never joins them" (v1 contrast: decisions/T1). The ignorable set as +# regex classes, one per setting of the two strip switches, so a region +# is scanned for its separators in C rather than asked character by +# character. These classes ARE the definition: _ignorable and _stripped +# match against them, and the boundary pattern _tokenize_region scans +# with is the ignorable class plus the characters that separate only by +# context (the commas, which record an offset, and the transcription +# dots, whose flanks decide). One ignorable character is context too: +# U+202F, whitespace, which rules/T6 has join two Mongolian letters, so +# the scan's three branches (comma, U+202F, dot) are the whole of +# context. `\s` matches exactly the characters str.isspace() accepts +# (checked over every code point). +_ALWAYS = r'\s' + _chars(_NAME_DOT_SEPARATORS | _INVISIBLE_SEPARATORS) +_SWITCHES = ((False, False), (False, True), (True, False), (True, True)) + + +def _strip_class(strip_bidi: bool, strip_emoji: bool) -> str: + return ((_BIDI_CHARS if strip_bidi else '') + + (_EMOJI_CHARS if strip_emoji else '')) + + +_STRIPPED = {key: re.compile(f'[{_strip_class(*key)}]') if any(key) else None + for key in _SWITCHES} +_IGNORABLE = {key: re.compile(f'[{_ALWAYS}{_strip_class(*key)}]') + for key in _SWITCHES} +_BOUNDARY = {key: re.compile( + f'[{_ALWAYS}{_strip_class(*key)}' + f'{_chars(COMMA_CHARS | _TRANSCRIPTION_DOTS)}]') for key in _SWITCHES} + + def _mongolian_letter(ch: str) -> bool: """A character of the Mongolian block that is not its punctuation (U+1800-U+180A) or a digit (U+1810-U+1819): its letters, free @@ -94,24 +138,17 @@ def _mongolian_letter(ch: str) -> bool: and not 0x1800 <= cp <= 0x180A and not 0x1810 <= cp <= 0x1819) -def _is_emoji(ch: str) -> bool: - cp = ord(ch) - return any(lo <= cp <= hi for lo, hi in _EMOJI_RANGES) - - -# rules/T1: "an ignorable character separates its neighbors and -# never joins them" (v1 contrast: decisions/T1) def _stripped(ch: str, policy: Policy) -> bool: - """True when the strip policy removes `ch` from the token stream. - The ONE definition of that set, shared by _ignorable and _flank: a - character the strip policy removes must not occupy a flank - position either, so a new strip class added here stays transparent - to the interpunct guard by construction. The separators (whitespace, - the name dots and _INVISIBLE_SEPARATORS) are not in this set and do - occupy one.""" - if policy.strip_bidi and _BIDI.match(ch): - return True - return bool(policy.strip_emoji and _is_emoji(ch)) + """True when the strip policy removes `ch` from the token stream: + the reader _flank asks of the set _strip_class defines, which also + feeds _IGNORABLE and _BOUNDARY. A character the strip policy removes + must not occupy a flank position either, so a new strip class added + to _strip_class stays transparent to the interpunct guard by + construction. The separators + (whitespace, the name dots and _INVISIBLE_SEPARATORS) are not in + this set and do occupy one.""" + pattern = _STRIPPED[policy.strip_bidi, policy.strip_emoji] + return pattern is not None and pattern.match(ch) is not None def _flank(text: str, indices: range, policy: Policy) -> str | None: @@ -129,24 +166,15 @@ def _flank(text: str, indices: range, policy: Policy) -> str | None: def _ignorable(ch: str, state: ParseState) -> bool: - if ch.isspace(): - return True - if ch.isascii(): - # both strip classes and the unconditional separators are - # entirely non-ASCII (bidi >= U+061C, emoji >= U+2600, the - # invisibles >= U+200B, dividers >= U+1361): skip the checks below - # for every ASCII letter - return False - # unconditional, like whitespace -- not policy-gated, so this sits - # ahead of the policy check rather than in _tokenize_region beside - # COMMA_CHARS (commas RECORD an offset; these must not -- only the - # 间隔号 marks a transcription and records, #298, see - # _NAME_DOT_SEPARATORS above). The only load-bearing constraint is - # "after the isascii fast path" (every member of both sets is - # non-ASCII). - if ch in _NAME_DOT_SEPARATORS or ch in _INVISIBLE_SEPARATORS: - return True - return _stripped(ch, state.policy) + """True when `ch` separates the words beside it: whitespace, the + name dots, _INVISIBLE_SEPARATORS (all unconditional, not + policy-gated: commas RECORD an offset and these must not, and only + the 间隔号 marks a transcription, #298), and what the strip policy + removes. The one exception is decided in _tokenize_region, by + context: U+202F between two Mongolian letters joins them (rules/T6).""" + policy = state.policy + return _IGNORABLE[policy.strip_bidi, policy.strip_emoji].match( + ch) is not None def _tokenize_region(state: ParseState, start: int, end: int, @@ -154,21 +182,28 @@ def _tokenize_region(state: ParseState, start: int, end: int, tokens: list[WorkToken], commas: list[int], interpuncts: list[int]) -> None: text = state.original - tok_start: int | None = None - for i in range(start, end): + policy = state.policy + tok_start = start + # Only the characters that may separate are visited; a word's + # letters are passed over in C + for m in _BOUNDARY[policy.strip_bidi, policy.strip_emoji].finditer( + text, start, end): + i = m.start() ch = text[i] - # rules/N1: "a comma inside the clause is kept as written: the - # comma divides nothing there" -- the Then bullet on a comma: a - # comma separates only where it records a segmentation offset, - # the main stream; inside an extracted region it is the clause's - # text, like ';' (decisions/N1, #653) - is_separator = ((record_offsets and ch in COMMA_CHARS) - or _ignorable(ch, state)) - if (ch == _NARROW_NBSP and start < i < end - 1 - and _mongolian_letter(text[i - 1]) - and _mongolian_letter(text[i + 1])): - is_separator = False - if ch in _TRANSCRIPTION_DOTS: + if ch in COMMA_CHARS: + # rules/N1: "a comma inside the clause is kept as written: + # the comma divides nothing there" -- the Then bullet on a + # comma: a comma separates only where it records a + # segmentation offset, the main stream; inside an extracted + # region it is the clause's text, like ';' (decisions/N1, + # #653) + if not record_offsets: + continue + elif ch == _NARROW_NBSP: + if (start < i < end - 1 and _mongolian_letter(text[i - 1]) + and _mongolian_letter(text[i + 1])): + continue + elif ch in _TRANSCRIPTION_DOTS: # Region-local flanks: a dot at a region edge stays token # text, and the bound also stops a custom delimiter's # classified edge character from acting as a flank across @@ -177,26 +212,22 @@ def _tokenize_region(state: ParseState, start: int, end: int, # conjoining jamo a decomposed syllable is written in, so # decomposed hangul flanks as composed hangul does # (decisions/nfd-shapes, #1087). - left = _flank(text, range(i - 1, start - 1, -1), - state.policy) - if left is not None and _classified_char(left): - right = _flank(text, range(i + 1, end), state.policy) - is_separator = (right is not None - and _classified_char(right)) - if is_separator: - if tok_start is not None: - tokens.append(WorkToken(text[tok_start:i], - Span(tok_start, i), role=role)) - tok_start = None - if record_offsets: - if ch in COMMA_CHARS: - commas.append(i) - elif ch in _TRANSCRIPTION_DOTS: - interpuncts.append(i) - continue - if tok_start is None: - tok_start = i - if tok_start is not None: + left = _flank(text, range(i - 1, start - 1, -1), policy) + if left is None or not _classified_char(left): + continue + right = _flank(text, range(i + 1, end), policy) + if right is None or not _classified_char(right): + continue + if tok_start < i: + tokens.append(WorkToken(text[tok_start:i], + Span(tok_start, i), role=role)) + tok_start = i + 1 + if record_offsets: + if ch in COMMA_CHARS: + commas.append(i) + elif ch in _TRANSCRIPTION_DOTS: + interpuncts.append(i) + if tok_start < end: tokens.append(WorkToken(text[tok_start:end], Span(tok_start, end), role=role)) @@ -218,7 +249,9 @@ def tokenize(state: ParseState) -> ParseState: for role, inner in state.extracted: _tokenize_region(state, inner.start, inner.end, role, False, tokens, commas, interpuncts) - tokens.sort(key=lambda t: t.span) + # attrgetter, not a lambda: the key is read in C, with no frame + # per token + tokens.sort(key=operator.attrgetter("span")) # extract_delimited runs before tokens exist, so its ambiguities # carry a character offset instead of an index. Resolve them now # that the stray character has landed in a token ('"Nick', 'Smith)') diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index cd248c3c3..425a21bbd 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -107,7 +107,9 @@ # here DERIVES from it. (single_script's sweep was first written # per-char on the _EMOJI_RANGES precedent in _tokenize.py, on the # theory that a range test needs no regex; measured at token scale the -# compiled regex wins by 3-9x, and by roughly 70x on long tokens.) +# compiled regex wins by 3-9x, and by roughly 70x on long tokens. The +# precedent has since gone the same way: tokenize scans for its +# separators with a compiled class, #1436.) # Derived, never hand-written -- even the class construction goes # through the shared factory: one wholly-of predicate per script, in # the table's key order -- the FIRST-covering-entry rule _classify diff --git a/nameparser/_types.py b/nameparser/_types.py index 3f9ea7650..c1c68fae4 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -422,8 +422,11 @@ def __post_init__(self) -> None: ) if not self.text: raise ValueError("Token.text must be a non-empty string") - object.__setattr__( - self, "role", _coerce_enum(self.role, Role, "Role", "roles")) + # _coerce_enum returns a Role unchanged; asked only of what may + # need coercing, the parse's own tokens pay no frame for it + if not isinstance(self.role, Role): + object.__setattr__( + self, "role", _coerce_enum(self.role, Role, "Role", "roles")) if self.reading is not None: if not isinstance(self.reading, Reading): raise TypeError( @@ -438,36 +441,43 @@ def __post_init__(self) -> None: f"Reading(kind, role, source) with the token's role" ) if self.span is not None: - if not ( - isinstance(self.span, tuple) - and len(self.span) == 2 - # bool is an int subclass: (False, True) is a comparison - # result leaking into a coordinate slot, not a span - and all(isinstance(v, int) and not isinstance(v, bool) - for v in self.span) - ): + # Each coordinate tested in turn, not by all() over a + # generator, which cost three frames a token; the parse + # builds every token it returns through here + if not (isinstance(self.span, tuple) and len(self.span) == 2): raise TypeError( f"invalid span {self.span!r}: expected a (start, end) " "pair of ints or None" ) start, end = self.span + # bool is an int subclass: (False, True) is a comparison + # result leaking into a coordinate slot, not a span + if not (isinstance(start, int) and not isinstance(start, bool) + and isinstance(end, int) and not isinstance(end, bool)): + raise TypeError( + f"invalid span {self.span!r}: expected a (start, end) " + "pair of ints or None" + ) if start < 0 or end < start: raise ValueError( f"invalid span ({start}, {end}): need 0 <= start <= end" ) - object.__setattr__(self, "span", Span(start, end)) + if type(self.span) is not Span: + object.__setattr__(self, "span", Span(start, end)) # The same guards _normset applies to Lexicon vocabulary: a bare # string would become its character set, a mapping would silently # contribute only its keys. - if isinstance(self.tags, str): - raise TypeError( - "Token.tags must be an iterable of strings, " - "not a bare string" - ) - if isinstance(self.tags, Mapping): - raise TypeError( - "Token.tags must be an iterable of strings, not a mapping" - ) + # A frozenset is neither, and asking Mapping, an ABC, is a frame + if type(self.tags) is not frozenset: + if isinstance(self.tags, str): + raise TypeError( + "Token.tags must be an iterable of strings, " + "not a bare string" + ) + if isinstance(self.tags, Mapping): + raise TypeError( + "Token.tags must be an iterable of strings, not a mapping" + ) tags = frozenset(self.tags) for tag in tags: if not isinstance(tag, str): diff --git a/tests/v2/pipeline/test_tokenize.py b/tests/v2/pipeline/test_tokenize.py index 5a657f846..9fadb283c 100644 --- a/tests/v2/pipeline/test_tokenize.py +++ b/tests/v2/pipeline/test_tokenize.py @@ -1,4 +1,5 @@ import dataclasses +import re import unicodedata import pytest @@ -11,7 +12,7 @@ from nameparser import (FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, Parser, _policy) from nameparser._policy import Policy, Script -from nameparser._types import Role +from nameparser._types import Role, Span def _tokenized(text: str, policy: Policy | None = None) -> ParseState: @@ -94,6 +95,73 @@ def test_emoji_and_bidi_are_ignorable_by_policy() -> None: assert [t.text for t in out2.tokens] == ["John", "\U0001f600Smith"] + +def test_the_separator_classes_hold_their_parts_over_every_code_point() -> None: + # #1436: the ignorable set is a regex class built from its parts, so + # a region is scanned in C; held here, over every code point, to the + # parts as their own predicates state them (str.isspace, the two + # sets, the emoji ranges, _BIDI), for each setting of the two strip + # switches, and the boundary class to the ignorable one plus the + # commas and transcription dots. Each class is matched whole, in C. + # It guards how the classes are ASSEMBLED, not which characters + # belong: _BIDI is compiled from the same string the classes are, + # so a mark dropped from it passes here. + from nameparser._pipeline._state import COMMA_CHARS + everything = "".join(map(chr, range(0x110000))) + space = {c for c, ch in enumerate(everything) if ch.isspace()} + bidi = {m.start() for m in re.finditer(_tokenize._BIDI.pattern[:-1], + everything)} + emoji = {c for lo, hi in _tokenize._EMOJI_RANGES + for c in range(lo, hi + 1)} + always = space | {ord(ch) for ch in _tokenize._NAME_DOT_SEPARATORS + | _tokenize._INVISIBLE_SEPARATORS} + context = {ord(ch) for ch in COMMA_CHARS | _tokenize._TRANSCRIPTION_DOTS} + + def matched(pattern: re.Pattern[str] | None) -> set[int]: + return ({m.start() for m in pattern.finditer(everything)} + if pattern is not None else set()) + + for key in _tokenize._SWITCHES: + strip_bidi, strip_emoji = key + stripped = (bidi if strip_bidi else set()) | ( + emoji if strip_emoji else set()) + assert matched(_tokenize._STRIPPED[key]) == stripped, key + assert matched(_tokenize._IGNORABLE[key]) == always | stripped, key + assert matched(_tokenize._BOUNDARY[key]) == ( + always | stripped | context), key + # the two readers of the classes, one member of each part and a + # letter, under every setting: the mixed two are where a reader + # picking its class by the switches in the wrong order shows + for strip_bidi, strip_emoji in _tokenize._SWITCHES: + policy = Policy(strip_bidi=strip_bidi, strip_emoji=strip_emoji) + state = ParseState(original="", lexicon=Lexicon.empty(), + policy=policy) + for ch in (" ", "\u30fb", "\u200b", "\u200f", "\U0001f600", "a"): + assert _tokenize._ignorable(ch, state) == ( + ord(ch) in always or (ord(ch) in bidi and policy.strip_bidi) + or (ord(ch) in emoji and policy.strip_emoji)), (policy, ch) + assert _tokenize._stripped(ch, policy) == ( + (ord(ch) in bidi and policy.strip_bidi) + or (ord(ch) in emoji and policy.strip_emoji)), (policy, ch) + + +def _region_bound(text: str, masked: tuple[Span, ...]) -> tuple[ + list[str], tuple[int, ...]]: + state = tokenize(ParseState(original=text, lexicon=Lexicon.empty(), + policy=Policy(), masked=masked)) + return [t.text for t in state.tokens], state.interpunct_offsets + + +def test_a_context_test_reads_no_character_across_a_region_edge() -> None: + # the dot's flanks and U+202F's two letters are read inside the + # region: a classified or Mongolian character on the far side of a + # mask seam is no neighbour (each row fails with the bound dropped) + assert _region_bound("威·廉", (Span(0, 1),)) == (["·廉"], ()) + assert _region_bound("威·廉", (Span(2, 3),)) == (["威·"], ()) + assert _region_bound("ᠴ\u202fᠪᠠ", (Span(0, 1),)) == (["ᠪᠠ"], ()) + assert _region_bound("ᠴᠣ\u202fᠪ", (Span(3, 4),)) == (["ᠴᠣ"], ()) + + _KEEP_ALL = Policy(strip_emoji=False, strip_bidi=False) diff --git a/tests/v2/test_benchmark.py b/tests/v2/test_benchmark.py index df62da68d..770a4490e 100644 --- a/tests/v2/test_benchmark.py +++ b/tests/v2/test_benchmark.py @@ -151,12 +151,17 @@ #: 3 on 3.12-3.15 (#1170), to the figures below: three post_rules #: helpers stopped making calls they did not need, saving 7 and 6 from #: a tree that had drifted up inside the band (decisions/parse-cost). +#: Lowered 2026-11-18 by 88 on 3.11 and 89 on 3.12-3.15 (#1436), to +#: the figures below: tokenize scans a region for its separators in C +#: rather than asking each character, and `Token`'s validation, the +#: generators and an enum-keyed dict stopped costing frames on the +#: parse's own input (decisions/parse-cost). _CALL_BASELINE = { - (3, 11): {"parse": 285}, - (3, 12): {"parse": 277}, - (3, 13): {"parse": 277}, - (3, 14): {"parse": 277}, - (3, 15): {"parse": 277}, + (3, 11): {"parse": 197}, + (3, 12): {"parse": 188}, + (3, 13): {"parse": 188}, + (3, 14): {"parse": 188}, + (3, 15): {"parse": 188}, } _BAND = 0.02 @@ -1538,11 +1543,13 @@ def test_a_fixed_point_does_not_reread_what_it_has_read( # `_CALL_BASELINE` above. # # THE LONG RUN AND NOT THE SHORT ONE, and the arithmetic is the whole -# reason: +1 per link is 16 frames at `_CLAUSE_RUN_SMALL`, and 2% of -# 859 is 17.2, so the small end's band swallows the very regression -# this pin exists for (875 sits inside 842-876). At -# `_CLAUSE_RUN_LARGE` the same change is +64 against a 2% band of -# 51.7, and 2,651 sits outside 2,535-2,639. A tighter band on the +# reason, as measured 2026-09-22 on py3.11 on the clause run the pin +# then read: +1 per link is 16 frames at `_CLAUSE_RUN_SMALL`, and 2% +# of 859 was 17.2, so the small end's band swallowed the very +# regression this pin exists for (875 sat inside 842-876). At +# `_CLAUSE_RUN_LARGE` the same change was +64 against a 2% band of +# 51.7, and 2,651 sat outside 2,535-2,639. The pin's current band is +# given with `_LINK_BASELINE` below. A tighter band on the # short name would do it too and was not taken: the band is the one # `_CALL_BASELINE` uses, and a bespoke one here would need its own # argument every time the shape moved. @@ -1585,13 +1592,18 @@ def test_a_fixed_point_does_not_reread_what_it_has_read( # `join_connectives` builds its connective runs in one pass rather # than through `merge_pieces`, so the 64-letter run here no longer # pays that function's frame per member (decisions/parse-cost, -# 2026-11-03). +# 2026-11-03). 1,716 (py3.11) and 1,706 (py3.12-3.15) from +# 2026-11-18 (#1436), each measured there, against 2,404 and 2,395 on +# the tree before the change: frames paid per token or per character +# (`Token`'s checks 402 of the 688, tokenize's per-character call 143) +# are paid per link here (decisions/parse-cost, 2026-11-18). A per-link regression still +# shows: +64 against a 2% band of 34.3. _LINK_BASELINE = { - (3, 11): 2418, - (3, 12): 2398, - (3, 13): 2398, - (3, 14): 2398, - (3, 15): 2398, + (3, 11): 1716, + (3, 12): 1706, + (3, 13): 1706, + (3, 14): 1706, + (3, 15): 1706, } #: The same +-2% `_CALL_BASELINE` uses, and for the same reason: frame #: counts are deterministic for a given tree and interpreter, so the diff --git a/tests/v2/test_types.py b/tests/v2/test_types.py index dc6aae384..5da08f17a 100644 --- a/tests/v2/test_types.py +++ b/tests/v2/test_types.py @@ -68,6 +68,20 @@ def test_token_rejects_malformed_span_shapes() -> None: Token("x", 5, Role.GIVEN) # type: ignore[arg-type] + +@pytest.mark.parametrize("span", [ + [0, 4], (False, True), (0, True), (0, 4.0), (True, 4), (0.0, 4), + # a Span keeps its coordinates as given, so it is checked as a tuple + # is (#1436 stopped rebuilding one, and must not stop checking it) + Span(0.0, 4.0), Span(False, True), Span(0, True), # type: ignore[arg-type] +], ids=["list", "bools", "bool-end", "float-end", "bool-start", + "float-start", "span-floats", + "span-bools", "span-bool-end"]) +def test_token_rejects_a_span_of_anything_but_two_ints(span: object) -> None: + with pytest.raises(TypeError, match="expected a \\(start, end\\) pair"): + Token("x", span, Role.GIVEN) # type: ignore[arg-type] + + def test_token_rejects_non_string_text() -> None: with pytest.raises(TypeError, match="got None"): Token(None, None, Role.GIVEN) # type: ignore[arg-type]