diff --git a/docs/design/decisions/parse-cost.md b/docs/design/decisions/parse-cost.md index bd1866c17..5367d46ff 100644 --- a/docs/design/decisions/parse-cost.md +++ b/docs/design/decisions/parse-cost.md @@ -48,3 +48,4 @@ Open: none in this file. - 2026-11-07 #1157 — NO ROW MOVES; release checklist step 0's pass over post_rules and assign. O3's fold and R2's marks now share one `_positions` walk, which the fold updates, so a `middle_as_family` parse walks once fewer; P10's and W5's tag gates are one plain walk; a lone name word's field is decided where it is placed, so placing one piece makes no `_name_positions` call; S1's settling credential is asked only of a clause holding an unsettled member; N3's early nickname scan sits behind `state.extracted`. Measured 2026-11-07, py3.11, one warm parse counted with a `sys.setprofile` hook, before and after (dbc2fc22 and the change, and again on b43ad933, #1154's merge, and on c489c9a7, #1162's merge, after #1158's, each against the change rebased onto it, every count the same): the reference name 287 either way; `Smith` 128 → 125, `John (Ret.) Smith` 216 → 213, `John Smith (PhD, MA)` 253 → 250, `王李小明` 155 → 152; `John Smith` 130 and `Smith, John` 140 unchanged; no name rose under the default, `FAMILY_FIRST`, `middle_as_family`, `lone_name_word=FAMILY` or `two_family_names` policies. BEHAVIOR IDENTICAL: `snapshot.jsonl` byte-identical, and, for the switches the snapshot's default policy cannot see, dbc2fc22 and the change (2aeb6b95, before the rebase) read alike under a policy sweep, each tree loaded by path with `PYTHONSAFEPATH=1` and its `__file__` checked: every distinct text of the committed `tools/differential/corpus*.jsonl` files (the `name` key, or the line's string) plus 14 shapes these paths reach (`Juan Smith`, `Smith`, `de`, `María del Carmen García`, `王李小明`, `Smith, John`, `John (Ret.) Smith`, `John Smith (PhD, MA)`, `John Smith (MA)`, `John Smith (MA, Jr.)`, `(Bob)`, `Smith (Bob)`, `Ivan Petrovich Sidorov`, `García López, Juan`), 6,906 texts, each parsed by `Parser(policy=Policy(name_order=..., middle_as_family=..., lone_name_word=..., two_family_names=...))` under the three orders, each switch off and on and the lone field given or family (24 policies, every other field at its default), the comparator a line per parse holding the `repr` of the text, the policy's four values, `tokens` and `ambiguities`, the two files compared with `cmp`: 0 of 165,744 lines differ. The negative control, dbc2fc22 with `_assign._name_positions`' one-piece branch returning `[first]` where it returned `[lone]`, reads 3,157 (measured 2026-11-07 in #1157's review). `test_post_rules_walks_the_roles_once_per_reader` pins the `middle_as_family` row at 2 walks, against 3 before. - 2026-11-09 #1170 — EVERY ROW MOVES DOWN, by 4 on 3.11 and 3 on 3.12 to 3.15: `parse` 289 → 285 and 280 → 277, the rows re-taken where the tree measures rather than by the change alone, master having drifted to 291.80 and 282.80 inside the band since #1089. Three `post_rules` helpers that every parse calls stop making calls they need not make, the three cleanups #1157's pass declined only because the code was unchanged since its last pass: `_leading_name_piece` tests a piece's tokens in a plain inner loop where an `any()` generator cost a frame per resume; `post_rules` asks `_rotations_apply` only where `patronymic_rules` is set, the rotations' only readers testing a rule's membership; and `_mark_suffix_entries` collects the suffix positions in a plain loop and returns before `delimiter_cores` on a name holding fewer than two, where `zip` over them never ran. Measured 2026-11-09 on 9f6264e4 (#1295's merge) and on the change, each interpreter on its own (`uv run python tools/perf/call_count.py`, and with `UV_PROJECT_ENVIRONMENT= uv run --python 3.1x` for 3.12 to 3.15): 291.80 → 284.80 on 3.11 and 282.80 → 276.80 on each of 3.12 to 3.15. Per name, one warm parse counted with `_frames_for` in `tests/v2/test_benchmark.py`, py3.11, each name in a process of its own: `John Smith` 131 → 126, `Smith, John` 141 → 136, `John Smith Jr., PhD` 264 → 260, `Dr. John Smith Jr.` 241 → 234. A name's absolute count depends on what the process parsed before it, `_frames_for`'s one warm-up parse leaving some first-use work to the name, so the same tree read `Dr. John Smith Jr.` 224, 214 and 234 by the order its names ran in (two reviews of #1170); compare counts taken the same way, and read the difference, which held at 7 for that name in every order. BEHAVIOR IDENTICAL by construction, each change returning where the old code's result was already decided (a rotation needs its rule in `patronymic_rules`; `zip` over fewer than two suffixes is empty; the inner loop is the `any()` it replaces), and `snapshot.jsonl` byte-identical. `test_post_rules_helpers_return_before_their_frames` holds the two calls the band cannot see put back, with its recorded control (9f6264e4: 2 generator frames on `John Smith`, 1 `delimiter_cores` call), and `test_post_rules_walks_the_roles_once_per_reader`'s pin on the calls `post_rules` makes directly moves 7 → 6. By the four frame questions: every parse pays them (question 2) and they move the band (question 3); none is a copy of a predicate (question 4). Declined again, as #1157 declined them: a `stale` flag on the position lists and dropping the recompute after P6, each bringing back the stale-list shape #359 fixed. SUPERSEDES: the 2026-11-05 #1089 entry's rows. - 2026-11-11 #1352 — THE TIMED ROWS READ CPU TIME, AND THE COUNTS RUN IN PARALLEL; NO ROW, BOUND OR BASELINE MOVES. On a laptop running several agent sessions the wall-clock rows failed at random and passed alone. Measured on py3.11, 12 cores (6 performance, 6 efficiency), five `uv run pytest -n0 -m serial tests/v2/test_benchmark.py` runs beside 24 shell busy loops (load average 37-48): 20, 23, 21, 19 and 23 of the module's 62 tests failed on 970b54e5, at up to 10.0x against `_MAX_RATIO` 6.0; on the change, 0 of 34 in each of five runs at 24 busy loops (load 50-62) and at none, and in five at 12, which barely loaded the machine (the before tree failed none there either). Per row, three runs at 24 busy loops (load 18-75): `time.perf_counter()` read 27 of the 33 ratio rows at 6.0 or over at least once, up to 14.0x; `time.thread_time()`, measured back to back, none over 5.76x (as high as 5.76x with no busy loops too). CPU time is not load-immune -- interleaved best-of-7 read a clean row up to 5.6x, the thread charged for a slower core or a colder cache, and a user-interactive QoS hint did not help -- and noise can DEFLATE a ratio as well as inflate it: with more rounds taken only at or over the bound, a planted quadratic reading 6.7x quiet (`given_part_titles`) read 5.93x once at 24 busy loops and passed, and a loop stopping at its first reading under the bound could stop on a dip (both found by #1352's review). So `_ratio` interleaves the two inputs and, where its first 7 rounds read at or over 80% of the bound (`_NEAR`; `bound * _NEAR` is 4.800000000000001 in floating point), takes all 35 and judges the last minimums, each round only moving the two minimums toward the true costs; `test_ratio_takes_more_rounds_only_near_the_bound` holds that on a scripted clock, and fails against both earlier loops. Loaded, the serial pass takes 78-115s against 47s, the escalation's price; quiet it takes 18s, as before. The defects it guards are seen as before. Quiet, the old wall-clock `_best` and `_ratio` fail the same rows in every run under a planted C-level quadratic in the word count, every row within 0.35x. The recorded controls read on `_ratio`, quiet then at 24 busy loops: `link_run` 13.9-14.1 and 14.9-16.4 on b9ed1429; `given_part_run` 8.0 and 10.0-10.3 and `given_part_titles` 10.1 and 11.1-12.5 on 003b4962; `title_chain` 12.0 and 13.5-14.8, `pair_run` 10.0 and 10.2-12.8, `caps_pairs` 10.1-10.2 and 12.2-12.8 and `delimiter_cores` 10.4 and 11.8-13.1 on 9dd271b2; `connective_between_particles` 8.5 and 8.9-10.2 on 5752df69 (#939 before its review); and `slash_chain` 8.8-8.9 and 8.9-9.2 with #1103's per-step copy planted. That is against 3.97-4.19 quiet and up to 5.06 at 24 busy loops on 970b54e5. The weakest planted quadratic, 6.3-7.0x quiet on eleven rows, read 7.18x or more on every row in five runs at 24 busy loops. A planted backtracking regex reads 7.31 s of CPU over the thousand names, against 7.36 s wall. The 5s bound is CPU seconds now and no looser for work on the parse's own thread, CPU time being at most the wall time of a single-threaded parse; it no longer sees time spent blocked or on another thread, which a parse never spends. 962301f7's memory move reads 4.07-4.43 on the CPU clock and 4.07-4.50 on the wall clock, quiet, as the 2026-11-03 #1036 entry found for the wall clock; `_ShiftCountingList` holds it. The 28 untimed tests (the frame, call and shift counts and the table checks) lose the `serial` mark. Each passes alone in a fresh process, after the 21,065 tests of `test_cases.py` and `test_match.py` in one, in three full parallel runs, and under `--cov` on 3.11 and 3.14. The order dependence of an absolute count (the 2026-11-09 #1170 entry) stays inside a ratio, a band, or an exact count of one named function's calls (`only=`/`caller=`, `== 0` up to `== 6`), none of which counts an `lru_cache`'d function. +- 2026-11-18 #1435 — NO ROW MOVES; A RUN OF INITIALS BEHIND A FULL NAME ASKS NO COUNT. #1238's C16 block in `_comma.decide` put a dotted run spelling a listed credential, every letter a capital or the name in one case, into `maybe`, so `John Smith, M.D.` paid C4's `name_word_count` and the C11 books, which only a one-word head reads the run by. Where the run is the part's only entry in `maybe` (no word of the class and no caps-shape candidate beside it), `decide` now asks `_lone_word` first (once, its answer kept for the reading), and behind a head that is not one word skips the count and the books: the run stays in `maybe`, so the part is still read as the count's part and never takes the listing form's early return, and with nothing else in `maybe` the count licenses nothing and `head_named` meets no member. Frames, `test_benchmark._frames_for` for one parse in a fresh process, master (b78d5af2) then branch: `John Smith, M.D.` 221 → 199 on 3.11 and 203 → 185 on 3.15; `John Smith, MD` 190 and 178 and `Smith, John` 136 and 125 unchanged; `Smith, J.P.`, which still counts, 233 and 218 unchanged. The issue's table (209 for `John Smith, M.D.` on master) was taken in a process that had parsed other names first, which moves the absolute count and not the difference (the 2026-11-09 #1170 entry). The issue's proposed gate (append the run only beside another class word or behind a one-word head) was not taken: it sends such a part down the no-`maybe` path, equal only where the listing early return and C1's fallback agree, which nothing holds by construction. Checked against master: a grid of 215,280 parses (65 heads, from empty, nickname, title, particle and title-particle heads alone and in pairs to full and one-case names, × 46 tails of dotted runs, listed and ambiguous credentials, paired initials and their company × three comma spellings and a later part × the three orders × six policies) byte-identical, the issue's gate also identical there, and the harness shown to see a move (on the same grid, a `_lone_word` always true moves 34,848 rows, and skipping the count behind a one-word head too moves 4,536, `Smith van, M.D.` reading given 'M.', middle 'D.', family 'Smith van' where master reads given 'Smith', family 'van', suffix 'M.D.', `_lone_word` reading that head as one word while the count finds two); `snapshot.jsonl` byte-identical; and `snapshot.py --local ctgov-officials` and `ctgov-contacts` against master, no move. `test_a_run_of_initials_behind_a_full_name_takes_no_count` holds the saving, its control recorded in its docstring, and `test_the_count_c16_skips_decides_no_reading` (tests/v2/pipeline/test_comma.py) the readings a wrong gate moves. Under each of three mutants (the gate removed; the count skipped behind a one-word head too; `run_alone` forced true, so a run beside another entry of `maybe` such as `John Smith, M.D. M.A` skips the count as well) the frame test fails, and under the last two the reading test does too. A run beside `MA` (`John Smith, M.D. MA`) never joins `maybe`, `MA` being the run's company (C16), so it reaches none of this. diff --git a/nameparser/_pipeline/_comma.py b/nameparser/_pipeline/_comma.py index 017bf2b60..680aff927 100644 --- a/nameparser/_pipeline/_comma.py +++ b/nameparser/_pipeline/_comma.py @@ -762,11 +762,18 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState: # rules/C16: "behind one word, initials run together are the given # name whatever they spell" -- the runs that spell a listed # credential ('J.P.', 'M.A.'), joined to the class so C4's count is - # taken (decisions/C16 (#1238)). Gathered only + # taken where the head may be one word (decisions/C16 (#1238), + # #1435). Gathered only # where the part holds a full stop, so 'Smith, John' pays no frame; # a part whose other words are all credentials is C8's run # ('Smith, M.D. Ph.D.'). initials_at: int | None = None + # whether the run is the part's only entry in `maybe` (no word of + # the class and no caps-shape candidate beside it), and, asked only + # then, whether the head is one word as C16 reads it (`_lone_word`; + # None where not asked, the reading's own test asking it instead) + run_alone = False + lone_head: bool | None = None for at, p in enumerate(p1): if len(p) == 1 and "." in (text := texts[p[0]]): tags = tokens[p[0]].tags @@ -801,6 +808,7 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState: for p, pt in rest): initials_at = None elif initials_at not in maybe: + run_alone = not maybe maybe.append(initials_at) # rules/C8: "a part that is nothing but suffix words is the # credential run and reads as suffixes, whole" -- the reading below. @@ -844,7 +852,17 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState: # a pair a later comma part's credential spoke for (rules/C5), whose # flip is reported as the count's, dotted or not later = False - if maybe: + # C16's run alone in `maybe`, behind a head that is not one word, + # asks no count and keeps no books (#1435): the run reads as + # initials only behind one word, and with nothing else in `maybe` + # the count licenses nothing, `head_named` meets no member and the books find + # none, so the reading is the one the count would give ('John + # Smith, M.D.'). The run stays in `maybe`, so the part is still read + # below as the count's part, never the listing form's + if run_alone: + lone_head = _lone_word(state, seg0) + counted = bool(maybe) and lone_head is not False + if counted: # `_members` inline: a frame here is paid by every name with the # ambiguous class after its comma ('John Smith, MA') split_at: set[int] = set() @@ -966,7 +984,8 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState: caps.discard(i) if maybe: lone = (initials_at is not None and names < 2 - and _lone_word(state, seg0)) + and (lone_head if lone_head is not None + else _lone_word(state, seg0))) reading = segment_suffix_reading( p1, pt1, tokens, pol.lenient_comma_suffixes, state.one_case, anchored, absorbed, licensed, head_named=names >= 2, @@ -1060,7 +1079,7 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState: members: list[int] = [] listed: list[int] = [] settled = pair_only = False - if maybe: + if counted: members = [p[0] for p in p1 if len(p) == 1 and AMBIGUOUS_ACRONYM_TAG in tokens[p[0]].tags] listed = [i for i in members diff --git a/tests/v2/pipeline/test_comma.py b/tests/v2/pipeline/test_comma.py index db204338c..fa6f93f58 100644 --- a/tests/v2/pipeline/test_comma.py +++ b/tests/v2/pipeline/test_comma.py @@ -690,3 +690,32 @@ def test_the_written_short_except_reads_past_a_credentials_own_case_mask() -> No # condition, whatever the mask (#1395's third review) got = parser.parse("Smith, PHD") assert (got.given, got.suffix) == ("", "PHD") + + +@pytest.mark.parametrize("text, fields, kinds", [ + # behind a full name: the count is skipped and the run reads as the + # credential it spells + ("John Smith, M.D.", ("", "John", "Smith", "M.D."), ()), + # a head `_lone_word` reads as one word where the count finds two + # (a trailing lower-case particle): the count is still taken, and + # decides the reading. Recorded as it reads before and after #1435, + # not endorsed + ("Smith van, M.D.", ("", "Smith", "van", "M.D."), ()), + ("Smith van, J.P.", ("", "Smith", "van", "J.P."), ()), + # the run beside another entry of `maybe` (a dotted member not read + # as company): the count is taken for that member + ("John Smith, M.D. M.A", ("", "John", "Smith", "M.D. M.A"), ()), + ("John Smith, M.A M.A M.D.", ("", "John", "Smith", "M.A M.A M.D."), + ("suffix-or-name",)), +]) +def test_the_count_c16_skips_decides_no_reading( + text: str, fields: tuple[str, str, str, str], + kinds: tuple[str, ...]) -> None: + # #1435: `decide` skips C4's count only where C16's run is alone in + # `maybe` behind a head that is not one word. Each row is read the + # same with and without the skip; a gate that skipped the count + # behind a one-word head too ('Smith van'), or beside another + # member, moves them + got = parse(text) + assert (got.middle, got.given, got.family, got.suffix) == fields, text + assert tuple(str(a.kind) for a in got.ambiguities) == kinds, text diff --git a/tests/v2/test_benchmark.py b/tests/v2/test_benchmark.py index 4b68e2a07..df62da68d 100644 --- a/tests/v2/test_benchmark.py +++ b/tests/v2/test_benchmark.py @@ -1186,6 +1186,39 @@ def test_a_listing_comma_with_nothing_to_decide_skips_the_counts() -> None: assert _frames_for("John Smith, MA", only="name_word_count") == 1 +def test_a_run_of_initials_behind_a_full_name_takes_no_count() -> None: + """rules/C16's run of initials reads as the given name only behind + one word, so where it is the part's only entry in `maybe` (no word + of the ambiguous class and no caps-shape candidate beside it) and + the head is not one word, `_comma.decide` asks C4's count nothing + (#1435). Recorded negative control (2026-11-18, py3.11, this + module's `_frames_for` in a fresh process): with the gate's + `lone_head is not False` arm deleted no output moves, and 'John + Smith, M.D.' enters `name_word_count` once and costs 23 more frames + (222 against 199; master before the gate, which did not ask + `_lone_word` there, 221).""" + if sys.getprofile() is not None: + pytest.skip("a profile hook is already installed; this test owns it") + only = "name_word_count" + for text in ("John Smith, M.D.", "Dr. Mr., M.D.", "Bin Zhang, J.P."): + # REACHABILITY: C16 found the run and asked about the head, so a + # zero below means the count was skipped rather than unreached + assert _frames_for(text, only="_lone_word") == 1, text + assert _frames_for(text, only=only) == 0, text + # behind one word the count is still taken, once, and `_lone_word` + # is asked once, not again for the reading + assert _frames_for("Smith, J.P.", only=only) == 1 + assert _frames_for("Smith, J.P.", only="_lone_word") == 1 + # a head `_lone_word` reads as one word where the count finds two + # still takes it + assert _frames_for("Smith van, M.D.", only=only) == 1 + # another entry of `maybe` beside the run still takes the count, and + # `_lone_word` is not asked there (REACHABILITY: the run joined + # `maybe` behind a member, not reset as company, which 'M.D. MA' is) + assert _frames_for("John Smith, M.D. M.A", only=only) == 1 + assert _frames_for("John Smith, M.D. M.A", only="_lone_word") == 0 + + def test_two_capitals_ask_the_later_parts_only_where_one_stands() -> None: """rules/C5's later credential (#1392) is read only where the pair is the whole part and a part stands behind it, so a pair alone