Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/design/decisions/parse-cost.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,4 @@ Open: none in this file.
- 2026-11-07 #1157 — NO ROW MOVES; release checklist step 0's pass over post_rules and assign. O3's fold and R2's marks now share one `_positions` walk, which the fold updates, so a `middle_as_family` parse walks once fewer; P10's and W5's tag gates are one plain walk; a lone name word's field is decided where it is placed, so placing one piece makes no `_name_positions` call; S1's settling credential is asked only of a clause holding an unsettled member; N3's early nickname scan sits behind `state.extracted`. Measured 2026-11-07, py3.11, one warm parse counted with a `sys.setprofile` hook, before and after (dbc2fc22 and the change, and again on b43ad933, #1154's merge, and on c489c9a7, #1162's merge, after #1158's, each against the change rebased onto it, every count the same): the reference name 287 either way; `Smith` 128 → 125, `John (Ret.) Smith` 216 → 213, `John Smith (PhD, MA)` 253 → 250, `王李小明` 155 → 152; `John Smith` 130 and `Smith, John` 140 unchanged; no name rose under the default, `FAMILY_FIRST`, `middle_as_family`, `lone_name_word=FAMILY` or `two_family_names` policies. BEHAVIOR IDENTICAL: `snapshot.jsonl` byte-identical, and, for the switches the snapshot's default policy cannot see, dbc2fc22 and the change (2aeb6b95, before the rebase) read alike under a policy sweep, each tree loaded by path with `PYTHONSAFEPATH=1` and its `__file__` checked: every distinct text of the committed `tools/differential/corpus*.jsonl` files (the `name` key, or the line's string) plus 14 shapes these paths reach (`Juan Smith`, `Smith`, `de`, `María del Carmen García`, `王李小明`, `Smith, John`, `John (Ret.) Smith`, `John Smith (PhD, MA)`, `John Smith (MA)`, `John Smith (MA, Jr.)`, `(Bob)`, `Smith (Bob)`, `Ivan Petrovich Sidorov`, `García López, Juan`), 6,906 texts, each parsed by `Parser(policy=Policy(name_order=..., middle_as_family=..., lone_name_word=..., two_family_names=...))` under the three orders, each switch off and on and the lone field given or family (24 policies, every other field at its default), the comparator a line per parse holding the `repr` of the text, the policy's four values, `tokens` and `ambiguities`, the two files compared with `cmp`: 0 of 165,744 lines differ. The negative control, dbc2fc22 with `_assign._name_positions`' one-piece branch returning `[first]` where it returned `[lone]`, reads 3,157 (measured 2026-11-07 in #1157's review). `test_post_rules_walks_the_roles_once_per_reader` pins the `middle_as_family` row at 2 walks, against 3 before.
- 2026-11-09 #1170 — EVERY ROW MOVES DOWN, by 4 on 3.11 and 3 on 3.12 to 3.15: `parse` 289 → 285 and 280 → 277, the rows re-taken where the tree measures rather than by the change alone, master having drifted to 291.80 and 282.80 inside the band since #1089. Three `post_rules` helpers that every parse calls stop making calls they need not make, the three cleanups #1157's pass declined only because the code was unchanged since its last pass: `_leading_name_piece` tests a piece's tokens in a plain inner loop where an `any()` generator cost a frame per resume; `post_rules` asks `_rotations_apply` only where `patronymic_rules` is set, the rotations' only readers testing a rule's membership; and `_mark_suffix_entries` collects the suffix positions in a plain loop and returns before `delimiter_cores` on a name holding fewer than two, where `zip` over them never ran. Measured 2026-11-09 on 9f6264e4 (#1295's merge) and on the change, each interpreter on its own (`uv run python tools/perf/call_count.py`, and with `UV_PROJECT_ENVIRONMENT=<a scratch folder> uv run --python 3.1x` for 3.12 to 3.15): 291.80 → 284.80 on 3.11 and 282.80 → 276.80 on each of 3.12 to 3.15. Per name, one warm parse counted with `_frames_for` in `tests/v2/test_benchmark.py`, py3.11, each name in a process of its own: `John Smith` 131 → 126, `Smith, John` 141 → 136, `John Smith Jr., PhD` 264 → 260, `Dr. John Smith Jr.` 241 → 234. A name's absolute count depends on what the process parsed before it, `_frames_for`'s one warm-up parse leaving some first-use work to the name, so the same tree read `Dr. John Smith Jr.` 224, 214 and 234 by the order its names ran in (two reviews of #1170); compare counts taken the same way, and read the difference, which held at 7 for that name in every order. BEHAVIOR IDENTICAL by construction, each change returning where the old code's result was already decided (a rotation needs its rule in `patronymic_rules`; `zip` over fewer than two suffixes is empty; the inner loop is the `any()` it replaces), and `snapshot.jsonl` byte-identical. `test_post_rules_helpers_return_before_their_frames` holds the two calls the band cannot see put back, with its recorded control (9f6264e4: 2 generator frames on `John Smith`, 1 `delimiter_cores` call), and `test_post_rules_walks_the_roles_once_per_reader`'s pin on the calls `post_rules` makes directly moves 7 → 6. By the four frame questions: every parse pays them (question 2) and they move the band (question 3); none is a copy of a predicate (question 4). Declined again, as #1157 declined them: a `stale` flag on the position lists and dropping the recompute after P6, each bringing back the stale-list shape #359 fixed. SUPERSEDES: the 2026-11-05 #1089 entry's rows.
- 2026-11-11 #1352 — THE TIMED ROWS READ CPU TIME, AND THE COUNTS RUN IN PARALLEL; NO ROW, BOUND OR BASELINE MOVES. On a laptop running several agent sessions the wall-clock rows failed at random and passed alone. Measured on py3.11, 12 cores (6 performance, 6 efficiency), five `uv run pytest -n0 -m serial tests/v2/test_benchmark.py` runs beside 24 shell busy loops (load average 37-48): 20, 23, 21, 19 and 23 of the module's 62 tests failed on 970b54e5, at up to 10.0x against `_MAX_RATIO` 6.0; on the change, 0 of 34 in each of five runs at 24 busy loops (load 50-62) and at none, and in five at 12, which barely loaded the machine (the before tree failed none there either). Per row, three runs at 24 busy loops (load 18-75): `time.perf_counter()` read 27 of the 33 ratio rows at 6.0 or over at least once, up to 14.0x; `time.thread_time()`, measured back to back, none over 5.76x (as high as 5.76x with no busy loops too). CPU time is not load-immune -- interleaved best-of-7 read a clean row up to 5.6x, the thread charged for a slower core or a colder cache, and a user-interactive QoS hint did not help -- and noise can DEFLATE a ratio as well as inflate it: with more rounds taken only at or over the bound, a planted quadratic reading 6.7x quiet (`given_part_titles`) read 5.93x once at 24 busy loops and passed, and a loop stopping at its first reading under the bound could stop on a dip (both found by #1352's review). So `_ratio` interleaves the two inputs and, where its first 7 rounds read at or over 80% of the bound (`_NEAR`; `bound * _NEAR` is 4.800000000000001 in floating point), takes all 35 and judges the last minimums, each round only moving the two minimums toward the true costs; `test_ratio_takes_more_rounds_only_near_the_bound` holds that on a scripted clock, and fails against both earlier loops. Loaded, the serial pass takes 78-115s against 47s, the escalation's price; quiet it takes 18s, as before. The defects it guards are seen as before. Quiet, the old wall-clock `_best` and `_ratio` fail the same rows in every run under a planted C-level quadratic in the word count, every row within 0.35x. The recorded controls read on `_ratio`, quiet then at 24 busy loops: `link_run` 13.9-14.1 and 14.9-16.4 on b9ed1429; `given_part_run` 8.0 and 10.0-10.3 and `given_part_titles` 10.1 and 11.1-12.5 on 003b4962; `title_chain` 12.0 and 13.5-14.8, `pair_run` 10.0 and 10.2-12.8, `caps_pairs` 10.1-10.2 and 12.2-12.8 and `delimiter_cores` 10.4 and 11.8-13.1 on 9dd271b2; `connective_between_particles` 8.5 and 8.9-10.2 on 5752df69 (#939 before its review); and `slash_chain` 8.8-8.9 and 8.9-9.2 with #1103's per-step copy planted. That is against 3.97-4.19 quiet and up to 5.06 at 24 busy loops on 970b54e5. The weakest planted quadratic, 6.3-7.0x quiet on eleven rows, read 7.18x or more on every row in five runs at 24 busy loops. A planted backtracking regex reads 7.31 s of CPU over the thousand names, against 7.36 s wall. The 5s bound is CPU seconds now and no looser for work on the parse's own thread, CPU time being at most the wall time of a single-threaded parse; it no longer sees time spent blocked or on another thread, which a parse never spends. 962301f7's memory move reads 4.07-4.43 on the CPU clock and 4.07-4.50 on the wall clock, quiet, as the 2026-11-03 #1036 entry found for the wall clock; `_ShiftCountingList` holds it. The 28 untimed tests (the frame, call and shift counts and the table checks) lose the `serial` mark. Each passes alone in a fresh process, after the 21,065 tests of `test_cases.py` and `test_match.py` in one, in three full parallel runs, and under `--cov` on 3.11 and 3.14. The order dependence of an absolute count (the 2026-11-09 #1170 entry) stays inside a ratio, a band, or an exact count of one named function's calls (`only=`/`caller=`, `== 0` up to `== 6`), none of which counts an `lru_cache`'d function.
- 2026-11-18 #1435 — NO ROW MOVES; A RUN OF INITIALS BEHIND A FULL NAME ASKS NO COUNT. #1238's C16 block in `_comma.decide` put a dotted run spelling a listed credential, every letter a capital or the name in one case, into `maybe`, so `John Smith, M.D.` paid C4's `name_word_count` and the C11 books, which only a one-word head reads the run by. Where the run is the part's only entry in `maybe` (no word of the class and no caps-shape candidate beside it), `decide` now asks `_lone_word` first (once, its answer kept for the reading), and behind a head that is not one word skips the count and the books: the run stays in `maybe`, so the part is still read as the count's part and never takes the listing form's early return, and with nothing else in `maybe` the count licenses nothing and `head_named` meets no member. Frames, `test_benchmark._frames_for` for one parse in a fresh process, master (b78d5af2) then branch: `John Smith, M.D.` 221 → 199 on 3.11 and 203 → 185 on 3.15; `John Smith, MD` 190 and 178 and `Smith, John` 136 and 125 unchanged; `Smith, J.P.`, which still counts, 233 and 218 unchanged. The issue's table (209 for `John Smith, M.D.` on master) was taken in a process that had parsed other names first, which moves the absolute count and not the difference (the 2026-11-09 #1170 entry). The issue's proposed gate (append the run only beside another class word or behind a one-word head) was not taken: it sends such a part down the no-`maybe` path, equal only where the listing early return and C1's fallback agree, which nothing holds by construction. Checked against master: a grid of 215,280 parses (65 heads, from empty, nickname, title, particle and title-particle heads alone and in pairs to full and one-case names, × 46 tails of dotted runs, listed and ambiguous credentials, paired initials and their company × three comma spellings and a later part × the three orders × six policies) byte-identical, the issue's gate also identical there, and the harness shown to see a move (on the same grid, a `_lone_word` always true moves 34,848 rows, and skipping the count behind a one-word head too moves 4,536, `Smith van, M.D.` reading given 'M.', middle 'D.', family 'Smith van' where master reads given 'Smith', family 'van', suffix 'M.D.', `_lone_word` reading that head as one word while the count finds two); `snapshot.jsonl` byte-identical; and `snapshot.py --local ctgov-officials` and `ctgov-contacts` against master, no move. `test_a_run_of_initials_behind_a_full_name_takes_no_count` holds the saving, its control recorded in its docstring, and `test_the_count_c16_skips_decides_no_reading` (tests/v2/pipeline/test_comma.py) the readings a wrong gate moves. Under each of three mutants (the gate removed; the count skipped behind a one-word head too; `run_alone` forced true, so a run beside another entry of `maybe` such as `John Smith, M.D. M.A` skips the count as well) the frame test fails, and under the last two the reading test does too. A run beside `MA` (`John Smith, M.D. MA`) never joins `maybe`, `MA` being the run's company (C16), so it reaches none of this.
27 changes: 23 additions & 4 deletions nameparser/_pipeline/_comma.py
Original file line number Diff line number Diff line change
Expand Up @@ -762,11 +762,18 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState:
# rules/C16: "behind one word, initials run together are the given
# name whatever they spell" -- the runs that spell a listed
# credential ('J.P.', 'M.A.'), joined to the class so C4's count is
# taken (decisions/C16 (#1238)). Gathered only
# taken where the head may be one word (decisions/C16 (#1238),
# #1435). Gathered only
# where the part holds a full stop, so 'Smith, John' pays no frame;
# a part whose other words are all credentials is C8's run
# ('Smith, M.D. Ph.D.').
initials_at: int | None = None
# whether the run is the part's only entry in `maybe` (no word of
# the class and no caps-shape candidate beside it), and, asked only
# then, whether the head is one word as C16 reads it (`_lone_word`;
# None where not asked, the reading's own test asking it instead)
run_alone = False
lone_head: bool | None = None
for at, p in enumerate(p1):
if len(p) == 1 and "." in (text := texts[p[0]]):
tags = tokens[p[0]].tags
Expand Down Expand Up @@ -801,6 +808,7 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState:
for p, pt in rest):
initials_at = None
elif initials_at not in maybe:
run_alone = not maybe
maybe.append(initials_at)
# rules/C8: "a part that is nothing but suffix words is the
# credential run and reads as suffixes, whole" -- the reading below.
Expand Down Expand Up @@ -844,7 +852,17 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState:
# a pair a later comma part's credential spoke for (rules/C5), whose
# flip is reported as the count's, dotted or not
later = False
if maybe:
# C16's run alone in `maybe`, behind a head that is not one word,
# asks no count and keeps no books (#1435): the run reads as
# initials only behind one word, and with nothing else in `maybe`
# the count licenses nothing, `head_named` meets no member and the books find
# none, so the reading is the one the count would give ('John
# Smith, M.D.'). The run stays in `maybe`, so the part is still read
# below as the count's part, never the listing form's
if run_alone:
lone_head = _lone_word(state, seg0)
counted = bool(maybe) and lone_head is not False
if counted:
# `_members` inline: a frame here is paid by every name with the
# ambiguous class after its comma ('John Smith, MA')
split_at: set[int] = set()
Expand Down Expand Up @@ -966,7 +984,8 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState:
caps.discard(i)
if maybe:
lone = (initials_at is not None and names < 2
and _lone_word(state, seg0))
and (lone_head if lone_head is not None
else _lone_word(state, seg0)))
reading = segment_suffix_reading(
p1, pt1, tokens, pol.lenient_comma_suffixes, state.one_case,
anchored, absorbed, licensed, head_named=names >= 2,
Expand Down Expand Up @@ -1060,7 +1079,7 @@ def decide(state: ParseState, splits: bool | None = True) -> ParseState:
members: list[int] = []
listed: list[int] = []
settled = pair_only = False
if maybe:
if counted:
members = [p[0] for p in p1 if len(p) == 1
and AMBIGUOUS_ACRONYM_TAG in tokens[p[0]].tags]
listed = [i for i in members
Expand Down
29 changes: 29 additions & 0 deletions tests/v2/pipeline/test_comma.py
Original file line number Diff line number Diff line change
Expand Up @@ -690,3 +690,32 @@ def test_the_written_short_except_reads_past_a_credentials_own_case_mask() -> No
# condition, whatever the mask (#1395's third review)
got = parser.parse("Smith, PHD")
assert (got.given, got.suffix) == ("", "PHD")


@pytest.mark.parametrize("text, fields, kinds", [
# behind a full name: the count is skipped and the run reads as the
# credential it spells
("John Smith, M.D.", ("", "John", "Smith", "M.D."), ()),
# a head `_lone_word` reads as one word where the count finds two
# (a trailing lower-case particle): the count is still taken, and
# decides the reading. Recorded as it reads before and after #1435,
# not endorsed
("Smith van, M.D.", ("", "Smith", "van", "M.D."), ()),
("Smith van, J.P.", ("", "Smith", "van", "J.P."), ()),
# the run beside another entry of `maybe` (a dotted member not read
# as company): the count is taken for that member
("John Smith, M.D. M.A", ("", "John", "Smith", "M.D. M.A"), ()),
("John Smith, M.A M.A M.D.", ("", "John", "Smith", "M.A M.A M.D."),
("suffix-or-name",)),
])
def test_the_count_c16_skips_decides_no_reading(
text: str, fields: tuple[str, str, str, str],
kinds: tuple[str, ...]) -> None:
# #1435: `decide` skips C4's count only where C16's run is alone in
# `maybe` behind a head that is not one word. Each row is read the
# same with and without the skip; a gate that skipped the count
# behind a one-word head too ('Smith van'), or beside another
# member, moves them
got = parse(text)
assert (got.middle, got.given, got.family, got.suffix) == fields, text
assert tuple(str(a.kind) for a in got.ambiguities) == kinds, text
33 changes: 33 additions & 0 deletions tests/v2/test_benchmark.py
Original file line number Diff line number Diff line change
Expand Up @@ -1186,6 +1186,39 @@ def test_a_listing_comma_with_nothing_to_decide_skips_the_counts() -> None:
assert _frames_for("John Smith, MA", only="name_word_count") == 1


def test_a_run_of_initials_behind_a_full_name_takes_no_count() -> None:
"""rules/C16's run of initials reads as the given name only behind
one word, so where it is the part's only entry in `maybe` (no word
of the ambiguous class and no caps-shape candidate beside it) and
the head is not one word, `_comma.decide` asks C4's count nothing
(#1435). Recorded negative control (2026-11-18, py3.11, this
module's `_frames_for` in a fresh process): with the gate's
`lone_head is not False` arm deleted no output moves, and 'John
Smith, M.D.' enters `name_word_count` once and costs 23 more frames
(222 against 199; master before the gate, which did not ask
`_lone_word` there, 221)."""
if sys.getprofile() is not None:
pytest.skip("a profile hook is already installed; this test owns it")
only = "name_word_count"
for text in ("John Smith, M.D.", "Dr. Mr., M.D.", "Bin Zhang, J.P."):
# REACHABILITY: C16 found the run and asked about the head, so a
# zero below means the count was skipped rather than unreached
assert _frames_for(text, only="_lone_word") == 1, text
assert _frames_for(text, only=only) == 0, text
# behind one word the count is still taken, once, and `_lone_word`
# is asked once, not again for the reading
assert _frames_for("Smith, J.P.", only=only) == 1
assert _frames_for("Smith, J.P.", only="_lone_word") == 1
# a head `_lone_word` reads as one word where the count finds two
# still takes it
assert _frames_for("Smith van, M.D.", only=only) == 1
# another entry of `maybe` beside the run still takes the count, and
# `_lone_word` is not asked there (REACHABILITY: the run joined
# `maybe` behind a member, not reset as company, which 'M.D. MA' is)
assert _frames_for("John Smith, M.D. M.A", only=only) == 1
assert _frames_for("John Smith, M.D. M.A", only="_lone_word") == 0


def test_two_capitals_ask_the_later_parts_only_where_one_stands() -> None:
"""rules/C5's later credential (#1392) is read only where the pair
is the whole part and a part stands behind it, so a pair alone
Expand Down
Loading