Skip to content

Skip C4's count for a run of initials behind a full name (#1435) - #1458

Merged
derek73 merged 3 commits into
masterfrom
fix/issue-1435-initials-run-count
Nov 18, 2026
Merged

derek73 merged 3 commits into
masterfrom
fix/issue-1435-initials-run-count

Conversation

@derek73

@derek73 derek73 commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Closes #1435

In _comma.decide, rules/C16's block put a dotted run of initials that spells a listed credential into maybe even behind a full name. So John Smith, M.D. paid C4's name_word_count and the C11 books, which only a one-word head reads the run by.

Where the run is the part's only entry in maybe (run_alone), decide now asks _lone_word first. Its answer is kept for the reading, so it is asked once. Behind a head that is not one word, decide skips the count and the books. The run stays in maybe, so the part is still read as the count's part and never takes the listing form's early return. This is why I did not take the issue's proposed gate: it sends the part down the no-maybe path, which matches only where the listing return and C1's fallback agree, and nothing guarantees that. With nothing else in maybe, the count licenses nothing, head_named meets no member, and the books find nothing. So the reading is the one the count would give.

Frames

One parse after a warm-up, in a fresh process, with test_benchmark._frames_for; master (b78d5af) → branch:

Name 3.11 3.15
John Smith, M.D. 221 → 199 203 → 185
John Smith, MD 190 → 190 178 → 178
Smith, John 136 → 136 125 → 125
Smith, J.P. (still counts) 233 → 233 218 → 218

The issue's 209 for John Smith, M.D. on master was taken in a process that had parsed other names first. That changes the absolute count, not the difference (decisions/parse-cost, 2026-11-09 #1170). The reference band holds; _CALL_BASELINE is not edited.

Evidence that no reading moves

  • tools/differential/snapshot.jsonl is byte-identical and untouched.
  • Grid: 215,280 parses, byte-identical between master and the branch (tokens, roles, tags, ambiguities, str(), initials()).
    • 65 heads: empty, nickname only, titles, particles, title-particles alone and in pairs (Freiherr St, St. Freiherr), one-word heads, Bin Zhang, de Waart, Smith van, one-case names and more.
    • × 46 tails: dotted runs, listed and ambiguous credentials, paired initials, the run's company, M.D.;, fullwidth.
    • × 4 comma spellings (, , ,, ,, and a later , PhD part) × the 3 name_orders × 6 policies (default, strict comma suffixes, dotted switch off, caps EVERYWHERE and OFF, an extra delimiter).
    • The harness can see a move. A planted _lone_word that always answers true moves 34,848 rows. Skipping the count behind a one-word head too moves 4,536: Smith van, M.D. reads given 'M.', middle 'D.', family 'Smith van', where master reads given 'Smith', family 'van', suffix 'M.D.'. The issue's own gate also moved 0 rows on this grid.
  • Local corpora against a master checkout, both "matches the base":
    • snapshot.py --local ctgov-officials --base <master>: 539,368 officials.
    • snapshot.py --local ctgov-contacts --base <master>: 522,688 contacts.

Tests

  • test_a_run_of_initials_behind_a_full_name_takes_no_count (tests/v2/test_benchmark.py) holds the saving with exact counts of name_word_count and _lone_word, with reachability probes. Its negative control is recorded: with the gate's lone_head is not False arm deleted, no reading moves and John Smith, M.D. costs 23 more frames. It passes alone with -n0, and under 3.15 in its own environment.
  • test_the_count_c16_skips_decides_no_reading (tests/v2/pipeline/test_comma.py) pins the readings a wrong gate moves. It records Smith van, M.D. as it reads today and does not endorse that reading.
  • I ran three mutants on copies of the tree:
    • gate removed: the frame test fails;
    • count skipped behind a one-word head too: both tests fail;
    • run_alone forced true: both tests fail.

docs/design/decisions/parse-cost.md gets a dated Log bullet; the baseline entry is not touched. There is no release-log line, following the earlier cost-only PRs (#1170, #1089, #1036).

Checks and reviews

  • uv run pytest (49,958 passed), uv run pytest -n0 -m serial, uv run mypy and uv run ruff check all pass.
  • pr-review-toolkit reviews run:
    • code-reviewer: tried to break the equivalence and could not.
    • comment-analyzer: found a reading attributed to master that was the mutant's, absolute frame figures that did not reproduce, and loose wording ("only word of the class"). All fixed.
    • pr-test-analyzer: found that the "another class word" row (M.D. MA) never reached the branch, because MA is the run's company. Replaced with M.D. M.A, and the reading pins were added.
    • code-reviewer on the fix commit: found two mutant move counts that did not reproduce, the one-case arm of the letter test left out of the log, and a stale C16 comment. All fixed in fe08159.

Follow-up worth a look, not touched here: master reads Smith van, M.D. as given 'Smith', family 'van', suffix 'M.D.', and John Smith, M.A M.D. as given 'M.', middle 'A', family 'John Smith'.

🤖 Generated with Claude Code

derek73 and others added 3 commits November 18, 2026 14:27
rules/C16's run of initials reads as the given name only behind one
word, so where it is the comma part's only word of the ambiguous class,
_comma.decide now asks _lone_word first and, behind a head that is not
one word, skips name_word_count and the C11 books. The run stays in
`maybe`, so the part is read as before and never takes the listing
form's early return. 'John Smith, M.D.' 209 -> 187 frames on 3.11
(191 -> 173 on 3.15); no reading moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t the log

The 'another word of the class' row used 'M.D. MA', where MA is the
run's company and the run never joins maybe; 'M.D. M.A' reaches the
branch. Pin the readings an over-wide gate moves ('Smith van, M.D.',
'John Smith, M.D. M.A') in test_comma.py, and correct the log bullet:
master's reading of 'Smith van, M.D.', the frame figures re-taken in
fresh processes with _frames_for, and 'only entry in maybe'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-take the grid's mutation counts (34,848 and 4,536 of 215,280 rows),
name the one-case arm of the run's letter test, and say that C16's
count is taken only where the head may be one word.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added this to the 3.2 milestone Nov 18, 2026
@derek73 derek73 added commas Commas and name structure (rules/C) internals Refactors and parse cost with no behavior change labels Nov 18, 2026
@derek73 derek73 self-assigned this Nov 18, 2026
@derek73
derek73 merged commit 688991e into master Nov 18, 2026
14 checks passed
@derek73
derek73 deleted the fix/issue-1435-initials-run-count branch November 18, 2026 23:51
derek73 added a commit that referenced this pull request Nov 19, 2026
Measured on the branch rebased onto 688991e (#1458's merge), each
interpreter in an environment of its own: parse 285.72 -> 197.72 on
3.11 and 277.72 -> 188.72 on 3.12 to 3.15, so _CALL_BASELINE moves
285 -> 197 and 277 -> 188; the 64-link run 2,404 -> 1,716 and
2,395 -> 1,706, so _LINK_BASELINE moves 2,418 -> 1,716 and
2,398 -> 1,706. decisions/parse-cost records the move with per-commit
and per-name figures, the clock, and the checks that no reading moved.

Closes #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 added a commit that referenced this pull request Nov 19, 2026
…1459)

* Test the emoji ranges in a loop rather than a generator

_is_emoji ran any() over a generator, a frame per range tried, for
every non-ASCII character under strip_emoji (the default). Frames per
parse on py3.11: reference row 285.72 -> 285.72, 'Smith, John' 136 ->
136, the long row 473.86 -> 473.86, a Devanagari name 336.04 -> 271.04.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Scan a region for its separators in C, not character by character

_tokenize_region asked _ignorable of every character of the name, a
frame each. The ignorable set is now a regex class per setting of the
two strip switches, built from the same parts (whitespace, the name
dots, the invisible separators, _BIDI's class, the emoji ranges), and
the region is scanned with finditer for the characters that may
separate: those, the commas and the transcription dots, whose context
tests are unchanged. _ignorable and _stripped match the same classes,
so extract and the flank guard read the one definition; a new test
holds each class to its parts over every code point, and
_is_emoji goes, its ranges now read only through the class.

Frames per parse on py3.11: reference row 285.72 -> 258.72,
'Smith, John' 136 -> 126, the long row 473.86 -> 421.86, a Devanagari
name 271.04 -> 226.04. Tokenize output for every policy over a grid of
the separator candidates in context, and _ignorable/_stripped over
every code point, compared byte-identical with master in two trees.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sort tokenize's tokens by an attrgetter key

The lambda key was a frame per token. Frames per parse on py3.11:
reference row 258.72 -> 252.72, 'Smith, John' 126 -> 124, the long row
421.86 -> 411.86.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Build classify's tokens from a list, not a generator

tuple() over a generator resumed it once per token, a frame each.
Frames per parse on py3.11: reference row 252.72 -> 246.72,
'Smith, John' 124 -> 122, the long row 411.86 -> 401.86.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Freeze group's pieces with map rather than generators

Each segment's pieces and piece tags were frozen through a generator,
a frame per piece twice over; map over the type does it in C.
Frames per parse on py3.11: reference row 246.72 -> 236.72,
'Smith, John' 122 -> 114, the long row 401.86 -> 383.86.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Read group's tail start by identity, not from an enum-keyed dict

The dict was built on every parse, and Enum.__hash__ is a Python
method: three frames a parse. Frames per parse on py3.11: reference
row 236.72 -> 233.72, 'Smith, John' 114 -> 111, the long row 383.86 ->
380.86.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Check a Token's span without a generator, and keep a Span as given

Token.__post_init__ tested the two coordinates with all() over a
generator, three frames a token, and rebuilt the span as a Span even
when it was one, a frame for the NamedTuple's __new__. The checks and
their messages are unchanged; a span that is not exactly a Span is
still rebuilt as one. Frames per parse on py3.11: reference row
233.72 -> 209.72, 'Smith, John' 111 -> 103, the long row 380.86 ->
340.86.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip the str and Mapping refusals for a Token's frozenset tags

A frozenset is neither, and isinstance against Mapping, an ABC, enters
a Python frame; every token the parse returns carries a frozenset.
Frames per parse on py3.11: reference row 209.72 -> 203.72,
'Smith, John' 103 -> 101, the long row 340.86 -> 330.86.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Coerce a Token's role only when it is not already a Role

_coerce_enum hands a Role back unchanged, a frame a token for nothing
on every token the parse builds; a string role is still coerced, and
refused, by it. Frames per parse on py3.11: reference row 203.72 ->
197.72, 'Smith, John' 101 -> 99, the long row 330.86 -> 320.86.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bring the tokenize comments in line with the scan, and test its edges

Review of the first nine commits: the _stripped and _ignorable
docstrings and the module docstring still described the per-character
call, and none named U+202F as the one ignorable character context
decides. The every-code-point test now matches each class whole, in C
(0.16s, was about 3.5s across four parameters), and says what it cannot
guard. New rows hold what no test reached: the dot's flanks and
U+202F's letters are read inside the region (each fails with the bound
dropped), and a Token's span is checked coordinate by coordinate when
it is already a Span (each fails with the check moved behind the Span
test).

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Check both class readers under every strip setting

Review of 8392358: its reader rows ran under the all-on and all-off
policies only, where a reader taking the two switches in the wrong
order reads right; under every setting that mutation now fails. The
module docstring's claim that extract and tokenize cannot disagree is
narrowed to the characters that separate in every context, the two
context decides being tokenize's alone. And a span with a bad start
and a good end gets its rows.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Re-take the call baselines and the link pin after #1436's cuts

Measured on the branch rebased onto 688991e (#1458's merge), each
interpreter in an environment of its own: parse 285.72 -> 197.72 on
3.11 and 277.72 -> 188.72 on 3.12 to 3.15, so _CALL_BASELINE moves
285 -> 197 and 277 -> 188; the 64-link run 2,404 -> 1,716 and
2,395 -> 1,706, so _LINK_BASELINE moves 2,418 -> 1,716 and
2,398 -> 1,706. decisions/parse-cost records the move with per-commit
and per-name figures, the clock, and the checks that no reading moved.

Closes #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Date the link pin's old band arithmetic, and pin the recipe's base

Review of f13950e: the comment on why the pin reads the long run
mixed the 2026-09-22 clause-run figures, given a wrong date, with the
name-link run's current one, and kept the old band in the present
tense beside the new; it now states the old arithmetic as measured
then and points at the pin for the current band. The decisions entry's
recipe compared against origin/master, which reads 0 once this merges,
and now names 688991e; and it names both pairs of trees the
114,647-reading grid compared, re-run after the rebase, identical.

Refs #1436

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commas Commas and name structure (rules/C) internals Refactors and parse cost with no behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

John Smith, M.D. costs 35 more frames to parse than in 3.0.1

1 participant