diff --git a/AGENTS.md b/AGENTS.md index 8880729d..9b127180 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -78,6 +78,36 @@ not deferred (a 2026-08-16 sweep of eight weeks of specs recovered nine such items). The same-PR amendment rule above covers code-driven changes; this covers the design-driven ones. +**Triaging a design backlog: shape before value.** When a pile of +open questions has to be ordered, relatedness will not partition it — +in a parser nearly everything touches particles or suffixes. The line +that does is whether a decision changes WHAT THE MODEL CAN EXPRESS or +fills in a value in a shape already fixed. Settle the shape questions +first, as a batch: they are few, they are usually independent of each +other (so their own ordering does not matter), and each one collapses +or re-frames a run of the value questions below it. The rest are +leaves needing one measurement and one answer apiece, in any order. +Worked example, 2026-08-16 (#386): 30 open issues and ~13 open design +questions reduced to three shape decisions, which decided or +re-framed fourteen of them — and two of the three turned out smaller +than their issues claimed, because the issues had gone stale. Check +the age of an issue's premises before believing its cost estimate. + +**A count in a dated entry is evidence, not a live fact.** decisions.md +entries are snapshots by convention, so measurements belong in them — +but a reader wanting TODAY's number must not have to trust the +snapshot's date. Where an entry quotes something that drifts +(vocabulary sizes, corpus counts, set compositions), give the +one-liner that recomputes it, and phrase the argument so it survives +the digits moving — "the two shares differ by orders of magnitude" +outlives "58% vs 0.65%". A count that carries no argument is better +deleted than dated: "over every name in the corpus, no prefilter" +says what "over all 782 names" says, and cannot go stale. Do NOT +reach for a test asserting the count — that is the constant-content +pattern, and it fails on every legitimate vocabulary addition. #326 +is the cautionary case: it quoted a vocabulary composition, carried a +date, and was stale in five days. + **Primary-source review.** When doc content is distilled from a session's work, have that session (or its transcript) review its own sections before or soon after landing — attribution flattening @@ -86,6 +116,24 @@ state which tree each measurement ran on (stale fetches produced three rounds of already-fixed findings), and landed corrections are re-verified here before committing. +Two shapes, needing different instruments. CROSS-SESSION distillation +is what the rule above is written for — the #381 arc harvested nine +named sessions, and for each the source transcript is an artifact +independent of the reviewing session, so "review your own sections" +is a two-artifact comparison. SAME-SESSION design work (#386) has no +such artifact: spec and docs were written in one context, so the same +instruction degenerates into re-reading the working memory that +produced any error, and the rationale inverts — when the source is +NOW, the source is what got it wrong. There, compare against +artifacts instead, in this measured order of yield: re-derive every +number with a freshly written script (2 wrong claims on #386); +re-read the written spec against a checklist for FIDELITY, not just +presence (1); interrogate NAMED contested rule pairs one at a time +(2). Unaided prose reading found none — so a general "coherence read" +by the author is the one form to distrust, because a green report +from the weakest instrument manufactures confidence rather than +supplying it. + **Guard tests** SHOULD carry a recorded negative control — the answer with the guard off, stored as data (the _EXCLUSION_EFFECT shape; see mechanisms.md's Verification shapes). diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 9e27ac5f..c7f532a1 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -77,15 +77,9 @@ The counter-example set has since shrunk, and its shrinkage is the section's history: "Sir de Mesnil" fell to #367 (titles became transparent); "Juan de la Vega" under family-first — the whole chain in the given position — was called working-as-intended by -#359, but #368 SUPERSEDES that sentence: the recorded decision is -that the particle wins and a chain becomes the family name -whatever order was declared, so that case is now P1's tracked -deviation, not its boundary. The survivor is the degenerate bare -"de". The MIDDLE position is deliberately not a fold site — and -not merely unimplemented: the two family-first orders disagree -there ("Mesnil Garcia de" strands middle="de" under FAMILY_FIRST -and folds under FAMILY_FIRST_GIVEN_LAST, 464 measured inputs), -which is what makes #365 a decision rather than a gap. +#359, contested by #368, and is working-as-intended again as of +the 2026-08-16 entries below. The survivor is the degenerate bare +"de". - 2026-08 #359 — the opening site is read from joining structure (pieces), not from assigned roles, so the fold holds under every @@ -94,6 +88,71 @@ which is what makes #365 a decision rather than a gap. - 2026-08 #367 — titles are transparent to the fold: "Sir de Mesnil" now reads like "de Mesnil". Fixed by removing the title→particle chain in grouping, not by touching this rule. +- 2026-08-16 (order-precedence keystone; #364, #365, #368) — the + stage split is the decision, and the three issues are one + question seen from three angles. GROUPING is vocabulary's job and + is order-independent: a particle joins forward through + consecutive particles and stops at the first non-particle, and no + name_order moves that stopping point. ASSIGNMENT is name_order's + job: groups take roles by the declared order. The bugs existed + because the implementation runs the two in the opposite + dependency — `assign` hands out positions from `_effective_order` + and `post_rules` then inspects a fixed list of ROLES, so P1's + fold sites and P2's chain had coverage that varied with the + declared order by accident. + Consequences, each recorded in its own right below: the fold + takes only the particle's own group (#364); the middle position + needs no third site once grouping is order-independent (#365); + and #368 reverses. +- 2026-08-16 #364 — the fold takes the particle RUN and the ONE + name word it attaches to, not every remaining word. + "de Mesnil Juan" is family="de Mesnil" plus given="Juan". Run, + not particle: "de la Vega" is two particles onto one word, and an + earlier wording here said "the particle", which its own example + contradicted (rule-vs-decision-record review). Nothing ever argued for + "takes everything"; it was the shape of v1's + handle_non_first_name_prefix, not a decision. + Measured before deciding, over every name in the three + differential corpora with NO string prefilter: exactly ONE family + holds words beyond its particle's group — "de Mesnil Garcia". + #364's own body warns the change "breaks the v1 parity + tools/differential protects" and that "each ledger would need + re-examining"; measured, it is one name and one ledger entry. + A shape filter (no comma, leading never-given particle, three or + more words) returns THREE candidates, of which two do not move: + "de la Vega" is one group start to finish, and "de Mesnil Jr." + has only two name words because Jr. is a suffix. The pre-merge + fact-check caught this stated as "the filter gives one name", + which it does not. + Measurement trap, recorded because the fact-check fell into it + twice: the particle group runs through ANY particle, not only the + never-given ones — "de la Vega" chains never-given "de" through + AMBIGUOUS "la" onto "Vega". A detector that walks only the + never-given run splits the group after "de la" and reports 50 + false movers. +- 2026-08-16 #368 REVERSED — shipped behavior is correct. + "Juan de la Vega" under FAMILY_FIRST groups [Juan][de la Vega] + and assigns family="Juan", given="de la Vega". The earlier + decision ("the particle wins ... whatever order was declared") + was made before the grouping/assignment split was stated and + cannot survive it: a mid-name chain HAS a head word and is + positioned like any other group. What NON_GIVEN_NAME_PARTICLES + guarantees is that the bare word never reads as a given name, not + that no name part may begin with one. + The asymmetry with the leading case is P4's, not an exception + invented here: a leading particle chains nothing, so without the + fold pure position makes the BARE particle the given name. That + is measurable today on the ambiguous half, where no fold fires — + "van Mesnil Juan" gives given="van", middle="Mesnil", + family="Juan". given="de" is the reading the vocabulary exists to + forbid, and the fold is what prevents it. + The all-orders agreement in P1 is deliberate and is W4's shape: a + wholly-hangul name reads family="김" under every declared order + because the script carries a signal the order does not override, + and a leading never-given particle is the Latin-script analogue. + decisions.md#O4 already draws the line — "Words no vocabulary has + claimed read by position" — so name_order governs the unclaimed + remainder, which is most inputs. Declined: @@ -106,13 +165,20 @@ Declined: "St John Smith" into one given name and broke test_add_title (which adds "te", also a particle). The shipped predicate is "not a title or a prefix". - -Open: [#364](https://github.com/derek73/python-nameparser/issues/364) -how much the fold takes · -[#365](https://github.com/derek73/python-nameparser/issues/365) -should the middle position be a third site · -[#360](https://github.com/derek73/python-nameparser/issues/360) -which particles count as never-given. +- 2026-08-16 — deleting P4 so a leading particle chains and is then + positioned, which is the only way to make "de Mesnil Juan" vary + by declared order. It avoids given="de" (the group would be + [de Mesnil]) but breaks "de la Vega": measured, a single group is + positioned by the declared order — "Cher" reads given under + GIVEN_FIRST — so "de la Vega" would read given="de la Vega" + unless a further rule forced a particle-headed group into the + family. Add that rule and [de Mesnil][Juan] yields the #364 + reading anyway, so the deletion buys nothing and costs P4. + +Open: [#360](https://github.com/derek73/python-nameparser/issues/360) +which particles count as never-given (the criterion is settled at +decisions.md#vocabulary-collisions; the 39-member application is +not). ### P2 — particles join forward @@ -132,6 +198,94 @@ which particles count as never-given. the same shape with opposing wants, which is why the combined reading lives in the surnames VIEW and the split in the fields. +### P6 — the trailing orphan particle + +- 2026-08-16 (order-precedence keystone; #379, #380, #365) — a + particle ending the name has nothing to link forward to, and no + particle is a name by itself, so it attaches to the family name + standing beside it and renders BEFORE it. The distinction from a + chain is what makes this a rule rather than an exception: a + chained group has a head word and can be positioned; an orphan + has no head, so position has nothing to work with. +- Scope: the COMMA form only, deliberately. "Jong, Anke de" is + unambiguous — the comma has already named the family. Without the + comma the written shape is not settled: "Jong Anke de" may be a + misformatted listing (arguably a missing comma under a declared + family-first order) and "Jong de" may be a given name beside a + particle. Those keep their positional reading and are not tracked + as deviations. +- The words-to-spare guard is load-bearing, not incidental. #379's + own subject is "van", which is in the AMBIGUOUS half — so a rule + keyed to never-given particles alone would not fix the issue it + was filed for, while a rule with no guard breaks Vietnamese + "Nguyen, Van" (given="Van") by eating the only given word. The + guard is S2's shape, reused: consume only when the name has words + to spare. +- Rendering before the family has precedent in rules.md#R1 — folded + family words under O3 already "render before the rest of the + family wherever they stood in the string". The surnames view + renders backwards today ("Vega de la", "Jong de") and flips with + this rule. +- Measured under this rule's OWN comma scope, over 782 corpus names + (245 of them comma-bearing): exactly one moves, + "Vega, Juan de la" → family="de la Vega". A pre-merge fact-check + corrected an earlier count of three here — "Smith van der" and + "Sander van" have no comma, so the rule as scoped does not reach + them; they were measured before the comma scoping was chosen and + carried forward unfiltered. +- What that number means is the opposite of reassuring. The corpus + holds 245 comma names and exactly ONE with a trailing particle, + so it is very nearly blind to the shape this rule governs — + Dutch and Flemish listings are barely represented. Treat the one + as evidence about the corpus, not about the blast radius, and + run tools/differential at all three baselines in the implementing + PR regardless of how small this looks. +- 2026-08-16 (pre-merge coherence pass) — P6 and S2 CONTEST + "Berg, Jan vd": today S2 wins and reports suffix="vd", while + P6's marker asserts family="vd Berg". P6 wins, and the rule says + so in its statement rather than leaving the pair to file order. + The reason is C-ii's, not a new judgement: vd as the British + Volunteer Decoration is rarer than vd as van der, and a trailing + abbreviation AFTER A FAMILY COMMA is the tussenvoegsel position + specifically. + Scope check on the precedence, so it cannot creep: it reaches + only words that are both particle and suffix vocabulary, in the + trailing-orphan position, under a family comma, with a given word + to spare. "John Smith, PhD" and "Smith, Jr." are untouched (not + particles), and "Jong, vd" is untouched (no given word remains), + which is why that row stays out of scope rather than becoming a + counter-example. + Recorded because the pair was declared in `interacts:` and left + unresolved — `interacts:` is advisory by design and pins nothing, + so a declared interaction is a prompt to state the outcome, not a + substitute for stating it. +- 2026-08-16 — P6 is the first rule in rules.md that nothing + implements. Legitimate per the preamble (the document is + normative, and a gap is a tracked deviation), but it left the + rule pointing at nothing, so the shape got a pointer rather than + an exception: `tracked: #379, #380` in place of `implemented:`, + with exactly one of the two required of every rule. An + unimplemented rule can no longer sit here untracked, and a + shipped rule cannot keep a stale tracking pointer after its + issues close. Mutation-tested three ways before being believed + (drop the pointer, carry both, malformed refs); each fails. +- The RATIONALE first shipped here was wrong and is corrected: it + read "no particle is a name by itself", which is true only of the + never-given half — decisions.md#vocabulary-collisions says the + opposite in as many words ("most particles are short words that + double as names"). The error mattered rather than merely reading + badly: the words-to-spare guard exists BECAUSE the rule reaches + ambiguous particles (#379's own subject is "van"), so the + rationale undercut its own guard. Caught in review, after a + coherence pass that interrogated rule-vs-rule pairs and never + checked rule-vs-decision-record — which is the gap to close next + time, not a one-off. + +Open: [#380](https://github.com/derek73/python-nameparser/issues/380) +covers "Berg, Jan vd" under this rule, but the vd reading itself is +decisions.md#vocabulary-collisions (C-ii); and the no-given-word +case "Jong, vd" is deliberately unresolved — see the scope note. + ### M2 — the maiden-marker rule - 2026-07-03 (maiden-bucket design; #274 filed 2026-07-07, landed @@ -195,6 +349,122 @@ Declined: unambiguously). Documented side effect: parenthesized bare "(MA)"/"(DO)" no longer escape to suffix as in 1.x. +### vocabulary-collisions — when a word earns the ambiguous marking + +The mechanism shipped twice before anyone wrote down its criterion. +Sizes as of 2.2.0dev: particles 67 with 39 ambiguous (58%, +PARTICLE_OR_GIVEN); suffix_acronyms 613 with 4 ambiguous (0.65%, +SUFFIX_OR_NAME); titles 711 with no ambiguous subset and no +AmbiguityKind at all. Those counts are evidence from the decision +date, not live facts — they drift with every vocabulary addition. +The argument does not depend on the digits (it depends on the two +shares differing by orders of magnitude), but recompute before +quoting them: + + uv run python -c "from nameparser import Parser; L=Parser().lexicon; print({s: len(getattr(L,s)) for s in ('particles','particles_ambiguous','suffix_words','suffix_acronyms','suffix_acronyms_ambiguous','titles')})" + +- 2026-08-16 (collision keystone; #348, #360, #342, #385) — the + 58%-vs-0.65% gap is BASE RATE, not disagreement. Both sets apply + the same test; most particles are short words that double as + names (van, bin, le, do, bar, mac) while most credential acronyms + are not (abpp, acp). Recorded because the gap reads as an + inconsistency and is not one — a reviewer who "harmonizes" the + two shares will break one of them. +- **C-i, vocabulary vs. name.** A word belongs in its set's + ambiguous subset iff it is also borne as an ordinary name (given + or family) in some tradition. Under uncertainty, default to + AMBIGUOUS. This generalizes the evidence standard already stated + in NON_GIVEN_NAME_PARTICLES' docstring — a wrong unambiguous + claim misparses a real person, a wrong ambiguous marking only + adds a flag — from "which set" to "which subset". Applies + uniformly to particles, suffix_acronyms and titles. +- **C-ii, vocabulary vs. vocabulary.** Where two sets claim a word + and NEITHER reading is a name, precedence is a frequency judgment + recorded per word. vd is the live case: never-given particle AND + credential acronym (the British Volunteer Decoration), neither of + them a name. Decision: the Dutch van der reading, as the more + common. That is what unblocks #380, whose trailing-orphan half is + a separate decision recorded under its own rule. +- The concrete demonstration is "do", which three vocabularies + claim — titles, particles/ambiguous, suffix_acronyms/ambiguous. + Two mark it ambiguous; the third, TITLES, is the one that + actually decides "Do Quang Minh" (title="Do", given="Quang") and + reports nothing. Same word behind #385's "Anh Do". +- Applications, each still its own work: #360 (mc, ste — neither is + a borne given name, so both leave the ambiguous half); #342 (rai + — Rai IS a borne surname, so it earns the marking rather than + moving); #385 (do — resolved at decisions.md#R2). +- C-ii's per-word framing versus a rule stated for a SHAPE: measured + 2026-08-16, the words that are both particle and suffix + vocabulary are vd, do and mc — three, not the one this criterion + adjudicated. rules.md#P6 states its precedence for the shape, so + do and mc inherit vd's answer without being weighed. Recorded + rather than papered over: stating a per-word judgement as a + general clause is how an unexamined word acquires a decision, and + the two are named here so the next reader knows which one was + actually argued. +- Caution when applying C-i to the particle set: TITLES ∩ ambiguous + == {do, freiherr, st} is load-bearing, per the Excluded note in + the W2 section. Emptying it makes the particle-or-given emitter + dead code. + +Open: [#348](https://github.com/derek73/python-nameparser/issues/348) +applying C-i to the 711 title entries, then titles_ambiguous plus a +TITLE_OR_GIVEN kind. Blocked on data, not on judgement — the census +needs a given-name frequency corpus this repo does not have, which +is why the criterion is recorded here and the census is not +attempted. + +### suffix-field-composition — three kinds of thing in one field + +- 2026-08-16 (suffix keystone; #326) — measured composition of + suffix_words (40 entries): 11 generational (i, ii, iii, iv, v, + jr, jnr, sr, snr, junior, 2), 5 neither (dr, esq, esquire, ret, + vet), and 24 POSTNOMINAL HONORIFICS — 20 CJK (さん, さま, くん, + ちゃん, 様, 殿, 氏, 先生, 博士, 教授, 女士, 小姐, 씨, 양, 군, 님, + 박사, 박사님, 교수님, 선생님) and 4 Hebrew (ז"ל, ז״ל, שליט"א, + שליט״א). The honorifics are the LARGEST group. +- Decision: do NOT split the field. Record the composition; #296, + #291, #325 and #289 proceed on their own terms rather than + waiting on it. +- Why #326 cannot be taken at face value: it argues the split is + tractable because "the vocabulary is already split — + suffix_acronyms is credentials, entirely; suffix_words is + generational plus a handful". That was true when written + (2026-08-02); the 2.1.0 East Asian work landed 2026-08-07 and put + the 24 honorifics in the same set. CLDR's two buckets do not + cover what the set now holds, so "adopt CLDR's model" is not + available as the cheap answer. The issue's table is corrected on + the issue. +- The unexamined question this leaves: whether the honorific bucket + belongs in `suffix` at all. rules.md#W3 already argues an + honorific is "no part of the name on either side" — the same + language the H section uses for prenominal titles — so the + conflation with PhD may be worse than the generation/credentials + one #326 was filed about. Deliberately not decided here. +- Recorded as a documentation failure mode, not just a fact: + `suffix` started generational, absorbed credentials, then + absorbed postnominal honorifics, each step locally reasonable and + none recorded as a widening of the field's MEANING. rules.md#S2's + Background still calls it "two different things", which was + accurate when written. The rules doc pinned the behavior + faithfully; what slipped is the field's definition, which no rule + states because no rule owns it. Field definitions need the same + discipline rule statements get. +- The 11/5/24 split above is this entry's own version of the hazard + it describes: a quoted vocabulary composition, dated, exactly as + #326 quoted one. A date does not stop rot — #326 carried one too + — so before relying on the split, check the set still looks like + it: + + uv run python -c "from nameparser import Parser; print(sorted(Parser().lexicon.suffix_words))" + + What is durable here is the SHAPE of the finding — three kinds of + thing, the honorifics the largest — not the three integers. A + test asserting the counts is deliberately not the answer: that is + the constant-content pattern, and it would fail on every + legitimate vocabulary addition. + ### deviates-registry — packs stay pure data (option C) - 2026-07-18 (d4aaafa; the DEVIATES design note) — a `deviates` @@ -1041,11 +1311,34 @@ Declined: v2 core does not (parse("Anh Do").family_base == "", family_particles == "Do"), and the surname vanishes from initials (parse("Anh Do").initials() == "A."). - -Open: -[#385](https://github.com/derek73/python-nameparser/issues/385) -whether an all-particle family should have an empty base (three -options weighed in the issue). +- 2026-08-16 #385 RESOLVED by the collision criterion, not on its + own terms: the issue's option 3 ("guard the view only when a word + is vocabulary-ambiguous") is what + decisions.md#vocabulary-collisions produces when applied here. + "Do" is borne as an ordinary surname, so it is ambiguous + vocabulary and anchors the base; "van der" is never anyone's + name, so an all-particle family there genuinely has no base. The + two rows of the issue's table were never one case. + This is the keystone's clearest payoff: #385 was filed as a leaf + with three options and no way to choose between them, and the + criterion picks one without arguing about family_base at all. +- Still open inside the resolution: whether "Do" remains in + family_particles once it is also the base. Recorded here rather + than left to the implementing PR to decide by accident. +- 2026-08-16 (pre-merge coherence pass) — the resolution moves R3 + too, and R3 now carries its own marker. Initials read the BASE + family word, so anchoring "Do" changes + parse("Anh Do").initials() from "A." to "A. D." while + "Juan van der" stays "J." (no borne name, no base, and initials + of a bare particle run would be nonsense). + The general lesson, worth more than this instance: a deviates: + marker gets written on the rule whose STATEMENT changed, but a + rule can change another rule's OUTPUT without touching its + statement, and nothing looks for that — the runner asserts per + example line, so an unmarked downstream rule stays green + precisely because its own examples avoid the affected input. + When adding a marker, walk the changed rule's `interacts:` + targets and ask whether any of THEIR examples move. ### removed-v1-surface diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 495c9037..3f86b207 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -536,6 +536,21 @@ decline, not delete the test. down, and an unrecognized word is by definition outside the vocabulary — a green run over the corpus proves nothing about such a rule. +- The corpus can be near-blind to a WRITING CONVENTION even where + the vocabulary is well covered, and a small honest count then + reads as a small blast radius. Measured for rules.md#P6: 245 of + 782 corpus names carry a comma and exactly ONE ends in a + particle, so the Dutch trailing-tussenvoegsel listing the rule + exists for is essentially unsampled. Before reporting "N names + move", report the size of the population that COULD move; when + that is ~1, the number is evidence about the corpus. +- A detector that re-implements a rule's grouping will get the + grouping wrong. Derive the boundary from the same vocabulary the + rule reads, not from the half you happen to be thinking about: + walking a particle run over the NEVER-GIVEN set alone (the rule + chains through ANY particle) split "de la Vega" after "de la" + and reported 50 false movers for #364, where the true count is + one. Both wrong answers were plausible and printed cleanly. - Guard the whole family, parametrize over it: a defect on one of N parallel entry points hides behind a per-example test — three times in one session (a guard on one class of two, a decode hint diff --git a/docs/design/rules.md b/docs/design/rules.md index f5f4a36a..210d353d 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -22,7 +22,12 @@ decisions.md. Cross-references use the anchor form `decisions.md#P2` / `mechanisms.md#SPANS`; a bare ID is never a citation. The `interacts:` field on a pointer line is advisory — the citation-integrity test checks the ID exists, not that the -interaction is real. +interaction is real. `implemented:` and `tracked:` are not +advisory: every rule carries exactly one. `implemented:` names the +modules honoring the rule, checked against the modules that cite +it; `tracked:` names the issues that would ship a rule nothing +implements yet, so a wholly-aspirational rule cannot sit here +untracked, and a shipped rule cannot keep a stale tracking pointer. Every example line is EXECUTABLE. The grammar (its executable definition is `tests/v2/rules_doc.py`; `tests/v2/test_rules_doc.py` @@ -163,19 +168,33 @@ P1. Rationale: a never-given particle standing alone cannot be one as the given name, is a surname written out in full. A never-given particle standing alone where the given name would go — or opening the name — marks the name as surname-only: the - given and middle words fold into the family. It needs another - name word to fold into. An ambiguous particle keeps whatever - reading its position gives it. Whether the fold should stop at - the particle group instead of taking everything is open (#364). + particle run and the one name word it attaches to are the + family, and any name words beyond that read by position. It + needs another name word to attach to. The run is every particle + in sequence, never-given and ambiguous alike ("de la Vega" is + one group, not "de" plus a separate "la Vega"). An ambiguous particle keeps + whatever reading its position gives it. The reading holds under + every declared name order: a never-given particle is evidence + about how the name is written, and a declared order governs only + what no vocabulary has claimed (O4) — the same precedence the + script license takes in W4. "de la Vega" → family="de la Vega" "Sir de Mesnil" → family="de Mesnil" "Mesnil de" family-first → family="Mesnil de" - "Juan de la Vega" family-first → family="de la Vega" deviates: #368 (today: family="Juan") + "de Mesnil Juan" → family="de Mesnil" deviates: #364 (today: family="de Mesnil Juan") + "de Mesnil Juan" → given="Juan" deviates: #364 (today: given="") "van Gogh" → given="van" · boundary Accepted: a bare "de" stays the given name — there is nothing to fold into, and inventing a surname would be worse. "de" → given="de" - history: decisions.md#P1 · interacts: P2 · implemented: nameparser/_pipeline/_post_rules.py + Accepted: only the OPENING position is this rule's subject. A + particle chain standing inside the name is grouped normally (P2) + and positioned by the declared order, so a family-first reading + may report it as the given name; what the vocabulary forbids is + the bare particle reading as a given name, not any name part + that begins with one. + "Juan de la Vega" family-first → family="Juan" + history: decisions.md#P1 · interacts: P2, P4, P6 · implemented: nameparser/_pipeline/_post_rules.py P2. Rationale: a particle is written as part of the surname it precedes, and a title stands outside the name entirely. @@ -234,6 +253,41 @@ P5. Rationale: some given-name words are incomplete alone — "abdul" "mohamad ali smith" → given="mohamad" · boundary history: decisions.md#P5 · implemented: nameparser/_pipeline/_group.py +P6. Rationale: a particle ending the name has nothing to link + forward to, so it is not doing a particle's work there. A + never-given particle in that position cannot be a name at all + and must belong to the family written beside it; an ambiguous + particle could genuinely be the name (Vietnamese "Van"), which + is what the words-to-spare test below is for, not an + afterthought to it. Dutch and Flemish names are listed exactly + this way ("Beethoven, Ludwig van"), the tussenvoegsel trailing + the given name but belonging to the surname. + Where a family comma has already named the family, a particle + ending the name attaches to that family name and is written + before it — provided at least one given word remains, so that a + name whose only given word is the particle keeps it (the + words-to-spare test S2 applies to ambiguous suffixes). Where + the word is BOTH a particle and suffix vocabulary, this + attachment outranks the suffix reading (S2): a trailing + abbreviation after a family comma is the tussenvoegsel far more + often than the decoration it collides with. + "Jong, Anke de" → family="de Jong" deviates: #379 (today: family="Jong") + "Beethoven, Ludwig van" → family="van Beethoven" deviates: #379 (today: family="Beethoven") + "Berg, Jan vd" → family="vd Berg" deviates: #380 (today: family="Berg") + "Nguyen, Van" → given="Van" · boundary + Accepted: without a family comma the name's written shape is not + settled — "Jong Anke de" may be a misformatted listing, and a + bare "Jong de" may be a given name beside a particle — so the + attachment is scoped to the comma form, and the comma-less + shapes keep their positional reading. + "Jong Anke de" → family="de" + Accepted: the precedence over S2 is stated for the shape, so it + sweeps in every word that is both particle and suffix + vocabulary — today vd, do and mc. Only vd's reading was + weighed; the other two inherit it, which is the shape's cost + and is tracked with the other contested memberships. + history: decisions.md#P6 · interacts: C1, P1, S2 · tracked: #379, #380 + ## Suffixes: generational & credentials (S) Background: what follows a name is one of two different things — @@ -709,10 +763,13 @@ R2. Rationale: callers need the surname with and without its "Dr. Juan Q. Xavier de la Vega III" → family_base="Vega" "Dr. Juan Q. Xavier de la Vega III" → family_particles="de la" "Sean O'Connor" → family_base="O'Connor" · boundary - Accepted: an all-particle family reads an empty base today; - whether it should is #385. - "Anh Do" → family_base="" - history: decisions.md#R2 · implemented: nameparser/_types.py + A family name written wholly out of particle vocabulary still + has a base where one of those words is itself borne as an + ordinary surname: that word anchors the base, and only the words + that are never anyone's name stay particles. + "Anh Do" → family_base="Do" deviates: #385 (today: family_base="") + "Juan van der" → family_base="" + history: decisions.md#R2 · interacts: R3 · implemented: nameparser/_types.py R3. Rationale: initials abbreviate the person's name words; titles, suffixes, particles and nicknames are not name words. @@ -720,8 +777,14 @@ R3. Rationale: initials abbreviate the person's name words; titles, family word; titles, suffixes, particles and nicknames contribute nothing. "Dr. Juan Q. Xavier de la Vega III" → initials="J. Q. X. V." + "Anh Do" → initials="A. D." deviates: #385 (today: initials="A.") "Sean O'Connor" → initials="S. O." · boundary - implemented: nameparser/_render.py + Accepted: a family that is ALL particles contributes nothing, + so the initials are the given words alone — "van der" has no + borne name to anchor a base (R2), and initials of a bare + particle run would be nonsense. + "Juan van der" → initials="J." + history: decisions.md#R2 · interacts: R2 · implemented: nameparser/_render.py R4. Rationale: case repair is a display concern, applied only on request and never destructively. diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index cca18904..906db723 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -95,9 +95,18 @@ def post_rules(state: ParseState) -> ParseState: # rules.md#P1: "a never-given particle standing alone where the # given name would go — or opening the name — marks the name as - # surname-only: the given and middle words fold into the family. - # It needs another name word to fold into." (v1 - # handle_non_first_name_prefix; history: decisions.md#P1) + # surname-only: the particle run and the one name word it + # attaches to are the family, and any name words beyond that + # read by position." (v1 handle_non_first_name_prefix; history: + # decisions.md#P1) + # DEVIATION #364: the fold below still takes every remaining name + # word, not just the particle run's own -- de Mesnil Juan gives + # family=de Mesnil Juan where the rule says family=de Mesnil plus + # given=Juan. Pinned by the deviates: markers on P1. + # Values written unquoted deliberately: this note sits INSIDE the + # citation block above (# decisions.md#P1) does not close it -- + # _CITE_RE wants a colon after the ID), and the excerpt check + # takes the first quoted span in the block. # Code-local: a lone PIECE is the test at both sites, so a # particle group already chained forward is not a lone particle, # and rule H1 above cannot be what produces the fold's family diff --git a/tests/v2/rules_doc.py b/tests/v2/rules_doc.py index 76c9e1d1..cf2781e0 100644 --- a/tests/v2/rules_doc.py +++ b/tests/v2/rules_doc.py @@ -8,6 +8,10 @@ plus per-rule ``no-boundary: reason`` lines and the trailing pointer line ``history: ... · interacts: A1, B2 · implemented: path, path``. +A rule nothing implements yet carries ``tracked: #N, #M`` in place of +``implemented:`` — the issues that would ship it. Exactly one of the +two is required, so a normative rule always points either at the code +that honors it or at the work that will. Inside a rule block, any line whose first non-space character is a double quote (or an opening bracket, the D-section subject form) is an @@ -35,8 +39,9 @@ rf"(?P[a-z_]+)=(?P{_VALUE})\))?" r"\s*$") _NO_BOUNDARY_RE = re.compile(r"^\s*no-boundary:\s+(?P\S.*)$") -_POINTER_RE = re.compile(r"^\s*(history|interacts|implemented):") -_POINTER_PART_RE = re.compile(r"(history|interacts|implemented):\s*([^·]+)") +_POINTER_RE = re.compile(r"^\s*(history|interacts|implemented|tracked):") +_POINTER_PART_RE = re.compile( + r"(history|interacts|implemented|tracked):\s*([^·]+)") ASSERTABLE_FIELDS = frozenset({ "title", "given", "middle", "family", "suffix", "nickname", "maiden", @@ -64,6 +69,10 @@ class Rule: no_boundary: str | None = None interacts: tuple[str, ...] = () implemented: tuple[str, ...] = () + #: Issues that would ship a rule nothing implements yet. Mutually + #: exclusive with ``implemented:`` -- a rule points at code or at + #: the issues that will produce it, never at neither. + tracked: tuple[str, ...] = () def has_boundary_or_waiver(self) -> bool: return self.no_boundary is not None or any( @@ -184,4 +193,6 @@ def parse_rules_doc(text: str) -> list[Rule]: current.interacts = items elif key == "implemented": current.implemented = items + elif key == "tracked": + current.tracked = items return rules diff --git a/tests/v2/test_doc_citations.py b/tests/v2/test_doc_citations.py index 227b14e7..7153c45d 100644 --- a/tests/v2/test_doc_citations.py +++ b/tests/v2/test_doc_citations.py @@ -102,13 +102,23 @@ def test_implemented_matches_citing_modules() -> None: citing.setdefault(cid, set()).add(str(path.relative_to(REPO))) problems = [] for rule in parse_rules_doc(RULES_DOC.read_text(encoding="utf-8")): + actual = citing.get(rule.rule_id, set()) if rule.implemented: - actual = citing.get(rule.rule_id, set()) declared = set(rule.implemented) if actual != declared: problems.append( f"{rule.rule_id}: implemented: says {sorted(declared)} " f"but citations found in {sorted(actual)}") + elif rule.tracked and actual: + # The other half of test_rules_doc.py's exactly-one-pointer + # rule: that test cannot see code, so a rule that SHIPPED + # while keeping tracked: would pass it. Without this branch + # the stale pointer is invisible -- the loop above skips + # any rule with no implemented: at all. + problems.append( + f"{rule.rule_id}: declares tracked: {sorted(rule.tracked)} " + f"but code cites it in {sorted(actual)}; swap tracked: for " + f"implemented: now that something implements it") assert not problems, "\n".join(problems) diff --git a/tests/v2/test_rules_doc.py b/tests/v2/test_rules_doc.py index 48776256..83bc151e 100644 --- a/tests/v2/test_rules_doc.py +++ b/tests/v2/test_rules_doc.py @@ -8,6 +8,7 @@ from __future__ import annotations import importlib.util +import re import pytest @@ -32,6 +33,29 @@ def test_every_rule_has_examples_and_boundary(rule: Rule) -> None: f"'no-boundary: '") +@pytest.mark.parametrize("rule", RULES, ids=lambda r: r.rule_id) +def test_every_rule_points_at_code_or_at_the_work(rule: Rule) -> None: + """A normative rule names the code honoring it, or the issues that + would ship it -- never neither, so an unimplemented rule cannot sit + in the doc untracked, and never both. + + Scope, precisely: this test reads the DOC only, so it cannot tell + that a tracked: rule has since been implemented -- adding the code + and its citation while leaving tracked: in place passes here. That + half is test_doc_citations.py::test_implemented_matches_citing_ + modules, which sees the citing modules. Neither test alone makes a + stale pointer unrepresentable; the pair does. + """ + assert rule.implemented or rule.tracked, ( + f"{rule.rule_id}: add 'implemented: ' or, if nothing " + f"implements it yet, 'tracked: #N' naming the issues that would") + assert not (rule.implemented and rule.tracked), ( + f"{rule.rule_id}: has both implemented: and tracked:; drop " + f"tracked: once the rule ships") + bad = [t for t in rule.tracked if not re.fullmatch(r"#\d+", t)] + assert not bad, f"{rule.rule_id}: tracked: wants #N issue refs, got {bad}" + + def _check_diagnostic(example: Example) -> None: import re as _re from collections.abc import Callable