diff --git a/README.rst b/README.rst index 7f7119318..72ff022de 100644 --- a/README.rst +++ b/README.rst @@ -184,7 +184,7 @@ the patronymic in Russian. See `Locale packs >>> ivan.given, ivan.patronymic, ivan.family ('Иван', 'Петрович', 'Сидоров') >>> locales.available() # every pack that ships - ('ar', 'bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'he', 'hi', 'hr', 'ja', 'mn', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'vi', 'zh') + ('ar', 'bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'he', 'hi', 'hr', 'ja', 'mn', 'ne', 'nl', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'vi', 'zh') When a reading is wrong for your data ------------------------------------- diff --git a/docs/customize.rst b/docs/customize.rst index 439f20594..9cb311745 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -71,6 +71,8 @@ and names the ``remove`` above. Both calls return a new :class:`~nameparser.Lexicon`, the one you started from (here, ``Lexicon.default()``) never being mutated. +.. _adding-a-word: + Adding a word ^^^^^^^^^^^^^ @@ -87,19 +89,19 @@ list is one reading per word: 'LEED AP' A surname particle is added the same way. The defaults leave out the -Dutch ``ten`` of ``ten Brink``, which as a default would take a middle +Dutch ``in`` of ``in 't Veld``, which as a default would take a middle name spelled like it into the surname of names in every language; add it when your data is Dutch: .. doctest:: - >>> parse("Marieke ten Brink").middle, parse("Marieke ten Brink").family - ('ten', 'Brink') - >>> dutch_ten = Lexicon.default().add("ten", kind="preposition") - >>> Parser(lexicon=dutch_ten).parse("Marieke ten Brink").family - 'ten Brink' - >>> Parser(lexicon=dutch_ten).parse("Park Ten Hwan").family - 'Ten Hwan' + >>> parse("Marieke in 't Veld").middle, parse("Marieke in 't Veld").family + ('in', "'t Veld") + >>> dutch_in = Lexicon.default().add("in", kind="preposition") + >>> Parser(lexicon=dutch_in).parse("Marieke in 't Veld").family + "in 't Veld" + >>> Parser(lexicon=dutch_in).parse("Park In Hwan").family + 'In Hwan' The last line is the cost: an edit applies to every name the parser reads. A particle added this way holds no ``given`` reading, so @@ -157,7 +159,7 @@ named by language, which a reading you added leaves empty: ... print(van_reading.kind, van_reading.category, van_reading.languages) preposition particle ('nl',) given name ('nl',) - >>> dutch_ten.readings_of("ten")[0].languages + >>> dutch_in.readings_of("in")[0].languages () :meth:`~nameparser.Lexicon.words` goes the other way: it lists the diff --git a/docs/design/decisions/vocabulary-collisions.md b/docs/design/decisions/vocabulary-collisions.md index 7a5862a02..0a6f9a38b 100644 --- a/docs/design/decisions/vocabulary-collisions.md +++ b/docs/design/decisions/vocabulary-collisions.md @@ -28,6 +28,8 @@ Elsewhere: C-i reaches the titles through a `given` reading, which makes a title - `ait`, `aït` and `nait` (ber.toml) and `ouled` (ar.toml) are never-given clan particles on by default; `ould` (ar.toml) is a never-given patronymic on by default, filling the patronymic view in an Algerian fixed family name too; Latin `ben` is a clan particle written `default = false` in ar.toml and he.toml, which `locales.AR` and `locales.HE` turn on; Hebrew `בן` is a clan particle that may be given, `אבו` reads as `abu` and `אל` as `al`, each on by default; `בת` is unchanged — 2026-11-12 #1257. - Hebrew `בן` opening a family name written family first with no comma, under a family-first order, gives `בן` alone to the family, reported, a cost of its given reading accepted on #1209's measurement (8 Knesset family names, if written family first) — 2026-11-12 #1257. - `el` (ar.toml) is an article that may be given, as `al` is, on by default: a decided exception to the Latin-spelling half of the rule above, which it fails, on its fixed/broken table, Algeria's compound given names the accepted cost — 2026-11-12 #1257. +- `oude` (nl.toml) is a particle on by default, of the generic kind `particle`, no kind naming an adjective, that may be given (nam_dict's rare Dutch given name), with its case mask `Oude`: it passes rule 2 of the 2026-11-12 #1257 entry — 2026-11-18 #1455. +- `ten` (nl.toml) is a preposition written `default = false`, which `locales.NL`, a pack choosing `nl` alone, turns on; it fails rule 2's mid-name test (`Kheng Ten Choo`), and zh.toml declares the Chinese given-name syllable it opens, so a plain Dutch choice keeps a leading `Ten` the given name, reported, and a strict one reads it as the family name — 2026-11-18 #1455. ## Log @@ -89,3 +91,10 @@ Elsewhere: C-i reaches the titles through a `given` reading, which makes a title - **What moved**, the PR's head against master's readings rebuilt in the tree as `expr:Parser(lexicon=Lexicon.default().remove('ait','aït','nait','ouled', kind='clan').remove('ould', kind='patronymic').remove('el', kind='article').remove('el', kind='given').remove('בן', kind='clan').remove('בן', kind='given').add('בן', kind='patronymic').remove('אבו').remove('אל'))`, whose compiled sets equal those of master `161ebd28` and of the merge base `73718fd9` (36 sets compared, 2026-11-12): the committed snapshot gains eight rule examples (`git diff 73718fd9 HEAD -- tools/differential/corpus_rules.jsonl`) and moves three names, the recipe below counting the eleven, each built on `Zein El Dine` (Zayn al-Din, a compound given name also borne as a family name; `Mohamad Zein El Dine` → family 'El Dine', where it read 'Dine'), neither reading the whole name: unclear rather than wrong, #1378's shape; JRC-Names moves 3,483 of 1,315,051 spellings and opensanctions-parliaments 16 of 24,192 (`config_diff.py diff "expr:Parser(lexicon=...)" default plain:AR plain:HE --committed --local jrc-names --local opensanctions-parliaments`). Before `el` was read by default the same diff moved 287 JRC-Names spellings and 15 parliament names, the snapshot no name. Scored (`config_diff.py score "expr:Parser(lexicon=...)" default plain:AR plain:HE --group MA --group DZ --group TN --group MR --group EG --group LB --group IL --group SA --group FR --group BE --group NL --group ES --group GB --group US --group Moroccan --group Algerian --group Tunisian --group Mauritanian --group Egyptian --group Israeli --group French --group British`), the default fixes ORCID Morocco 2,275 to 34 broken, Lebanon 432 to 6, Egypt 286 to 20, Algeria 132 to 142, Tunisia 92 to 14 and Mauritania 38 to 6, PSC's Moroccan filers 452 to 16 and Algerian 5 to 11; `locales.AR` and `locales.HE` each fix ORCID Tunisia 1,056 to 17, Israel 164 to 4, Morocco 2,403 to 34, PSC's Tunisian filers 54 to 5, and PatentsView Israel 146 to 61, most of those breaks filers who typed `Ben` in the given box. Agreement in every box, ORCID: Tunisia 87.6% by default, 96.1% under either pack; Israel 92.7% and 93.5%; Morocco 84.0% on master, 95.5% by default, 96.1% under either pack. The CLDR locale view's `ar` and `he` files read alike with and without each pack (20/24, 18/24). - **Accepted:** a listing written family first without a comma reads whole as the family where a never-given `ait` or `ould` opens it (rules/P1); a family name written with Hebrew `בן` first and no comma under a family-first order gives `בן` alone to the family (`בן חיים יעל` → family 'בן', rules/P1; #1209 counted 8 Knesset family names written so), accepted the same day as a cost of the given reading; an Arab name transcribed in Hebrew script with `בן` for the father leaves the patronymic view empty (rules/R9); a family name written alone with `בן` reads given `בן` (#1209); `אל` inside a given name gives the article and the rest to the family; an Algerian compound given name holding `El`, or a bound given name before it, gives the article and the rest to the family (rules/P2); a choice of `ar` drops en's credential ABD for ar's bound given name `abd` (`Jane Smith, ABD` → given 'ABD' under `locales.AR`), which its docstring and docs/locales.rst say; the packs read a register-style chain's last `ben` into the family (`Karim ben Ali ben Salah Haddad` → family 'ben Salah Haddad'; #1248 measured Tunisia's register-style list at 140 of 158 right by default, 24 with `ben` read), being for surnames; under `strict=True` the Arabic pack reads a leading `Ben` as the family, English's given reading not chosen, and drops ber.toml's `ait` and `nait` unless `ber` is chosen too. - **Left out:** in #1378 (with its 2026-11-12 comment), Hebrew `עבד` and Latin `abd`, `abdel`, `abdul`, `abu` with rules/P12's `completed_by` for the names after the article, the Arabic compound given names behind `el` and `al` as rules/P10's `after_particle` words (`houda`, `amine`, `islam`, `din`, `zahraa`), and the Arabic-script `ايت` and `ولد`; and `N'Ait`, which #1249 lists and which reads today as one word, given "N'Ait" opening a name and a middle name inside one (`Leila N'Ait Haddou` → middle "N'Ait"); `בת` stays a never-given patronymic; `בר` stays excluded in he.toml (Bar, a common Israeli given name), rule 1 not applied to it here, the word not reopened. +- 2026-11-18 #1455 — DUTCH `OUDE` IS READ BY DEFAULT AND `TEN` UNDER A DUTCH CHOICE; `LOCALES.NL` SHIPS. Decided on the issue (its two 2026-11-18 decision comments), on the vetting comment there: `oude` a default particle with a case mask, `ten` an nl.toml particle written `default = false` with a `given` reading beside it, and a shipped pack. The builder's choices, open to review on the PR: `oude`'s kind, `oude`'s `given` reading, and which file holds `ten`'s, which decides what a strict Dutch choice reads. + - **`oude`.** Passes rule 2 of the 2026-11-12 #1257 entry: every family box holding it writes `Oude`, with a capital, never alone, and before another surname word in all but two, which end the box after the article (`DE Oude`, `De Oude`) (ORCID 10, PatentsView 8, PSC 1, by a scratch probe over `local_agreement.records` for the three fielded corpora, 2026-11-18), no given box opens with it, and no real name holds it as a middle given name. nam_dict lists `Oude` as a given name, band 1 (rare), in the Netherlands alone (`uv run python tools/name_usage.py oude`, 2026-11-18), so it carries a `given` reading under C-i's attested-bearer side and is a particle that may be given: opening a name it stays the given name and reports the fork (`Oude Luttikhuis` → given 'Oude', rules/P1's opening site, as `van`). Its kind is `particle`, the generic one: it is an adjective ("old", the Twente and Achterhoek farm names), and the README's particle kinds name what a word does in a name (a preposition, an article, a father, a lineage), none an adjective; a kind of its own was weighed for the class it heads (Low Saxon farm-name qualifiers, `Groot`, `Klein`, `Lütke`) and not built, none of those being vetted. Its `cased` mask keeps the capital case repair would take from a particle (`PIET OUDE LUTTIKHUIS` → `Piet Oude Luttikhuis`). + - **`ten`.** Fails rule 2's mid-name test: a Chinese given name writes the syllable between two words (`Kheng Ten Choo`), which the particle takes into the family name, and ORCID and PatentsView open given boxes with it 8 times (Singapore 4, Bhutan 2, Taiwan 2) where no Dutch filer means a given name. So `default = false` in nl.toml, the preposition (te den, as `ter` is te der), and the `given` reading the decision asks for in zh.toml, the file whose names carry it (no file holds Dzongkha), as `ben`'s given reading is in en.toml and `sushri`'s in or.toml, rather than in nl.toml, Dutch names bearing no given name `Ten`: the plain `nl` choice keeps a leading `Ten` the given name, reported, and the strict one, reading no zh.toml, reads it as the family name (`Ten Brink` → family 'Ten Brink'), so `ten` joins tests/v2/test_reading_index.py's reviewed table of the words strict ungates. It carries no case mask, taking a particle's lower case, which is how Dutch writes it after a given name (`Jan ten Brink`); a family box holding the surname alone is written both ways (opening such a box, ORCID NL `ten` 28 to `Ten` 3, PatentsView NL 8 to 20, PSC's Dutch filers 0 to 7). `locales.NL` chooses `nl` and nothing else, so its choice also drops en's credential VD for nl's `vd` (`John Smith VD` → family 'VD'), which its docstring and docs/languages.rst say. + - **What moved**, against master `688991e7`'s readings rebuilt in the tree as `expr:Parser(lexicon=Lexicon.default().remove('oude'))`, whose 36 compiled sets equal master's (2026-11-18; its case mask stays, which no compared field reads): `uv run python tools/differential/config_diff.py diff "expr:Parser(lexicon=Lexicon.default().remove('oude'))" default plain:NL strict:NL --committed --local all --match '(?i)(^|[\s,.(])(ten|oude|vd)([\s,.)]|$)'`, 370 names at the PR's head. The default moves 26, every one `oude` from the middle name into the family: 25 in the local corpora (the vetting's 22 and 3 in ctgov-contacts, a corpus fetched since, #1454) and 1 in the committed corpus, the rules/R4 example this change adds; no name holding `ten` moves by default. `plain:NL` moves 250 names and `strict:NL` 254, holding `ten` 206 and 208, `oude` 26 each, `vd` 18 and 20; the two read 34 names differently, 24 of them holding `ten`: a leading `Ten` (`Ten Kao Liu`) or one opening the given part after a family comma (`SWACKHAMER, TEN EYCK`, `Nick HT Hacken, ten, MD, PhD`) reads as the family under strict, a `ten` ending the given part after a family comma is reported under plain only, and one name differs by a report on its credentials alone. The committed snapshot moves no name it held before; the rules corpus gains the examples (rules/P2, rules/R4); the CLDR default records do not move, and the locale view's `nl` file reads 22/24 with and without the pack. `uv run python tools/differential/suggest_measure.py --base ` moves no cell. `uv run python tools/differential/match_diff.py --base ` moved none of 690,382 rows before tests/v2/test_match.py held a pair with `oude`; with its two pairs, 1,254 of 690,426 rows move (57 under each of the 22 settings), every one a pair holding `Piet Oude Luttikhuis`, whose keys gain `oudeluttikhuis;p` and whose middle name, `Oude` on master, is now empty: `Piet Oude Luttikhuis` and `Piet Oudeluttikhuis` are consistent, the family joined, where master read a conflict, and against `Piet Luttikhuis` the family sets the particle aside, which a caller who drops `particles-aside` now reads as a conflict where master read the pair consistent (`Oude` a middle name), as `Jan van Berg` against `Jan Berg` already reads; no consistent pair shares no key. + - **Scored** against the filed boxes, `uv run python tools/differential/config_diff.py score "expr:Parser(lexicon=Lexicon.default().remove('oude'))" default plain:NL strict:NL --cap 20000`, ORCID and PatentsView by country and PSC by nationality, the 434 groups of 100 records or more, each capped at 20,000 (2026-11-18): the default fixes 16 to 1 broken (ORCID Netherlands 10, PatentsView Netherlands 5 and Germany 1; the break a PatentsView filer who typed `Oude` at the end of the given box), the vetting's figure. `plain:NL` fixes 102 to 10, so `ten` under the pack 86 to 9: the Netherlands and its filers 66 to 4 (ORCID 31 to 0, PatentsView 28 to 4, PSC's Dutch 7 to 0), Belgium 2 to 0, elsewhere 18 to 5. Read one by one with `local_agreement.read` (a scratch probe, the same day), two of the nine breaks are the mid-name syllable (PatentsView Malaysia `Kheng Ten Choo` and Singapore `Terence Ten Teck Hong`), the other seven filers who typed `ten` at the end of the given box (`Nils Ten` / `Hoeve`). The 20 fixes outside the Netherlands are the Dutch-surnamed people the decision gives up for the default caller. `uv run python tools/differential/config_diff.py gaps nl` (the same day) reads the strict choice against the plain one over the Dutch-language groups: 456 broken to 54 fixed, other languages' Latin particles dropped (`el`, `di`, `al`, `le`, `da`), `ten` not among its twenty words. + - **Accepted:** under the pack, a Chinese given name holding `Ten` between two words gives it and the rest to the family (rules/P2's Accepted); a surname-only `Oude Luttikhuis` reads given 'Oude', reported; under a strict Dutch choice a leading `Ten` is the family (`Ten Kao Liu` → family 'Ten Kao Liu'), and so is one opening the given part after a family comma (`Brink, Ten Jan` → family 'Brink Ten Jan', `, Ten Brink` → family 'Ten Brink'), the given name lost; a family name of two Dutch names whose second opens with `ten` keeps only the second under the pack (`Jutte van der Werff ten Bosch` → middle 'van der Werff'), as a second name opening with `ter` does by default; the pack drops the credential VD (`John Smith VD` → family 'VD', and after a comma the whole name, `John Smith, VD` → family 'John Smith VD'). + - **Left out:** a kind for Dutch and Low German farm-name qualifiers, and the other qualifiers themselves; `Ten` as a Korean (Koryo-saram), Chinese and Turkish family name written alone, which no reading moves; Dutch `in` (`in 't Veld`), which docs/customize.rst now uses as its example of an unlisted particle; a possibly-given particle opening a name before a suffix comma folds into the family unreported (`Ten Smith, Jr.` under the pack, `Van Smith, Jr.` by default), a shape not settled here. diff --git a/docs/design/rules/P2-particles-join-forward.md b/docs/design/rules/P2-particles-join-forward.md index 990fa33ac..74da4db1f 100644 --- a/docs/design/rules/P2-particles-join-forward.md +++ b/docs/design/rules/P2-particles-join-forward.md @@ -85,6 +85,11 @@ "Abd El Kader Haddou" → family="El Kader Haddou" "Abu El Hassan Haddou" → given="Abu" ``` +- Under a Dutch choice `ten` is a particle (te den), and a Chinese given name written with its syllable between two words joins the family from it, as the Korean and Vietnamese syllables PR #191 kept out of the particles would: the mid-name cost that keeps the preposition off by default, two of the nine filed names the pack breaks among ORCID's, PatentsView's and PSC's records, against 86 it fixes (2026-11-18; decisions/vocabulary-collisions, #1455). The default reads `ten` as a name word. + ``` + "Kheng Ten Choo" [nl] → family="Ten Choo" + "Kheng Ten Choo" → family="Choo" · boundary + ``` - A caller wanting the combined double-surname reading (#132's ask) has it as the surnames view rather than the family field. ``` "Vincent van Gogh van Beethoven" → surnames="van Gogh van Beethoven" diff --git a/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md b/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md index 02d65d71c..5f0c4ed85 100644 --- a/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md +++ b/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md @@ -56,6 +56,7 @@ - **A particle is written in lowercase**, but a particle the vocabulary records a casing for takes that casing, the entry being how the word is written wherever it stands, and a particle of a kind written with a capital is written as a name word is: a saint's name (`St Clair`, `San Martín`) and a rank written inside the surname (`Freiherr von Richthofen`) take their capital in every convention that writes them. So is a particle opening a hyphenated word, which the hyphen joins to the name after it (`Ter-Petrosyan`, `Al-Ahmari`): it is written as a name word is, while a particle standing apart keeps its lowercase, save one R12 gives a capital as it opens the family name (`Ter Petrosyan`). ``` "SEÁN Ó MURCHÚ" → capitalized="Seán Ó Murchú" + "PIET OUDE LUTTIKHUIS" → capitalized="Piet Oude Luttikhuis" "IVOI, PAUL D'" → capitalized="Paul d' Ivoi" "zaʿīm, ḥusnī az-" → capitalized="Ḥusnī az- Zaʿīm" "jacobus henricus van 't hoff" → capitalized="Jacobus Henricus van 't Hoff" diff --git a/docs/languages.rst b/docs/languages.rst index 6888e8bcd..4debde7b8 100644 --- a/docs/languages.rst +++ b/docs/languages.rst @@ -18,8 +18,8 @@ subset with its own parser, or pass several packs to ``parser_for``: The sections below cover Czech, Slovak, Slovenian and Croatian names, Spanish and Portuguese titles and family names, romanized Indic titles, -Vietnamese, Mongolian and Thai names, and Maghrebi, Amazigh and Hebrew -particles. Other languages are described where their mechanism is: +Vietnamese, Mongolian and Thai names, and Maghrebi, Amazigh, Hebrew and +Dutch particles. Other languages are described where their mechanism is: - **East Slavic patronymics** (``ru``): :ref:`east-slavic-patronymics`. - **Turkic patronymic markers** (``tr_az``): @@ -1089,3 +1089,91 @@ family box, the names agreeing in both boxes rise from 87.6% by default to 96.1% under either pack in Tunisia, and from 92.7% to 93.5% in Israel. The dated figures are in ``docs/design/decisions/vocabulary-collisions.md``. + +.. _dutch-particles: + +Dutch particles +--------------- + +Turn on ``locales.NL`` when your data is Dutch or Flemish names written +with ``ten``, the one particle the pack adds. The other Dutch particles +the vocabulary lists are read by default; one it does not list, such as +the ``in`` of ``in 't Veld``, is added as :ref:`adding-a-word` shows. + +Read by default: ``van``, ``de``, ``ter``, ``Oude`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A Dutch family name often opens with prepositions and articles written +in lower case (``van der Berg``, ``ter Horst``, ``'t Hooft``). The +default reads them as part of the family name, and reads ``Oude`` +("old") the same way, which opens the farm names of Twente and the +Achterhoek and is written with a capital (``Oude Luttikhuis``): + +.. doctest:: + + >>> parse("Jan van der Berg").family + 'van der Berg' + >>> luttikhuis = parse("PIET OUDE LUTTIKHUIS") + >>> luttikhuis.family, str(luttikhuis.capitalized()) + ('OUDE LUTTIKHUIS', 'Piet Oude Luttikhuis') + +``Oude`` is also a rare Dutch given name, so where it opens a name it +stays the given name and reports the fork, as ``van`` does. + +``ten``: the Dutch pack +~~~~~~~~~~~~~~~~~~~~~~~ + +``ten`` ("at the") opens family names the same way (``ten Brink``), but +it is also a syllable of Chinese given names, written between two words +(``Kheng Ten Choo``), where the particle would take it into the family +name. So the default leaves it to position, and ``locales.NL``, which +chooses Dutch, reads it as the start of the family name: + +.. doctest:: + + >>> parse("Jan ten Brink").family + 'Brink' + >>> dutch = parser_for(locales.NL) + >>> dutch.parse("Jan ten Brink").family + 'ten Brink' + >>> dutch.parse("Ten Brink").given + 'Ten' + +Under the pack, a ``Ten`` opening a name stays the given name and +reports the fork, save under a strict Dutch choice (``strict=True``), which drops the +Chinese given name and reads a leading ``Ten`` as the family name. + +What the pack costs +~~~~~~~~~~~~~~~~~~~ + +- **A Chinese given-name syllable inside a name.** ``Kheng Ten Choo`` + gives family ``Ten Choo`` under the pack. Among ORCID's, + PatentsView's and the UK company register's records, ``ten`` read as + a particle fixes 86 names for 9 it breaks, 66 to 4 among Dutch + records, and two of the breaks are names of this shape + (``docs/design/decisions/vocabulary-collisions.md``, :issue:`1455`). +- **Two particle runs.** A family name of two Dutch names, each opening + with a particle, keeps only the second as the family name when the + second opens with ``ten`` (``Jutte van der Werff ten Bosch`` gives + middle ``van der Werff``, family ``ten Bosch``), as one whose second + name opens with ``ter`` does by default. +- **The credential VD.** Choosing Dutch costs English's reading of + ``vd``, which Dutch reads as ``van der`` written short, so the + credential VD (the Volunteer Decoration) is a name word under + ``locales.NL``: ``John Smith VD`` gives family ``VD``, and ``John + Smith, VD`` reads whole as the family name. Choose ``en`` too to keep + it. +- **A leading Ten, under a strict Dutch choice.** A strict Dutch + choice drops the Chinese given name, so a ``Ten`` opening the name, + or opening the given name after a family comma, joins the family + (``Brink, Ten Jan`` gives family ``Brink Ten Jan``). +- **Other languages' particles, under a strict Dutch choice.** A + strict Dutch choice drops the particles of other languages written in + Latin letters (``Maria da Silva`` gives middle ``da``). + +.. doctest:: + + >>> english = Locale(code="en", lexicon=Lexicon.empty(), + ... languages=frozenset({"en"})) + >>> parser_for(locales.NL, english).parse("John Smith VD").suffix + 'VD' diff --git a/docs/locales.rst b/docs/locales.rst index df2a31d31..2e22cf3ce 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -182,7 +182,7 @@ look one up dynamically by its lowercase code with .. doctest:: >>> locales.available() - ('ar', 'bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'he', 'hi', 'hr', 'ja', 'mn', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'vi', 'zh') + ('ar', 'bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'he', 'hi', 'hr', 'ja', 'mn', 'ne', 'nl', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'vi', 'zh') >>> locales.get("ru") is locales.RU True @@ -300,6 +300,11 @@ says which parts each pack is made of. on ``initials_as_family``. See :ref:`nepali-given-names`. - | ``श्री कृष्ण श्रेष्ठ`` → given ``श्री`` | ``Ram Bahadur B K`` → family ``B K`` + * - ``nl`` + - Dutch: family names opening with ``ten`` + - Chooses ``nl``, which reads ``ten`` inside a name as the particle + opening the family name. See :ref:`dutch-particles`. + - ``Jan ten Brink`` → family ``ten Brink`` * - ``pt`` - Portuguese - Chooses ``pt``, which settles Portuguese's clashes its way @@ -365,8 +370,8 @@ says what it can change: - Chooses a language - Policy fields - Word list - * - ``ar``, ``bn``, ``ga``, ``gd``, ``he``, ``hi``, ``hr``, ``pt``, - ``pt_br``, ``sl`` + * - ``ar``, ``bn``, ``ga``, ``gd``, ``he``, ``hi``, ``hr``, ``nl``, + ``pt``, ``pt_br``, ``sl`` - yes - none - none diff --git a/docs/migrate.rst b/docs/migrate.rst index cd068e692..9f6f6dd43 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -4,7 +4,7 @@ Migrating to 3.0 nameparser 3.0 removes the 1.x compatibility layer: ``HumanName``, ``Constants`` and everything around them, which 2.x kept working on top of the core parser and 2.8 deprecated. This page is for code moving to -3.0 or 3.1 from 1.x or from any 2.x release, whether it called +3.0, 3.1 or 3.2 from 1.x or from any 2.x release, whether it called ``HumanName`` or the 2.x core API. It says what is gone and what replaces each piece, one row per old name, and lists the names whose reading changed, release by release. @@ -40,11 +40,11 @@ Which part is yours applies * - edits ``CONSTANTS`` or builds a ``Constants`` - the :ref:`migrate-config-map` - * - uses the 2.x ``Lexicon`` and ``Policy`` and is going to 3.0 or - 3.1 + * - uses the 2.x ``Lexicon`` and ``Policy`` and is going to 3.0, + 3.1 or 3.2 - :ref:`migrate-lexicon-readings`, :ref:`migrate-constants`, - :ref:`migrate-3-0` and :ref:`migrate-3-1`; the attribute map - and the ``Constants`` tables are not yours + :ref:`migrate-3-0`, :ref:`migrate-3-1` and :ref:`migrate-3-2`; + the attribute map and the ``Constants`` tables are not yours * - compares names with ``==`` - :ref:`migrate-comparison` * - pickles parsers, lexicons or parsed names @@ -857,10 +857,10 @@ Behavior changes ----------------- Beyond the API surface mapped above, parse *outputs* differ between 1.4 -and 3.1 for specific input shapes. The ones listed here, release by +and 3.2 for specific input shapes. The ones listed here, release by release, are the shapes worth checking your fixtures for; the full list, with reasoning, is in each release's section of -:doc:`release_log`, the 3.0 and 3.1 ones included. +:doc:`release_log`, the 3.0, 3.1 and 3.2 ones included. .. list-table:: :header-rows: 1 @@ -908,13 +908,16 @@ list, with reasoning, is in each release's section of scripts, case repair, titles and particles in other scripts, ``Md``, Thai names, credentials after a comma - :ref:`migrate-3-1` + * - 3.2 + - Dutch ``Oude`` + - :ref:`migrate-3-2` Each entry says what it compares with. The lists of readings through 3.0 give what 1.4 read, so they are written for code coming from 1.x: a release between may have read the name a third way, which the release log gives. 3.0's rendering and initials sections compare with 2.8, and -each 3.1 entry names the releases that read the name the old way. Coming -from 2.8, read the 3.0 and 3.1 lists, and the release log's 3.0 section +each 3.1 and 3.2 entry names the releases that read the name the old way. +Coming from 2.8, read the 3.0, 3.1 and 3.2 lists, and the release log's 3.0 section too: a reading that a 2.x release moved and 3.0 moved back to 1.4's is in the log and not in these lists. @@ -2127,3 +2130,30 @@ is not run here: >>> ja = parser_for(locales.JA, segmenter=locales.ja_segmenter()) >>> ja.parse("あいさわいちろう").family 'あいさわいちろう' + +.. _migrate-3-2: + +Changed in 3.2 +~~~~~~~~~~~~~~ + +Each entry names the releases that read the name the old way. + +Dutch Oude +^^^^^^^^^^ + +``Oude`` starts the family name it opens, where 1.4 through 3.1 left it +in the middle name, after a family comma too (``Luttikhuis, Piet +Oude``); opening a name it stays the given name, as before, and now +reports the fork, as ``van`` does: + +.. doctest:: + + >>> parse("Piet Oude Luttikhuis").family + 'Oude Luttikhuis' + >>> oude_first = parse("Oude Luttikhuis") + >>> oude_first.given, [str(report.kind) for report in oude_first.ambiguities] + ('Oude', ['particle-or-given']) + +``locales.NL``, new in 3.2, reads ``ten`` the same way (``Jan ten +Brink`` gives family ``ten Brink``); the default parse of a name holding +``ten`` is unchanged. diff --git a/docs/release_log.rst b/docs/release_log.rst index 74f051242..900a55df9 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -4,6 +4,9 @@ Release Log nameparser 3.2 is under development. + - **Fix Dutch Oude, which read as a middle name.** ``parse("Piet Oude Luttikhuis")`` gives family ``Oude Luttikhuis``, where 1.4.0 through 3.1.0 gave middle ``Oude``, family ``Luttikhuis``; ``Oude`` opening a name stays the given name, reported, being a rare Dutch given name too. :issue:`1455` + - **Add locales.NL, which reads ten as a family-name particle.** ``parser_for(locales.NL).parse("Jan ten Brink")`` gives family ``ten Brink``, where the default, as 1.4.0 through 3.1.0, gives middle ``ten``; a ``Ten`` opening a name stays the given name, reported, save under ``strict=True``. It also takes a Chinese given-name syllable inside a name into the family (``Kheng Ten Choo`` gives family ``Ten Choo``), and choosing Dutch reads ``VD`` as ``van der`` rather than the credential (``John Smith, VD`` gives family ``John Smith VD``). See :ref:`dutch-particles`. :issue:`1455` + * 3.1.0 - November 16, 2026 **nameparser 3.1** introduces two new tools for working with lists of names: one for configuration and one for matching. ``Parser.suggest()`` suggests the ``Policy`` settings for the list of names you give it, and ``Parser.matching()`` provides tools for joining records from different sources. It also reads family names written in capitals (``DUPONT Jean``), Thai titles, and a given name written as a run of initials, and adds Mongolian, Arabic and Hebrew packs. :doc:`migrate` lists the names whose reading moved, and which pickles load where. Highlights: diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index efc080927..40a0f7fc5 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -2358,6 +2358,7 @@ 'nait', 'ní', 'op', + 'oude', 'ould', 'ouled', 'prinz', @@ -3767,6 +3768,7 @@ 'mvdr': 'MVDr', 'nait': 'Nait', 'ní': 'Ní', + 'oude': 'Oude', 'ould': 'Ould', 'ouled': 'Ouled', 'paeddr': 'PaedDr', diff --git a/nameparser/config/_index.py b/nameparser/config/_index.py index 280d55d4f..6b1a88652 100644 --- a/nameparser/config/_index.py +++ b/nameparser/config/_index.py @@ -2093,6 +2093,7 @@ ('otr', (('en', 'credential', ()),)), ('otr/l', (('en', 'credential', ()),)), ('otrl', (('en', 'credential', (('same_as', 'otr/l'),)),)), + ('oude', (('nl', 'particle', (('cased', 'Oude'),)), ('nl', 'given', ()))), ('ould', (('ar', 'patronymic', (('cased', 'Ould'),)),)), ('ouled', (('ar', 'clan', (('cased', 'Ouled'),)),)), ('owura', (('ak', 'salutation', (('after_given', True),)),)), @@ -2614,6 +2615,7 @@ ('teddy', (('en', 'familiar', (('short_for', ('theodore',)),)),)), ('television', (('en', 'modifier', ()),)), ('temi', (('en', 'familiar', (('short_for', ('temitope',)),)),)), + ('ten', (('nl', 'preposition', (('default', False),)), ('zh', 'given', ()))), ('tenor', (('en', 'occupation', ()),)), ('ter', (('nl', 'preposition', (('capital_opening', True),)),)), ('teri', (('en', 'familiar', (('short_for', ('teresa', 'theresa')),)),)), diff --git a/nameparser/locales/__init__.py b/nameparser/locales/__init__.py index 89a2032f6..83d1012ac 100644 --- a/nameparser/locales/__init__.py +++ b/nameparser/locales/__init__.py @@ -36,6 +36,7 @@ "JA": ("nameparser.locales.ja", "JA"), "MN": ("nameparser.locales.mn", "MN"), "NE": ("nameparser.locales.ne", "NE"), + "NL": ("nameparser.locales.nl", "NL"), "PT": ("nameparser.locales.pt", "PT"), "PT_BR": ("nameparser.locales.pt_br", "PT_BR"), "RU": ("nameparser.locales.ru", "RU"), diff --git a/nameparser/locales/nl.py b/nameparser/locales/nl.py new file mode 100644 index 000000000..4a92058e8 --- /dev/null +++ b/nameparser/locales/nl.py @@ -0,0 +1,42 @@ +"""The Dutch locale pack: a choice of languages, ``nl``, and nothing +else -- no lexicon fragment and no policy patch +(mechanisms.md#LOCALE-PACKS-PURE-DATA; #1455). Choosing it turns on +nameparser/vocab/nl.toml's readings written ``default = false``: the +preposition ``ten`` (te den), so ``Jan ten Brink`` gives family +``ten Brink`` where the default gives middle ``ten``, family ``Brink``. +The default leaves ``ten`` to position because it is also a syllable of +Chinese given names, written between two words (``Kheng Ten Choo``), +which the particle would take into the family name under the pack too +(decisions/vocabulary-collisions, 2026-11-18 #1455). A ``Ten`` opening +a name stays the given name under the pack and reports the fork +(``Ten Kao Liu``), zh.toml declaring the given name, save under +``strict=True``, where zh.toml is not chosen and a leading ``Ten`` +reads as the family name (``Ten Brink`` → family ``Ten Brink``). + +The Dutch particles read by default (``van``, ``de``, ``ter``, ``oude`` +and the rest) read the same with or without the pack. + +On a word nl.toml holds a vocabulary reading of, other languages' +vocabulary readings leave: ``vd`` (van der, abbreviated) is then +Dutch's particle alone and no longer English's credential VD (the +Volunteer Decoration), so ``John Smith VD`` gives middle ``Smith``, +family ``VD`` where the default gives suffix ``VD``, and ``John Smith, +VD`` reads whole as the family name, its given name lost. A caller whose names also +carry the credential chooses ``en`` too. + +Data sources: the readings' own entries in nameparser/vocab/nl.toml, +each with its issue and the vetting on its note; nothing is listed +here. + +Deviations: derived, not declared. The words a choice of ``nl`` +can move are ``_reading_index.changed_words``, which the +non-interference gate in tests/v2/test_locales.py reads +(decisions/deviates-registry), so this module carries no DEVIATES. +""" +from __future__ import annotations + +from nameparser._lexicon import Lexicon +from nameparser._locale import Locale + +NL = Locale(code="nl", lexicon=Lexicon.empty(), + languages=frozenset({"nl"})) diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index cf27af796..969e6546c 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -95,7 +95,7 @@ The build checks the order and never rewrites a file, because a comment above an ## Kinds -A reading's key is its **kind**. The table lists the kinds in their canonical order, which is the [word order](#word-order)'s first key and `CATEGORY`'s order in `nameparser/_reading_index.py`, the compiler the build shares with the package. Each kind belongs to one **category**, and the category decides which compiled set the reading feeds. For title, suffix, particle and conjunction the category's own name is also a kind (`title = {}`: a title not yet classified), so classifying a word later is a data edit that moves no compiled set. The one exception: a generic `suffix = {}` classified as `credential` moves from `SUFFIX_WORDS` to `SUFFIX_ACRONYMS`, because a credential's `acronym` is true unwritten; write `acronym = false` to keep it where it was. The suffix and particle categories are classified (the last two generic particles, `bon` and `dí` in `und.toml`, lost their particle readings in #766); so are the titles (#750); the others carry their generic kind. +A reading's key is its **kind**. The table lists the kinds in their canonical order, which is the [word order](#word-order)'s first key and `CATEGORY`'s order in `nameparser/_reading_index.py`, the compiler the build shares with the package. Each kind belongs to one **category**, and the category decides which compiled set the reading feeds. For title, suffix, particle and conjunction the category's own name is also a kind (`title = {}`: a title not yet classified), so classifying a word later is a data edit that moves no compiled set. The one exception: a generic `suffix = {}` classified as `credential` moves from `SUFFIX_WORDS` to `SUFFIX_ACRONYMS`, because a credential's `acronym` is true unwritten; write `acronym = false` to keep it where it was. The suffix, particle and title categories are classified (the last two generic particles, `bon` and `dí` in `und.toml`, lost their particle readings in #766; the titles in #750), save one particle no kind names, Dutch `oude`, an adjective ("old"), which `nl.toml` writes as a generic `particle` reading (#1455); the others carry their generic kind. | Kind | Category | Compiled into | |---|---|---| @@ -167,7 +167,8 @@ The particle kinds say what the word does in the names of the file's language: - `clan`: a lineage prefix, where the word after it is a family name shared by the lineage rather than a parent's name (`mac`, `mc`, `ó`, `ní`, `آل`, and `ben` and `בן`, whose family names no longer name a father); - `rank`: a former rank written inside the surname (`freiherr`, `freiherrin`, `graf`, `gräfin`, `fürst`, `prinz`); - `saint`: "Saint" opening a surname (`san`, `santa`, `st`, `ste`, `sta`, `sto`); -- `alias`: a word joining two surnames of one person (`vel`). +- `alias`: a word joining two surnames of one person (`vel`); +- `particle`, the generic kind: a particle no kind above names (Dutch `oude`, "old", an adjective: `Oude Luttikhuis`, #1455). Unlike the suffix kinds, one word may take a different particle kind in each language: `de` is a preposition in `fr.toml` and an article in `nl.toml`. Whether a particle may be given is the word's, not the reading's: its `given` reading, in any file, decides it for every particle reading (check 11). On a particle reading, `expands` is the word's meaning in that language, and `note` is reserved for vetting. A never-given particle may rest on its kind and `expands` alone: one attested bearer settles the may-be-given side, but no amount of searching settles absence, so what is recorded is what the word IS (an article, a preposition) rather than a failed search. And a word with a `given` reading whose particle readings carry no `note` in any file sits on the conservative default and nobody has examined it, and so does one whose note says so, as `heer`'s does: counts on the given reading were taken for phase 2's derivation (2026-10-25), not read against the particle position. The test is per word, because a word's note sits in one of its files, not in each: `mac` is examined though only `en.toml` holds its note. diff --git a/nameparser/vocab/nl.toml b/nameparser/vocab/nl.toml index 868d1c840..0c367043b 100644 --- a/nameparser/vocab/nl.toml +++ b/nameparser/vocab/nl.toml @@ -53,6 +53,15 @@ family = { us_census_2010 = 2007, wikidata = 113, as_of = 2026-10-27, note = "Wi reason = "a rank particle ('lord', the nobiliary 'heer van') added from #18's test \"Johan de heer Van Kampen\" (family 'de heer Van Kampen'), which appears to be two real names run together (Johannes de Heer, a Dutch Protestant figure, and the surname Van Kampen): the string occurs elsewhere only in another name parser's test data (hex.pm full_name_splitter 1.0.3). No real name supports the particle reading, and Heer is borne as a given name and as a surname (the counts on its name readings)" issue = 766 +[[word]] +text = "oude" +given = { note = "nam_dict band 1 (rare) in the Netherlands, its only country; no Wikidata bearer, and no given box opens with it in ORCID, PatentsView or PSC (2026-11-18, #1455)" } +[word.particle] +issue = 1455 +expands = "old: \"Oude Luttikhuis\", \"Oude Elferink\", the Twente and Achterhoek farm names" +note = "an adjective, which no particle kind names; written Oude, with a capital, in every family box that holds it (ORCID 10, PatentsView 8, PSC 1, 2026-11-18), never alone, and before another surname word in all but two, which end the box after the article (De Oude). As a particle it fixes 16 filed names to 1 broken, a filer who typed Oude at the end of the given box (config_diff.py score, ORCID, PatentsView and PSC, 20,000 records a group, 2026-11-18, #1455)" +cased = "Oude" + [[word]] text = "aan" given = { wikidata = 1, as_of = 2026-10-25, note = "Wikidata is a lower bound" } @@ -77,6 +86,14 @@ issue = 18 expands = "at: \"te Riele\"" note = "Te Awanui-a-Rangi: a Māori given-name element, and in the corpus. Looks like the Dutch preposition 'te', which is why attestation rather than etymology decides" +[[word]] +text = "ten" +[word.preposition] +default = false +issue = 1455 +expands = "at the, te den contracted: \"ten Brink\", \"ten Cate\"" +note = "off by default (decisions/vocabulary-collisions, 2026-11-12 #1257, rule 2): it fails the mid-name test, a Chinese given-name syllable written between two words (Kheng Ten Choo), and zh.toml declares the given name it opens (Ten Kao Liu). Under a Dutch choice it fixes ORCID's, PatentsView's and PSC's Dutch records 66 to 4 broken, 20 more Dutch-surnamed people filed under other countries left unfixed by default (2026-11-18, #1455)" + [[word]] text = "ter" preposition = { may_be_given = false, capital_opening = true, expands = "at the, te der contracted: \"ter Horst\"" } diff --git a/nameparser/vocab/zh.toml b/nameparser/vocab/zh.toml index b3af2d3d5..60f46653e 100644 --- a/nameparser/vocab/zh.toml +++ b/nameparser/vocab/zh.toml @@ -160,3 +160,7 @@ excluded = { attached = "the simplified twin of 部長, declined with it (#707)" [[word]] text = "hon" given = { default = false, note = "a Cantonese given name (Hon Wah Yeung, Hon Chan) as well as the style (en.toml's Hon, the Honourable): ORCID records whose given box opens with it 202 (Hong Kong 33, Malaysia 10), PatentsView 198 inventors (US 72, China 68, Singapore 22), PSC forename boxes 31 against 13 title boxes (2026-11-03); off by default, where UK registers write the style bare at the head (Hon Raymond Bonham-Carter, 10 of the 11 ukec-individuals names opening with it, the eleventh the phrase Hon Mrs), and turned on by a Chinese choice, which makes the style one only inside a title phrase (rules/H9); no words-to-spare count separates the two, a Cantonese two-syllable given name leaving two words after it as a titled name often does (#999)" } + +[[word]] +text = "ten" +given = { note = "a given-name syllable of Chinese and of Bhutanese names (Ten Kao Liu, Ten Dorji; no file holds Dzongkha): ORCID researchers open a given box with it 4 times (Singapore 2, Bhutan 2), PatentsView inventors 4 (Taiwan 2, Singapore 2), and it stands inside Malaysian and Singaporean given names; declared so that nl.toml's preposition, which a Dutch choice turns on, leaves a name opening with it the given name unless the choice is strict (2026-11-18, #1455)" } diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 6cb3e5743..c2d2a9d05 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -957,6 +957,88 @@ def _check_cjk_shape_purity(self) -> None: notes="a word before the father's name (a clan name, not read " "as one) leaves the given name last, the reason the pack " "declares FAMILY_FIRST_GIVEN_LAST rather than FAMILY_FIRST"), + # #1455: the Dutch oude and ten, each row checked against the 1.4.0 + # and 3.1.0 wheels on 2026-11-18 + Case("a_dutch_oude_starts_the_family_name", + "Piet Oude Luttikhuis", + {"given": "Piet", "family": "Oude Luttikhuis"}, + classification="fix(#1455)", + notes="1.4.0 through 3.1.0 read middle 'Oude'"), + Case("a_dutch_oude_ending_the_given_part_joins_the_family", + "Luttikhuis, Piet Oude", + {"given": "Piet", "family": "Oude Luttikhuis"}, + classification="fix(#1455)", + ambiguities=("particle-or-given",), + notes="rules/P6's trailing attachment; 1.4.0 through 3.1.0 read " + "middle 'Oude'"), + Case("a_dutch_oude_opening_a_name_stays_given", + "Oude Luttikhuis", + {"given": "Oude", "family": "Luttikhuis"}, + classification="parity", + ambiguities=("particle-or-given",), + notes="nam_dict's rare Dutch given name Oude keeps it given where " + "it opens the name (rules/P1), the fork reported; 1.4.0 " + "reads the same fields"), + Case("ten_stays_a_middle_name_by_default", + "Jan ten Brink", + {"given": "Jan", "middle": "ten", "family": "Brink"}, + classification="parity", + notes="the negative control for #1455: ten is nl.toml's " + "preposition written default = false; 1.4.0 reads the same"), + Case("the_dutch_pack_reads_ten_as_the_family", + "Jan ten Brink", + {"given": "Jan", "family": "ten Brink"}, + locale="nl", classification="feat(#1455)", + notes="1.4.0 through 3.1.0 read middle 'ten'"), + Case("the_dutch_pack_reads_ten_ending_the_given_part", + "Brink, Jan ten", + {"given": "Jan", "family": "ten Brink"}, + locale="nl", classification="feat(#1455)", + ambiguities=("particle-or-given",), + notes="rules/P6's trailing attachment; 1.4.0 through 3.1.0 read " + "middle 'ten'"), + Case("the_dutch_pack_keeps_ten_opening_a_name_given", + "Ten Kao Liu", + {"given": "Ten", "middle": "Kao", "family": "Liu"}, + locale="nl", classification="parity", + ambiguities=("particle-or-given",), + notes="zh.toml's given-name syllable where the particle opens a " + "name, the fork reported; 1.4.0 reads the same fields"), + Case("a_double_dutch_name_with_ten_reads_whole_by_default", + "Jutte van der Werff ten Bosch", + {"given": "Jutte", "family": "van der Werff ten Bosch"}, + classification="parity", + notes="the negative control for the pack's two-run cost; 1.4.0 " + "reads the same"), + Case("the_dutch_pack_splits_a_double_name_at_ten", + "Jutte van der Werff ten Bosch", + {"given": "Jutte", "middle": "van der Werff", "family": "ten Bosch"}, + locale="nl", classification="feat(#1455)", + notes="a second particle run opening with ten takes the family, as " + "one opening with ter does by default (docs/languages.rst's " + "costs); 1.4.0 through 3.1.0 read family 'van der Werff ten " + "Bosch'"), + Case("the_dutch_pack_reads_vd_as_van_der_not_the_credential", + "John Smith VD", + {"given": "John", "middle": "Smith", "family": "VD"}, + locale="nl", classification="feat(#1455)", + notes="nl's choice turns off en's credential VD for its own vd " + "(van der); 1.4.0 through 3.1.0 read suffix 'VD'"), + Case("the_dutch_pack_reads_vd_after_a_comma_as_the_family", + "John Smith, VD", + {"family": "John Smith VD"}, + locale="nl", classification="feat(#1455)", + notes="the never-given particle with nothing behind it after the " + "comma, as 'John Smith, de' reads by default; the cost the " + "pack's docstring and docs/languages.rst state; 1.4.0 " + "through 3.1.0 read suffix 'VD'"), + Case("the_dutch_pack_takes_a_chinese_syllable_into_the_family", + "Kheng Ten Choo", + {"given": "Kheng", "family": "Ten Choo"}, + locale="nl", classification="feat(#1455)", + notes="the mid-name cost that keeps ten off by default " + "(rules/P2's Accepted); 1.4.0 through 3.1.0 read middle " + "'Ten'"), # #1257: the Maghrebi and Hebrew particles, each row checked against # the 1.4.0 wheel and master 161ebd28 on 2026-11-12 Case("an_amazigh_ait_starts_the_family_name", diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index def0b04e7..89be22320 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -488,10 +488,12 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # 2086, the coordinator's review: 'Jane Doe, MH, Mc, PhD' and 'Jane # Doe, VI, PhD' under three orders (the vi row is not swept); 2086 # -> 2098, the third review's four rows under three orders; 2098 -> - # 2101, 'Jane Doe, A.L., PhD' under three orders. - assert claims == 2101, ( - f"the field sweep checks {claims} claims, recorded as 2101 on " - f"2026-11-15") + # 2101, 'Jane Doe, A.L., PhD' under three orders; 2101 -> 2107 on + # 2026-11-18, #1455's 'Oude Luttikhuis' and 'Luttikhuis, Piet Oude', + # each reporting particle-or-given under three orders. + assert claims == 2107, ( + f"the field sweep checks {claims} claims, recorded as 2107 on " + f"2026-11-18") #: The kinds that report a fork between two readings of a word, so carry @@ -953,8 +955,10 @@ def test_the_alternative_sweep_sees_the_fields_it_checks() -> None: # as the undotted ones do, and one boundary row's stand-in lands in # the other field; 453 -> 456, the second code review's glued row; # 456 -> 462, the coordinator's review's two rows; 462 -> 474, the - # third review's four; 474 -> 477, the 'A.L.' row - assert (checked, skipped) == (595, 477), (checked, skipped) + # third review's four; 474 -> 477, the 'A.L.' row; 595 -> 598 on + # 2026-11-18, #1455's 'Luttikhuis, Piet Oude' under the three orders, + # P6's report, each stand-in landing in the alternative's field + assert (checked, skipped) == (598, 477), (checked, skipped) def test_every_case_name_is_unique() -> None: diff --git a/tests/v2/test_locales.py b/tests/v2/test_locales.py index 7d113ecea..736509996 100644 --- a/tests/v2/test_locales.py +++ b/tests/v2/test_locales.py @@ -21,7 +21,7 @@ from nameparser import _reading_index as _ri from nameparser._lexicon import _VOCAB_FIELDS, Lexicon from nameparser._policy import ( - FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, UNSET, PatronymicRule, Policy, + FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST, GIVEN_FIRST, UNSET, PatronymicRule, Policy, PolicyPatch, Script, _SCRIPT_RANGES, ) from nameparser._types import AmbiguityKind, Role @@ -118,10 +118,10 @@ def test_locales_get_and_available() -> None: assert locales.get("ru") is locales.RU assert set(locales.available()) == { "ar", "bn", "cs", "es", "es_ar", "ga", "gd", "he", "hi", "hr", "ja", "mn", - "ne", "pt", "pt_br", "ru", "sk", "sl", "tr_az", "vi", "zh"} + "ne", "nl", "pt", "pt_br", "ru", "sk", "sl", "tr_az", "vi", "zh"} with pytest.raises( KeyError, - match="ar, bn, cs, es, es_ar, ga, gd, he, hi, hr, ja, mn, ne, pt, pt_br, ru, sk, sl, tr_az, vi, zh"): + match="ar, bn, cs, es, es_ar, ga, gd, he, hi, hr, ja, mn, ne, nl, pt, pt_br, ru, sk, sl, tr_az, vi, zh"): locales.get("xx") # the segmenter FACTORY is not a locale: it is an ordinary function # on the package, so it must not appear among the codes @@ -149,6 +149,105 @@ def test_ne_pack_contents() -> None: "guru", "गुरु", "श्री"} +def test_nl_pack_is_its_choice_alone() -> None: + # #1455: it turns on `ten`'s preposition, written default = false, and + # its choice drops en's credential VD for vd's particle + import pickle + from nameparser._reading_index import changed_words, selected + assert locales.NL == Locale(code="nl", lexicon=Lexicon.empty(), + languages=frozenset({"nl"})) + assert changed_words(selected(locales.NL.languages)) == {"ten", "vd"} + assert pickle.loads(pickle.dumps(locales.NL)) == locales.NL + parser = pickle.loads(pickle.dumps(parser_for(locales.NL))) + name = parser.parse("Jan ten Brink") + assert (name.middle, name.family) == ("", "ten Brink") + name = parse("Jan ten Brink") + assert (name.middle, name.family) == ("ten", "Brink") + + +@pytest.mark.parametrize("strict, text, given, family, reported", [ + # #1455: a `Ten` opening a name is zh.toml's given name, so the plain + # choice keeps it given and reports the fork; a strict choice reads + # no zh.toml, and the particle opens the family name (rules/P1) + (False, "Ten Kao Liu", "Ten", "Liu", True), + (True, "Ten Kao Liu", "", "Ten Kao Liu", False), + (False, "Ten Brink", "Ten", "Brink", True), + (True, "Ten Brink", "", "Ten Brink", False), + # inside the name both choices read the particle, the Chinese + # syllable joining the family name: the cost that keeps it off by + # default (decisions/vocabulary-collisions, 2026-11-18 #1455) + (False, "Kheng Ten Choo", "Kheng", "Ten Choo", False), + (True, "Kheng Ten Choo", "Kheng", "Ten Choo", False), +]) +def test_nl_reads_a_leading_ten_by_its_given_reading_unless_strict( + strict: bool, text: str, given: str, family: str, reported: bool) -> None: + name = parser_for(locales.NL, strict=strict).parse(text) + assert (name.given, name.family) == (given, family) + assert any(a.kind is AmbiguityKind.PARTICLE_OR_GIVEN + for a in name.ambiguities) is reported + + +def test_a_strict_dutch_choice_keeps_a_leading_oude_given() -> None: + # #1455: oude's given reading is in nl.toml itself, so a strict Dutch + # choice keeps it, where ten's (in zh.toml) leaves + name = parser_for(locales.NL, strict=True).parse("Oude Luttikhuis") + assert (name.given, name.family) == ("Oude", "Luttikhuis") + assert [a.kind for a in name.ambiguities] == [AmbiguityKind.PARTICLE_OR_GIVEN] + + +@pytest.mark.parametrize("order", [GIVEN_FIRST, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST]) +@pytest.mark.parametrize("text", [ + "Ten Kao Liu", "Kheng Ten Choo", "Ten", "Liu, Ten", "Ten, Kheng", + ", Ten Kao", "Ten Kao Liu, Jr.", "TEN KAO LIU", "Liu, Kheng Ten", + "Dr. Ten Liu", "Jan ten Brink", "Brink, Jan ten", +]) +def test_the_given_reading_of_ten_moves_no_default_parse( + text: str, order: tuple[Role, Role, Role]) -> None: + # #1455: zh.toml's given reading of ten is on by default, but with + # nl.toml's preposition off a lone name reading compiles to no set, + # so the default reads every name holding ten as without it + policy = Policy(name_order=order) + with_it = Parser(policy=policy).parse(text) + without = Parser(policy=policy, lexicon=Lexicon.default().remove( + "ten", kind="given")).parse(text) + assert with_it.as_dict() == without.as_dict() + assert [a.kind for a in with_it.ambiguities] == [ + a.kind for a in without.ambiguities] + + +def test_the_given_reading_of_ten_moves_a_parse_where_the_particle_is_on() -> None: + # the negative control for the test above: the same comparison sees + # the reading under the pack, where the preposition is on + nl = parser_for(locales.NL) + without = parser_for(locales.NL, base=Parser( + lexicon=Lexicon.default().remove("ten", kind="given"))) + assert nl.parse("Ten Kao Liu").given == "Ten" + assert without.parse("Ten Kao Liu").family == "Ten Kao Liu" + + +@pytest.mark.parametrize("code, text, repaired", [ + # #1455: oude keeps the capital its family names are written with + # (its `cased` mask) where case repair would lowercase a particle; + # ten takes a particle's lower case, as Dutch writes it after a given + # name (a family box holding the surname alone often capitalizes it) + (None, "PIET OUDE LUTTIKHUIS", "Piet Oude Luttikhuis"), + (None, "piet oude luttikhuis", "Piet Oude Luttikhuis"), + (None, "Piet Oude Luttikhuis", "Piet Oude Luttikhuis"), + (None, "JAN DE OUDE BRINK", "Jan de Oude Brink"), + ("nl", "PIET OUDE LUTTIKHUIS", "Piet Oude Luttikhuis"), + ("nl", "JAN TEN BRINK", "Jan ten Brink"), + ("nl", "jan ten brink", "Jan ten Brink"), + # by default ten is a name word, title-cased + (None, "JAN TEN BRINK", "Jan Ten Brink"), +]) +def test_a_dutch_particle_keeps_its_case( + code: str | None, text: str, repaired: str) -> None: + parser = parser_for(locales.get(code)) if code else Parser() + name = parser.parse(text) + assert str(parser.capitalized(name)) == repaired + assert str(name.capitalized(parser.lexicon, force=True)) == repaired + + def test_ar_and_he_packs_are_their_choice_alone() -> None: # #1257: each turns on `ben`'s clan reading, written default = false; # ar's choice also drops en's credential ABD for abd's bound given name @@ -1901,6 +2000,18 @@ def _default_corpus() -> list[str]: "YAEL BEN ARIEH", # ben, in one case "Arieh, Yael Ben", # ben ending the given part ] +# nl (#1455): ten mid-name, in each case and after a family comma (a +# leading Ten keeps the default's fields under the plain choice, only +# reporting the fork: test_nl_reads_a_leading_ten_by_its_given_reading_ +# unless_strict); the credential VD its choice drops for vd's particle +_ROTATORS["nl"] = [ + "Jan ten Brink", # ten + "JAN TEN BRINK", # ten, in one case + "jan ten brink", # ten, lower case + "Brink, Jan ten", # ten ending the given part + "John Smith VD", # vd, the credential leaving + "John Smith, VD", # vd after a comma +] # hi and bn: each romanized title at the head, in each case, behind a # listed title, after a family comma and before one (rules/H6); a dotted # one where it ends the given part after a family comma (rules/H8) or diff --git a/tests/v2/test_match.py b/tests/v2/test_match.py index c3cf701f8..60abdb924 100644 --- a/tests/v2/test_match.py +++ b/tests/v2/test_match.py @@ -556,6 +556,22 @@ def test_the_armenian_ter_prefix_is_a_particle_set_aside() -> None: "պետրոսյան;լ", "տերպետրոսյան;լ", "տեր-պետրոսյան;լ"} +@pytest.mark.parametrize("a, b, family", [ + ("Piet Oude Luttikhuis", "Piet Oudeluttikhuis", Outcome.JOINED), + ("Piet Oude Luttikhuis", "Piet Luttikhuis", Outcome.PARTICLES_ASIDE), +]) +def test_the_dutch_oude_is_a_particle_to_the_match( + a: str, b: str, family: Outcome) -> None: + # oude reads by default as the particle it is (#1455): written apart + # or run together with its surname the family joins, where 3.1.0 read + # a conflict, and set aside against the bare surname, where 3.1.0 read + # a middle name on one side; neither side then holds a middle name + for x, y in ((a, b), (b, a)): + r = m.check(x, y) + assert r.consistent and r["family"].outcome is family, (x, y) + assert r["middle"].outcome is Outcome.SAME, (x, y) + + @pytest.mark.parametrize("a, b", [ ("Juan de la Vega", "Juan Marcos Vega"), ("Զավեն Տեր Եղիյան", "Զավեն Մարկոս Եղիյան"), diff --git a/tests/v2/test_reading_index.py b/tests/v2/test_reading_index.py index 4606386c1..54399399c 100644 --- a/tests/v2/test_reading_index.py +++ b/tests/v2/test_reading_index.py @@ -389,7 +389,7 @@ def test_a_file_holding_only_familiar_readings_is_chosen_only_strictly( "hr": {"st"}, "it": {"del", "la", "san", "santa"}, "ms": {"bin", "sultan"}, - "nl": {"den", "tho"}, + "nl": {"den", "ten", "tho"}, "pt": {"da", "do", "santa"}, "pt-BR": {"da", "do", "santa"}, "sk": {"st"}, diff --git a/tools/differential/cldr_agreement.py b/tools/differential/cldr_agreement.py index 52fb57172..fdfa8464e 100644 --- a/tools/differential/cldr_agreement.py +++ b/tools/differential/cldr_agreement.py @@ -164,7 +164,7 @@ "ga": "GA", "gd": "GD", "pt": "PT_BR", "es": "ES", # hi_Latn is the romanized Hindi the pack's Latin readings are for "hi": "HI", "hi_Latn": "HI", "bn": "BN", "vi": "VI", - "mn": "MN", "ar": "AR", "he": "HE"} + "mn": "MN", "ar": "AR", "he": "HE", "nl": "NL"} #: Packs with no CLDR file to measure them on. CLDR's pt is Brazilian, #: read with PT_BR, which chooses PT's language and more; CLDR 48 has no #: pt_PT names file, and no es_AR one (its es is read with ES). diff --git a/tools/differential/cldr_agreement_locales.json b/tools/differential/cldr_agreement_locales.json index ba79c6411..37fa04f06 100644 --- a/tools/differential/cldr_agreement_locales.json +++ b/tools/differential/cldr_agreement_locales.json @@ -1438,6 +1438,28 @@ }, "text": "वुस्टर बर्ट्राम विलबरफोर्स हेनरी रोबर्ट एमपी" }, + "nl/foreignFull/surnameFirst": { + "given_names": { + "expected": "Ada Cornelia César Martín", + "got": "mevrouw Ada Cornelia César Martín" + }, + "text": "von Brühl mevrouw Ada Cornelia César Martín jr., PhD", + "title": { + "expected": "mevrouw", + "got": "" + } + }, + "nl/nativeFull/surnameFirst": { + "given_names": { + "expected": "Bertus Wevers Harry Robert", + "got": "dhr. Bertus Wevers Harry Robert" + }, + "text": "de Jong dhr. Bertus Wevers Harry Robert jr., mp", + "title": { + "expected": "dhr.", + "got": "" + } + }, "pt/foreignFull/givenFirst": { "suffix": { "expected": "Jr. Dr. Ph.D", diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index d51fb89df..1e9a9ae49 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -1151,6 +1151,7 @@ "Kavitha S" "Kavitha S MA" "Kh. O. Bezirdzhyan" +"Kheng Ten Choo" "Kilang.Manah" "Kim Yu.A." "Kim, AE-RA" @@ -1380,6 +1381,7 @@ "PETER Okello MUWANGA" "PETROSYAN, LEVON TER" "PHUOC T DO" +"PIET OUDE LUTTIKHUIS" "PIETER 'T-HART" "Paulo de Tarso Ribeiro" "Pavlović, Antonio mlađi" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 2e68c75c5..f9cde4ef3 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -2716,6 +2716,7 @@ {"name": "Kevin", "order": "FAMILY_FIRST", "given": "Kevin", "ambiguities": ["GIVEN_OR_FAMILY"], "alternatives": ["GIVEN_OR_FAMILY: name as family (position)"], "initials": "K."} {"name": "Kh. O. Bezirdzhyan", "given": "Kh.", "middle": "O.", "family": "Bezirdzhyan", "initials": "K. O. B."} {"name": "Khaled el-Hassani ben el-Hachemi", "given": "Khaled", "middle": "el-Hassani ben", "family": "el-Hachemi", "initials": "K. e. b. e.", "capitalized_force": "Khaled El-Hassani Ben El-Hachemi"} +{"name": "Kheng Ten Choo", "given": "Kheng", "middle": "Ten", "family": "Choo", "initials": "K. T. C."} {"name": "Kho Liang Ie", "given": "Kho", "middle": "Liang", "family": "Ie", "initials": "K. L. I."} {"name": "Khun Malee", "given": "Khun", "family": "Malee", "initials": "K. M."} {"name": "Khun Sa", "given": "Khun", "family": "Sa", "initials": "K. S."} @@ -3481,6 +3482,7 @@ {"name": "PETER Okello MUWANGA", "given": "PETER", "middle": "Okello", "family": "MUWANGA", "initials": "P. O. M.", "capitalized_force": "Peter Okello Muwanga"} {"name": "PETROSYAN, LEVON TER", "given": "LEVON", "family": "TER PETROSYAN", "initials": "L. P.", "capitalized": "Levon Ter Petrosyan"} {"name": "PHUOC T DO", "given": "PHUOC", "family": "T", "suffix": "DO", "ambiguities": ["ORDER", "SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: name as family (declared)"], "initials": "P. T.", "capitalized": "Phuoc T DO"} +{"name": "PIET OUDE LUTTIKHUIS", "given": "PIET", "family": "OUDE LUTTIKHUIS", "initials": "P. L.", "capitalized": "Piet Oude Luttikhuis"} {"name": "PIETER 'T-HART", "given": "PIETER", "family": "'T-HART", "initials": "P. '.", "capitalized": "Pieter 't-Hart"} {"name": "Pablo Emilio Riveros Reina", "given": "Pablo", "middle": "Emilio Riveros", "family": "Reina", "initials": "P. E. R. R."} {"name": "Pablo Pérez Miguel Ángel Juan Antonio II", "order": "FAMILY_FIRST", "given": "Pérez", "middle": "Miguel Ángel Juan Antonio", "family": "Pablo", "suffix": "II", "initials": "P. M. Á. J. A. P.", "generation": "II"}