Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ the patronymic in Russian. See `Locale packs
>>> ivan.given, ivan.patronymic, ivan.family
('Иван', 'Петрович', 'Сидоров')
>>> locales.available() # every pack that ships
('ar', 'bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'he', 'hi', 'hr', 'ja', 'mn', 'ne', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'vi', 'zh')
('ar', 'bn', 'cs', 'es', 'es_ar', 'ga', 'gd', 'he', 'hi', 'hr', 'ja', 'mn', 'ne', 'nl', 'pt', 'pt_br', 'ru', 'sk', 'sl', 'tr_az', 'vi', 'zh')

When a reading is wrong for your data
-------------------------------------
Expand Down
20 changes: 11 additions & 9 deletions docs/customize.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ and names the ``remove`` above.
Both calls return a new :class:`~nameparser.Lexicon`, the one you
started from (here, ``Lexicon.default()``) never being mutated.

.. _adding-a-word:

Adding a word
^^^^^^^^^^^^^

Expand All @@ -87,19 +89,19 @@ list is one reading per word:
'LEED AP'

A surname particle is added the same way. The defaults leave out the
Dutch ``ten`` of ``ten Brink``, which as a default would take a middle
Dutch ``in`` of ``in 't Veld``, which as a default would take a middle
name spelled like it into the surname of names in every language; add
it when your data is Dutch:

.. doctest::

>>> parse("Marieke ten Brink").middle, parse("Marieke ten Brink").family
('ten', 'Brink')
>>> dutch_ten = Lexicon.default().add("ten", kind="preposition")
>>> Parser(lexicon=dutch_ten).parse("Marieke ten Brink").family
'ten Brink'
>>> Parser(lexicon=dutch_ten).parse("Park Ten Hwan").family
'Ten Hwan'
>>> parse("Marieke in 't Veld").middle, parse("Marieke in 't Veld").family
('in', "'t Veld")
>>> dutch_in = Lexicon.default().add("in", kind="preposition")
>>> Parser(lexicon=dutch_in).parse("Marieke in 't Veld").family
"in 't Veld"
>>> Parser(lexicon=dutch_in).parse("Park In Hwan").family
'In Hwan'

The last line is the cost: an edit applies to every name the parser
reads. A particle added this way holds no ``given`` reading, so
Expand Down Expand Up @@ -157,7 +159,7 @@ named by language, which a reading you added leaves empty:
... print(van_reading.kind, van_reading.category, van_reading.languages)
preposition particle ('nl',)
given name ('nl',)
>>> dutch_ten.readings_of("ten")[0].languages
>>> dutch_in.readings_of("in")[0].languages
()

:meth:`~nameparser.Lexicon.words` goes the other way: it lists the
Expand Down
9 changes: 9 additions & 0 deletions docs/design/decisions/vocabulary-collisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ Elsewhere: C-i reaches the titles through a `given` reading, which makes a title
- `ait`, `aït` and `nait` (ber.toml) and `ouled` (ar.toml) are never-given clan particles on by default; `ould` (ar.toml) is a never-given patronymic on by default, filling the patronymic view in an Algerian fixed family name too; Latin `ben` is a clan particle written `default = false` in ar.toml and he.toml, which `locales.AR` and `locales.HE` turn on; Hebrew `בן` is a clan particle that may be given, `אבו` reads as `abu` and `אל` as `al`, each on by default; `בת` is unchanged — 2026-11-12 #1257.
- Hebrew `בן` opening a family name written family first with no comma, under a family-first order, gives `בן` alone to the family, reported, a cost of its given reading accepted on #1209's measurement (8 Knesset family names, if written family first) — 2026-11-12 #1257.
- `el` (ar.toml) is an article that may be given, as `al` is, on by default: a decided exception to the Latin-spelling half of the rule above, which it fails, on its fixed/broken table, Algeria's compound given names the accepted cost — 2026-11-12 #1257.
- `oude` (nl.toml) is a particle on by default, of the generic kind `particle`, no kind naming an adjective, that may be given (nam_dict's rare Dutch given name), with its case mask `Oude`: it passes rule 2 of the 2026-11-12 #1257 entry — 2026-11-18 #1455.
- `ten` (nl.toml) is a preposition written `default = false`, which `locales.NL`, a pack choosing `nl` alone, turns on; it fails rule 2's mid-name test (`Kheng Ten Choo`), and zh.toml declares the Chinese given-name syllable it opens, so a plain Dutch choice keeps a leading `Ten` the given name, reported, and a strict one reads it as the family name — 2026-11-18 #1455.

## Log

Expand Down Expand Up @@ -89,3 +91,10 @@ Elsewhere: C-i reaches the titles through a `given` reading, which makes a title
- **What moved**, the PR's head against master's readings rebuilt in the tree as `expr:Parser(lexicon=Lexicon.default().remove('ait','aït','nait','ouled', kind='clan').remove('ould', kind='patronymic').remove('el', kind='article').remove('el', kind='given').remove('בן', kind='clan').remove('בן', kind='given').add('בן', kind='patronymic').remove('אבו').remove('אל'))`, whose compiled sets equal those of master `161ebd28` and of the merge base `73718fd9` (36 sets compared, 2026-11-12): the committed snapshot gains eight rule examples (`git diff 73718fd9 HEAD -- tools/differential/corpus_rules.jsonl`) and moves three names, the recipe below counting the eleven, each built on `Zein El Dine` (Zayn al-Din, a compound given name also borne as a family name; `Mohamad Zein El Dine` → family 'El Dine', where it read 'Dine'), neither reading the whole name: unclear rather than wrong, #1378's shape; JRC-Names moves 3,483 of 1,315,051 spellings and opensanctions-parliaments 16 of 24,192 (`config_diff.py diff "expr:Parser(lexicon=...)" default plain:AR plain:HE --committed --local jrc-names --local opensanctions-parliaments`). Before `el` was read by default the same diff moved 287 JRC-Names spellings and 15 parliament names, the snapshot no name. Scored (`config_diff.py score "expr:Parser(lexicon=...)" default plain:AR plain:HE --group MA --group DZ --group TN --group MR --group EG --group LB --group IL --group SA --group FR --group BE --group NL --group ES --group GB --group US --group Moroccan --group Algerian --group Tunisian --group Mauritanian --group Egyptian --group Israeli --group French --group British`), the default fixes ORCID Morocco 2,275 to 34 broken, Lebanon 432 to 6, Egypt 286 to 20, Algeria 132 to 142, Tunisia 92 to 14 and Mauritania 38 to 6, PSC's Moroccan filers 452 to 16 and Algerian 5 to 11; `locales.AR` and `locales.HE` each fix ORCID Tunisia 1,056 to 17, Israel 164 to 4, Morocco 2,403 to 34, PSC's Tunisian filers 54 to 5, and PatentsView Israel 146 to 61, most of those breaks filers who typed `Ben` in the given box. Agreement in every box, ORCID: Tunisia 87.6% by default, 96.1% under either pack; Israel 92.7% and 93.5%; Morocco 84.0% on master, 95.5% by default, 96.1% under either pack. The CLDR locale view's `ar` and `he` files read alike with and without each pack (20/24, 18/24).
- **Accepted:** a listing written family first without a comma reads whole as the family where a never-given `ait` or `ould` opens it (rules/P1); a family name written with Hebrew `בן` first and no comma under a family-first order gives `בן` alone to the family (`בן חיים יעל` → family 'בן', rules/P1; #1209 counted 8 Knesset family names written so), accepted the same day as a cost of the given reading; an Arab name transcribed in Hebrew script with `בן` for the father leaves the patronymic view empty (rules/R9); a family name written alone with `בן` reads given `בן` (#1209); `אל` inside a given name gives the article and the rest to the family; an Algerian compound given name holding `El`, or a bound given name before it, gives the article and the rest to the family (rules/P2); a choice of `ar` drops en's credential ABD for ar's bound given name `abd` (`Jane Smith, ABD` → given 'ABD' under `locales.AR`), which its docstring and docs/locales.rst say; the packs read a register-style chain's last `ben` into the family (`Karim ben Ali ben Salah Haddad` → family 'ben Salah Haddad'; #1248 measured Tunisia's register-style list at 140 of 158 right by default, 24 with `ben` read), being for surnames; under `strict=True` the Arabic pack reads a leading `Ben` as the family, English's given reading not chosen, and drops ber.toml's `ait` and `nait` unless `ber` is chosen too.
- **Left out:** in #1378 (with its 2026-11-12 comment), Hebrew `עבד` and Latin `abd`, `abdel`, `abdul`, `abu` with rules/P12's `completed_by` for the names after the article, the Arabic compound given names behind `el` and `al` as rules/P10's `after_particle` words (`houda`, `amine`, `islam`, `din`, `zahraa`), and the Arabic-script `ايت` and `ولد`; and `N'Ait`, which #1249 lists and which reads today as one word, given "N'Ait" opening a name and a middle name inside one (`Leila N'Ait Haddou` → middle "N'Ait"); `בת` stays a never-given patronymic; `בר` stays excluded in he.toml (Bar, a common Israeli given name), rule 1 not applied to it here, the word not reopened.
- 2026-11-18 #1455 — DUTCH `OUDE` IS READ BY DEFAULT AND `TEN` UNDER A DUTCH CHOICE; `LOCALES.NL` SHIPS. Decided on the issue (its two 2026-11-18 decision comments), on the vetting comment there: `oude` a default particle with a case mask, `ten` an nl.toml particle written `default = false` with a `given` reading beside it, and a shipped pack. The builder's choices, open to review on the PR: `oude`'s kind, `oude`'s `given` reading, and which file holds `ten`'s, which decides what a strict Dutch choice reads.
- **`oude`.** Passes rule 2 of the 2026-11-12 #1257 entry: every family box holding it writes `Oude`, with a capital, never alone, and before another surname word in all but two, which end the box after the article (`DE Oude`, `De Oude`) (ORCID 10, PatentsView 8, PSC 1, by a scratch probe over `local_agreement.records` for the three fielded corpora, 2026-11-18), no given box opens with it, and no real name holds it as a middle given name. nam_dict lists `Oude` as a given name, band 1 (rare), in the Netherlands alone (`uv run python tools/name_usage.py oude`, 2026-11-18), so it carries a `given` reading under C-i's attested-bearer side and is a particle that may be given: opening a name it stays the given name and reports the fork (`Oude Luttikhuis` → given 'Oude', rules/P1's opening site, as `van`). Its kind is `particle`, the generic one: it is an adjective ("old", the Twente and Achterhoek farm names), and the README's particle kinds name what a word does in a name (a preposition, an article, a father, a lineage), none an adjective; a kind of its own was weighed for the class it heads (Low Saxon farm-name qualifiers, `Groot`, `Klein`, `Lütke`) and not built, none of those being vetted. Its `cased` mask keeps the capital case repair would take from a particle (`PIET OUDE LUTTIKHUIS` → `Piet Oude Luttikhuis`).
- **`ten`.** Fails rule 2's mid-name test: a Chinese given name writes the syllable between two words (`Kheng Ten Choo`), which the particle takes into the family name, and ORCID and PatentsView open given boxes with it 8 times (Singapore 4, Bhutan 2, Taiwan 2) where no Dutch filer means a given name. So `default = false` in nl.toml, the preposition (te den, as `ter` is te der), and the `given` reading the decision asks for in zh.toml, the file whose names carry it (no file holds Dzongkha), as `ben`'s given reading is in en.toml and `sushri`'s in or.toml, rather than in nl.toml, Dutch names bearing no given name `Ten`: the plain `nl` choice keeps a leading `Ten` the given name, reported, and the strict one, reading no zh.toml, reads it as the family name (`Ten Brink` → family 'Ten Brink'), so `ten` joins tests/v2/test_reading_index.py's reviewed table of the words strict ungates. It carries no case mask, taking a particle's lower case, which is how Dutch writes it after a given name (`Jan ten Brink`); a family box holding the surname alone is written both ways (opening such a box, ORCID NL `ten` 28 to `Ten` 3, PatentsView NL 8 to 20, PSC's Dutch filers 0 to 7). `locales.NL` chooses `nl` and nothing else, so its choice also drops en's credential VD for nl's `vd` (`John Smith VD` → family 'VD'), which its docstring and docs/languages.rst say.
- **What moved**, against master `688991e7`'s readings rebuilt in the tree as `expr:Parser(lexicon=Lexicon.default().remove('oude'))`, whose 36 compiled sets equal master's (2026-11-18; its case mask stays, which no compared field reads): `uv run python tools/differential/config_diff.py diff "expr:Parser(lexicon=Lexicon.default().remove('oude'))" default plain:NL strict:NL --committed --local all --match '(?i)(^|[\s,.(])(ten|oude|vd)([\s,.)]|$)'`, 370 names at the PR's head. The default moves 26, every one `oude` from the middle name into the family: 25 in the local corpora (the vetting's 22 and 3 in ctgov-contacts, a corpus fetched since, #1454) and 1 in the committed corpus, the rules/R4 example this change adds; no name holding `ten` moves by default. `plain:NL` moves 250 names and `strict:NL` 254, holding `ten` 206 and 208, `oude` 26 each, `vd` 18 and 20; the two read 34 names differently, 24 of them holding `ten`: a leading `Ten` (`Ten Kao Liu`) or one opening the given part after a family comma (`SWACKHAMER, TEN EYCK`, `Nick HT Hacken, ten, MD, PhD`) reads as the family under strict, a `ten` ending the given part after a family comma is reported under plain only, and one name differs by a report on its credentials alone. The committed snapshot moves no name it held before; the rules corpus gains the examples (rules/P2, rules/R4); the CLDR default records do not move, and the locale view's `nl` file reads 22/24 with and without the pack. `uv run python tools/differential/suggest_measure.py --base <a checkout of 688991e7>` moves no cell. `uv run python tools/differential/match_diff.py --base <the same>` moved none of 690,382 rows before tests/v2/test_match.py held a pair with `oude`; with its two pairs, 1,254 of 690,426 rows move (57 under each of the 22 settings), every one a pair holding `Piet Oude Luttikhuis`, whose keys gain `oudeluttikhuis;p` and whose middle name, `Oude` on master, is now empty: `Piet Oude Luttikhuis` and `Piet Oudeluttikhuis` are consistent, the family joined, where master read a conflict, and against `Piet Luttikhuis` the family sets the particle aside, which a caller who drops `particles-aside` now reads as a conflict where master read the pair consistent (`Oude` a middle name), as `Jan van Berg` against `Jan Berg` already reads; no consistent pair shares no key.
- **Scored** against the filed boxes, `uv run python tools/differential/config_diff.py score "expr:Parser(lexicon=Lexicon.default().remove('oude'))" default plain:NL strict:NL --cap 20000`, ORCID and PatentsView by country and PSC by nationality, the 434 groups of 100 records or more, each capped at 20,000 (2026-11-18): the default fixes 16 to 1 broken (ORCID Netherlands 10, PatentsView Netherlands 5 and Germany 1; the break a PatentsView filer who typed `Oude` at the end of the given box), the vetting's figure. `plain:NL` fixes 102 to 10, so `ten` under the pack 86 to 9: the Netherlands and its filers 66 to 4 (ORCID 31 to 0, PatentsView 28 to 4, PSC's Dutch 7 to 0), Belgium 2 to 0, elsewhere 18 to 5. Read one by one with `local_agreement.read` (a scratch probe, the same day), two of the nine breaks are the mid-name syllable (PatentsView Malaysia `Kheng Ten Choo` and Singapore `Terence Ten Teck Hong`), the other seven filers who typed `ten` at the end of the given box (`Nils Ten` / `Hoeve`). The 20 fixes outside the Netherlands are the Dutch-surnamed people the decision gives up for the default caller. `uv run python tools/differential/config_diff.py gaps nl` (the same day) reads the strict choice against the plain one over the Dutch-language groups: 456 broken to 54 fixed, other languages' Latin particles dropped (`el`, `di`, `al`, `le`, `da`), `ten` not among its twenty words.
- **Accepted:** under the pack, a Chinese given name holding `Ten` between two words gives it and the rest to the family (rules/P2's Accepted); a surname-only `Oude Luttikhuis` reads given 'Oude', reported; under a strict Dutch choice a leading `Ten` is the family (`Ten Kao Liu` → family 'Ten Kao Liu'), and so is one opening the given part after a family comma (`Brink, Ten Jan` → family 'Brink Ten Jan', `, Ten Brink` → family 'Ten Brink'), the given name lost; a family name of two Dutch names whose second opens with `ten` keeps only the second under the pack (`Jutte van der Werff ten Bosch` → middle 'van der Werff'), as a second name opening with `ter` does by default; the pack drops the credential VD (`John Smith VD` → family 'VD', and after a comma the whole name, `John Smith, VD` → family 'John Smith VD').
- **Left out:** a kind for Dutch and Low German farm-name qualifiers, and the other qualifiers themselves; `Ten` as a Korean (Koryo-saram), Chinese and Turkish family name written alone, which no reading moves; Dutch `in` (`in 't Veld`), which docs/customize.rst now uses as its example of an unlisted particle; a possibly-given particle opening a name before a suffix comma folds into the family unreported (`Ten Smith, Jr.` under the pack, `Van Smith, Jr.` by default), a shape not settled here.
5 changes: 5 additions & 0 deletions docs/design/rules/P2-particles-join-forward.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,11 @@
"Abd El Kader Haddou" → family="El Kader Haddou"
"Abu El Hassan Haddou" → given="Abu"
```
- Under a Dutch choice `ten` is a particle (te den), and a Chinese given name written with its syllable between two words joins the family from it, as the Korean and Vietnamese syllables PR #191 kept out of the particles would: the mid-name cost that keeps the preposition off by default, two of the nine filed names the pack breaks among ORCID's, PatentsView's and PSC's records, against 86 it fixes (2026-11-18; decisions/vocabulary-collisions, #1455). The default reads `ten` as a name word.
```
"Kheng Ten Choo" [nl] → family="Ten Choo"
"Kheng Ten Choo" → family="Choo" · boundary
```
- A caller wanting the combined double-surname reading (#132's ask) has it as the surnames view rather than the family field.
```
"Vincent van Gogh van Beethoven" → surnames="van Gogh van Beethoven"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
- **A particle is written in lowercase**, but a particle the vocabulary records a casing for takes that casing, the entry being how the word is written wherever it stands, and a particle of a kind written with a capital is written as a name word is: a saint's name (`St Clair`, `San Martín`) and a rank written inside the surname (`Freiherr von Richthofen`) take their capital in every convention that writes them. So is a particle opening a hyphenated word, which the hyphen joins to the name after it (`Ter-Petrosyan`, `Al-Ahmari`): it is written as a name word is, while a particle standing apart keeps its lowercase, save one R12 gives a capital as it opens the family name (`Ter Petrosyan`).
```
"SEÁN Ó MURCHÚ" → capitalized="Seán Ó Murchú"
"PIET OUDE LUTTIKHUIS" → capitalized="Piet Oude Luttikhuis"
"IVOI, PAUL D'" → capitalized="Paul d' Ivoi"
"zaʿīm, ḥusnī az-" → capitalized="Ḥusnī az- Zaʿīm"
"jacobus henricus van 't hoff" → capitalized="Jacobus Henricus van 't Hoff"
Expand Down
Loading