You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b78d5af
Browse filesBrowse the repository at this point in the historyBrowse files
Record a country on ctgov-officials and add ctgov-contacts (#1454)
* Record a country on ctgov-officials and add ctgov-contacts (#1451)
An overall official takes the ISO code of the one country every site of
its study is in, and none otherwise; the study's site contacts go in a
sibling corpus, ctgov-contacts, each with its site's country. The
registry's country names are mapped to ISO 3166-1 alpha-2 codes, and an
unmapped name fails the fetch, naming it. Both corpora declare country
as their group, so config_diff.py --group keeps one country, and a
corpus fetched before the country is refused on load by a read version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Refuse a ctgov fetch whose countries a moved key would empty
Review of the country change: a renamed "country" or "locations" key
would have emptied every record's country without failing, since the
corpus-key guard reaches only names. Both fetches now refuse a corpus
holding no country, and one past a measured share of officials with no
site or of records at a site stating no country; a country or role that
is not text fails the fetch naming the study. The docs now give the
contacts' repeat key as name, role, study and country, the precedence of
the officials' site buckets, and where the counts are printed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Date the registry-wide shares behind the ctgov gates
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: tools/differential/README.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -291,7 +291,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage
291
291
292
292
A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names.
293
293
294
-
`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_govuk.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`, `local_volby.py`, `local_dof.py`, `local_uit.py`, `local_vnassembly.py`, `local_parliaments.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this).
294
+
`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_govuk.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`, `local_volby.py`, `local_dof.py`, `local_uit.py`, `local_vnassembly.py`, `local_parliaments.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built, and `CTGOV_READ_VERSION` likewise for the ctgov corpora's country; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this).
295
295
296
296
```
297
297
uv run python tools/differential/local_corpora.py fetch ctgov-officials
@@ -320,7 +320,8 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w
320
320
321
321
| Source | What it is | Blind to |
322
322
|---|---|---|
323
-
|`ctgov-officials`| every overall official of every study on ClinicalTrials.gov, whatever the role (principal investigators, study directors and chairs, and a few sub-investigators and officials with no role), the name as the sponsor's staff typed it into a free-text field: credentials run together (`MBBS,FCPS`), job titles in the name, all-caps and all-lowercase names, typos. Fetched from API v2 (NLM) | names outside clinical research, which is mostly medicine and heavily credentialed; names nobody typed by hand |
323
+
| `ctgov-officials` | every overall official of every study on ClinicalTrials.gov, whatever the role (principal investigators, study directors and chairs, and a few sub-investigators and officials with no role), the name as the sponsor's staff typed it into a free-text field: credentials run together (`MBBS,FCPS`), job titles in the name, all-caps and all-lowercase names, typos. Fetched from API v2 (NLM). Each record carries its study, its role and a `country` (#1451): the ISO 3166-1 alpha-2 code of the one country every site of its study is in, and empty where the sites are in several, one states none or the study lists none -- where the study ran, not where the official is from, the affiliation text left unread. The country is the group `config_diff.py --group` keeps (`--group KR`); `snapshot.py --local` reads the whole corpus | names outside clinical research, which is mostly medicine and heavily credentialed; names nobody typed by hand; the country of an official whose study ran in several countries or lists no site |
324
+
| `ctgov-contacts` | the sibling of `ctgov-officials` (#1451): the named contacts under each study's sites (mostly `CONTACT`, `PRINCIPAL_INVESTIGATOR` and `SUB_INVESTIGATOR`, the role as the registry writes it), typed into the same registry's free-text fields, each with its study, role and its site's `country` code, the group `--group` keeps, empty where the site states none. One record per name, role, study and country: a name listed again with the same role at another site of one study in one country is kept once and counted in `meta.json`'s detail (`repeated_in_study_and_country`). A corpus of its own so that every count quoted from `ctgov-officials` stays true. Central contacts are left out, belonging to no site. Both corpora map the registry's country names (`Turkey (Türkiye)`, `South Korea`) to codes by `local_ctgov.CTGOV_COUNTRIES`, so the groups line up with ORCID's and PatentsView's, and a name it does not hold fails the fetch, naming it. A fetch holding no country at all, or past a share of records without one (`CTGOV_MAX_NO_SITE`, `CTGOV_MAX_COUNTRYLESS`), is refused, as a renamed key would leave it; a corpus fetched before the country, or read by another `CTGOV_READ_VERSION`, is refused on load with the fetch that replaces it | as `ctgov-officials`; studies with no named site contact |
324
325
|`fec-individuals`| the distinct names of every individual contributor in the FEC's bulk file for the 2026 cycle (`indiv26.zip`, downloaded once into the cache and read without unpacking; `fetch --refresh` downloads it again). Mostly a filer's software writing a form's fields in a fixed order: of its 1,893,964 names on 2026-10-18 every one was in capitals and 1,893,961 held a comma, with hand-typed text inside the fields -- 3,306 held a second comma, for a credential run or a suffix in a part of its own (`DOE, JR., JOHN`). The population for the one-case family-comma shape. The FEC's terms (52 U.S.C. 30111(a)(4)) bar using the names to solicit contributions or for any commercial purpose | case, since every name is in capitals; names written in their spoken order, or without a comma; small donors, since a committee need itemize only a donor whose gifts to it pass $200; names outside the US; names in any script but Latin |
325
326
| `tse-donors` | every individual donor to a candidate in Brazil's 2022 general and 2024 municipal elections (the TSE's candidates' accounts, rows from `Recursos de pessoas físicas`), a record per distinct pair of the name as the campaign filed it (`name`) and the federal tax registry's name for the same donor (`registry`): a typed form with its reference form beside it. Of 854,531 records fetched 2026-10-22 (848,351 distinct filed names), 205,925 differ from their registry name -- an agnomen dropped or abbreviated (`JR` filed for the registry's `JUNIOR` in 1,615, `JÚNIOR` for `JUNIOR` in 975), a word left out. Nearly all in capitals (845,498 filed names), in spoken order with no comma (184 records hold one), and full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO` ends 10,955 distinct filed names, `JUNIOR` 13,784, `NETO` 8,410). Published by the TSE under CC BY | case, mostly; the family-comma shape; credentials and titles, which a donation form has no place for; donors to parties rather than candidates; names outside Brazil; names in any script but Latin |
326
327
| `tw-donors` | the distinct names of every individual donor in every campaign finance report Taiwan's Control Yuan publishes on its political donation platform (ardata.cy.gov.tw, whose zips data.gov.tw also lists): 454 reports of 64 elections on 2026-10-22, presidential, legislative and local: every report from 2018's elections on, and the residual reports of 2014's and 2016's. Each report's zip is cached under the name the platform lists it by, which carries the date it was built, so a refresh downloads only the reports published or rebuilt since. Names as a candidate's staff typed them into a filing, published in full where the ID, address and phone beside them are masked. Of the 308,667 names dated 2026-10-05, 296,820 were three Han characters, 5,449 four and 5,260 two; the population for unspaced Han names written family first, and for the married double surname (冠夫姓) a four-character name often is. Under the Open Government Data License, version 1.0, with the platform's own bar on using the data for profit, credit investigation or another improper purpose | names in spoken order; Latin script (80 of the names dated 2026-10-05 held a Latin letter); the comma (3 of them held a fullwidth one, none an ASCII one); case; names outside Taiwan; donors whose names the report does not publish, such as anonymous gifts; Japanese kanji names, which the Han script cannot tell from Chinese |
@@ -395,7 +396,7 @@ uv run python tools/differential/measure.py --base ../master-checkout
395
396
396
397
The steps: the committed snapshot (`snapshot.py`, the check `tests/v2/test_snapshot.py` makes), `cldr_agreement.py`, `snapshot.py --local --base` for every fetched local corpus, `local_agreement.py SOURCE --base` for every fetched source whose records carry their fields (`psc-individuals`, `orcid-researchers`, `patentsview-inventors`, `cz-candidates`, `uit-vinames`, `vn-assembly`), and `local_corpora.py alike --base` for every fetched source that pairs spellings. Only the local steps read `--base`: the first two compare the tree with the records committed on its branch, so on a branch that has re-recorded `snapshot.jsonl` the committed snapshot says no moves and the moves are in the PR's diff of that file, and the summary labels each step with what it compared against. A local corpus that is not fetched gets no step, the summary names it, and a run in which no step read `--base` exits 1. Each step's whole output goes to a file of its own under `--out`, by default a folder named for the time under the cache's `measure/`, outside the repository, since a local snapshot lists its moves by name. The summary, `summary.txt` beside them and printed, gives each step's status (no moves, moves, or failed with its exit code), its time, and its unindented lines (the first few and the last, where the verdicts are, up to `SUMMARY_LINES`, and any line saying part of a check was not measured), never the indented ones where the tools list names; a crash's traceback is indented below a line naming only its type, since its message may hold a name. A step that fails makes the run exit 1; one that finds moves does not. Each tool refuses with `SystemExit("message")`, which exits 1 as a move does, so measure runs every step through its own `--step` wrapper, which turns a message exit or an uncaught exception into exit 2: a refused base or a damaged corpus is reported as a failure, never as moves.
397
398
398
-
**Comparing configurations in one tree.** `snapshot.py --local` compares two trees under one configuration and `local_agreement.py` scores one configuration against the filed boxes; `config_diff.py diff` compares two or more CONFIGURATIONS in the tree it runs in, over the committed corpora, any fetched local corpus (`--local SOURCE`, or `all`), a fielded corpus's countries or nationalities as it writes them, or `opensanctions-parliaments`' datasets (`--group`, which needs `--local` and refuses a group no record holds; a corpus with no fields names its group key in its `Source`'s `group`), the names a regular expression finds (`--match`) and a hash sample (`--sample N`), each name rewritten first where `--spelling` asks (`stripped`, `upper`, `lower`, comma-separated: `--spelling stripped,upper` reads a Vietnamese population as plain-ASCII data in capitals writes it, #1108). A configuration is `default`, `plain:<items>` or `strict:<items>` (`parser_for` over the items joined by `+`: a shipped pack in capitals, `ES`, `PT_BR`, a bare language tag in lower case, `pt-BR`), or `expr:<python>` for any other lexicon or policy; the first is the baseline. It compares the seven roles, the ambiguity kinds and 3.0's views (`generation`, `credentials`, `honorifics`, `patronymic`, `surname2`), and prints, per configuration, the names that moved by input and grouped by the fields that changed, and the words whose token changed role, ranked, each with an example; `--json` keeps the counts and the moved names, `--readings` every name's readings one line a name, so a run in each of two trees compares line by line. Every parse runs in `--jobs` worker processes (`snapshot.parallel`). Reach for it before writing a script that parses the corpora under two configurations: #1000, #1005, #1017 and #1019 each wrote one.
399
+
**Comparing configurations in one tree.** `snapshot.py --local` compares two trees under one configuration and `local_agreement.py` scores one configuration against the filed boxes; `config_diff.py diff` compares two or more CONFIGURATIONS in the tree it runs in, over the committed corpora, any fetched local corpus (`--local SOURCE`, or `all`), a fielded corpus's countries or nationalities as it writes them, `opensanctions-parliaments`' datasets, or the ctgov corpora's ISO country codes (`--group`, which needs `--local` and refuses a group no record holds; a corpus with no fields names its group key in its `Source`'s `group`), the names a regular expression finds (`--match`) and a hash sample (`--sample N`), each name rewritten first where `--spelling` asks (`stripped`, `upper`, `lower`, comma-separated: `--spelling stripped,upper` reads a Vietnamese population as plain-ASCII data in capitals writes it, #1108). A configuration is `default`, `plain:<items>` or `strict:<items>` (`parser_for` over the items joined by `+`: a shipped pack in capitals, `ES`, `PT_BR`, a bare language tag in lower case, `pt-BR`), or `expr:<python>` for any other lexicon or policy; the first is the baseline. It compares the seven roles, the ambiguity kinds and 3.0's views (`generation`, `credentials`, `honorifics`, `patronymic`, `surname2`), and prints, per configuration, the names that moved by input and grouped by the fields that changed, and the words whose token changed role, ranked, each with an example; `--json` keeps the counts and the moved names, `--readings` every name's readings one line a name, so a run in each of two trees compares line by line. Every parse runs in `--jobs` worker processes (`snapshot.parallel`). Reach for it before writing a script that parses the corpora under two configurations: #1000, #1005, #1017 and #1019 each wrote one.
0 commit comments