Skip to content

Commit b78d5af

Browse files
derek73claude
andauthored
Record a country on ctgov-officials and add ctgov-contacts (#1454)
* Record a country on ctgov-officials and add ctgov-contacts (#1451) An overall official takes the ISO code of the one country every site of its study is in, and none otherwise; the study's site contacts go in a sibling corpus, ctgov-contacts, each with its site's country. The registry's country names are mapped to ISO 3166-1 alpha-2 codes, and an unmapped name fails the fetch, naming it. Both corpora declare country as their group, so config_diff.py --group keeps one country, and a corpus fetched before the country is refused on load by a read version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Refuse a ctgov fetch whose countries a moved key would empty Review of the country change: a renamed "country" or "locations" key would have emptied every record's country without failing, since the corpus-key guard reaches only names. Both fetches now refuse a corpus holding no country, and one past a measured share of officials with no site or of records at a site stating no country; a country or role that is not text fails the fetch naming the study. The docs now give the contacts' repeat key as name, role, study and country, the precedence of the officials' site buckets, and where the counts are printed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Date the registry-wide shares behind the ctgov gates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent c878443 commit b78d5af

10 files changed

Lines changed: 587 additions & 60 deletions

File tree

‎tests/v2/test_config_diff.py‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,7 @@ def test_a_source_holding_none_of_a_group_says_so(
249249

250250
def test_a_group_needs_a_grouped_corpus() -> None:
251251
with pytest.raises(SystemExit, match="needs a grouped corpus"):
252-
cd.local_names("ctgov-officials", ["US"])
252+
cd.local_names("fec-individuals", ["US"])
253253

254254

255255
def test_local_all_with_a_group_reads_the_grouped_corpora_only(
@@ -277,6 +277,21 @@ def test_a_group_keeps_a_grouped_corpus_s_records_by_its_source_s_key(
277277
assert [t for t, _o, _i in got] == ["Ana Souza"]
278278

279279

280+
@pytest.mark.parametrize("source", ["ctgov-officials", "ctgov-contacts"])
281+
def test_a_group_keeps_a_ctgov_corpus_s_records_by_country(
282+
monkeypatch: pytest.MonkeyPatch, source: str) -> None:
283+
# A record with no country (a study's sites in several) is in no group.
284+
records = [{"name": name, "study": "NCT1", "role": "", "country": country}
285+
for name, country in (("Ana Souza", "BR"), ("Bo Lima", "KR"),
286+
("Cy Dias", ""))]
287+
plant_corpus(monkeypatch, cd.local_corpora, source, records, "2026-11-18", "officials",
288+
{"read_version": importlib.import_module("local_ctgov").CTGOV_READ_VERSION})
289+
assert cd.group_keys()[source] == "country"
290+
assert cd.local_names(source, ["KR"]) == (["Bo Lima"], {"KR"})
291+
got = cd.gather(_args(local=[source], group=["BR"]))
292+
assert [t for t, _o, _i in got] == ["Ana Souza"]
293+
294+
280295
@needs_checkout
281296
def test_main_runs_a_diff_end_to_end(monkeypatch: pytest.MonkeyPatch,
282297
capsys: pytest.CaptureFixture[str], tmp_path: Path) -> None:

‎tests/v2/test_local_corpora.py‎

Lines changed: 251 additions & 7 deletions
Large diffs are not rendered by default.

‎tests/v2/test_measure.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ def run(step: object) -> object:
101101
assert "Zyx" not in summary
102102
assert "none failed" in summary
103103
assert "== committed snapshot (against this branch's committed record): moves" in summary
104-
assert "not measured, not fetched: ctgov-officials," in summary
104+
assert "not measured, not fetched: ctgov-contacts, ctgov-officials," in summary
105105
assert (out / "01-committed-snapshot.txt").read_text().endswith("Ann Zyx\n")
106106
codes["CLDR and GND agreement"] = 2
107107
assert measure.main(["--base", str(base), "--out", str(out)]) == 1

‎tests/v2/test_word_report.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
the output. The corpus names are invented, and each carries a word no
99
grid shape holds, so that one in the output is a quoted name.
1010
"""
11+
import importlib
1112
import importlib.util
1213
import sys
1314
from collections import Counter
@@ -60,7 +61,8 @@ def _no_usage(monkeypatch: pytest.MonkeyPatch) -> None:
6061
def _plant(monkeypatch: pytest.MonkeyPatch, names: list[str] = _NAMES) -> None:
6162
plant_corpus(monkeypatch, corpora, "ctgov-officials",
6263
[{"name": n, "nct": str(i)} for i, n in enumerate(names)],
63-
"2026-10-21T09:00:05", "officials")
64+
"2026-10-21T09:00:05", "officials",
65+
{"read_version": importlib.import_module("local_ctgov").CTGOV_READ_VERSION})
6466

6567

6668
def test_the_grid_spells_every_shape_three_ways_under_every_order() -> None:

‎tools/differential/README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -291,7 +291,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage
291291

292292
A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names.
293293

294-
`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_govuk.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`, `local_volby.py`, `local_dof.py`, `local_uit.py`, `local_vnassembly.py`, `local_parliaments.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this).
294+
`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_govuk.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`, `local_volby.py`, `local_dof.py`, `local_uit.py`, `local_vnassembly.py`, `local_parliaments.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built, and `CTGOV_READ_VERSION` likewise for the ctgov corpora's country; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this).
295295

296296
```
297297
uv run python tools/differential/local_corpora.py fetch ctgov-officials
@@ -320,7 +320,8 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w
320320

321321
| Source | What it is | Blind to |
322322
|---|---|---|
323-
| `ctgov-officials` | every overall official of every study on ClinicalTrials.gov, whatever the role (principal investigators, study directors and chairs, and a few sub-investigators and officials with no role), the name as the sponsor's staff typed it into a free-text field: credentials run together (`MBBS,FCPS`), job titles in the name, all-caps and all-lowercase names, typos. Fetched from API v2 (NLM) | names outside clinical research, which is mostly medicine and heavily credentialed; names nobody typed by hand |
323+
| `ctgov-officials` | every overall official of every study on ClinicalTrials.gov, whatever the role (principal investigators, study directors and chairs, and a few sub-investigators and officials with no role), the name as the sponsor's staff typed it into a free-text field: credentials run together (`MBBS,FCPS`), job titles in the name, all-caps and all-lowercase names, typos. Fetched from API v2 (NLM). Each record carries its study, its role and a `country` (#1451): the ISO 3166-1 alpha-2 code of the one country every site of its study is in, and empty where the sites are in several, one states none or the study lists none -- where the study ran, not where the official is from, the affiliation text left unread. The country is the group `config_diff.py --group` keeps (`--group KR`); `snapshot.py --local` reads the whole corpus | names outside clinical research, which is mostly medicine and heavily credentialed; names nobody typed by hand; the country of an official whose study ran in several countries or lists no site |
324+
| `ctgov-contacts` | the sibling of `ctgov-officials` (#1451): the named contacts under each study's sites (mostly `CONTACT`, `PRINCIPAL_INVESTIGATOR` and `SUB_INVESTIGATOR`, the role as the registry writes it), typed into the same registry's free-text fields, each with its study, role and its site's `country` code, the group `--group` keeps, empty where the site states none. One record per name, role, study and country: a name listed again with the same role at another site of one study in one country is kept once and counted in `meta.json`'s detail (`repeated_in_study_and_country`). A corpus of its own so that every count quoted from `ctgov-officials` stays true. Central contacts are left out, belonging to no site. Both corpora map the registry's country names (`Turkey (Türkiye)`, `South Korea`) to codes by `local_ctgov.CTGOV_COUNTRIES`, so the groups line up with ORCID's and PatentsView's, and a name it does not hold fails the fetch, naming it. A fetch holding no country at all, or past a share of records without one (`CTGOV_MAX_NO_SITE`, `CTGOV_MAX_COUNTRYLESS`), is refused, as a renamed key would leave it; a corpus fetched before the country, or read by another `CTGOV_READ_VERSION`, is refused on load with the fetch that replaces it | as `ctgov-officials`; studies with no named site contact |
324325
| `fec-individuals` | the distinct names of every individual contributor in the FEC's bulk file for the 2026 cycle (`indiv26.zip`, downloaded once into the cache and read without unpacking; `fetch --refresh` downloads it again). Mostly a filer's software writing a form's fields in a fixed order: of its 1,893,964 names on 2026-10-18 every one was in capitals and 1,893,961 held a comma, with hand-typed text inside the fields -- 3,306 held a second comma, for a credential run or a suffix in a part of its own (`DOE, JR., JOHN`). The population for the one-case family-comma shape. The FEC's terms (52 U.S.C. 30111(a)(4)) bar using the names to solicit contributions or for any commercial purpose | case, since every name is in capitals; names written in their spoken order, or without a comma; small donors, since a committee need itemize only a donor whose gifts to it pass $200; names outside the US; names in any script but Latin |
325326
| `tse-donors` | every individual donor to a candidate in Brazil's 2022 general and 2024 municipal elections (the TSE's candidates' accounts, rows from `Recursos de pessoas físicas`), a record per distinct pair of the name as the campaign filed it (`name`) and the federal tax registry's name for the same donor (`registry`): a typed form with its reference form beside it. Of 854,531 records fetched 2026-10-22 (848,351 distinct filed names), 205,925 differ from their registry name -- an agnomen dropped or abbreviated (`JR` filed for the registry's `JUNIOR` in 1,615, `JÚNIOR` for `JUNIOR` in 975), a word left out. Nearly all in capitals (845,498 filed names), in spoken order with no comma (184 records hold one), and full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO` ends 10,955 distinct filed names, `JUNIOR` 13,784, `NETO` 8,410). Published by the TSE under CC BY | case, mostly; the family-comma shape; credentials and titles, which a donation form has no place for; donors to parties rather than candidates; names outside Brazil; names in any script but Latin |
326327
| `tw-donors` | the distinct names of every individual donor in every campaign finance report Taiwan's Control Yuan publishes on its political donation platform (ardata.cy.gov.tw, whose zips data.gov.tw also lists): 454 reports of 64 elections on 2026-10-22, presidential, legislative and local: every report from 2018's elections on, and the residual reports of 2014's and 2016's. Each report's zip is cached under the name the platform lists it by, which carries the date it was built, so a refresh downloads only the reports published or rebuilt since. Names as a candidate's staff typed them into a filing, published in full where the ID, address and phone beside them are masked. Of the 308,667 names dated 2026-10-05, 296,820 were three Han characters, 5,449 four and 5,260 two; the population for unspaced Han names written family first, and for the married double surname (冠夫姓) a four-character name often is. Under the Open Government Data License, version 1.0, with the platform's own bar on using the data for profit, credit investigation or another improper purpose | names in spoken order; Latin script (80 of the names dated 2026-10-05 held a Latin letter); the comma (3 of them held a fullwidth one, none an ASCII one); case; names outside Taiwan; donors whose names the report does not publish, such as anonymous gifts; Japanese kanji names, which the Han script cannot tell from Chinese |
@@ -395,7 +396,7 @@ uv run python tools/differential/measure.py --base ../master-checkout
395396

396397
The steps: the committed snapshot (`snapshot.py`, the check `tests/v2/test_snapshot.py` makes), `cldr_agreement.py`, `snapshot.py --local --base` for every fetched local corpus, `local_agreement.py SOURCE --base` for every fetched source whose records carry their fields (`psc-individuals`, `orcid-researchers`, `patentsview-inventors`, `cz-candidates`, `uit-vinames`, `vn-assembly`), and `local_corpora.py alike --base` for every fetched source that pairs spellings. Only the local steps read `--base`: the first two compare the tree with the records committed on its branch, so on a branch that has re-recorded `snapshot.jsonl` the committed snapshot says no moves and the moves are in the PR's diff of that file, and the summary labels each step with what it compared against. A local corpus that is not fetched gets no step, the summary names it, and a run in which no step read `--base` exits 1. Each step's whole output goes to a file of its own under `--out`, by default a folder named for the time under the cache's `measure/`, outside the repository, since a local snapshot lists its moves by name. The summary, `summary.txt` beside them and printed, gives each step's status (no moves, moves, or failed with its exit code), its time, and its unindented lines (the first few and the last, where the verdicts are, up to `SUMMARY_LINES`, and any line saying part of a check was not measured), never the indented ones where the tools list names; a crash's traceback is indented below a line naming only its type, since its message may hold a name. A step that fails makes the run exit 1; one that finds moves does not. Each tool refuses with `SystemExit("message")`, which exits 1 as a move does, so measure runs every step through its own `--step` wrapper, which turns a message exit or an uncaught exception into exit 2: a refused base or a damaged corpus is reported as a failure, never as moves.
397398

398-
**Comparing configurations in one tree.** `snapshot.py --local` compares two trees under one configuration and `local_agreement.py` scores one configuration against the filed boxes; `config_diff.py diff` compares two or more CONFIGURATIONS in the tree it runs in, over the committed corpora, any fetched local corpus (`--local SOURCE`, or `all`), a fielded corpus's countries or nationalities as it writes them, or `opensanctions-parliaments`' datasets (`--group`, which needs `--local` and refuses a group no record holds; a corpus with no fields names its group key in its `Source`'s `group`), the names a regular expression finds (`--match`) and a hash sample (`--sample N`), each name rewritten first where `--spelling` asks (`stripped`, `upper`, `lower`, comma-separated: `--spelling stripped,upper` reads a Vietnamese population as plain-ASCII data in capitals writes it, #1108). A configuration is `default`, `plain:<items>` or `strict:<items>` (`parser_for` over the items joined by `+`: a shipped pack in capitals, `ES`, `PT_BR`, a bare language tag in lower case, `pt-BR`), or `expr:<python>` for any other lexicon or policy; the first is the baseline. It compares the seven roles, the ambiguity kinds and 3.0's views (`generation`, `credentials`, `honorifics`, `patronymic`, `surname2`), and prints, per configuration, the names that moved by input and grouped by the fields that changed, and the words whose token changed role, ranked, each with an example; `--json` keeps the counts and the moved names, `--readings` every name's readings one line a name, so a run in each of two trees compares line by line. Every parse runs in `--jobs` worker processes (`snapshot.parallel`). Reach for it before writing a script that parses the corpora under two configurations: #1000, #1005, #1017 and #1019 each wrote one.
399+
**Comparing configurations in one tree.** `snapshot.py --local` compares two trees under one configuration and `local_agreement.py` scores one configuration against the filed boxes; `config_diff.py diff` compares two or more CONFIGURATIONS in the tree it runs in, over the committed corpora, any fetched local corpus (`--local SOURCE`, or `all`), a fielded corpus's countries or nationalities as it writes them, `opensanctions-parliaments`' datasets, or the ctgov corpora's ISO country codes (`--group`, which needs `--local` and refuses a group no record holds; a corpus with no fields names its group key in its `Source`'s `group`), the names a regular expression finds (`--match`) and a hash sample (`--sample N`), each name rewritten first where `--spelling` asks (`stripped`, `upper`, `lower`, comma-separated: `--spelling stripped,upper` reads a Vietnamese population as plain-ASCII data in capitals writes it, #1108). A configuration is `default`, `plain:<items>` or `strict:<items>` (`parser_for` over the items joined by `+`: a shipped pack in capitals, `ES`, `PT_BR`, a bare language tag in lower case, `pt-BR`), or `expr:<python>` for any other lexicon or policy; the first is the baseline. It compares the seven roles, the ambiguity kinds and 3.0's views (`generation`, `credentials`, `honorifics`, `patronymic`, `surname2`), and prints, per configuration, the names that moved by input and grouped by the fields that changed, and the words whose token changed role, ranked, each with an example; `--json` keeps the counts and the moved names, `--readings` every name's readings one line a name, so a run in each of two trees compares line by line. Every parse runs in `--jobs` worker processes (`snapshot.parallel`). Reach for it before writing a script that parses the corpora under two configurations: #1000, #1005, #1017 and #1019 each wrote one.
399400

400401
```
401402
uv run python tools/differential/config_diff.py diff default strict:PT_BR --local orcid-researchers --group BR

0 commit comments

Comments
 (0)