From 1f1446ca533d5e0d34e503ed2acf05a8f00a2ded Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 15:39:00 +1000 Subject: [PATCH 1/2] mpd2020.xlsx: record the repoint Flips the tracker once both consuming lectures read this repo, and records them in the manifest's consumers list so a future correction knows what to rebuild. Held as a draft deliberately. The strict audit has no green state for a dataset whose consumers have only partly moved, so this must not merge before QuantEcon/lecture-python-intro#823 and QuantEcon/lecture-wasm#52. Verified both ways: against the lecture repos' current main this fails with "marked repointed but consumers still read via ['own-repo', 'sibling']"; against their repoint branches it passes clean, and the orphan count falls 35 -> 34 as wasm's mirror copy goes. mpd2020 is the first dataset to exercise the two-repo rule end to end. intro's copy could not be deleted while wasm fetched that exact blob by URL, so the pair moved together and this flip trails both. Co-Authored-By: Claude Opus 5 (1M context) --- CATALOG.md | 4 ++-- lectures/mpd2020.xlsx.yml | 6 +++++- migration.yml | 12 ++++++++++-- 3 files changed, 17 insertions(+), 5 deletions(-) diff --git a/CATALOG.md b/CATALOG.md index 8d6f9bc..d8ec2bc 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -6,7 +6,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md). -**18 datasets** · 10 read by lectures today, 8 awaiting repoint · 5.3 MB total · 17 permitted / 1 restricted redistribution +**18 datasets** · 11 read by lectures today, 7 awaiting repoint · 5.3 MB total · 17 permitted / 1 restricted redistribution | Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | @@ -24,7 +24,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)
Japan — population by single year of age, 2024 | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) | | [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)
Pacific Coast lingcod — biomass and fishing pressure relative to MSY | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) | | [**longprices.xls**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/longprices.xls)
Price levels in four hard-currency countries, 1600-2000 | verbatim | [Sargent and Velde, "The Big Problem of Small Change" — supporting data](https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 379.5 KB | — | -| [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)
Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ⇄ diverged | ⚠️ unrecovered | 1.7 MB | — | +| [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)
Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ⇄ diverged | ⚠️ unrecovered | 1.7 MB | [lecture-python-intro · long_run_growth.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/long_run_growth.md)
[lecture-wasm · long_run_growth.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/long_run_growth.md) | | [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)
French Revolution — monthly nominal assignat balances, 1789-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | — | | [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)
OECD real minimum wages — 32 countries, 2006–2016 | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | | [**us_adult_heights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_adult_heights.csv)
United States — adult standing height by sex, NHANES 2015-2018 | constructed | [National Health and Nutrition Examination Survey (NHANES), US Centers for Disease Control and Prevention, National Center for Health Statistics](https://www.cdc.gov/nchs/nhanes/index.htm) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 123.1 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md)
[lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | diff --git a/lectures/mpd2020.xlsx.yml b/lectures/mpd2020.xlsx.yml index 14f4ec5..a40b39e 100644 --- a/lectures/mpd2020.xlsx.yml +++ b/lectures/mpd2020.xlsx.yml @@ -137,7 +137,11 @@ schema: # Empty until the Track A repoint lands. lecture-python-intro and lecture-wasm # both read this file today, but from intro's own copy, not from here. -consumers: [] +consumers: + - repo: QuantEcon/lecture-python-intro + file: lectures/long_run_growth.md + - repo: QuantEcon/lecture-wasm + file: lectures/long_run_growth.md # --------------------------------------------------------------------------- # Builder diff --git a/migration.yml b/migration.yml index 885704a..ca1a2cd 100644 --- a/migration.yml +++ b/migration.yml @@ -182,12 +182,20 @@ datasets: mpd2020.xlsx: pilot: A1 - status: landed + status: repointed prior_pattern: own-repo landed: pr: QuantEcon/data-lectures#5 date: 2025-02-14 - repoints: [] + # Two consuming repos, moved together: intro's copy could not be deleted + # while wasm fetched that exact blob by URL (PLAN, "Repoint rules" 1). + repoints: + - repo: QuantEcon/lecture-python-intro + pr: QuantEcon/lecture-python-intro#823 + date: 2026-08-06 + - repo: QuantEcon/lecture-wasm + pr: QuantEcon/lecture-wasm#52 + date: 2026-08-06 cutover: null longprices.xls: From fbd2889d13746c3d64a16e34976bbab7cdfdc8d8 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 15:55:58 +1000 Subject: [PATCH 2/2] Make the consumers comment durable instead of point-in-time Copilot caught mpd2020's Consumers comment still reading "Empty until the Track A repoint lands ... lectures read this file from intro's own copy" while sitting directly above a populated consumers list naming both repos reading it from here. It flatly contradicted the data beneath it. The mistake is structural rather than a typo: the comment described the field's CURRENT VALUE, so it was guaranteed to go stale the moment its own PR landed. The identical comment was in all eight manifests from #38, so it would have gone stale again on repoint sets 2 and 3 and been flagged both times. All eight now describe what the field MEANS, which stays true in either state: every lecture that reads this file from this repo, empty not implying the file is unused since it may still be read from a lecture repo's own copy -- which is exactly what a repoint moves -- and populated in the repoint PR alongside the migration.yml flip. Comments only. Verified: every manifest still parses, consumers values are untouched (mpd2020 two, the other seven zero), CATALOG.md regenerates byte-identical, and the strict audit stays green. Co-Authored-By: Claude Opus 5 (1M context) --- lectures/assignat.xlsx.yml | 6 +++++- lectures/caron.npy.yml | 6 +++++- lectures/chapter_3.xlsx.yml | 6 +++++- lectures/dette.xlsx.yml | 6 +++++- lectures/fig_3.xlsx.yml | 6 +++++- lectures/longprices.xls.yml | 6 +++++- lectures/mpd2020.xlsx.yml | 8 +++++--- lectures/nom_balances.npy.yml | 6 +++++- 8 files changed, 40 insertions(+), 10 deletions(-) diff --git a/lectures/assignat.xlsx.yml b/lectures/assignat.xlsx.yml index 078e326..c00161d 100644 --- a/lectures/assignat.xlsx.yml +++ b/lectures/assignat.xlsx.yml @@ -103,7 +103,11 @@ schema: row_count_floor: null # per-range, declared above known_nulls: {} -consumers: [] # empty until the Track A repoint lands +# Every lecture that reads this file FROM THIS REPO. Empty does not mean the +# file is unused -- it may still be consumed from a lecture repo's own copy, +# which is exactly what a repoint moves. Populated in the repoint PR, at the +# same time migration.yml flips to `repointed` (PLAN, "Repoint rules"). +consumers: [] builder: null builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/caron.npy.yml b/lectures/caron.npy.yml index 6534684..5f69c78 100644 --- a/lectures/caron.npy.yml +++ b/lectures/caron.npy.yml @@ -90,7 +90,11 @@ schema: of this array with rows 14:77 of nom_balances.npy. Changing the length of either silently misaligns the other. -consumers: [] # empty until the Track A repoint lands +# Every lecture that reads this file FROM THIS REPO. Empty does not mean the +# file is unused -- it may still be consumed from a lecture repo's own copy, +# which is exactly what a repoint moves. Populated in the repoint PR, at the +# same time migration.yml flips to `repointed` (PLAN, "Repoint rules"). +consumers: [] # --------------------------------------------------------------------------- # Builder diff --git a/lectures/chapter_3.xlsx.yml b/lectures/chapter_3.xlsx.yml index f7d0a0c..6d157ac 100644 --- a/lectures/chapter_3.xlsx.yml +++ b/lectures/chapter_3.xlsx.yml @@ -90,7 +90,11 @@ schema: map to strip whitespace and handle footnote markers), so dtypes are not stable across sheets and are deliberately not asserted here. -consumers: [] # empty until the Track A repoint lands +# Every lecture that reads this file FROM THIS REPO. Empty does not mean the +# file is unused -- it may still be consumed from a lecture repo's own copy, +# which is exactly what a repoint moves. Populated in the repoint PR, at the +# same time migration.yml flips to `repointed` (PLAN, "Repoint rules"). +consumers: [] # --------------------------------------------------------------------------- # Builder diff --git a/lectures/dette.xlsx.yml b/lectures/dette.xlsx.yml index c9421f5..95dfdd8 100644 --- a/lectures/dette.xlsx.yml +++ b/lectures/dette.xlsx.yml @@ -108,7 +108,11 @@ schema: sparse layouts in a working spreadsheet, and the lecture indexes specific columns out of them rather than consuming the block whole. -consumers: [] # empty until the Track A repoint lands +# Every lecture that reads this file FROM THIS REPO. Empty does not mean the +# file is unused -- it may still be consumed from a lecture repo's own copy, +# which is exactly what a repoint moves. Populated in the repoint PR, at the +# same time migration.yml flips to `repointed` (PLAN, "Repoint rules"). +consumers: [] builder: null builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/fig_3.xlsx.yml b/lectures/fig_3.xlsx.yml index 87702f7..cef1997 100644 --- a/lectures/fig_3.xlsx.yml +++ b/lectures/fig_3.xlsx.yml @@ -83,7 +83,11 @@ schema: row_count_floor: 30 # exact by design -- a frozen 30-row block known_nulls: {} # declared per column above -consumers: [] # empty until the Track A repoint lands +# Every lecture that reads this file FROM THIS REPO. Empty does not mean the +# file is unused -- it may still be consumed from a lecture repo's own copy, +# which is exactly what a repoint moves. Populated in the repoint PR, at the +# same time migration.yml flips to `repointed` (PLAN, "Repoint rules"). +consumers: [] builder: null builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/longprices.xls.yml b/lectures/longprices.xls.yml index 9c81a1e..c177eb1 100644 --- a/lectures/longprices.xls.yml +++ b/lectures/longprices.xls.yml @@ -88,7 +88,11 @@ schema: row_count_floor: 401 # frozen historical extract: exact by design known_nulls: {} # declared per sheet above -consumers: [] # empty until the Track A repoint lands +# Every lecture that reads this file FROM THIS REPO. Empty does not mean the +# file is unused -- it may still be consumed from a lecture repo's own copy, +# which is exactly what a repoint moves. Populated in the repoint PR, at the +# same time migration.yml flips to `repointed` (PLAN, "Repoint rules"). +consumers: [] builder: null builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/mpd2020.xlsx.yml b/lectures/mpd2020.xlsx.yml index a40b39e..80f6201 100644 --- a/lectures/mpd2020.xlsx.yml +++ b/lectures/mpd2020.xlsx.yml @@ -132,10 +132,12 @@ schema: known_nulls: {} # declared per sheet above # --------------------------------------------------------------------------- -# Consumers +# Consumers -- how a correction knows what to rebuild # --------------------------------------------------------------------------- -# Empty until the Track A repoint lands. lecture-python-intro and lecture-wasm -# both read this file today, but from intro's own copy, not from here. +# Every lecture that reads this file FROM THIS REPO. Empty does not mean the +# file is unused -- it may still be consumed from a lecture repo's own copy, +# which is exactly what a repoint moves. Populated in the repoint PR, at the +# same time migration.yml flips to `repointed` (PLAN, "Repoint rules"). consumers: - repo: QuantEcon/lecture-python-intro diff --git a/lectures/nom_balances.npy.yml b/lectures/nom_balances.npy.yml index 8d2ad73..0a260e0 100644 --- a/lectures/nom_balances.npy.yml +++ b/lectures/nom_balances.npy.yml @@ -85,7 +85,11 @@ schema: (late 1789 through 1790) exist to extend the circulation series before the price series begins, and must not be trimmed. -consumers: [] # empty until the Track A repoint lands +# Every lecture that reads this file FROM THIS REPO. Empty does not mean the +# file is unused -- it may still be consumed from a lecture repo's own copy, +# which is exactly what a repoint moves. Populated in the repoint PR, at the +# same time migration.yml flips to `repointed` (PLAN, "Repoint rules"). +consumers: [] # --------------------------------------------------------------------------- # Builder