Write manifests for the eight intro statics, and check them against upstream by mmcky · Pull Request #38 · QuantEcon/data-lectures

mmcky · 2026-08-06T04:11:26Z

Manifests for the eight static files that landed in the Feb 2025 migration (#5) and have sat in lectures/ without metadata ever since. They are Track A's repoint targets, and a repoint needs a manifest — so the metadata research lands here as one PR, leaving the three repoint sets purely mechanical.

Status is landed, consumers empty. The files are here and documented, but no lecture reads them from this repo yet. The strict audit is green in exactly that state — I verified that before writing anything, rather than assuming it.

Two findings from actually checking

`mpd2020.xlsx` is not a pristine upstream copy

I re-fetched it from the Maddison release and compared cell by cell across all six sheets. Every data value matches, including all 21,683 rows of Full data. Three header labels on Regional data are local edits:

Cell	Ours	Upstream
row0 col1	`gdppc_2011`	`GDP pc 2011 prices`
row0 col9	`pop`	`Population`
row0 col18	`gdppc_2011`	(empty)

The lecture reads that sheet with header=(0,1,2), so those renames are load-bearing — replacing this file with a clean upstream copy would silently break long_run_growth, with no error, just different output. That is precisely the trap a manifest exists to catch, and it would have been invisible if I had transcribed the audit's "verbatim (Maddison Project 2020)" instead of downloading the file.

Consequences: classified constructed rather than verbatim (a file we have modified is not "republished as distributed"), and the delta is enumerated in integrity.upstream.note so recovery is cheap rather than lost. Licence confirmed CC BY 4.0 against the release page, and the workbook's own Notes sheet turns out to carry a citation policy requiring the original source papers be cited when data is shown graphically or fewer than twelve countries are used — the lecture does both, so it is a live obligation. Recorded as source.citation_policy.

`caron.npy` and `nom_balances.npy` have no recorded provenance at all

The lecture's own Data Sources section names three spreadsheets and omits both files. So I tested the obvious hypothesis — that they are dumps of columns in the committed Sargent–Velde workbooks. Neither is. caron's terminal values (96.696, 0.431) appear nowhere in assignat.xlsx, dette.xlsx or fig_3.xlsx. nom_balances' final value 33,555.59 does occur in several assignat.xlsx sheets, but no column holds the 81-value series contiguously and its maximum 37,540.9 appears nowhere.

So they cannot be re-derived from what we hold, and the manifests say that plainly instead of inventing a source. The filename caron most likely points to Pierre Caron's published assignat depreciation tables, but that is an inference from a filename and is recorded as a lead explicitly flagged as inference, not promoted to a source claim.

These two are also the local-path reads (np.load('datasets/caron.npy')) that break downloaded and Colab notebooks — the same failure mode P1 fixed for msy_fishery, and the reason this batch is worth doing.

Classification calls

File	Class	Why
`mpd2020.xlsx`	constructed	modified upstream release — three header edits
`longprices.xls`, `assignat.xlsx`, `dette.xlsx`, `fig_3.xlsx`	verbatim	the authors' own files republished unmodified; `unverifiable` upstream because there is no publisher distribution point to re-fetch from
`chapter_3.xlsx`	constructed	the lecture states it was transcribed by hand from printed appendix tables — our processing, and one that cannot ship a builder
`caron.npy`, `nom_balances.npy`	constructed	derived arrays, origin unknown; `builder_status: unrecovered`

Two schema notes for #14

These are the first multi-sheet workbooks to get manifests, and they exercise the schema in ways the flat-table sketch does not cover.

A sheets: list replaces a single flat columns: list. The meaningful contract for a workbook is which sheets and cell ranges the lectures read, plus what else ships alongside. mpd2020.xlsx has 6 sheets of which 2 are read; dette.xlsx has 27 of which 5 ranges across 2 are read. Each entry records read_as (the exact pandas call), resulting shape, dtypes and null counts.

The French Revolution files are read positionally, and this is the sharper finding. Every read is header=None with explicit usecols/skiprows/nrows — for example pd.read_excel(dette_url, sheet_name='Militspe', usecols='M:X', skiprows=7, nrows=102, header=None). Inserting or deleting a row or column anywhere above or left of a range silently shifts what the lecture plots, with no error at all. That is a much stronger fragility than a named-column read, and the manifests flag it with positional_reads: true plus a warning. It also means those cell ranges, not just the filenames, are the public interface.

Also here

The catalog headline would otherwise have read "18 datasets migrated" when 8 of them are landed but not repointed. It now counts what lectures actually read, derived from consumers rather than from manifest count: "18 datasets · 10 read by lectures today, 8 awaiting repoint". The empty Used by column is documented as meaning "here and documented, not yet read from this repo".

Verification

Strict audit green against current origin/main of all 8 lecture repos — 41 static files, 35 orphans, 22 live-API lectures, unchanged. All YAML parses and every manifest carries the filename key load_manifests() requires. CATALOG.md regenerated.

Byte-compare gates for the repoints are pre-recorded in each manifest: all eight were confirmed identical to lecture-python-intro's copies by git blob hash on 2026-08-06, so the Phase 7 prerequisite is already satisfied for the whole batch.

…pstream These files landed in the Feb 2025 migration (#5) and have sat in lectures/ without metadata ever since. They are Track A's repoint targets, and a repoint needs a manifest, so the metadata research lands here as one PR and leaves the three repoint sets purely mechanical. Status is `landed`, consumers empty: the files are here and documented, but no lecture reads them from this repo yet. The strict audit is green in that state -- verified before writing, not assumed. Two findings came out of actually checking rather than transcribing. mpd2020.xlsx is NOT a pristine upstream copy. Re-fetched from the Maddison release and compared cell by cell across all six sheets: every data value matches, including all 21,683 rows of `Full data` -- but three header labels on `Regional data` are local edits ("GDP pc 2011 prices" -> "gdppc_2011", "Population" -> "pop", and a "gdppc_2011" added where upstream is empty). The lecture reads that sheet with header=(0,1,2), so those renames are load-bearing: replacing the file with a clean upstream copy would silently break long_run_growth. Classified `constructed` rather than `verbatim` as a result, and the delta is enumerated in the manifest so recovery is cheap. caron.npy and nom_balances.npy have no recorded provenance anywhere -- the lecture's own Data Sources section names three spreadsheets and omits both. Checked whether either is a dump of a column in the committed Sargent-Velde workbooks: neither is. caron's terminal values appear nowhere in them, and while nom_balances' final value does occur in several assignat.xlsx sheets, no column holds the 81-value series contiguously. So they cannot be re-derived from what we hold, and the manifests say so instead of guessing a source. The filename "caron" is recorded as a lead, explicitly flagged as inference. The three French Revolution workbooks and longprices.xls are the authors' own files republished unmodified, so `verbatim` with `unverifiable` upstream status -- there is no publisher distribution point to re-fetch from. chapter_3.xlsx is `constructed`: the lecture states it was transcribed by hand from printed appendix tables, which is our processing and cannot ship a builder. Two schema notes worth carrying into #14. These are multi-sheet workbooks, so the manifests record a `sheets:` list -- the sheets and cell ranges lectures actually read -- rather than one flat column list. And the French Revolution files are read POSITIONALLY (header=None with explicit usecols/skiprows/nrows), so a row or column inserted above or left of a range silently changes what the lecture plots. The manifests flag that with `positional_reads: true`. Also fixes the catalog headline, which would otherwise have claimed "18 datasets migrated" when 8 of them are landed but not yet repointed. It now counts what lectures actually read, derived from `consumers`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR backfills sidecar manifests for the eight “intro statics” that already live in lectures/, updates the migration tracker to record them as landed, and adjusts the generated catalog headline to distinguish “manifested” from “currently consumed by lectures”.

Changes:

Add manifests for the eight inherited static intro files in lectures/ (with consumers: [] pending Track A repoints).
Extend migration.yml with Track A entries for these eight datasets, marked status: landed.
Update scripts/build_catalog.py (and regenerate CATALOG.md) so the catalog headline reflects “read by lectures” based on consumers.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
scripts/build_catalog.py	Updates the catalog intro/headline to count “in use” datasets via `consumers`, not manifest count.
migration.yml	Adds Track A “intro static batch” records (status `landed`) for the eight files.
lectures/mpd2020.xlsx.yml	New manifest for `mpd2020.xlsx` (constructed, upstream verified with documented header edits).
lectures/longprices.xls.yml	New manifest for `longprices.xls` (verbatim, upstream unverifiable).
lectures/chapter_3.xlsx.yml	New manifest for `chapter_3.xlsx` (constructed; manual transcription, builder unrecovered).
lectures/assignat.xlsx.yml	New manifest for `assignat.xlsx` (verbatim; positional reads flagged).
lectures/dette.xlsx.yml	New manifest for `dette.xlsx` (verbatim; positional reads flagged).
lectures/fig_3.xlsx.yml	New manifest for `fig_3.xlsx` (verbatim; positional reads flagged).
lectures/caron.npy.yml	New manifest for `caron.npy` (constructed; provenance unrecorded, builder unrecovered).
lectures/nom_balances.npy.yml	New manifest for `nom_balances.npy` (constructed; provenance unrecorded, builder unrecovered).
CATALOG.md	Regenerated catalog reflecting the new manifests and updated headline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mmcky · 2026-08-06T05:06:35Z

+      Re-fetched from the Maddison release and compared cell by cell across all
+      six sheets. Every data value matches, including all 21,683 rows of `Full


Fixed in 16ea8da. Good catch — both numbers were individually correct and measured differently, which is exactly why it read as a contradiction: 21,683 is the raw sheet including the header row (header=None, how I ran the upstream comparison), 21,682 is data rows as the lecture reads it (header=0, how I recorded the schema). A manifest that asserts both without explaining which is which is useless as a contract, so the note now says 21,682 data rows throughout, matching schema.sheets[].shape and row_count_floor.

…hain The manifest asserted the three Regional-data header differences were local edits. That was an inference, and it had an untested alternative: Maddison could have renamed its own headers after we took the file, which would make our copy a faithful older vintage rather than a modified one -- and would flip the class back to verbatim. Ruled out. The Internet Archive holds the upstream file with one unchanging content digest (4OWZWHTE5HXBBF4XLCCQTGDXY4HXOGOK) across every snapshot from 2021-01-10 to 2026-01-02, and the copy fetched on 2026-08-06 matches that digest exactly. Upstream was byte-stable two years before these bytes were committed to lecture-python-intro on 2023-03-23, so the edits are ours. Also records the chain these bytes actually travelled, which the manifest did not make explicit: they did not come here from Maddison. They were downloaded, edited, and committed to lecture-python-intro in #120 (2023-03-23, blob unchanged since), then copied into this repo by #5 in Feb 2025. The immediate source is the lecture series; Maddison is the ultimate source. One useful corollary, now recorded: because upstream never moved, the data held here corresponds to the CURRENT Maddison release. There is no vintage ambiguity and the three header labels are the only delta. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

mmcky · 2026-08-06T04:30:45Z

Provenance chain nailed down: these bytes came from the lecture series, not from Maddison

Following a question on where this copy actually originated. The manifest asserted the three Regional data header differences were local edits, and that was an inference with an untested alternative worth taking seriously: Maddison could have renamed its own headers after we took the file, which would make our copy a faithful older vintage rather than a modified one — and would flip the class back to verbatim.

Ruled out. The Internet Archive holds the upstream file with a single unchanging content digest, 4OWZWHTE5HXBBF4XLCCQTGDXY4HXOGOK, across every snapshot from 2021-01-10 to 2026-01-02. The copy I fetched on 2026-08-06 matches that digest exactly. Upstream was already byte-stable two years before these bytes were committed, so the edits are ours and constructed stands.

The chain, now recorded in the manifest rather than left implicit:

Step	What
Maddison publishes the file	content stable since at least 2021-01-10
Downloaded, three headers edited, committed to the lecture repo	2023-03-23, QuantEcon/lecture-python-intro#120 — blob never changed since
Copied into this repo	#5, Feb 2025
This repo's copy	byte-identical to intro's

So the immediate source is the lecture series; Maddison is the ultimate source. Worth being precise about, because "verbatim (Maddison Project 2020)" — how the audit describes it — is true of the data and false of the file.

One useful corollary now recorded too: because upstream never moved, the data held here corresponds to the current Maddison release. No vintage ambiguity, and those three header labels are the only delta — which is what makes the recovery note actionable rather than a warning about an unknown difference.

retrieved stays null. The 2023-03-23 commit date records when QuantEcon committed the file, not when it was retrieved from the source, and AGENTS.md is explicit that reconstructing one from the other is false precision. The commit date is recorded in the provenance note as a bound instead.

Strict audit still green.

(Reposting — backticks in the previous comment were mangled by shell expansion; this is the intended text.)

The mpd2020 delta raised a question the plan never answered: what happens when a migration discovers that the file the lectures use is not what upstream publishes today? Answering it per-file, mid-repoint, would turn every migration into a content review. The rule is that a migration moves the copy the lectures ALREADY consume, validated byte-identical, and nothing else. That property is what makes a repoint safe to merge -- it provably cannot change a figure. Adopting a newer upstream vintage is a different change: it does alter lecture output, needs figures re-reviewed, and is an author's decision rather than an infrastructure one. So a discovered delta is recorded, not resolved -- in the dataset's manifest so it shows in the catalog immediately, and in the register (#39) where it gets reasoned about once the migration completes. Two kinds of delta look alike and need opposite responses, so both are named: upstream moved (adopt via a NEW filename, so consumers opt in) versus our copy diverges (reconcile the edit). mpd2020 is the first recorded instance of the second kind, and shows why it cannot be a mechanical fix -- its local edits are load-bearing for long_run_growth, so file and lecture must move together. Detecting these automatically rather than by accident is proposed in #40. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…t, and a silent YAML truncation bug it did not catch Two of Copilot's three comments were correct and are fixed. The mpd2020 manifest cited 21,683 rows in its integrity note while its schema said 21,682. Both numbers were right and measured differently -- raw sheet including the header row, versus data rows as the lecture reads it -- but a manifest that asserts both without explaining is useless as a contract. Now 21,682 throughout, matching schema. chapter_3's Table3.1 sample declared shape [6, 6] and listed five columns. That was my error: the inspection script printed list(columns)[:5] and I transcribed the truncated output. The sixth column is "Percentage of expenditures covered by new issues of paper money". Added, with a note that these lists are complete rather than illustrative, since a sample that does not match its own shape is worse than no sample. Copilot's third comment -- that `dtype: str` should be `string`/`object` for consistency -- is declined on the merits and answered in the thread. Under pandas 3, which is the anaconda=2026.07 pin the lectures run on, str(df[col].dtype) for a text column IS `str`; a validator comparing against `string` or `object` is what would break. The repo-wide inconsistency it correctly notices is real and is #14's to settle. Two bugs Copilot missed, found while checking its claims: `source.data_url` was a name I invented for a field the repo already has as `source.file_url`, used by six existing manifests. Renamed -- straight schema drift, caught before it set a precedent. More seriously, unquoted descriptions containing a comma inside a YAML flow mapping silently parse as a new key: {description: mid-year population, thousands} yields a `thousands` key and a description truncated at the comma. This affected 15 columns across 7 manifests, four of them predating this PR (epl_match_goals, japan_earthquakes, japan_population_by_age, employ, realwage). All quoted, and verified: 72 column entries now parse with no spurious keys and no dropped descriptions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…onest home mpd2020 recorded `integrity.upstream.status: verified` while differing from upstream in three header labels. That is a claim a reader would act on: anyone refreshing the file from the source would find it silently changes lecture output. The status vocabulary simply had no value for the outcome that actually occurred -- the check ran in full and the file did not match. `failing` implies something is broken, and it is not. So `diverged` joins the vocabulary, with four fields the register (#39) needs: delta_kind local-edit | upstream-moved -- the two look alike and need opposite responses, so the distinction is machine-readable rather than buried in prose delta what differs, precisely enough that resolving it later needs no re-investigation delta_evidence how delta_kind was ESTABLISHED rather than assumed. Ruling out the other kind is the whole work: for mpd2020 it is what separates "our copy was edited" (constructed) from "we hold an older vintage" (verbatim), so it determines the file's class, not just its status register the issue tracking resolution This replaces three fields invented ad hoc for mpd2020 in an earlier commit (edits_are_local, edits_evidence, upstream_vintage_note) -- undocumented precedent set by one file, which is how a schema rots. Its upstream block now uses exactly the documented field set and nothing else. `diverged` is deliberately not a defect state. A migration moves the copy the lectures already consume (PLAN "Repoint rules" 3), so a known delta against today's upstream is an expected result to be recorded, not a problem to solve mid-repoint. The catalog gives it its own mark (⇄) so it cannot be misread as either a clean pass or a failure. Scope note: the schema sketch is behind the manifests in other ways that predate this PR -- consumers[].repo/.file is used by 10 manifests and documented in none, as are source.doi, source.note, source.version and source.file_url -- plus the multi-sheet vocabulary added here. That sweep belongs with the schema decisions in #14, against a decided convention, rather than piecemeal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot caught mpd2020's Consumers comment still reading "Empty until the Track A repoint lands ... lectures read this file from intro's own copy" while sitting directly above a populated consumers list naming both repos reading it from here. It flatly contradicted the data beneath it. The mistake is structural rather than a typo: the comment described the field's CURRENT VALUE, so it was guaranteed to go stale the moment its own PR landed. The identical comment was in all eight manifests from #38, so it would have gone stale again on repoint sets 2 and 3 and been flagged both times. All eight now describe what the field MEANS, which stays true in either state: every lecture that reads this file from this repo, empty not implying the file is unused since it may still be read from a lecture repo's own copy -- which is exactly what a repoint moves -- and populated in the repoint PR alongside the migration.yml flip. Comments only. Verified: every manifest still parses, consumers values are untouched (mpd2020 two, the other seven zero), CATALOG.md regenerates byte-identical, and the strict audit stays green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

) * mpd2020.xlsx: record the repoint Flips the tracker once both consuming lectures read this repo, and records them in the manifest's consumers list so a future correction knows what to rebuild. Held as a draft deliberately. The strict audit has no green state for a dataset whose consumers have only partly moved, so this must not merge before QuantEcon/lecture-python-intro#823 and QuantEcon/lecture-wasm#52. Verified both ways: against the lecture repos' current main this fails with "marked repointed but consumers still read via ['own-repo', 'sibling']"; against their repoint branches it passes clean, and the orphan count falls 35 -> 34 as wasm's mirror copy goes. mpd2020 is the first dataset to exercise the two-repo rule end to end. intro's copy could not be deleted while wasm fetched that exact blob by URL, so the pair moved together and this flip trails both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make the consumers comment durable instead of point-in-time Copilot caught mpd2020's Consumers comment still reading "Empty until the Track A repoint lands ... lectures read this file from intro's own copy" while sitting directly above a populated consumers list naming both repos reading it from here. It flatly contradicted the data beneath it. The mistake is structural rather than a typo: the comment described the field's CURRENT VALUE, so it was guaranteed to go stale the moment its own PR landed. The identical comment was in all eight manifests from #38, so it would have gone stale again on repoint sets 2 and 3 and been flagged both times. All eight now describe what the field MEANS, which stays true in either state: every lecture that reads this file from this repo, empty not implying the file is unused since it may still be read from a lecture repo's own copy -- which is exactly what a repoint moves -- and populated in the repoint PR alongside the migration.yml flip. Comments only. Verified: every manifest still parses, consumers values are untouched (mpd2020 two, the other seven zero), CATALOG.md regenerates byte-identical, and the strict audit stays green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI lite review requested due to automatic review settings August 6, 2026 04:11

Copilot started reviewing on behalf of mmcky August 6, 2026 04:12 View session

mmcky mentioned this pull request Aug 6, 2026

Builder architecture: four-stage template, traceback-clean failures, notebook policy (Phase 5) #14

Open

Copilot AI reviewed Aug 6, 2026

View reviewed changes

This was referenced Aug 6, 2026

Upstream-delta register: migrate what the lectures use, review data updates after the migration #39

Open

Upstream-freshness dashboard: when was each dataset last checked against its source, and does it still match? #40

Open

mmcky and others added 3 commits August 6, 2026 14:48

mmcky merged commit 9f353c3 into main Aug 6, 2026
4 checks passed

mmcky deleted the manifests/intro-statics branch August 6, 2026 05:33

mmcky mentioned this pull request Aug 6, 2026

VALIDATION: independent review of the 2026-08-06 migration work #45

Closed

38 tasks

Navigation Menu

Uh oh!

Uh oh!

Write manifests for the eight intro statics, and check them against upstream - #38

Write manifests for the eight intro statics, and check them against upstream#38
mmcky merged 5 commits into
mainfrom
manifests/intro-statics

mmcky commented Aug 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

mmcky Aug 6, 2026

Uh oh!

Uh oh!

Uh oh!

mmcky commented Aug 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		Re-fetched from the Maddison release and compared cell by cell across all
		six sheets. Every data value matches, including all 21,683 rows of `Full

Uh oh!

Uh oh!

Conversation

mmcky commented Aug 6, 2026

Two findings from actually checking

mpd2020.xlsx is not a pristine upstream copy

caron.npy and nom_balances.npy have no recorded provenance at all

Classification calls

Two schema notes for #14

Also here

Verification

Next

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mmcky Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mmcky commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Provenance chain nailed down: these bytes came from the lecture series, not from Maddison

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`mpd2020.xlsx` is not a pristine upstream copy

`caron.npy` and `nom_balances.npy` have no recorded provenance at all

mmcky commented Aug 6, 2026 •

edited

Loading