From d10a7931b8c5cf02b6ae55e01c8b17465c5d7b2a Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 2 Sep 2026 15:21:50 -0700 Subject: [PATCH 1/3] Pin the 202609 pipeline toolchain (duckdb/pandas/pyarrow/rdflib) + isolated venv MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 202609 rebuild's reproducibility was verified twice (8/28, 8/28→8/31) but only by luck: only duckdb was pinned in scripts/requirements.txt, and nothing actually installed FROM that file into an isolated environment — scripts ran against whatever was in the shared pyenv `myenv`, used by many other projects. Between 8/28 and 8/31, an unrelated `myenv` upgrade moved duckdb 1.4.4 -> 1.5.5, which changed step 2's output bytes and hard-crashed step 4 (provenance/isamples_202609/step9_clean_rerun_2026-08-31.md in the isamples-suite provenance repo). pandas/pyarrow/rdflib were only ever floor-pinned (>=), so even duckdb's pin wasn't protecting the full chain. This exact-pins scripts/requirements.txt to the versions the 8/28 reference build actually used (recorded in that build's own manifest.json files), and adds scripts/setup_pipeline_venv.sh to install them into a project-local `scripts/.venv` via `uv` rather than into any shared interpreter — mirroring the hermetic pattern the pqg repo's step 1 already uses via its own uv.lock, applied here as a pinned requirements.txt since these are one-shot scripts rather than an installable package. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01NdwkgvUJhJ3G9cJNj9oUuZ --- DATA_PROVENANCE.md | 2 +- scripts/requirements.txt | 25 ++++++++++++++++++++----- scripts/setup_pipeline_venv.sh | 30 ++++++++++++++++++++++++++++++ 3 files changed, 51 insertions(+), 6 deletions(-) create mode 100755 scripts/setup_pipeline_venv.sh diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index 06138335..9460f222 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -79,7 +79,7 @@ Eric Kansa maintains OpenContext PQG **independently** on GCS (`storage.googleap - **Geometry-agnostic input.** The `geometry` column may be **WKB BLOB** (e.g. `isamples_202604_wide`) or DuckDB **GEOMETRY** (e.g. `isamples_202601_wide`, the Zenodo wide). The builder detects the type at runtime — earlier ad-hoc SQL assumed BLOB and threw `BinderException` on GEOMETRY wides. - **Material selection (#265/#271).** `material` = the **first NON-ROOT** concept in `p__has_material_category` (the root `.../material/1.0/material` "Material" can sit at any array position). Samples tagged only at the root get `NULL` material (excluded from the facet). This is **NOT leaf/most-specific** selection — the arrays are not clean SKOS paths. `context`/`object_type` use `[1]`; their root-dropping is deferred. - **Determinism.** Every COPY has `ORDER BY`; `dominant_source` ties break on source name (ASC); center lat/lng rounded to 6 dp. -- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB pinned in `scripts/requirements.txt`. +- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB, pandas, pyarrow, and rdflib are all exact-pinned in `scripts/requirements.txt` — install into an isolated env with `bash scripts/setup_pipeline_venv.sh` (never into a shared/ambient interpreter such as pyenv's `myenv`, which other projects also use and which can silently upgrade a pin nobody re-installed against; this happened for real 2026-08-28→31, see `provenance/isamples_202609/step9_clean_rerun_2026-08-31.md` and `step0_toolchain_pin_2026-09-02.md` in the isamples-suite provenance repo). - **Tested.** `tests/test_frontend_derived.py` (fixtures, CI via `.github/workflows/pipeline-tests.yml`) + `scripts/validate_frontend_derived.py` (algebraic: `facet_summaries == GROUP BY sample_facets_v2`, `facet_cross_filter == conditional GROUP BY`, `facets.pid == map_lite.pid`, pid uniqueness, H3 sums). `make test` / `make all`. - **`sample_facet_index_meta` (#313 P1) is paired with `sample_facet_index` and MUST be deployed together.** It's a tiny per-source-histogram manifest built DIRECTLY from `samp_geo` (never by reading back `sample_facet_index.parquet` — that would make the "staleness check" self-referential) so the explorer's boot-time `facetIndexReady` preflight can validate the index without a live 6M-row `GROUP BY` scan. `--only sample_facet_index_meta` alone builds just the meta file (no forced `sample_facet_index` rebuild) for re-pairing a meta file with an already-published index of the same `build_id`; a normal build or `--only sample_facet_index,sample_facet_index_meta` builds both together. **R2 upload must always publish the two files together with matching `build_id`** — see `SERIALIZATIONS.md` §4.13. Independently validated by `validate_frontend_derived.py --index ... --index-meta ...`, which recomputes the histogram/build_id/schema_version/row-count from the actual on-disk index file (not from the meta file's own claims). diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 0015d6da..fa148890 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -1,11 +1,26 @@ # Dependencies for one-shot scripts in this directory (NOT the Quarto site -# build — that uses ../requirements.txt). Install with: -# pip install -r scripts/requirements.txt +# build — that uses ../requirements.txt). Do NOT `pip install` this into a +# shared/ambient interpreter (e.g. a pyenv env used by other projects) — +# that's exactly how the pin below drifted silently in the past (see +# scripts/setup_pipeline_venv.sh). Install into an isolated venv instead: +# bash scripts/setup_pipeline_venv.sh +# which is equivalent to: +# uv venv scripts/.venv && uv pip install -p scripts/.venv -r scripts/requirements.txt + +# All versions below are exact pins, not floors. Reproducible-build scripts +# (build_vocab_labels.py, build_frontend_derived.py, enrich_wide_with_oc_*.py, +# ingest_oc_records.py, tools/build_search_index.py) depend on byte-identical +# output across runs; an unpinned "latest compatible" install silently +# changes output bytes (and, for DuckDB major-version bumps, can outright +# crash a step — see provenance/isamples_202609/step9_clean_rerun_2026-08-31.md +# in the isamples-suite provenance repo). Versions below are the exact ones +# the 2026-08-28 isamples_202609 reference build used (recorded in that +# build's own *.manifest.json "environment" fields). # build_vocab_labels.py -rdflib>=6.3 -pandas>=2.0 -pyarrow>=14 +rdflib==6.3.2 +pandas==2.3.3 +pyarrow==21.0.0 # build_frontend_derived.py + validate_frontend_derived.py + tests/ # DuckDB pinned for reproducible builds; the h3 (community) and spatial diff --git a/scripts/setup_pipeline_venv.sh b/scripts/setup_pipeline_venv.sh new file mode 100755 index 00000000..1d332def --- /dev/null +++ b/scripts/setup_pipeline_venv.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Create (or refresh) an isolated Python environment for the one-shot +# reproducible-build scripts in this directory, pinned exactly per +# scripts/requirements.txt. Never installs into a shared/ambient +# interpreter (e.g. pyenv's `myenv`, used by other projects) — that's how +# scripts/requirements.txt's duckdb==1.4.4 pin silently drifted to 1.5.5 +# between 2026-08-28 and 2026-08-31 (see +# provenance/isamples_202609/step9_clean_rerun_2026-08-31.md in the +# isamples-suite provenance repo): nothing actually installed FROM this +# file into an isolated env, so ambient package upgrades unrelated to this +# project changed the pipeline's output bytes without anyone touching this +# repo. +# +# Usage: +# bash scripts/setup_pipeline_venv.sh +# scripts/.venv/bin/python scripts/enrich_wide_with_oc_thumbnails.py ... +# +# Requires `uv` (https://docs.astral.sh/uv/). Mirrors the hermetic pattern +# the `pqg` repo's step 1 already uses (its own `uv.lock`), applied here via +# a pinned requirements.txt instead of a uv project, since these are +# one-shot scripts rather than an installable package. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")/.." + +VENV=scripts/.venv +uv venv "$VENV" --quiet +uv pip install --python "$VENV/bin/python" --quiet -r scripts/requirements.txt + +echo "Pipeline venv ready: $VENV" +"$VENV/bin/python" -c "import duckdb, pandas, pyarrow, rdflib; print(f'duckdb {duckdb.__version__} pandas {pandas.__version__} pyarrow {pyarrow.__version__} rdflib {rdflib.__version__}')" From 710adf70f229438ea70b4fe33f0994d42c8a91c9 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 2 Sep 2026 15:42:25 -0700 Subject: [PATCH 2/3] Address Codex review of #358: wire the venv into make, tighten claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex review (High): the Makefile's `PY ?= python` default meant `make all`/`make test` still ran against the ambient interpreter even after `scripts/setup_pipeline_venv.sh` created the pinned venv — the exact mistake this PR set out to close, just reachable through a different door. `PY` now defaults to `scripts/.venv/bin/python` (overridable, e.g. `PY=python` for CI, which already installs scripts/requirements.txt into its own ephemeral runner); a Makefile warning fires if that venv is missing. Verified `make test` (62/62) against the pinned venv both before and after this change. build_vocab_labels.py's docstring usage block updated to match. Codex review (Medium): scripts/requirements.txt and setup_pipeline_venv.sh overstated hermeticity — pandas/pyarrow/rdflib's transitive deps, the Python interpreter version, and `uv` itself are still resolver-chosen, so this is narrower than pqg's step-1 uv.lock. Reworded both to claim only what's actually pinned (the 4 output-critical direct deps), and to say so explicitly rather than "all versions ... exact pins." Codex review (Low): fixed "major-version bump" -> correct term for 1.4->1.5; "the pin drifted" -> "the ambient install drifted, the pin was never applied" (the file's pin didn't move); setup_pipeline_venv.sh now prints package versions before "ready" instead of after (so an import-time failure can't look like a successful run); DATA_PROVENANCE.md's toolchain-pin bullet now explicitly distinguishes itself from the pre-existing "still unresolved" data-lineage reproducibility gap immediately below it, instead of reading as a contradiction. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01NdwkgvUJhJ3G9cJNj9oUuZ --- DATA_PROVENANCE.md | 2 +- Makefile | 17 +++++++++++++---- scripts/build_vocab_labels.py | 6 +++--- scripts/requirements.txt | 32 ++++++++++++++++++++------------ scripts/setup_pipeline_venv.sh | 27 +++++++++++++++------------ 5 files changed, 52 insertions(+), 32 deletions(-) diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index 9460f222..4ab1a021 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -79,7 +79,7 @@ Eric Kansa maintains OpenContext PQG **independently** on GCS (`storage.googleap - **Geometry-agnostic input.** The `geometry` column may be **WKB BLOB** (e.g. `isamples_202604_wide`) or DuckDB **GEOMETRY** (e.g. `isamples_202601_wide`, the Zenodo wide). The builder detects the type at runtime — earlier ad-hoc SQL assumed BLOB and threw `BinderException` on GEOMETRY wides. - **Material selection (#265/#271).** `material` = the **first NON-ROOT** concept in `p__has_material_category` (the root `.../material/1.0/material` "Material" can sit at any array position). Samples tagged only at the root get `NULL` material (excluded from the facet). This is **NOT leaf/most-specific** selection — the arrays are not clean SKOS paths. `context`/`object_type` use `[1]`; their root-dropping is deferred. - **Determinism.** Every COPY has `ORDER BY`; `dominant_source` ties break on source name (ASC); center lat/lng rounded to 6 dp. -- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB, pandas, pyarrow, and rdflib are all exact-pinned in `scripts/requirements.txt` — install into an isolated env with `bash scripts/setup_pipeline_venv.sh` (never into a shared/ambient interpreter such as pyenv's `myenv`, which other projects also use and which can silently upgrade a pin nobody re-installed against; this happened for real 2026-08-28→31, see `provenance/isamples_202609/step9_clean_rerun_2026-08-31.md` and `step0_toolchain_pin_2026-09-02.md` in the isamples-suite provenance repo). +- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB, pandas, pyarrow, and rdflib — the four packages the scripts' output bytes are actually sensitive to — are exact-pinned in `scripts/requirements.txt`; `make` (see its `PY` var) and `bash scripts/setup_pipeline_venv.sh` install them into an isolated, project-local env rather than a shared/ambient interpreter such as pyenv's `myenv` (which other projects also use, and which can silently move a pin nobody re-installed against — this happened for real 2026-08-28→31, see `provenance/isamples_202609/step9_clean_rerun_2026-08-31.md` and `step0_toolchain_pin_2026-09-02.md` in the isamples-suite provenance repo, both private/not in this repo). This is **narrower than a full lockfile**: transitive dependencies, the Python interpreter version, and `uv` itself are still resolver-chosen, so a clean checkout can still drift there (unlike `pqg`'s step 1, which is fully `uv.lock`-hermetic). This bullet is about that *toolchain-pin* gap specifically — a different, now-closed issue from the *data-lineage* "reproducibility gap" the snapshot note directly below still describes as open. - **Tested.** `tests/test_frontend_derived.py` (fixtures, CI via `.github/workflows/pipeline-tests.yml`) + `scripts/validate_frontend_derived.py` (algebraic: `facet_summaries == GROUP BY sample_facets_v2`, `facet_cross_filter == conditional GROUP BY`, `facets.pid == map_lite.pid`, pid uniqueness, H3 sums). `make test` / `make all`. - **`sample_facet_index_meta` (#313 P1) is paired with `sample_facet_index` and MUST be deployed together.** It's a tiny per-source-histogram manifest built DIRECTLY from `samp_geo` (never by reading back `sample_facet_index.parquet` — that would make the "staleness check" self-referential) so the explorer's boot-time `facetIndexReady` preflight can validate the index without a live 6M-row `GROUP BY` scan. `--only sample_facet_index_meta` alone builds just the meta file (no forced `sample_facet_index` rebuild) for re-pairing a meta file with an already-published index of the same `build_id`; a normal build or `--only sample_facet_index,sample_facet_index_meta` builds both together. **R2 upload must always publish the two files together with matching `build_id`** — see `SERIALIZATIONS.md` §4.13. Independently validated by `validate_frontend_derived.py --index ... --index-meta ...`, which recomputes the histogram/build_id/schema_version/row-count from the actual on-disk index file (not from the meta file's own claims). diff --git a/Makefile b/Makefile index 8f2f152a..542b6a32 100644 --- a/Makefile +++ b/Makefile @@ -13,10 +13,19 @@ # Override on the command line, e.g.: # make all-272 TAG=isamples_202606 # -# Requirements: python with `pip install -r scripts/requirements.txt`, plus -# network access on first run (DuckDB pulls the h3 community extension). - -PY ?= python +# Requirements: an isolated pipeline venv (bash scripts/setup_pipeline_venv.sh, +# which installs the exact pins in scripts/requirements.txt — the targets +# below use it by default), plus network access on first run (DuckDB pulls +# the h3 community extension). Override with PY=python (or any interpreter) +# to use something else — e.g. CI installs scripts/requirements.txt into its +# own ephemeral runner via plain `pip`, so it isn't exposed to ambient +# ("myenv") drift the way a local run is; see scripts/requirements.txt's +# header for why the isolated-venv default matters for LOCAL runs. + +PY ?= scripts/.venv/bin/python +ifeq ($(wildcard $(PY)),) +$(warning $(PY) not found -- run: bash scripts/setup_pipeline_venv.sh (or pass PY= to use something else, e.g. in CI)) +endif WIDE_URL ?= https://data.isamples.org/isamples_202604_wide.parquet OC_WIDE_URL ?= https://storage.googleapis.com/opencontext-parquet/oc_isamples_pqg_wide.parquet OUTDIR ?= build/derived diff --git a/scripts/build_vocab_labels.py b/scripts/build_vocab_labels.py index a23f0844..e1c4aeb2 100644 --- a/scripts/build_vocab_labels.py +++ b/scripts/build_vocab_labels.py @@ -34,9 +34,9 @@ Issue: https://github.com/isamplesorg/isamplesorg.github.io/issues/148 Usage: - pip install -r scripts/requirements.txt - python scripts/build_vocab_labels.py # writes ./vocab_labels.parquet - python scripts/build_vocab_labels.py -o /tmp/v.parquet + bash scripts/setup_pipeline_venv.sh # once, isolated env (see scripts/requirements.txt) + scripts/.venv/bin/python scripts/build_vocab_labels.py # writes ./vocab_labels.parquet + scripts/.venv/bin/python scripts/build_vocab_labels.py -o /tmp/v.parquet """ from __future__ import annotations diff --git a/scripts/requirements.txt b/scripts/requirements.txt index fa148890..2d0af984 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -1,21 +1,27 @@ # Dependencies for one-shot scripts in this directory (NOT the Quarto site # build — that uses ../requirements.txt). Do NOT `pip install` this into a -# shared/ambient interpreter (e.g. a pyenv env used by other projects) — -# that's exactly how the pin below drifted silently in the past (see -# scripts/setup_pipeline_venv.sh). Install into an isolated venv instead: +# shared/ambient interpreter (e.g. a pyenv env used by other projects): this +# file being pinned does nothing by itself if nothing installs FROM it into +# an isolated env — that's exactly how the pipeline ran against an +# ambient-drifted duckdb in the past (see scripts/setup_pipeline_venv.sh and +# `make` targets, which default to that same isolated venv). Install with: # bash scripts/setup_pipeline_venv.sh # which is equivalent to: # uv venv scripts/.venv && uv pip install -p scripts/.venv -r scripts/requirements.txt -# All versions below are exact pins, not floors. Reproducible-build scripts -# (build_vocab_labels.py, build_frontend_derived.py, enrich_wide_with_oc_*.py, -# ingest_oc_records.py, tools/build_search_index.py) depend on byte-identical -# output across runs; an unpinned "latest compatible" install silently -# changes output bytes (and, for DuckDB major-version bumps, can outright -# crash a step — see provenance/isamples_202609/step9_clean_rerun_2026-08-31.md -# in the isamples-suite provenance repo). Versions below are the exact ones -# the 2026-08-28 isamples_202609 reference build used (recorded in that -# build's own *.manifest.json "environment" fields). +# duckdb/pandas/pyarrow/rdflib below are exact pins (the four packages the +# reproducible-build scripts' OUTPUT BYTES are sensitive to), not floors — +# an unpinned "latest compatible" install can silently change output bytes, +# or (for a DuckDB release-line bump, e.g. 1.4->1.5) outright crash a step; +# see provenance/isamples_202609/step9_clean_rerun_2026-08-31.md in the +# isamples-suite provenance repo. Versions below are the exact ones the +# 2026-08-28 isamples_202609 reference build used (recorded in that build's +# own *.manifest.json "environment" fields). This is NOT a fully hermetic +# lockfile: transitive dependencies (numpy, python-dateutil, isodate, ...), +# the Python interpreter version, and `uv` itself are still resolver-chosen +# and can still drift between clean checkouts. A `uv.lock`-based lockfile +# (as the pqg repo's step 1 uses) would close that; not done here as it's a +# bigger change than this pin. # build_vocab_labels.py rdflib==6.3.2 @@ -27,4 +33,6 @@ pyarrow==21.0.0 # extensions are version-bound to this DuckDB release and installed at runtime # (the build manifest records the exact resolved extension hashes). duckdb==1.4.4 + +# test runner only — not part of any script's output, so left floating pytest>=8.0 diff --git a/scripts/setup_pipeline_venv.sh b/scripts/setup_pipeline_venv.sh index 1d332def..9ba0f1bc 100755 --- a/scripts/setup_pipeline_venv.sh +++ b/scripts/setup_pipeline_venv.sh @@ -1,30 +1,33 @@ #!/usr/bin/env bash # Create (or refresh) an isolated Python environment for the one-shot -# reproducible-build scripts in this directory, pinned exactly per -# scripts/requirements.txt. Never installs into a shared/ambient -# interpreter (e.g. pyenv's `myenv`, used by other projects) — that's how -# scripts/requirements.txt's duckdb==1.4.4 pin silently drifted to 1.5.5 -# between 2026-08-28 and 2026-08-31 (see +# reproducible-build scripts in this directory (also used by `make`, see +# Makefile's PY var), pinned exactly per scripts/requirements.txt. Never +# installs into a shared/ambient interpreter (e.g. pyenv's `myenv`, used by +# other projects): the pin in scripts/requirements.txt didn't itself drift +# between 2026-08-28 and 2026-08-31 — nothing had ever installed FROM it +# into an isolated env, so the pipeline ran against whatever duckdb `myenv` +# happened to have (1.4.4, then silently 1.5.5, from an unrelated project's +# upgrade), and that ambient drift changed the pipeline's output bytes. See # provenance/isamples_202609/step9_clean_rerun_2026-08-31.md in the -# isamples-suite provenance repo): nothing actually installed FROM this -# file into an isolated env, so ambient package upgrades unrelated to this -# project changed the pipeline's output bytes without anyone touching this -# repo. +# isamples-suite provenance repo (private supporting evidence, not in this +# repo) for the incident. # # Usage: # bash scripts/setup_pipeline_venv.sh +# make derived TAG=isamples_dev # Makefile defaults PY to this venv # scripts/.venv/bin/python scripts/enrich_wide_with_oc_thumbnails.py ... # # Requires `uv` (https://docs.astral.sh/uv/). Mirrors the hermetic pattern # the `pqg` repo's step 1 already uses (its own `uv.lock`), applied here via # a pinned requirements.txt instead of a uv project, since these are -# one-shot scripts rather than an installable package. +# one-shot scripts rather than an installable package. Not a full lockfile: +# transitive dependencies, the Python version, and `uv` itself are still +# resolver-chosen (see scripts/requirements.txt's header comment). set -euo pipefail cd "$(dirname "${BASH_SOURCE[0]}")/.." VENV=scripts/.venv uv venv "$VENV" --quiet uv pip install --python "$VENV/bin/python" --quiet -r scripts/requirements.txt - -echo "Pipeline venv ready: $VENV" "$VENV/bin/python" -c "import duckdb, pandas, pyarrow, rdflib; print(f'duckdb {duckdb.__version__} pandas {pandas.__version__} pyarrow {pyarrow.__version__} rdflib {rdflib.__version__}')" +echo "Pipeline venv ready: $VENV" From f8c05afe3c052bf8c31bcdd286e88d9a130297eb Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 2 Sep 2026 15:48:06 -0700 Subject: [PATCH 3/3] Fix 2 low-severity nits from Codex's second-pass review of #358 - Makefile: only warn about a missing PY when PY looks like a path (contains "/") -- `wildcard` doesn't search $PATH, so PY=python was producing a false "not found" warning even though the shell resolves it fine. - DATA_PROVENANCE.md: `make` doesn't install anything -- it just uses the venv scripts/setup_pipeline_venv.sh already installed into. Reworded so only the setup script is credited with the install step. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01NdwkgvUJhJ3G9cJNj9oUuZ --- DATA_PROVENANCE.md | 2 +- Makefile | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index 4ab1a021..9acca432 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -79,7 +79,7 @@ Eric Kansa maintains OpenContext PQG **independently** on GCS (`storage.googleap - **Geometry-agnostic input.** The `geometry` column may be **WKB BLOB** (e.g. `isamples_202604_wide`) or DuckDB **GEOMETRY** (e.g. `isamples_202601_wide`, the Zenodo wide). The builder detects the type at runtime — earlier ad-hoc SQL assumed BLOB and threw `BinderException` on GEOMETRY wides. - **Material selection (#265/#271).** `material` = the **first NON-ROOT** concept in `p__has_material_category` (the root `.../material/1.0/material` "Material" can sit at any array position). Samples tagged only at the root get `NULL` material (excluded from the facet). This is **NOT leaf/most-specific** selection — the arrays are not clean SKOS paths. `context`/`object_type` use `[1]`; their root-dropping is deferred. - **Determinism.** Every COPY has `ORDER BY`; `dominant_source` ties break on source name (ASC); center lat/lng rounded to 6 dp. -- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB, pandas, pyarrow, and rdflib — the four packages the scripts' output bytes are actually sensitive to — are exact-pinned in `scripts/requirements.txt`; `make` (see its `PY` var) and `bash scripts/setup_pipeline_venv.sh` install them into an isolated, project-local env rather than a shared/ambient interpreter such as pyenv's `myenv` (which other projects also use, and which can silently move a pin nobody re-installed against — this happened for real 2026-08-28→31, see `provenance/isamples_202609/step9_clean_rerun_2026-08-31.md` and `step0_toolchain_pin_2026-09-02.md` in the isamples-suite provenance repo, both private/not in this repo). This is **narrower than a full lockfile**: transitive dependencies, the Python interpreter version, and `uv` itself are still resolver-chosen, so a clean checkout can still drift there (unlike `pqg`'s step 1, which is fully `uv.lock`-hermetic). This bullet is about that *toolchain-pin* gap specifically — a different, now-closed issue from the *data-lineage* "reproducibility gap" the snapshot note directly below still describes as open. +- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB, pandas, pyarrow, and rdflib — the four packages the scripts' output bytes are actually sensitive to — are exact-pinned in `scripts/requirements.txt`; `bash scripts/setup_pipeline_venv.sh` installs them into an isolated, project-local env (`scripts/.venv`), which `make` then uses by default (see its `PY` var) — rather than a shared/ambient interpreter such as pyenv's `myenv` (which other projects also use, and which can silently move a pin nobody re-installed against — this happened for real 2026-08-28→31, see `provenance/isamples_202609/step9_clean_rerun_2026-08-31.md` and `step0_toolchain_pin_2026-09-02.md` in the isamples-suite provenance repo, both private/not in this repo). This is **narrower than a full lockfile**: transitive dependencies, the Python interpreter version, and `uv` itself are still resolver-chosen, so a clean checkout can still drift there (unlike `pqg`'s step 1, which is fully `uv.lock`-hermetic). This bullet is about that *toolchain-pin* gap specifically — a different, now-closed issue from the *data-lineage* "reproducibility gap" the snapshot note directly below still describes as open. - **Tested.** `tests/test_frontend_derived.py` (fixtures, CI via `.github/workflows/pipeline-tests.yml`) + `scripts/validate_frontend_derived.py` (algebraic: `facet_summaries == GROUP BY sample_facets_v2`, `facet_cross_filter == conditional GROUP BY`, `facets.pid == map_lite.pid`, pid uniqueness, H3 sums). `make test` / `make all`. - **`sample_facet_index_meta` (#313 P1) is paired with `sample_facet_index` and MUST be deployed together.** It's a tiny per-source-histogram manifest built DIRECTLY from `samp_geo` (never by reading back `sample_facet_index.parquet` — that would make the "staleness check" self-referential) so the explorer's boot-time `facetIndexReady` preflight can validate the index without a live 6M-row `GROUP BY` scan. `--only sample_facet_index_meta` alone builds just the meta file (no forced `sample_facet_index` rebuild) for re-pairing a meta file with an already-published index of the same `build_id`; a normal build or `--only sample_facet_index,sample_facet_index_meta` builds both together. **R2 upload must always publish the two files together with matching `build_id`** — see `SERIALIZATIONS.md` §4.13. Independently validated by `validate_frontend_derived.py --index ... --index-meta ...`, which recomputes the histogram/build_id/schema_version/row-count from the actual on-disk index file (not from the meta file's own claims). diff --git a/Makefile b/Makefile index 542b6a32..403fde6e 100644 --- a/Makefile +++ b/Makefile @@ -23,9 +23,14 @@ # header for why the isolated-venv default matters for LOCAL runs. PY ?= scripts/.venv/bin/python +# Only check for a missing file when PY looks like a path (contains "/") — +# `wildcard` doesn't search $$PATH, so a bare command (PY=python) would +# otherwise warn falsely even though the shell can find it fine. +ifneq ($(findstring /,$(PY)),) ifeq ($(wildcard $(PY)),) $(warning $(PY) not found -- run: bash scripts/setup_pipeline_venv.sh (or pass PY= to use something else, e.g. in CI)) endif +endif WIDE_URL ?= https://data.isamples.org/isamples_202604_wide.parquet OC_WIDE_URL ?= https://storage.googleapis.com/opencontext-parquet/oc_isamples_pqg_wide.parquet OUTDIR ?= build/derived