Pin 202609 pipeline toolchain (duckdb/pandas/pyarrow/rdflib) + isolated venv

DATA_PROVENANCE.md

@@ -79,7 +79,7 @@ Eric Kansa maintains OpenContext PQG **independently** on GCS (`storage.googleap
 - **Geometry-agnostic input.** The `geometry` column may be **WKB BLOB** (e.g. `isamples_202604_wide`) or DuckDB **GEOMETRY** (e.g. `isamples_202601_wide`, the Zenodo wide). The builder detects the type at runtime — earlier ad-hoc SQL assumed BLOB and threw `BinderException` on GEOMETRY wides.
 - **Material selection (#265/#271).** `material` = the **first NON-ROOT** concept in `p__has_material_category` (the root `.../material/1.0/material` "Material" can sit at any array position). Samples tagged only at the root get `NULL` material (excluded from the facet). This is **NOT leaf/most-specific** selection — the arrays are not clean SKOS paths. `context`/`object_type` use `[1]`; their root-dropping is deferred.
 - **Determinism.** Every COPY has `ORDER BY`; `dominant_source` ties break on source name (ASC); center lat/lng rounded to 6 dp.
-- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB pinned in `scripts/requirements.txt`.
+- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB, pandas, pyarrow, and rdflib — the four packages the scripts' output bytes are actually sensitive to — are exact-pinned in `scripts/requirements.txt`; `bash scripts/setup_pipeline_venv.sh` installs them into an isolated, project-local env (`scripts/.venv`), which `make` then uses by default (see its `PY` var) — rather than a shared/ambient interpreter such as pyenv's `myenv` (which other projects also use, and which can silently move a pin nobody re-installed against — this happened for real 2026-08-28→31, see `provenance/isamples_202609/step9_clean_rerun_2026-08-31.md` and `step0_toolchain_pin_2026-09-02.md` in the isamples-suite provenance repo, both private/not in this repo). This is **narrower than a full lockfile**: transitive dependencies, the Python interpreter version, and `uv` itself are still resolver-chosen, so a clean checkout can still drift there (unlike `pqg`'s step 1, which is fully `uv.lock`-hermetic). This bullet is about that *toolchain-pin* gap specifically — a different, now-closed issue from the *data-lineage* "reproducibility gap" the snapshot note directly below still describes as open.
 - **Tested.** `tests/test_frontend_derived.py` (fixtures, CI via `.github/workflows/pipeline-tests.yml`) + `scripts/validate_frontend_derived.py` (algebraic: `facet_summaries == GROUP BY sample_facets_v2`, `facet_cross_filter == conditional GROUP BY`, `facets.pid == map_lite.pid`, pid uniqueness, H3 sums). `make test` / `make all`.
 - **`sample_facet_index_meta` (#313 P1) is paired with `sample_facet_index` and MUST be deployed together.** It's a tiny per-source-histogram manifest built DIRECTLY from `samp_geo` (never by reading back `sample_facet_index.parquet` — that would make the "staleness check" self-referential) so the explorer's boot-time `facetIndexReady` preflight can validate the index without a live 6M-row `GROUP BY` scan. `--only sample_facet_index_meta` alone builds just the meta file (no forced `sample_facet_index` rebuild) for re-pairing a meta file with an already-published index of the same `build_id`; a normal build or `--only sample_facet_index,sample_facet_index_meta` builds both together. **R2 upload must always publish the two files together with matching `build_id`** — see `SERIALIZATIONS.md` §4.13. Independently validated by `validate_frontend_derived.py --index ... --index-meta ...`, which recomputes the histogram/build_id/schema_version/row-count from the actual on-disk index file (not from the meta file's own claims).
 

Makefile

@@ -13,10 +13,24 @@
 # Override on the command line, e.g.:
 #   make all-272 TAG=isamples_202606
 #
-# Requirements: python with `pip install -r scripts/requirements.txt`, plus
-# network access on first run (DuckDB pulls the h3 community extension).
-
-PY      ?= python
+# Requirements: an isolated pipeline venv (bash scripts/setup_pipeline_venv.sh,
+# which installs the exact pins in scripts/requirements.txt — the targets
+# below use it by default), plus network access on first run (DuckDB pulls
+# the h3 community extension). Override with PY=python (or any interpreter)
+# to use something else — e.g. CI installs scripts/requirements.txt into its
+# own ephemeral runner via plain `pip`, so it isn't exposed to ambient
+# ("myenv") drift the way a local run is; see scripts/requirements.txt's
+# header for why the isolated-venv default matters for LOCAL runs.
+
+PY      ?= scripts/.venv/bin/python
+# Only check for a missing file when PY looks like a path (contains "/") —
+# `wildcard` doesn't search $$PATH, so a bare command (PY=python) would
+# otherwise warn falsely even though the shell can find it fine.
+ifneq ($(findstring /,$(PY)),)
+ifeq ($(wildcard $(PY)),)
+$(warning $(PY) not found -- run: bash scripts/setup_pipeline_venv.sh (or pass PY=<interpreter> to use something else, e.g. in CI))
+endif
+endif
 WIDE_URL ?= https://data.isamples.org/isamples_202604_wide.parquet
 OC_WIDE_URL ?= https://storage.googleapis.com/opencontext-parquet/oc_isamples_pqg_wide.parquet
 OUTDIR  ?= build/derived

scripts/build_vocab_labels.py

@@ -34,9 +34,9 @@
 Issue: https://github.com/isamplesorg/isamplesorg.github.io/issues/148
 
 Usage:
-    pip install -r scripts/requirements.txt
-    python scripts/build_vocab_labels.py              # writes ./vocab_labels.parquet
-    python scripts/build_vocab_labels.py -o /tmp/v.parquet
+    bash scripts/setup_pipeline_venv.sh                # once, isolated env (see scripts/requirements.txt)
+    scripts/.venv/bin/python scripts/build_vocab_labels.py              # writes ./vocab_labels.parquet
+    scripts/.venv/bin/python scripts/build_vocab_labels.py -o /tmp/v.parquet
 """
 from __future__ import annotations
 

scripts/requirements.txt

@@ -1,15 +1,38 @@
 # Dependencies for one-shot scripts in this directory (NOT the Quarto site
-# build — that uses ../requirements.txt). Install with:
-#   pip install -r scripts/requirements.txt
+# build — that uses ../requirements.txt). Do NOT `pip install` this into a
+# shared/ambient interpreter (e.g. a pyenv env used by other projects): this
+# file being pinned does nothing by itself if nothing installs FROM it into
+# an isolated env — that's exactly how the pipeline ran against an
+# ambient-drifted duckdb in the past (see scripts/setup_pipeline_venv.sh and
+# `make` targets, which default to that same isolated venv). Install with:
+#   bash scripts/setup_pipeline_venv.sh
+# which is equivalent to:
+#   uv venv scripts/.venv && uv pip install -p scripts/.venv -r scripts/requirements.txt
+
+# duckdb/pandas/pyarrow/rdflib below are exact pins (the four packages the
+# reproducible-build scripts' OUTPUT BYTES are sensitive to), not floors —
+# an unpinned "latest compatible" install can silently change output bytes,
+# or (for a DuckDB release-line bump, e.g. 1.4->1.5) outright crash a step;
+# see provenance/isamples_202609/step9_clean_rerun_2026-08-31.md in the
+# isamples-suite provenance repo. Versions below are the exact ones the
+# 2026-08-28 isamples_202609 reference build used (recorded in that build's
+# own *.manifest.json "environment" fields). This is NOT a fully hermetic
+# lockfile: transitive dependencies (numpy, python-dateutil, isodate, ...),
+# the Python interpreter version, and `uv` itself are still resolver-chosen
+# and can still drift between clean checkouts. A `uv.lock`-based lockfile
+# (as the pqg repo's step 1 uses) would close that; not done here as it's a
+# bigger change than this pin.
 
 # build_vocab_labels.py
-rdflib>=6.3
-pandas>=2.0
-pyarrow>=14
+rdflib==6.3.2
+pandas==2.3.3
+pyarrow==21.0.0
 
 # build_frontend_derived.py + validate_frontend_derived.py + tests/
 # DuckDB pinned for reproducible builds; the h3 (community) and spatial
 # extensions are version-bound to this DuckDB release and installed at runtime
 # (the build manifest records the exact resolved extension hashes).
 duckdb==1.4.4
+
+# test runner only — not part of any script's output, so left floating
 pytest>=8.0

scripts/setup_pipeline_venv.sh

@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+# Create (or refresh) an isolated Python environment for the one-shot
+# reproducible-build scripts in this directory (also used by `make`, see
+# Makefile's PY var), pinned exactly per scripts/requirements.txt. Never
+# installs into a shared/ambient interpreter (e.g. pyenv's `myenv`, used by
+# other projects): the pin in scripts/requirements.txt didn't itself drift
+# between 2026-08-28 and 2026-08-31 — nothing had ever installed FROM it
+# into an isolated env, so the pipeline ran against whatever duckdb `myenv`
+# happened to have (1.4.4, then silently 1.5.5, from an unrelated project's
+# upgrade), and that ambient drift changed the pipeline's output bytes. See
+# provenance/isamples_202609/step9_clean_rerun_2026-08-31.md in the
+# isamples-suite provenance repo (private supporting evidence, not in this
+# repo) for the incident.
+#
+# Usage:
+#   bash scripts/setup_pipeline_venv.sh
+#   make derived TAG=isamples_dev          # Makefile defaults PY to this venv
+#   scripts/.venv/bin/python scripts/enrich_wide_with_oc_thumbnails.py ...
+#
+# Requires `uv` (https://docs.astral.sh/uv/). Mirrors the hermetic pattern
+# the `pqg` repo's step 1 already uses (its own `uv.lock`), applied here via
+# a pinned requirements.txt instead of a uv project, since these are
+# one-shot scripts rather than an installable package. Not a full lockfile:
+# transitive dependencies, the Python version, and `uv` itself are still
+# resolver-chosen (see scripts/requirements.txt's header comment).
+set -euo pipefail
+cd "$(dirname "${BASH_SOURCE[0]}")/.."
+
+VENV=scripts/.venv
+uv venv "$VENV" --quiet
+uv pip install --python "$VENV/bin/python" --quiet -r scripts/requirements.txt
+"$VENV/bin/python" -c "import duckdb, pandas, pyarrow, rdflib; print(f'duckdb {duckdb.__version__}  pandas {pandas.__version__}  pyarrow {pyarrow.__version__}  rdflib {rdflib.__version__}')"
+echo "Pipeline venv ready: $VENV"