diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index 0613833..9acca43 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -79,7 +79,7 @@ Eric Kansa maintains OpenContext PQG **independently** on GCS (`storage.googleap - **Geometry-agnostic input.** The `geometry` column may be **WKB BLOB** (e.g. `isamples_202604_wide`) or DuckDB **GEOMETRY** (e.g. `isamples_202601_wide`, the Zenodo wide). The builder detects the type at runtime — earlier ad-hoc SQL assumed BLOB and threw `BinderException` on GEOMETRY wides. - **Material selection (#265/#271).** `material` = the **first NON-ROOT** concept in `p__has_material_category` (the root `.../material/1.0/material` "Material" can sit at any array position). Samples tagged only at the root get `NULL` material (excluded from the facet). This is **NOT leaf/most-specific** selection — the arrays are not clean SKOS paths. `context`/`object_type` use `[1]`; their root-dropping is deferred. - **Determinism.** Every COPY has `ORDER BY`; `dominant_source` ties break on source name (ASC); center lat/lng rounded to 6 dp. -- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB pinned in `scripts/requirements.txt`. +- **Reproducibility & build identity.** Each run writes `{tag}_manifest.json` (input + per-output sha256, argv, git SHA, DuckDB + extension versions). DuckDB, pandas, pyarrow, and rdflib — the four packages the scripts' output bytes are actually sensitive to — are exact-pinned in `scripts/requirements.txt`; `bash scripts/setup_pipeline_venv.sh` installs them into an isolated, project-local env (`scripts/.venv`), which `make` then uses by default (see its `PY` var) — rather than a shared/ambient interpreter such as pyenv's `myenv` (which other projects also use, and which can silently move a pin nobody re-installed against — this happened for real 2026-08-28→31, see `provenance/isamples_202609/step9_clean_rerun_2026-08-31.md` and `step0_toolchain_pin_2026-09-02.md` in the isamples-suite provenance repo, both private/not in this repo). This is **narrower than a full lockfile**: transitive dependencies, the Python interpreter version, and `uv` itself are still resolver-chosen, so a clean checkout can still drift there (unlike `pqg`'s step 1, which is fully `uv.lock`-hermetic). This bullet is about that *toolchain-pin* gap specifically — a different, now-closed issue from the *data-lineage* "reproducibility gap" the snapshot note directly below still describes as open. - **Tested.** `tests/test_frontend_derived.py` (fixtures, CI via `.github/workflows/pipeline-tests.yml`) + `scripts/validate_frontend_derived.py` (algebraic: `facet_summaries == GROUP BY sample_facets_v2`, `facet_cross_filter == conditional GROUP BY`, `facets.pid == map_lite.pid`, pid uniqueness, H3 sums). `make test` / `make all`. - **`sample_facet_index_meta` (#313 P1) is paired with `sample_facet_index` and MUST be deployed together.** It's a tiny per-source-histogram manifest built DIRECTLY from `samp_geo` (never by reading back `sample_facet_index.parquet` — that would make the "staleness check" self-referential) so the explorer's boot-time `facetIndexReady` preflight can validate the index without a live 6M-row `GROUP BY` scan. `--only sample_facet_index_meta` alone builds just the meta file (no forced `sample_facet_index` rebuild) for re-pairing a meta file with an already-published index of the same `build_id`; a normal build or `--only sample_facet_index,sample_facet_index_meta` builds both together. **R2 upload must always publish the two files together with matching `build_id`** — see `SERIALIZATIONS.md` §4.13. Independently validated by `validate_frontend_derived.py --index ... --index-meta ...`, which recomputes the histogram/build_id/schema_version/row-count from the actual on-disk index file (not from the meta file's own claims). diff --git a/Makefile b/Makefile index 8f2f152..403fde6 100644 --- a/Makefile +++ b/Makefile @@ -13,10 +13,24 @@ # Override on the command line, e.g.: # make all-272 TAG=isamples_202606 # -# Requirements: python with `pip install -r scripts/requirements.txt`, plus -# network access on first run (DuckDB pulls the h3 community extension). - -PY ?= python +# Requirements: an isolated pipeline venv (bash scripts/setup_pipeline_venv.sh, +# which installs the exact pins in scripts/requirements.txt — the targets +# below use it by default), plus network access on first run (DuckDB pulls +# the h3 community extension). Override with PY=python (or any interpreter) +# to use something else — e.g. CI installs scripts/requirements.txt into its +# own ephemeral runner via plain `pip`, so it isn't exposed to ambient +# ("myenv") drift the way a local run is; see scripts/requirements.txt's +# header for why the isolated-venv default matters for LOCAL runs. + +PY ?= scripts/.venv/bin/python +# Only check for a missing file when PY looks like a path (contains "/") — +# `wildcard` doesn't search $$PATH, so a bare command (PY=python) would +# otherwise warn falsely even though the shell can find it fine. +ifneq ($(findstring /,$(PY)),) +ifeq ($(wildcard $(PY)),) +$(warning $(PY) not found -- run: bash scripts/setup_pipeline_venv.sh (or pass PY= to use something else, e.g. in CI)) +endif +endif WIDE_URL ?= https://data.isamples.org/isamples_202604_wide.parquet OC_WIDE_URL ?= https://storage.googleapis.com/opencontext-parquet/oc_isamples_pqg_wide.parquet OUTDIR ?= build/derived diff --git a/scripts/build_vocab_labels.py b/scripts/build_vocab_labels.py index a23f084..e1c4aeb 100644 --- a/scripts/build_vocab_labels.py +++ b/scripts/build_vocab_labels.py @@ -34,9 +34,9 @@ Issue: https://github.com/isamplesorg/isamplesorg.github.io/issues/148 Usage: - pip install -r scripts/requirements.txt - python scripts/build_vocab_labels.py # writes ./vocab_labels.parquet - python scripts/build_vocab_labels.py -o /tmp/v.parquet + bash scripts/setup_pipeline_venv.sh # once, isolated env (see scripts/requirements.txt) + scripts/.venv/bin/python scripts/build_vocab_labels.py # writes ./vocab_labels.parquet + scripts/.venv/bin/python scripts/build_vocab_labels.py -o /tmp/v.parquet """ from __future__ import annotations diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 0015d6d..2d0af98 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -1,15 +1,38 @@ # Dependencies for one-shot scripts in this directory (NOT the Quarto site -# build — that uses ../requirements.txt). Install with: -# pip install -r scripts/requirements.txt +# build — that uses ../requirements.txt). Do NOT `pip install` this into a +# shared/ambient interpreter (e.g. a pyenv env used by other projects): this +# file being pinned does nothing by itself if nothing installs FROM it into +# an isolated env — that's exactly how the pipeline ran against an +# ambient-drifted duckdb in the past (see scripts/setup_pipeline_venv.sh and +# `make` targets, which default to that same isolated venv). Install with: +# bash scripts/setup_pipeline_venv.sh +# which is equivalent to: +# uv venv scripts/.venv && uv pip install -p scripts/.venv -r scripts/requirements.txt + +# duckdb/pandas/pyarrow/rdflib below are exact pins (the four packages the +# reproducible-build scripts' OUTPUT BYTES are sensitive to), not floors — +# an unpinned "latest compatible" install can silently change output bytes, +# or (for a DuckDB release-line bump, e.g. 1.4->1.5) outright crash a step; +# see provenance/isamples_202609/step9_clean_rerun_2026-08-31.md in the +# isamples-suite provenance repo. Versions below are the exact ones the +# 2026-08-28 isamples_202609 reference build used (recorded in that build's +# own *.manifest.json "environment" fields). This is NOT a fully hermetic +# lockfile: transitive dependencies (numpy, python-dateutil, isodate, ...), +# the Python interpreter version, and `uv` itself are still resolver-chosen +# and can still drift between clean checkouts. A `uv.lock`-based lockfile +# (as the pqg repo's step 1 uses) would close that; not done here as it's a +# bigger change than this pin. # build_vocab_labels.py -rdflib>=6.3 -pandas>=2.0 -pyarrow>=14 +rdflib==6.3.2 +pandas==2.3.3 +pyarrow==21.0.0 # build_frontend_derived.py + validate_frontend_derived.py + tests/ # DuckDB pinned for reproducible builds; the h3 (community) and spatial # extensions are version-bound to this DuckDB release and installed at runtime # (the build manifest records the exact resolved extension hashes). duckdb==1.4.4 + +# test runner only — not part of any script's output, so left floating pytest>=8.0 diff --git a/scripts/setup_pipeline_venv.sh b/scripts/setup_pipeline_venv.sh new file mode 100755 index 0000000..9ba0f1b --- /dev/null +++ b/scripts/setup_pipeline_venv.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Create (or refresh) an isolated Python environment for the one-shot +# reproducible-build scripts in this directory (also used by `make`, see +# Makefile's PY var), pinned exactly per scripts/requirements.txt. Never +# installs into a shared/ambient interpreter (e.g. pyenv's `myenv`, used by +# other projects): the pin in scripts/requirements.txt didn't itself drift +# between 2026-08-28 and 2026-08-31 — nothing had ever installed FROM it +# into an isolated env, so the pipeline ran against whatever duckdb `myenv` +# happened to have (1.4.4, then silently 1.5.5, from an unrelated project's +# upgrade), and that ambient drift changed the pipeline's output bytes. See +# provenance/isamples_202609/step9_clean_rerun_2026-08-31.md in the +# isamples-suite provenance repo (private supporting evidence, not in this +# repo) for the incident. +# +# Usage: +# bash scripts/setup_pipeline_venv.sh +# make derived TAG=isamples_dev # Makefile defaults PY to this venv +# scripts/.venv/bin/python scripts/enrich_wide_with_oc_thumbnails.py ... +# +# Requires `uv` (https://docs.astral.sh/uv/). Mirrors the hermetic pattern +# the `pqg` repo's step 1 already uses (its own `uv.lock`), applied here via +# a pinned requirements.txt instead of a uv project, since these are +# one-shot scripts rather than an installable package. Not a full lockfile: +# transitive dependencies, the Python version, and `uv` itself are still +# resolver-chosen (see scripts/requirements.txt's header comment). +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")/.." + +VENV=scripts/.venv +uv venv "$VENV" --quiet +uv pip install --python "$VENV/bin/python" --quiet -r scripts/requirements.txt +"$VENV/bin/python" -c "import duckdb, pandas, pyarrow, rdflib; print(f'duckdb {duckdb.__version__} pandas {pandas.__version__} pyarrow {pyarrow.__version__} rdflib {rdflib.__version__}')" +echo "Pipeline venv ready: $VENV"