Skip to content

ci(big-endian): resolve dependencies from pyproject.toml instead of trusting Debian - #411

Open
d-v-b wants to merge 7 commits into
mainfrom
ci/big-endian-respect-pins
Open

d-v-b wants to merge 7 commits into
mainfrom
ci/big-endian-respect-pins

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

The big-endian (s390x) job installed zarr with pip install --no-deps into an image whose numpy and numcodecs come from Debian sid, so Debian's package versions defined the test environment and the constraints in pyproject.toml were never checked. Debian sid currently ships numcodecs 0.17.0, which zarr now excludes (GHSA-jqc8-qjfv-5m6r, see zarr-developers#4465), and the job reported that as three errors and a failure in the pickle opt-in tests rather than as a dependency problem.

This separates the two roles. Debian remains a source of prebuilt s390x binaries, which PyPI does not have; pyproject.toml decides which versions are acceptable. The workflow now installs zarr with its dependencies resolved and runs pip check, so an image that falls outside the constraints fails at install time with a message naming the package. The Dockerfile builds a satisfying numcodecs from source into the venv, where it shadows Debian's package. It is built without isolation against Debian's meson, Cython and numpy, so nothing else compiles under emulation, and the compile is cached as an image layer rather than repeated per job. That line can go once Debian ships a version zarr accepts.

Verified locally with Docker. Natively on arm64 and under QEMU for s390x, the image builds, the venv's numcodecs is 0.17.1 and shadows Debian's 0.17.0, pip install --no-build-isolation . and pip check succeed on the zarr-developers#4465 tree, and its pickle opt-in tests pass (byteorder big on s390x). With the venv's numcodecs removed so only Debian's is visible, pip check fails with zarr 3.4.2 has requirement numcodecs!=0.17.0,>=0.16.6, but you have numcodecs 0.17.0+ds, which is the message the old --no-deps install could never produce. The numcodecs source build took about 3 minutes under QEMU on this machine, and it is cached as an image layer.

🤖 Generated with Claude Code

dependabot Bot and others added 3 commits October 20, 2026 22:12
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…rs#4535)

Bumps [certifi](https://github.com/certifi/python-certifi) from 2026.4.22 to 2026.10.19.
- [Commits](certifi/python-certifi@2026.04.22...2026.10.19)

---
updated-dependencies:
- dependency-name: certifi
  dependency-version: 2026.10.19
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…rusting Debian

The s390x job installed zarr with --no-deps into an image whose numpy and numcodecs come from Debian sid, so Debian's package versions defined the test environment and zarr's own constraints were never checked. Debian currently ships numcodecs 0.17.0, which zarr excludes (GHSA-jqc8-qjfv-5m6r), and the job failed through the pickle opt-in tests instead of saying so.

Install zarr with its dependencies resolved and run pip check, so an image that does not satisfy pyproject.toml fails at install time. Debian stays a source of prebuilt s390x binaries only. Build a satisfying numcodecs from source in the image, where the compile is cached as a layer, using Debian's meson, Cython and numpy so nothing else compiles under emulation.

Assisted-by: ClaudeCode:claude-fable-5.1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
d-v-b and others added 2 commits October 21, 2026 18:01
Assisted-by: ClaudeCode:claude-fable-5.1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
d-v-b and others added 2 commits October 21, 2026 18:43
…r-developers#4543)

* ci(docs): fix the Read the Docs sub-site builds and skip filters

The zarr-indexing, zarr-metadata and zarr-http-server docs sites install their docs group with pip on Read the Docs, without a lockfile, so pymdown-extensions floated to 12.3, whose Highlight extension API the pinned mkdocstrings 1.0.6 cannot drive (Highlight.__init__() missing argument 'md'). No zarr-indexing build has succeeded since. Pin pymdown-extensions to 11.0.1, the version the repo-root uv.lock resolves to, and refresh the package lockfiles.

The post_checkout skip filters compared the PR head against the tip of main with a two-dot diff, so any PR that was behind main appeared to touch every package and built every site. Compare against the merge base instead, after fetching full history into the shallow checkout. Also exclude zarr-http-server from the root site's filter, as the other two packages already were.

Assisted-by: ClaudeCode:claude-fable-5.1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs: add changelog entry for zarr-developers#4543

Assisted-by: ClaudeCode:claude-fable-5.1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant