Skip to content

ci(big-endian): resolve dependencies from pyproject.toml instead of trusting Debian - #4547

Merged
d-v-b merged 4 commits into
zarr-developers:mainfrom
d-v-b:ci/big-endian-respect-pins
Oct 21, 2026
Merged

d-v-b merged 4 commits into
zarr-developers:mainfrom
d-v-b:ci/big-endian-respect-pins

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

this PR changes how our big-endian CI job runs, ensuring that we actually get the dependencies we declare, or error.

closes #4546

🤖 ai text below! 🤖
The s390x job installed zarr with --no-deps into an image whose numpy and numcodecs come from Debian sid, so Debian's package versions defined the test environment and zarr's own constraints were never checked. Debian currently ships numcodecs 0.17.0, which zarr excludes (GHSA-jqc8-qjfv-5m6r), and the job failed through the pickle opt-in tests instead of saying so.

Install zarr with its dependencies resolved and run pip check, so an image that does not satisfy pyproject.toml fails at install time. Debian stays a source of prebuilt s390x binaries only. Build a satisfying numcodecs from source in the image, where the compile is cached as a layer, using Debian's meson, Cython and numpy so nothing else compiles under emulation.

Assisted-by: ClaudeCode:claude-fable-5.1

Summary

[Describe what this PR changes and why, in your own words.]

For reviewers

[What would you most value a second look at? What are you already confident in? For a refactor, say whether behavior is meant to be unchanged.]

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

…rusting Debian

The s390x job installed zarr with --no-deps into an image whose numpy and numcodecs come from Debian sid, so Debian's package versions defined the test environment and zarr's own constraints were never checked. Debian currently ships numcodecs 0.17.0, which zarr excludes (GHSA-jqc8-qjfv-5m6r), and the job failed through the pickle opt-in tests instead of saying so.

Install zarr with its dependencies resolved and run pip check, so an image that does not satisfy pyproject.toml fails at install time. Debian stays a source of prebuilt s390x binaries only. Build a satisfying numcodecs from source in the image, where the compile is cached as a layer, using Debian's meson, Cython and numpy so nothing else compiles under emulation.

Assisted-by: ClaudeCode:claude-fable-5.1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@d-v-b d-v-b added this to the 3.4.2 milestone Oct 21, 2026
@github-actions github-actions Bot added the needs release notes Automatically applied to PRs which haven't added release notes label Oct 21, 2026
@d-v-b
d-v-b marked this pull request as ready for review October 21, 2026 16:01
Assisted-by: ClaudeCode:claude-fable-5.1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions github-actions Bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Oct 21, 2026
Comment thread ci/big-endian/Dockerfile

RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
build-essential cmake git python3 python3-dev python3-pip python3-venv \
build-essential cmake cython3 git meson ninja-build pkgconf python3 python3-dev \

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need this stuff to compile numcodecs, if compilation is necessary (e.g., because debian doesn't ship the version of numcodecs we need)

@d-v-b

d-v-b commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

after this change, compiling dependencies for the big-endian CI job can take a long time. since the alternative was to install the wrong dependencies, I think the wait is worth it.

I will be self-merging this when checks clear

@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.71%. Comparing base (1dca23f) to head (d0588b2).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4547   +/-   ##
=======================================
  Coverage   94.71%   94.71%           
=======================================
  Files          94       94           
  Lines       13663    13663           
=======================================
  Hits        12941    12941           
  Misses        722      722           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@d-v-b
d-v-b merged commit 816aa32 into zarr-developers:main Oct 21, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

big endian tests ignore pyproject.toml

1 participant