From ed40125a530cf341597002f642421818e6ffb048 Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Thu, 16 Jul 2026 12:29:14 -0400 Subject: [PATCH 1/2] chore: cleanup indentation and guidance for release --- .github/ISSUE_TEMPLATE/release-checklist.md | 7 +- .github/workflows/prepare_release.yml | 74 ------ ci/lint_docs.py | 67 ++++- docs/contributing.md | 21 +- docs/release-notes.md | 258 ++++++++++---------- docs/user-guide/data_types.md | 20 +- docs/user-guide/v3_migration.md | 73 +++--- 7 files changed, 250 insertions(+), 270 deletions(-) delete mode 100644 .github/workflows/prepare_release.yml diff --git a/.github/ISSUE_TEMPLATE/release-checklist.md b/.github/ISSUE_TEMPLATE/release-checklist.md index 5e073cb59f..309c76b4dc 100644 --- a/.github/ISSUE_TEMPLATE/release-checklist.md +++ b/.github/ISSUE_TEMPLATE/release-checklist.md @@ -18,8 +18,8 @@ assignees: '' - [ ] Check [SPEC 0](https://scientific-python.org/specs/spec-0000/#support-window) to see if the minimum supported version of Python or NumPy needs bumping. - [ ] Verify that the latest CI workflows on `main` are passing: [Tests](https://github.com/zarr-developers/zarr-python/actions/workflows/test.yml), [GPU Tests](https://github.com/zarr-developers/zarr-python/actions/workflows/gpu_test.yml), [Hypothesis](https://github.com/zarr-developers/zarr-python/actions/workflows/hypothesis.yaml), [Docs](https://github.com/zarr-developers/zarr-python/actions/workflows/docs.yml), [Lint](https://github.com/zarr-developers/zarr-python/actions/workflows/lint.yml), [Wheels](https://github.com/zarr-developers/zarr-python/actions/workflows/releases.yml). -- [ ] Run the ["Prepare release" workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) with the target version. This will build the changelog and open a release PR with the `run-downstream` label. -- [ ] Verify that the [downstream tests](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) (triggered automatically by the `run-downstream` label) pass on the release PR. +- [ ] Run the [downstream tests](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) against `main`: go to the workflow page, click "Run workflow", and select the `main` branch. Verify that the Xarray and numcodecs integration tests pass. +- [ ] Open a release PR with the changelog entries for the upcoming release, generated with `uv run --only-group release towncrier build --version x.y.z`. - [ ] Review the release PR and verify the changelog in `docs/release-notes.md` looks correct. - [ ] Merge the release PR. @@ -45,7 +45,8 @@ In rare cases (e.g. patch releases for an older minor version), you may need to - Create the release branch from the appropriate tag if it doesn't already exist. - Cherry-pick or backport the necessary commits onto the branch. -- Run `towncrier build --version x.y.z` and commit the result to the release branch instead of `main`. +- Run `towncrier build --version x.y.z` and open the release PR against the release branch instead of `main`. +- Run the downstream tests against the release branch instead of `main`. - When drafting the GitHub Release, set the target to the release branch instead of `main`. - After the release, ensure any relevant changelog updates are also reflected on `main`. diff --git a/.github/workflows/prepare_release.yml b/.github/workflows/prepare_release.yml deleted file mode 100644 index d558779bf2..0000000000 --- a/.github/workflows/prepare_release.yml +++ /dev/null @@ -1,74 +0,0 @@ -name: Prepare release notes - -on: - workflow_dispatch: - inputs: - version: - description: 'Release version notes (e.g. 3.2.0)' - required: true - type: string - target_branch: - description: 'Branch to target' - required: false - default: 'main' - type: string - -permissions: - contents: write - pull-requests: write - -jobs: - prepare: - name: Build changelog and open PR - runs-on: ubuntu-latest - steps: - - name: Validate inputs - run: | - if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+([-\.][a-zA-Z0-9]+)*$ ]]; then - echo "::error::Invalid version format: '$VERSION'" - exit 1 - fi - if [[ ! "$TARGET_BRANCH" =~ ^[a-zA-Z0-9._/-]+$ ]]; then - echo "::error::Invalid branch name: '$TARGET_BRANCH'" - exit 1 - fi - env: - VERSION: ${{ inputs.version }} - TARGET_BRANCH: ${{ inputs.target_branch }} - - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - with: - ref: ${{ inputs.target_branch }} - fetch-depth: 0 - persist-credentials: false - - - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - - - name: Build changelog - # Use the pinned towncrier from the `release` dependency group (single - # source of truth) rather than an unpinned standalone install. - run: uv run --only-group release towncrier build --version "$VERSION" --yes - env: - VERSION: ${{ inputs.version }} - - - name: Create pull request - uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1 - with: - branch: release/v${{ inputs.version }} - base: ${{ inputs.target_branch }} - title: "Release v${{ inputs.version }}" - body: | - Automated release preparation for v${{ inputs.version }}. - - This PR was generated by the "Prepare release" workflow. It includes: - - Rendered changelog via `towncrier build --version ${{ inputs.version }}` - - Removal of consumed changelog fragments from `changes/` - - ## Checklist - - - [ ] Review the rendered changelog in `docs/release-notes.md` - - [ ] Downstream tests pass (see [downstream workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml)) - - [ ] Merge this PR, then [draft a GitHub Release](https://github.com/zarr-developers/zarr-python/releases/new) targeting `${{ inputs.target_branch }}` with tag `v${{ inputs.version }}` - commit-message: "chore: build changelog for v${{ inputs.version }}" - labels: run-downstream - delete-branch: true diff --git a/ci/lint_docs.py b/ci/lint_docs.py index ff7b56649b..a847e8aa9d 100644 --- a/ci/lint_docs.py +++ b/ci/lint_docs.py @@ -13,7 +13,7 @@ fast, source-level check that needs no docs build. Checks fall into two groups -- RST markup that silently fails under MkDocs/mkdocstrings, -and a Markdown structural problem that renders as valid-but-wrong HTML (so `mkdocs build` +and Markdown structural problems that render as valid-but-wrong HTML (so `mkdocs build` emits no warning): sphinx-role :class:`X`, :func:`X`, :py:meth:`X` -> [`X`][zarr.X] @@ -21,6 +21,8 @@ rst-field :param x:, :returns:, :rtype: -> numpydoc Parameters/Returns/Raises rst-link `text `_ -> [text](https://example) list-break unindented code fence between list items -> indent the fence under its item + list-indent continuation block indented < 4 spaces -> indent it 4 spaces + list-blank list item directly after indented block -> blank line before the item The ``list-break`` check catches a fenced code block at column 0 placed *between* two list items: because the fence is not indented into the preceding item, Markdown ends the list at @@ -29,6 +31,15 @@ notices this for sequentially-numbered ordered lists; lazily-numbered (1., 1.) and unordered lists slip past it, so this structural check covers the gap. +The ``list-indent`` and ``list-blank`` checks catch the two halves of Python-Markdown's +strict list-continuation rules, which differ from CommonMark. A blank-line-separated +block (paragraph, nested list, table) belongs to a list item only when indented at least +4 spaces; at the 2-space indent other renderers accept, Python-Markdown ends the list and +the block escapes to the top level (``list-indent``). And a new list item can not start +directly after an indented continuation block: without a blank line first, the ``- `` line +is lazily absorbed into the preceding paragraph as literal text (``list-blank``). Both +produced silently-broken changelog rendering in ``docs/release-notes.md``. + Usage: python ci/lint_docs.py [PATH ...] @@ -87,14 +98,16 @@ class Check(NamedTuple): fix: str -# ``list-break`` carries no pattern -- it is detected structurally in find_list_breaking_fences, -# not by scanning a single line -- but it appears here so it shares the remediation help. +# The ``list-*`` checks carry no pattern -- they are detected structurally, not by scanning +# a single line -- but they appear here so they share the remediation help. CHECKS = ( Check("sphinx-role", SPHINX_ROLE, ":class:`X`", "[`X`][zarr.X]"), Check("rst-directive", RST_DIRECTIVE, ".. note::", "MkDocs admonition (!!! note)"), Check("rst-field", RST_FIELD, ":param x:", "numpydoc Parameters/Returns/Raises section"), Check("rst-link", RST_LINK, "`text `_", "[text](url)"), Check("list-break", None, "fence between items", "indent the fence 4 spaces to nest it"), + Check("list-indent", None, "2-space continuation", "indent the block 4 spaces under its item"), + Check("list-blank", None, "item after indented block", "add a blank line before the item"), ) @@ -211,9 +224,49 @@ def splits_a_list(open_i: int, close_i: int) -> bool: ] +def find_list_continuation_issues( + lines: list[str], in_code: set[int] +) -> list[tuple[int, str, str]]: + """Return ``(lineno, category, snippet)`` for list continuations Python-Markdown will + mis-render (see the module docstring): + + - ``list-indent``: a blank-line-separated block inside a list item indented 1-3 + spaces. Python-Markdown requires 4; at less, the block escapes the list. + - ``list-blank``: a top-level list item directly after a line indented 4+ spaces. + Without a blank line in between, the item is absorbed into the preceding paragraph + as literal ``- `` text. + + Lazy continuations (an indented line with no blank line before it) are valid at any + indent and are not flagged. Fenced-code lines are opaque: never flagged themselves, + but they keep the item scope open and their indent feeds the ``list-blank`` check so + an item directly after an indented fence is still caught.""" + findings: list[tuple[int, str, str]] = [] + in_item = False # inside a top-level list item's scope + prev_blank = True + prev_indent = 0 + for i, line in enumerate(lines): + stripped = line.strip() + if not stripped: + prev_blank = True + continue + indent = len(line) - len(line.lstrip(" ")) + if i not in in_code: + if indent == 0: + is_item = bool(LIST_ITEM.match(line)) + if is_item and in_item and not prev_blank and prev_indent >= 4: + findings.append((i + 1, "list-blank", line)) + in_item = is_item + elif in_item and prev_blank and indent < 4: + findings.append((i + 1, "list-indent", line)) + prev_blank = False + prev_indent = indent + return findings + + def lint_markdown(path: Path) -> list[Finding]: """Scan a Markdown file: RST residue in prose (skipping fenced code blocks), plus - fenced code blocks that break a list (see find_list_breaking_fences).""" + list-structure problems (see find_list_breaking_fences and + find_list_continuation_issues).""" lines = path.read_text(encoding="utf-8").splitlines() blocks = fenced_blocks(lines) in_code = {i for fence in blocks for i in range(fence.open, fence.close + 1)} @@ -228,7 +281,11 @@ def lint_markdown(path: Path) -> list[Finding]: Finding(path, lineno, "list-break", snippet) for lineno, snippet in find_list_breaking_fences(lines, blocks) ] - return prose + breaks + continuations = [ + Finding(path, lineno, category, snippet) + for lineno, category, snippet in find_list_continuation_issues(lines, in_code) + ] + return prose + breaks + continuations def iter_files(paths: tuple[Path, ...]) -> list[Path]: diff --git a/docs/contributing.md b/docs/contributing.md index df46f381de..aeb88e6ce1 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -354,18 +354,7 @@ This policy exists to lower the cost of routine work and to help newer core deve ### Release procedure -Open an issue on GitHub announcing the release using the release checklist template: -[https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md](https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md). The release checklist includes all steps necessary for the release. - -#### Preparing a release - -Releases are prepared using the ["Prepare release notes"](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) workflow. To run it: - -1. Go to the [workflow page](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) and click "Run workflow". -2. Enter the release version (e.g. `3.2.0`) and the target branch (defaults to `main`). -3. The workflow will run `towncrier build` to render the changelog, remove consumed fragments from `changes/`, and open a pull request on the `release/v` branch. -4. The release PR is automatically labeled `run-downstream`, which triggers the [downstream test workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) to run Xarray and numcodecs integration tests against the release branch. -5. Review the rendered changelog in `docs/release-notes.md` and verify downstream tests pass before merging. +To give the release visibility and a single place to track progress, open an issue on GitHub announcing the release using the [release checklist template](https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md). The release checklist includes all steps necessary for the release. ## Compatibility and versioning policies @@ -377,17 +366,17 @@ Releases are classified by the library changes contained in that release. This c * **major** releases (for example, `2.18.0` -> `3.0.0`) are for changes that will require extensive adaptation efforts from many users and downstream projects. For example, breaking changes to widely-used user-facing APIs should only be applied in a major release. - Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. + Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. * **minor** releases (for example, `3.0.0` -> `3.1.0`) are for changes that do not require significant effort from most users or downstream projects to respond to. API changes are possible in minor releases if the burden on users imposed by those changes is sufficiently small. - For example, a recently released API may need fixes or refinements that are breaking, but low impact due to the recency of the feature. Such API changes are permitted in a minor release. + For example, a recently released API may need fixes or refinements that are breaking, but low impact due to the recency of the feature. Such API changes are permitted in a minor release. - Minor releases are safe for most users and downstream projects to adopt. + Minor releases are safe for most users and downstream projects to adopt. * **patch** releases (for example, `3.1.0` -> `3.1.1`) are for changes that contain no breaking or behavior changes for downstream projects or users. Examples of changes suitable for a patch release are bugfixes and documentation improvements. - Users should always feel safe upgrading to the latest patch release. + Users should always feel safe upgrading to the latest patch release. Note that this versioning scheme is not consistent with [Semantic Versioning](https://semver.org/). Contrary to SemVer, the Zarr library may release breaking changes in `minor` releases, or even `patch` releases under exceptional circumstances. But we should strive to avoid doing so. diff --git a/docs/release-notes.md b/docs/release-notes.md index 7a5b12f59f..3fd8a5f360 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -51,32 +51,34 @@ `ClientSession` was left open until garbage collection, producing `"Unclosed client session"` `ResourceWarning`s from aiohttp. - The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when - `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` - when a sync→async conversion was performed). When `_owns_fs` is ``True``, - `store.close()` calls the new `_close_fs()` helper, which invokes - `fs.set_session()` and closes the returned client. Callers who supply their own - filesystem instance to `FsspecStore()` directly remain responsible for its - lifecycle; `_owns_fs` is ``False`` for those stores. - - **Scope note**: This fix closes the S3 client session that is active at the time - `store.close()` is called. Some S3-backed filesystem implementations (e.g. - s3fs with ``cache_regions=True``) may internally refresh and replace their - client during I/O operations, abandoning prior sessions before ``store.close()`` - is invoked. Those intermediate sessions are outside the scope of this fix and - are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/issues/4003)) + The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when + `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` + when a sync→async conversion was performed). When `_owns_fs` is ``True``, + `store.close()` calls the new `_close_fs()` helper, which invokes + `fs.set_session()` and closes the returned client. Callers who supply their own + filesystem instance to `FsspecStore()` directly remain responsible for its + lifecycle; `_owns_fs` is ``False`` for those stores. + + **Scope note**: This fix closes the S3 client session that is active at the time + `store.close()` is called. Some S3-backed filesystem implementations (e.g. + s3fs with ``cache_regions=True``) may internally refresh and replace their + client during I/O operations, abandoning prior sessions before ``store.close()`` + is invoked. Those intermediate sessions are outside the scope of this fix and + are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/issues/4003)) + - Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested. ([#4016](https://github.com/zarr-developers/zarr-python/issues/4016)) - Fixed `ObjectStore.list_dir` for object-store listings that include a directory-marker object matching the requested non-root prefix. ([#4032](https://github.com/zarr-developers/zarr-python/issues/4032)) - Prevents mutation of the attributes dict provided by the user by copying them instead of keeping the reference ([#4059](https://github.com/zarr-developers/zarr-python/issues/4059)) - Fixed several storage and codec bugs: - - Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. - - `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). - - `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). - - `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). - - `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. + - Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. + - `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). + - `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). + - `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). + - `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. + + ([#4074](https://github.com/zarr-developers/zarr-python/issues/4074)) - ([#4074](https://github.com/zarr-developers/zarr-python/issues/4074)) - Fixed writing Fortran-ordered (F-contiguous) arrays through the variable-length string and bytes codecs and through numcodecs array-array filters such as `Delta`, `FixedScaleOffset` and `PackBits`. Chunks are now passed to numcodecs as C-contiguous arrays, so elements are no longer stored in transposed order. ([#4116](https://github.com/zarr-developers/zarr-python/issues/4116)) - Fix silent byte-order corruption for structured dtypes with the `bytes` codec: multi-byte fields are now byte-swapped to the codec's configured `endian` on write and decoded honoring it on read, so non-native-endian structured data (e.g. big-endian fields, as produced by virtual references to external data) round-trips correctly. ([#4141](https://github.com/zarr-developers/zarr-python/issues/4141)) @@ -91,23 +93,24 @@ use only the public API, eliminating all non-public imports, illustrating what users should do. - To better support this, the following types and functions were made available - from public modules: - - | Type/Function | Non-public module | Public module | - | ------------------------- | ------------------------ | ------------- | - | `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | - | `JSON` | `zarr.core.common` | `zarr.types` | - | `ZarrFormat` | `zarr.core.common` | `zarr.types` | - | `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | - | `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | - | `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | - | `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | - - `DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from - `zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or - `zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining - types and functions are simply re-exported from the listed public module. ([#4052](https://github.com/zarr-developers/zarr-python/issues/4052)) + To better support this, the following types and functions were made available + from public modules: + + | Type/Function | Non-public module | Public module | + | ------------------------- | ------------------------ | ------------- | + | `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | + | `JSON` | `zarr.core.common` | `zarr.types` | + | `ZarrFormat` | `zarr.core.common` | `zarr.types` | + | `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | + | `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | + + `DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from + `zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or + `zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining + types and functions are simply re-exported from the listed public module. ([#4052](https://github.com/zarr-developers/zarr-python/issues/4052)) + - Document a self-merge policy in the contributor guide, describing when a core developer may merge their own pull request without a second reviewer and which changes warrant more caution. ([#4053](https://github.com/zarr-developers/zarr-python/issues/4053)) - Fixed many documentation errors found in a full review of the user guide, including prose contradicted by rendered example output on the performance page, invisible @@ -126,11 +129,12 @@ access, and will be removed in a future release. ``BloscCodec.cname`` and ``BloscCodec.shuffle`` are now plain strings rather than enum members. - Additional renames in ``zarr.codecs.blosc`` from the same change: the type - aliases ``Shuffle`` and ``CName`` are now ``BloscShuffleLiteral`` and - ``BloscCnameLiteral``, the constant ``SHUFFLE`` is now ``BLOSC_SHUFFLE`` - (with a new ``BLOSC_CNAME`` alongside it), and ``BloscShuffle.from_int`` - now returns a literal string rather than an enum member. ([#3963](https://github.com/zarr-developers/zarr-python/issues/3963)) + Additional renames in ``zarr.codecs.blosc`` from the same change: the type + aliases ``Shuffle`` and ``CName`` are now ``BloscShuffleLiteral`` and + ``BloscCnameLiteral``, the constant ``SHUFFLE`` is now ``BLOSC_SHUFFLE`` + (with a new ``BLOSC_CNAME`` alongside it), and ``BloscShuffle.from_int`` + now returns a literal string rather than an enum member. ([#3963](https://github.com/zarr-developers/zarr-python/issues/3963)) + - The ``Endian`` (``zarr.codecs.bytes.Endian``) and ``ShardingCodecIndexLocation`` (``zarr.codecs.ShardingCodecIndexLocation``) enums are now deprecated. Pass the equivalent literal string instead (e.g. ``"little"`` / ``"big"``, ``"start"`` / @@ -139,20 +143,21 @@ and ``ShardingCodec.index_location`` are now plain strings rather than enum members. - Two follow-on changes from this deprecation: + Two follow-on changes from this deprecation: - - ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or - ``"big"``) rather than an ``Endian`` member. Subclasses overriding this - property should update their return type. - - The module-level binding ``zarr.codecs.bytes.default_system_endian`` was - removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; - external callers that imported ``default_system_endian`` should use - ``sys.byteorder`` directly. + - ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or + ``"big"``) rather than an ``Endian`` member. Subclasses overriding this + property should update their return type. + - The module-level binding ``zarr.codecs.bytes.default_system_endian`` was + removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; + external callers that imported ``default_system_endian`` should use + ``sys.byteorder`` directly. - Additionally, the module-level function ``zarr.codecs.sharding.parse_index_location`` - was made private as part of this change. + Additionally, the module-level function ``zarr.codecs.sharding.parse_index_location`` + was made private as part of this change. + + ([#3968](https://github.com/zarr-developers/zarr-python/issues/3968)) - ([#3968](https://github.com/zarr-developers/zarr-python/issues/3968)) - Removed the NumPy 1.x implementation of the `VariableLengthUTF8` data type because NumPy 1.x is no longer supported under [SPEC0](https://scientific-python.org/specs/spec-0000/). ([#3973](https://github.com/zarr-developers/zarr-python/issues/3973)) ### Misc @@ -191,19 +196,19 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Add support for rectilinear (variable-sized) chunk grids. This feature is experimental and must be explicitly enabled via `zarr.config.set({'array.rectilinear_chunks': True})`. - Rectilinear chunks can be used through: + Rectilinear chunks can be used through: - - **Creating arrays**: Pass nested sequences (e.g., `[[10, 20, 30], [50, 50]]`) to `chunks` - in `zarr.create_array`, `zarr.from_array`, `zarr.zeros`, `zarr.ones`, `zarr.full`, - `zarr.open`, and related functions, or to `chunk_shape` in `zarr.create`. - - **Opening existing arrays**: Arrays stored with the `rectilinear` chunk grid are read - transparently via `zarr.open` and `zarr.open_array`. - - **Rectilinear sharding**: Shard boundaries can be rectilinear while inner chunks remain regular. + - **Creating arrays**: Pass nested sequences (e.g., `[[10, 20, 30], [50, 50]]`) to `chunks` + in `zarr.create_array`, `zarr.from_array`, `zarr.zeros`, `zarr.ones`, `zarr.full`, + `zarr.open`, and related functions, or to `chunk_shape` in `zarr.create`. + - **Opening existing arrays**: Arrays stored with the `rectilinear` chunk grid are read + transparently via `zarr.open` and `zarr.open_array`. + - **Rectilinear sharding**: Shard boundaries can be rectilinear while inner chunks remain regular. - **Breaking change**: The `validate` method on `BaseCodec` and `CodecPipeline` now receives - a `ChunkGridMetadata` instance instead of a `ChunkGrid` instance for the `chunk_grid` - parameter. Third-party codecs that override `validate` and inspect the chunk grid will need to - update their type annotations. No known downstream packages were using this parameter. ([#3802](https://github.com/zarr-developers/zarr-python/issues/3802)) + **Breaking change**: The `validate` method on `BaseCodec` and `CodecPipeline` now receives + a `ChunkGridMetadata` instance instead of a `ChunkGrid` instance for the `chunk_grid` + parameter. Third-party codecs that override `validate` and inspect the chunk grid will need to + update their type annotations. No known downstream packages were using this parameter. ([#3802](https://github.com/zarr-developers/zarr-python/issues/3802)) - Add `cast_value` and `scale_offset` codecs. ([#3874](https://github.com/zarr-developers/zarr-python/issues/3874)) @@ -394,52 +399,52 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Ensure that invocations of `create_array` use consistent keyword arguments, with consistent defaults. - [`zarr.api.synchronous.create_array`][] now takes a `write_data` keyword argument - The `Group.create_array` method takes `data` and `write_data` keyword arguments. - The functions [`zarr.api.asynchronous.create`][], [`zarr.api.asynchronous.create_array`] - and the methods `Group.create_array`, `Group.array`, had the default - `fill_value` changed from `0` to the `DEFAULT_FILL_VALUE` value, which instructs Zarr to - use the default scalar value associated with the array's data type as the fill value. These are - all functions or methods for array creation that mirror, wrap or are wrapped by, another function - that already has a default `fill_value` set to `DEFAULT_FILL_VALUE`. This change is necessary - to make these functions consistent across the entire codebase, but as this changes default values, - new data might have a different fill value than expected after this change. - - For data types where 0 is meaningful, like integers or floats, the default scalar is 0, so this - change should not be noticeable. For data types where 0 is ambiguous, like fixed-length unicode - strings, the default fill value might be different after this change. Users who were relying on how - Zarr interpreted `0` as a non-numeric scalar value should set their desired fill value explicitly - after this change. + [`zarr.api.synchronous.create_array`][] now takes a `write_data` keyword argument + The `Group.create_array` method takes `data` and `write_data` keyword arguments. + The functions [`zarr.api.asynchronous.create`][], [`zarr.api.asynchronous.create_array`] + and the methods `Group.create_array`, `Group.array`, had the default + `fill_value` changed from `0` to the `DEFAULT_FILL_VALUE` value, which instructs Zarr to + use the default scalar value associated with the array's data type as the fill value. These are + all functions or methods for array creation that mirror, wrap or are wrapped by, another function + that already has a default `fill_value` set to `DEFAULT_FILL_VALUE`. This change is necessary + to make these functions consistent across the entire codebase, but as this changes default values, + new data might have a different fill value than expected after this change. + + For data types where 0 is meaningful, like integers or floats, the default scalar is 0, so this + change should not be noticeable. For data types where 0 is ambiguous, like fixed-length unicode + strings, the default fill value might be different after this change. Users who were relying on how + Zarr interpreted `0` as a non-numeric scalar value should set their desired fill value explicitly + after this change. - Added public API for Buffer ABCs and implementations. - Use `zarr.buffer` to access buffer implementations, and - `zarr.abc.buffer` for the interface to implement new buffer types. + Use `zarr.buffer` to access buffer implementations, and + `zarr.abc.buffer` for the interface to implement new buffer types. - Users previously importing buffer from `zarr.core.buffer` should update their - imports to use `zarr.buffer`. As a reminder, all of `zarr.core` is - considered a private API that's not covered by zarr-python's versioning policy. ([#2871](https://github.com/zarr-developers/zarr-python/issues/2871)) + Users previously importing buffer from `zarr.core.buffer` should update their + imports to use `zarr.buffer`. As a reminder, all of `zarr.core` is + considered a private API that's not covered by zarr-python's versioning policy. ([#2871](https://github.com/zarr-developers/zarr-python/issues/2871)) - Adds zarr-specific data type classes. - This change adds a `ZDType` base class for Zarr V2 and Zarr V3 data types. Child classes are - defined for each NumPy data type. Each child class defines routines for `JSON` serialization. - New data types can be created and registered dynamically. + This change adds a `ZDType` base class for Zarr V2 and Zarr V3 data types. Child classes are + defined for each NumPy data type. Each child class defines routines for `JSON` serialization. + New data types can be created and registered dynamically. - Prior to this change, Zarr Python had two streams for handling data types. For Zarr V2 arrays, - we used NumPy data type identifiers. For Zarr V3 arrays, we used a fixed set of string enums. Both - of these systems proved hard to extend. + Prior to this change, Zarr Python had two streams for handling data types. For Zarr V2 arrays, + we used NumPy data type identifiers. For Zarr V3 arrays, we used a fixed set of string enums. Both + of these systems proved hard to extend. - This change is largely internal, but it does change the type of the `dtype` and `data_type` - fields on the `ArrayV2Metadata` and `ArrayV3Metadata` classes. Previously, `ArrayV2Metadata.dtype` - was a NumPy `dtype` object, and `ArrayV3Metadata.data_type` was an internally-defined `enum`. - After this change, both `ArrayV2Metadata.dtype` and `ArrayV3Metadata.data_type` are instances of - `ZDType`. A NumPy data type can be generated from a `ZDType` via the `ZDType.to_native_dtype()` - method. The internally-defined Zarr V3 `enum` class is gone entirely, but the `ZDType.to_json(zarr_format=3)` - method can be used to generate either a string, or dictionary that has a string `name` field, that - represents the string value previously associated with that `enum`. + This change is largely internal, but it does change the type of the `dtype` and `data_type` + fields on the `ArrayV2Metadata` and `ArrayV3Metadata` classes. Previously, `ArrayV2Metadata.dtype` + was a NumPy `dtype` object, and `ArrayV3Metadata.data_type` was an internally-defined `enum`. + After this change, both `ArrayV2Metadata.dtype` and `ArrayV3Metadata.data_type` are instances of + `ZDType`. A NumPy data type can be generated from a `ZDType` via the `ZDType.to_native_dtype()` + method. The internally-defined Zarr V3 `enum` class is gone entirely, but the `ZDType.to_json(zarr_format=3)` + method can be used to generate either a string, or dictionary that has a string `name` field, that + represents the string value previously associated with that `enum`. - For more on this new feature, see the [documentation](user-guide/data_types.md) ([#2874](https://github.com/zarr-developers/zarr-python/issues/2874)) + For more on this new feature, see the [documentation](user-guide/data_types.md) ([#2874](https://github.com/zarr-developers/zarr-python/issues/2874)) - Added `NDBuffer.empty` method for faster ndbuffer initialization. ([#3191](https://github.com/zarr-developers/zarr-python/issues/3191)) @@ -451,10 +456,10 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Fixes a variety of issues related to string data types. - - Brings the `VariableLengthUTF8` data type Zarr V3 identifier in alignment with Zarr Python 3.0.8 - - Disallows creation of 0-length fixed-length data types - - Adds a regression test for the `VariableLengthUTF8` data type that checks against version 3.0.8 - - Allows users to request the `VariableLengthUTF8` data type with `str`, `"str"`, or `"string"`. ([#3170](https://github.com/zarr-developers/zarr-python/issues/3170)) + - Brings the `VariableLengthUTF8` data type Zarr V3 identifier in alignment with Zarr Python 3.0.8 + - Disallows creation of 0-length fixed-length data types + - Adds a regression test for the `VariableLengthUTF8` data type that checks against version 3.0.8 + - Allows users to request the `VariableLengthUTF8` data type with `str`, `"str"`, or `"string"`. ([#3170](https://github.com/zarr-developers/zarr-python/issues/3170)) - Add human readable size for No. bytes stored to `info_complete` ([#3190](https://github.com/zarr-developers/zarr-python/issues/3190)) @@ -475,26 +480,26 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Add a description on how to create a RemoteStore of a specific filesystem to the `Remote Store` section in `docs/user-guide/storage.md`. State in the docstring of `FsspecStore.from_url` that the filesystem type is inferred from the URL scheme. - It should help a user handling the case when the type of FsspecStore doesn't match the URL scheme. ([#3212](https://github.com/zarr-developers/zarr-python/issues/3212)) + It should help a user handling the case when the type of FsspecStore doesn't match the URL scheme. ([#3212](https://github.com/zarr-developers/zarr-python/issues/3212)) ### Deprecations and Removals - Removes default chunk encoding settings (filters, serializer, compressors) from the global configuration object. - This removal is justified on the basis that storing chunk encoding settings in the config required - a brittle, confusing, and inaccurate categorization of array data types, which was particularly - unsuitable after the recent addition of new data types that didn't fit naturally into the - pre-existing categories. + This removal is justified on the basis that storing chunk encoding settings in the config required + a brittle, confusing, and inaccurate categorization of array data types, which was particularly + unsuitable after the recent addition of new data types that didn't fit naturally into the + pre-existing categories. - The default chunk encoding is the same (Zstandard compression, and the required object codecs for - variable length data types), but the chunk encoding is now generated by functions that cannot be - reconfigured at runtime. Users who relied on setting the default chunk encoding via the global configuration object should - instead specify the desired chunk encoding explicitly when creating an array. + The default chunk encoding is the same (Zstandard compression, and the required object codecs for + variable length data types), but the chunk encoding is now generated by functions that cannot be + reconfigured at runtime. Users who relied on setting the default chunk encoding via the global configuration object should + instead specify the desired chunk encoding explicitly when creating an array. - This change also adds an extra validation step to the creation of Zarr V2 arrays, which ensures that - arrays with a `VariableLengthUTF8` or `VariableLengthBytes` data type cannot be created without the - correct "object codec". ([#3228](https://github.com/zarr-developers/zarr-python/issues/3228)) + This change also adds an extra validation step to the creation of Zarr V2 arrays, which ensures that + arrays with a `VariableLengthUTF8` or `VariableLengthBytes` data type cannot be created without the + correct "object codec". ([#3228](https://github.com/zarr-developers/zarr-python/issues/3228)) - Removes support for passing keyword-only arguments positionally to the following functions and methods: `save_array`, `open`, `group`, `open_group`, `create`, `get_basic_selection`, `set_basic_selection`, @@ -546,8 +551,8 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr For large arrays this will result in smaller default chunk sizes. To retain previous behaviour, explicitly set the chunk shape to the data shape. - This fix matches the existing chunking behaviour of - `zarr.save_array` and `zarr.api.asynchronous.AsyncArray.create`. ([#3103](https://github.com/zarr-developers/zarr-python/issues/3103)) + This fix matches the existing chunking behaviour of + `zarr.save_array` and `zarr.api.asynchronous.AsyncArray.create`. ([#3103](https://github.com/zarr-developers/zarr-python/issues/3103)) - When `zarr.save` has an argument `path=some/path/` and multiple arrays in `args`, the path resulted in `some/path/some/path` due to using the `path` argument twice while building the array path. This is now fixed. ([#3127](https://github.com/zarr-developers/zarr-python/issues/3127)) @@ -556,12 +561,12 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Suppress `FileNotFoundError` when deleting non-existent keys in the `obstore` adapter. - When writing empty chunks (i.e. chunks where all values are equal to the array's fill value) to a zarr array, zarr - will delete those chunks from the underlying store. For zarr arrays backed by the `obstore` adapter, this will potentially - raise a `FileNotFoundError` if the chunk doesn't already exist. - Since whether or not a delete of a non-existing object raises an error depends on the behavior of the underlying store, - suppressing the error in all cases results in consistent behavior across stores, and is also what `zarr` seems to expect - from the store. ([#3140](https://github.com/zarr-developers/zarr-python/issues/3140)) + When writing empty chunks (i.e. chunks where all values are equal to the array's fill value) to a zarr array, zarr + will delete those chunks from the underlying store. For zarr arrays backed by the `obstore` adapter, this will potentially + raise a `FileNotFoundError` if the chunk doesn't already exist. + Since whether or not a delete of a non-existing object raises an error depends on the behavior of the underlying store, + suppressing the error in all cases results in consistent behavior across stores, and is also what `zarr` seems to expect + from the store. ([#3140](https://github.com/zarr-developers/zarr-python/issues/3140)) - Trying to open a StorePath/Array with `mode='r'` when the store is not read-only creates a read-only copy of the store. ([#3156](https://github.com/zarr-developers/zarr-python/issues/3156)) @@ -583,8 +588,9 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - It is now possible to specify no compressor when creating a zarr format 2 array. This can be done by passing `compressor=None` to the various array creation routines. - The default behaviour of automatically choosing a suitable default compressor remains if the compressor argument is not given. - To reproduce the behaviour in previous zarr-python versions when `compressor=None` was passed, pass `compressor='auto'` instead. ([#3039](https://github.com/zarr-developers/zarr-python/issues/3039)) + The default behaviour of automatically choosing a suitable default compressor remains if the compressor argument is not given. + To reproduce the behaviour in previous zarr-python versions when `compressor=None` was passed, pass `compressor='auto'` instead. ([#3039](https://github.com/zarr-developers/zarr-python/issues/3039)) + - Fixed the typing of `dimension_names` arguments throughout so that it now accepts iterables that contain `None` alongside `str`. ([#3045](https://github.com/zarr-developers/zarr-python/issues/3045)) - Using various functions to open data with `mode='a'` no longer deletes existing data in the store. ([#3062](https://github.com/zarr-developers/zarr-python/issues/3062)) - Internally use `typesize` constructor parameter for `numcodecs.blosc.Blosc` to improve compression ratios back to the v2-package levels. ([#2962](https://github.com/zarr-developers/zarr-python/issues/2962)) diff --git a/docs/user-guide/data_types.md b/docs/user-guide/data_types.md index ac966f885f..91f828a738 100644 --- a/docs/user-guide/data_types.md +++ b/docs/user-guide/data_types.md @@ -139,17 +139,17 @@ Zarr V3 introduced several key changes to how data types are represented: The basic data types are identified by strings like `"int8"`, `"int16"`, etc., and data types that require a configuration can be identified by a JSON object. - For example, this JSON object declares a datetime data type: - - ```json - { - "name": "numpy.datetime64", - "configuration": { - "unit": "s", - "scale_factor": 10 + For example, this JSON object declares a datetime data type: + + ```json + { + "name": "numpy.datetime64", + "configuration": { + "unit": "s", + "scale_factor": 10 + } } - } - ``` + ``` - Zarr V3 data types do not have endianness. This is a departure from Zarr V2, where multi-byte data types are defined with endianness information. Instead, Zarr V3 requires that the endianness diff --git a/docs/user-guide/v3_migration.md b/docs/user-guide/v3_migration.md index 6d9d516ebe..4d97963be2 100644 --- a/docs/user-guide/v3_migration.md +++ b/docs/user-guide/v3_migration.md @@ -42,40 +42,40 @@ the following actions in order: will be compatible in Zarr-Python 3. However, the following breaking API changes are planned: - - `numcodecs.*` will no longer be available in `zarr.*`. To migrate, import codecs - directly from `numcodecs`: - - ```python exec="false" reason="intentionally shows the old/incorrect import for contrast" - from numcodecs import Blosc - # instead of: - # from zarr import Blosc - ``` - - - The `zarr.v3_api_available` feature flag is being removed. In Zarr-Python 3 - the v3 API is always available, so you shouldn't need to use this flag. - - `zarr.errors` has been consolidated. Several exception classes from - Zarr-Python 2 (such as `zarr.errors.PathNotFoundError`) have been removed - or replaced. For example, missing nodes now raise `zarr.errors.NodeNotFoundError` - (which subclasses both `BaseZarrError` and `FileNotFoundError`) instead of - `zarr.errors.PathNotFoundError`. Review any code that catches exceptions - from `zarr.errors` after migrating. - - The following internal modules are being removed or significantly changed. If - your application relies on imports from any of the below modules, you will need - to either a) modify your application to no longer rely on these imports or b) - vendor the parts of the specific modules that you need. - - * `zarr.attrs` has gone, with no replacement - * `zarr.codecs` has changed, see "Codecs" section below for more information - * `zarr.context` has gone, with no replacement - * `zarr.core` remains but should be considered private API - * `zarr.hierarchy` has gone, with no replacement (use `zarr.Group` in place of `zarr.hierarchy.Group`) - * `zarr.indexing` has gone, with no replacement - * `zarr.meta` has gone, with no replacement - * `zarr.meta_v1` has gone, with no replacement - * `zarr.sync` has gone, with no replacement - * `zarr.types` has gone, with no replacement - * `zarr.util` has gone, with no replacement - * `zarr.n5` has gone, see below for an alternative N5 option + - `numcodecs.*` will no longer be available in `zarr.*`. To migrate, import codecs + directly from `numcodecs`: + + ```python exec="false" reason="intentionally shows the old/incorrect import for contrast" + from numcodecs import Blosc + # instead of: + # from zarr import Blosc + ``` + + - The `zarr.v3_api_available` feature flag is being removed. In Zarr-Python 3 + the v3 API is always available, so you shouldn't need to use this flag. + - `zarr.errors` has been consolidated. Several exception classes from + Zarr-Python 2 (such as `zarr.errors.PathNotFoundError`) have been removed + or replaced. For example, missing nodes now raise `zarr.errors.NodeNotFoundError` + (which subclasses both `BaseZarrError` and `FileNotFoundError`) instead of + `zarr.errors.PathNotFoundError`. Review any code that catches exceptions + from `zarr.errors` after migrating. + - The following internal modules are being removed or significantly changed. If + your application relies on imports from any of the below modules, you will need + to either a) modify your application to no longer rely on these imports or b) + vendor the parts of the specific modules that you need. + + * `zarr.attrs` has gone, with no replacement + * `zarr.codecs` has changed, see "Codecs" section below for more information + * `zarr.context` has gone, with no replacement + * `zarr.core` remains but should be considered private API + * `zarr.hierarchy` has gone, with no replacement (use `zarr.Group` in place of `zarr.hierarchy.Group`) + * `zarr.indexing` has gone, with no replacement + * `zarr.meta` has gone, with no replacement + * `zarr.meta_v1` has gone, with no replacement + * `zarr.sync` has gone, with no replacement + * `zarr.types` has gone, with no replacement + * `zarr.util` has gone, with no replacement + * `zarr.n5` has gone, see below for an alternative N5 option 3. Test that your package works with version 3. 4. Update the pin to include `zarr>=3,<4`. @@ -123,8 +123,9 @@ The following sections provide details on breaking changes in Zarr-Python 3. 2. The h5py compatibility methods `create_dataset` and `require_dataset` have been removed. Use the following replacements: - - [`zarr.Group.create_array`][] in place of `Group.create_dataset` - - [`zarr.Group.require_array`][] in place of `Group.require_dataset` + - [`zarr.Group.create_array`][] in place of `Group.create_dataset` + - [`zarr.Group.require_array`][] in place of `Group.require_dataset` + 3. Disallow "." syntax for getting group members. To get a member of a group named `foo`, use `group["foo"]` in place of `group.foo`. 4. The `zarr.storage.init_group` low-level helper function has been removed. Use From 066131db82689f08e2ecb6589c076cbc83144268 Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Thu, 16 Jul 2026 12:34:04 -0400 Subject: [PATCH 2/2] Make explanation a comment --- .github/PULL_REQUEST_TEMPLATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 1d12aa02eb..47adb4b19e 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -15,7 +15,7 @@ - [ ] I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct. -AI coding assistance is welcome, but a human must be the author and is responsible for the contents of the PR. The description and any review responses must be in your own words. Please read [AI-assisted contributions](https://zarr.readthedocs.io/en/stable/contributing/#ai-assisted-contributions) before opening. + ## TODO