Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
faa5ef5
feat: add IndexTransform library for composable, lazy coordinate mapp…
d-v-b Apr 14, 2026
732dddd
Merge branch 'main' into refactor/simplify-indexing
d-v-b Apr 14, 2026
79cd9c8
feat: add JSON serialization for IndexTransform (TensorStore-compatible)
d-v-b Apr 14, 2026
db31317
Merge branch 'main' into refactor/simplify-indexing
d-v-b Apr 15, 2026
797b25e
Merge branch 'main' into refactor/simplify-indexing
d-v-b Apr 16, 2026
9f50a4c
Merge branch 'main' into refactor/simplify-indexing
d-v-b Apr 21, 2026
53c0042
Merge branch 'main' into refactor/simplify-indexing
d-v-b Apr 22, 2026
1477f92
Merge branch 'main' into refactor/simplify-indexing
d-v-b May 6, 2026
e3927be
Merge branch 'main' of https://github.com/zarr-developers/zarr-python…
d-v-b May 6, 2026
2989756
Merge branch 'main' into refactor/simplify-indexing
d-v-b May 11, 2026
ed9167c
Merge branch 'main' into refactor/simplify-indexing
d-v-b May 18, 2026
0cf74f9
Merge branch 'main' into refactor/simplify-indexing
d-v-b May 19, 2026
ce78f48
rename accessor to lazy, and use result instead of resolve
d-v-b May 19, 2026
6bd22dc
Merge branch 'main' into refactor/simplify-indexing
d-v-b Jun 8, 2026
93e8c94
perf: cache Array shape/ndim and reuse chunk grid in transform resolvers
d-v-b Jun 29, 2026
ab795ca
perf: skip the transform resolver for eager (identity-transform) inde…
d-v-b Jun 29, 2026
b43023a
perf(lazy): resolve chunk_grid[coords] once per chunk in transform re…
d-v-b Jun 29, 2026
32cd3af
perf(lazy): single-pass sub_transform_to_selections with hoisted doma…
d-v-b Jun 29, 2026
f16ce13
fix(indexing): restore type narrowing and memoize shape on IndexDomain
d-v-b Jun 29, 2026
cd217c7
docs: update Array repr doctests for domain= suffix
d-v-b Jun 29, 2026
22db497
Merge branch 'main' into refactor/simplify-indexing
d-v-b Jun 29, 2026
a2091a3
test(lazy-indexing): dense parametrized matrix over 0-D / sharded / u…
d-v-b Jun 30, 2026
24aaaaf
test(lazy-indexing): model-based randomized round-trips over sharded/…
d-v-b Jun 30, 2026
c51ef58
fix(lazy): correct orthogonal (oindex) indexing across multiple array…
d-v-b Jun 30, 2026
6f1a829
fix(indexing): honor the transform for fancy/field-less selections on…
d-v-b Jun 30, 2026
d38205c
test(lazy-indexing): hypothesis property test for the lazy-view surface
d-v-b Jun 30, 2026
d9aad9f
test(indexing): comprehensive cross-path indexing parity harness
d-v-b Jun 30, 2026
3734357
test(strategies): reusable indexing-selection fixtures (indexers, win…
d-v-b Jun 30, 2026
889787a
fix(indexing): reject block selection on a lazy view instead of corru…
d-v-b Jun 30, 2026
2027a77
test(indexing): make the parity harness consumer-agnostic (eager-merg…
d-v-b Jun 30, 2026
8dc45e2
test(indexing): fold per-mode eager tests into the single parity oracle
d-v-b Jun 30, 2026
ab2e352
refactor(indexing): address review nits (comments, dead code, test co…
d-v-b Jun 30, 2026
57f3c00
fix(indexing): bounds-check lazy oindex/vindex array values
d-v-b Jun 30, 2026
ebf9f8f
test: type mode params and indexers return; consistent mode collections
d-v-b Jun 30, 2026
8aee921
test(indexing): split read/write parity; xfail the one known-unsuppor…
d-v-b Jun 30, 2026
734febd
test(indexing): filter writes whose targets collide after negative-in…
d-v-b Jun 30, 2026
dce5d46
test(indexing): stateful model-lockstep harness + bounds error-parity…
d-v-b Jun 30, 2026
93fb2d4
test(indexing): rename indexers -> numpy_array_indexers and clarify s…
d-v-b Jun 30, 2026
8153d48
docs: use single backticks in indexing test-infra docstrings
d-v-b Jun 30, 2026
04dab0a
docs: clarify why repeated-target writes are rejected in _write_is_un…
d-v-b Jun 30, 2026
a192ca4
test(indexing): use the Expect helper for test_write_is_unambiguous c…
d-v-b Jun 30, 2026
34d3b1b
feat(array): guard grid-describing members on lazy views (LazyViewError)
d-v-b Jul 1, 2026
16e96dc
feat(array): add view-aware chunk_projections partition API (Layer B)
d-v-b Jul 1, 2026
9b8da76
Merge branch 'main' into refactor/simplify-indexing
d-v-b Jul 1, 2026
8f51ead
test(indexing): fix the write filter to inspect the raw zarr selectio…
d-v-b Jul 1, 2026
97b0947
fix(lazy): make negative slice bounds literal — one consistent coordi…
d-v-b Jul 2, 2026
02a5c67
test(lazy): type the bounds-error trigger list for mypy
d-v-b Jul 2, 2026
767df8b
docs: add a lazy-indexing user guide (theory + executed patterns)
d-v-b Jul 2, 2026
76799cc
fix(transforms): slice bounds are literal coordinates at any domain o…
d-v-b Jul 2, 2026
e70056d
feat(lazy)!: TensorStore-parity domain semantics — preservation, one …
d-v-b Jul 2, 2026
816810d
docs(lazy): rewrite the lazy-indexing guide around TensorStore domain…
d-v-b Jul 2, 2026
ba97603
test(lazy): pin mask True-positions as absolute coordinates (roborev …
d-v-b Jul 2, 2026
8d273d2
docs: specify index transform corrections
d-v-b Jul 15, 2026
e1ba6f1
docs: add stateful indexing test design
d-v-b Jul 15, 2026
49fd195
docs: plan index transform corrections
d-v-b Jul 15, 2026
d0abd4b
fix(indexing): handle empty and negative view selections
d-v-b Jul 16, 2026
0804d4e
feat(lazy): add .lazy.shape and dask duck-typing compatibility
d-v-b Jul 15, 2026
c370f79
fix(indexing): preserve array map dependency axes
d-v-b Jul 16, 2026
3b182a5
fix(indexing): distinguish orthogonal array maps
d-v-b Jul 16, 2026
a0f48f9
fix(indexing): compose advanced lazy selections
d-v-b Jul 16, 2026
43bcf71
fix(indexing): validate boolean mask shapes at the lazy boundary
d-v-b Jul 16, 2026
8c6d152
fix(indexing): preserve transform result shapes
d-v-b Jul 16, 2026
6b20925
test(indexing): generate composed indexing programs
d-v-b Jul 16, 2026
15d1ffb
fix(lazy): guard the async surface against silent transform bypass
d-v-b Jul 16, 2026
cd33fb0
fix(indexing): correct sharded detection, chunk-coverage parity, and …
d-v-b Jul 17, 2026
06ff95a
chore: review housekeeping — truthiness, mkdocs docstrings, changelog…
d-v-b Jul 17, 2026
7ea8899
refactor(transforms): extract index-transform algebra into zarr-trans…
d-v-b Jul 17, 2026
f72df94
docs(transforms): note zarr-transforms package in 3906 changelog
d-v-b Jul 17, 2026
420956a
fix(coverage): measure zarr-transforms package source in coverage runs
d-v-b Jul 17, 2026
f6675be
fix(lazy): raise NotImplementedError for fancy-after-fancy composition
d-v-b Jul 17, 2026
0d390cb
ci(zarr-transforms): add test/release workflows and disclose eager ch…
d-v-b Jul 17, 2026
d6d4d3d
fix(zarr-transforms): expose __version__ for the release smoke test
d-v-b Jul 17, 2026
3c276b7
perf(zarr-transforms): enumerate only touched chunks in iter_chunk_tr…
d-v-b Jul 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/check_changelogs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,6 @@ jobs:

- name: Check zarr-metadata changelog entries
run: uv run --no-sync python ci/check_changelog_entries.py packages/zarr-metadata/changes

- name: Check zarr-transforms changelog entries
run: uv run --no-sync python ci/check_changelog_entries.py packages/zarr-transforms/changes
117 changes: 117 additions & 0 deletions .github/workflows/zarr-transforms-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
name: zarr-transforms release

on:
workflow_dispatch:
push:
tags:
- 'zarr_transforms-v*'

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false

jobs:
build:
name: Build wheel and sdist
runs-on: ubuntu-latest
defaults:
run:
shell: bash
working-directory: packages/zarr-transforms
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0 # hatch-vcs needs full history + tags

- name: Install Hatch
uses: pypa/hatch@257e27e51a6a5616ed08a39a408a21c35c9931bc
with:
version: '1.16.5'

- name: Build
run: hatch build

- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: zarr-transforms-dist
path: packages/zarr-transforms/dist

test_artifacts:
name: Test built artifacts
needs: [build]
runs-on: ubuntu-latest
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: zarr-transforms-dist
path: dist

- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: false

- name: Set up Python
run: uv python install 3.12

- name: Install built wheel and run import smoke test
run: |
wheel=$(ls dist/*.whl)
uv run --with "${wheel}" --python 3.12 --no-project \
python -c "import zarr_transforms; print('zarr_transforms', zarr_transforms.__version__)"

upload_pypi:
name: Upload to PyPI
needs: [build, test_artifacts]
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/zarr_transforms-v')
runs-on: ubuntu-latest
environment:
name: zarr-transforms-releases
url: https://pypi.org/p/zarr-transforms
permissions:
id-token: write # required for OIDC trusted publishing
attestations: write # required for artifact attestations
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: zarr-transforms-dist
path: dist

- name: Generate artifact attestation
uses: actions/attest@59d89421af93a897026c735860bf21b6eb4f7b26 # v4.1.0
with:
subject-path: dist/*

- name: Publish package to PyPI
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0

upload_testpypi:
name: Upload to TestPyPI
needs: [build, test_artifacts]
if: github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
environment:
name: zarr-transforms-releases-test
url: https://test.pypi.org/p/zarr-transforms
permissions:
id-token: write
attestations: write
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: zarr-transforms-dist
path: dist

- name: Generate artifact attestation
uses: actions/attest@59d89421af93a897026c735860bf21b6eb4f7b26 # v4.1.0
with:
subject-path: dist/*

- name: Publish package to TestPyPI
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
with:
repository-url: https://test.pypi.org/legacy/
102 changes: 102 additions & 0 deletions .github/workflows/zarr-transforms.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
name: zarr-transforms

on:
push:
branches: [main]
paths:
- 'packages/zarr-transforms/**'
- '.github/workflows/zarr-transforms.yml'
pull_request:
paths:
- 'packages/zarr-transforms/**'
- '.github/workflows/zarr-transforms.yml'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
test:
name: pytest py=${{ matrix.python-version }}
runs-on: ubuntu-latest
defaults:
run:
shell: bash
strategy:
fail-fast: false
matrix:
python-version: ['3.12', '3.13', '3.14']
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true
- name: Set up Python ${{ matrix.python-version }}
run: uv python install ${{ matrix.python-version }}
# The transform tests exercise chunk resolution against zarr's ChunkGrid,
# so they run from the repo root against the full workspace (which installs
# both `zarr` and the `zarr-transforms` workspace member) rather than in
# package isolation.
- name: Sync test dependency group
run: uv sync --group test --python ${{ matrix.python-version }}
- name: Run pytest
run: uv run --group test pytest packages/zarr-transforms/tests

ruff:
name: ruff
runs-on: ubuntu-latest
defaults:
run:
shell: bash
working-directory: packages/zarr-transforms
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
- name: Run ruff
run: uvx ruff check .

pyright:
name: pyright
runs-on: ubuntu-latest
defaults:
run:
shell: bash
working-directory: packages/zarr-transforms
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true
- name: Set up Python
run: uv python install 3.12
- name: Sync test dependency group
run: uv sync --group test --python 3.12
- name: Run pyright
run: uv run --group test --with pyright pyright src

zarr-transforms-complete:
name: zarr-transforms complete
needs: [test, ruff, pyright]
if: always()
runs-on: ubuntu-latest
steps:
- name: Check failure
if: |
contains(needs.*.result, 'failure') ||
contains(needs.*.result, 'cancelled')
run: exit 1
- name: Success
run: echo Success!
1 change: 1 addition & 0 deletions changes/3906.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added a `.lazy` accessor to `Array` for lazy (deferred) indexing. `array.lazy[...]`, `array.lazy.oindex[...]`, and `array.lazy.vindex[...]` return a new `Array` view composed on a TensorStore-style index-transform layer; slicing and indexing a view never triggers I/O, and views can be composed further before reading. Call `.result()` on a view to resolve it and return the underlying data. Regular (eager) indexing semantics on `Array` and `AsyncArray` are unchanged, with two deliberate exceptions on `Array`: its `repr` now carries a `domain={...}` suffix describing the view's coordinate box, and iterating a 0-d array now raises `TypeError("iteration over a 0-d array")` to match NumPy (previously it silently yielded nothing). Arrays also gain `Array.chunk_projections`, which enumerates the stored chunks a view (or a whole array) projects onto, for view-aware chunk partitioning. The index-transform machinery that powers this lives in a new NumPy-only `zarr-transforms` workspace package (import name `zarr_transforms`), which `zarr` now depends on at runtime.
1 change: 1 addition & 0 deletions docs/api/zarr/array.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
::: zarr.Array
::: zarr.AsyncArray
::: zarr.ChunkProjection
Loading