Skip to content

fix(store): copy buffers on write in MemoryStore#224

Open
d-v-b wants to merge 10 commits into
mainfrom
fix/memorystore-buffer-aliasing
Open

fix(store): copy buffers on write in MemoryStore#224
d-v-b wants to merge 10 commits into
mainfrom
fix/memorystore-buffer-aliasing

Conversation

@d-v-b

@d-v-b d-v-b commented Jul 16, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Fixes F1 from the v3.3.0 evaluation — silent wrong data on the default pipeline.

The bug

z = zarr.create_array(store=MemoryStore(), shape=(8,), chunks=(4,), dtype="i4", compressors=None)
data = np.arange(8, dtype="i4")
z[:] = data
data[:] = -1        # mutate the source *after* the write
z[:]                # -> [-1 -1 -1 -1 -1 -1 -1 -1]   the stored chunks changed

No error, no warning. Reproduces on both pipelines.

Root cause

Three zero-copy links in a row, ending in a store that retains what it is handed:

  1. _merge_chunk_array returns a view of the caller's value for complete chunks
  2. BytesCodec._encode_sync wraps that view without copying when there is no compressor
  3. MemoryStore.set stores the buffer by reference

Every other store serializes on write, so the alias dies at the syscall. MemoryStore is the one that keeps the caller's pages alive in a dict.

Why fix it here

The view itself is sound, and the existing contract around it is deliberate — _merge_chunk_array's docstring and test_merge_complete_chunk_returns_view_and_write_does_not_mutate_source both pin it. But that contract only reasoned about zarr's callers treating the view as read-only. The invariant a store needs is stronger: don't retain. So the defect is at the store boundary, and that is where this fixes it.

Copying in MemoryStore.set rather than narrowing the fast path means:

  • _merge_chunk_array is untouched — the pinned test still passes, and perf: use sync methods for chunk encoding / decoding zarr-developers/zarr-python#3885's win is intact for LocalStore/ZipStore/remote stores, which were never affected.
  • The single-chunk case is fixed too. Narrowing the fast path back to v3.2.1's value.shape == chunk_spec.shape would only shrink the blast radius: v3.2.1 aliased on single-chunk writes and silently corrupted just the same. This fixes both.

Cost

Measured on a 34 MB write to MemoryStore:

before after
uncompressed 22.6 ms 39.4 ms
compressed 52.0 ms 52.6 ms

The copy is only material on the uncompressed path — which is exactly the path that was aliasing. Compressed writes hand over a freshly-encoded, smaller buffer and are unchanged. Worth being explicit that part of zarr-developers#3885's uncompressed speedup came from eliminating a copy MemoryStore actually needed; v3.2.1 paid a comparable one via _merge_chunk_array's create/fill/copy.

Changes

  • MemoryStore.set / set_sync / set_if_not_exists copy via a new _copy_buffer helper. The byte_range branch already copies bytes into an existing buffer, so it is left alone.
  • ArrayLike protocol gains copy() (NDArrayLike already declares it; numpy and cupy both implement it).
  • MemoryStore docstring now states the ownership semantics.

Tests

  • test_set_does_not_retain_caller_buffer — store-level, across all three write methods.
  • test_write_does_not_alias_source_array — end-to-end, across both pipelines and the multi-chunk / multi-chunk-exact / single-chunk shapes.

Both fail on main and pass here. Full suite: 6739 passed, 0 failed. mypy clean.

ilan-gold and others added 6 commits July 14, 2026 15:13
* chore(deps): bump the actions group across 1 directory with 8 updates (#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: byte-order handling for structured dtypes in the bytes codec (#220)

* fix: byte-order handling for structured dtypes in the bytes codec

The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).

Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.

Closes zarr-developers#4141

Assisted-by: ClaudeCode:claude-fable-5

* test: fold structured byte-order cases into existing bytes codec tests

Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: rename stored_dtype to view_dtype in BytesCodec decode

The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.

Assisted-by: ClaudeCode:claude-fable-5

* docs: note that the decode-side byte-order conversion copies the chunk

Assisted-by: ClaudeCode:claude-fable-5

* docs: 3.3.0 release notes

Add missing changelog fragments for zarr-developers#3955 (datetime64/timedelta64 V3
metadata types), zarr-developers#3966 (writes to 0-dimensional sharded arrays), and the
public alias renames that accompanied the zarr-developers#3963/zarr-developers#3968 enum deprecations,
then build the 3.3.0 release notes with towncrier, consuming all
fragments accumulated since v3.2.1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* chore: CI worfklows for missing docs and RST docstrings

* Add linter for docs

* Add link checking workflow

* fix links

* fix: apply suggestions from code review

* refactor: readability improvements

* chore: switch lychee action to opening issue on failure

---------

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…4153)

* chore: cleanup indentation and guidance for release

* Make explanation a comment
Encoding an uncompressed chunk hands the store a zero-copy view of the
caller's array, and MemoryStore keeps whatever it is given alive in a
dict rather than serializing it. Mutating the source array after a write
therefore rewrote chunks already committed to the store, silently.

Stores that serialize on write (LocalStore, ZipStore, remote stores) are
unaffected, so they keep the full benefit of zarr-developers#3885. Only MemoryStore pays
the copy, and only where it was aliasing to begin with: an uncompressed
34 MB write goes from ~23 ms to ~39 ms, while compressed writes are
unchanged.

Copying at the store boundary rather than narrowing the fast path in
_merge_chunk_array also fixes the single-chunk case, which aliased in
v3.2.1 too.

Assisted-by: ClaudeCode:claude-opus-4.8
Assisted-by: ClaudeCode:claude-opus-4.8
…zarr-developers#4152)

Bumps the actions group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [actions/checkout](https://github.com/actions/checkout) | `6.0.3` | `7.0.0` |
| [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `8.2.0` | `8.3.2` |
| [CodSpeedHQ/action](https://github.com/codspeedhq/action) | `4.18.1` | `4.18.5` |
| [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics) | `4.2.8` | `5.0.0` |
| [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) | `2.8.0` | `2.9.0` |
| [actions/labeler](https://github.com/actions/labeler) | `6.1.0` | `6.2.0` |



Updates `actions/checkout` from 6.0.3 to 7.0.0
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v6.0.3...9c091bb)

Updates `astral-sh/setup-uv` from 8.2.0 to 8.3.2
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@fac544c...11f9893)

Updates `CodSpeedHQ/action` from 4.18.1 to 4.18.5
- [Release notes](https://github.com/codspeedhq/action/releases)
- [Changelog](https://github.com/CodSpeedHQ/action/blob/main/CHANGELOG.md)
- [Commits](CodSpeedHQ/action@a4a36bb...f99becd)

Updates `github-community-projects/issue-metrics` from 4.2.8 to 5.0.0
- [Release notes](https://github.com/github-community-projects/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@44173f9...df8c49d)

Updates `lycheeverse/lychee-action` from 2.8.0 to 2.9.0
- [Release notes](https://github.com/lycheeverse/lychee-action/releases)
- [Commits](lycheeverse/lychee-action@8646ba3...e747777)

Updates `actions/labeler` from 6.1.0 to 6.2.0
- [Release notes](https://github.com/actions/labeler/releases)
- [Commits](actions/labeler@f27b608...b8dd2d9)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/labeler
  dependency-version: 6.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: astral-sh/setup-uv
  dependency-version: 8.3.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: CodSpeedHQ/action
  dependency-version: 4.18.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github-community-projects/issue-metrics
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: lycheeverse/lychee-action
  dependency-version: 2.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants