Skip to content

buffer aliasing in memory storage w/ no compression #4156

Description

@d-v-b

From an investigation @maxrjones did with Claude Fable:

_merge_chunk_array returns a view value[out_selection] for any complete chunk (src/zarr/core/chunk_utils.py:134). With no compressor, BytesCodec._encode_sync produces a zero-copy buffer (src/zarr/codecs/bytes.py:155) and MemoryStore.set stores it by reference (src/zarr/storage/_memory.py:120). Net effect: after z[:] = data on an uncompressed MemoryStore array, mutating data mutates the stored chunks.

At v3.2.1 the zero-copy fast path fired only when a single chunk covered the entire write (that case was already aliased — pre-existing); this range widened it to every complete chunk, so ordinary multi-chunk writes are now affected. tests/test_fastpath_equivalence.py:157 pins the view as intentional but only tests the write-doesn't-mutate-source direction, not source-mutates-store. Reads are unaffected (scatter copies). Suggested fix: copy at the MemoryStore boundary, or narrow the view fast path, or document sharing semantics explicitly.

I think we need a policy that, by default, memory storage owns its buffers. Maybe there are some situations where we want memory storage to share mutable buffers with other objects, but that's probably not the default.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions