v3 structured dtype: per-field endianness still lost on metadata roundtrip (#4142 covers the write path only; breaks virtual/reference stores)

[emfdavid]

Follow-up to #4141 / #4142

First — thank you @d-v-b for the incredibly fast turnaround on #4142! It fixed the case I hit first (zarr-written big-endian structured arrays now round-trip correctly).

While validating it end-to-end I found that the underlying issue #4141 described — the serialization itself dropping per-field endianness — is still present. #4142 resolved the symptom on the write path by byte-swapping structured data to native on encode, but the metadata serialization still loses per-field byte order. That leaves one important case broken: stores whose chunk bytes are not written by zarr — i.e. virtual / reference stores (VirtualiZarr referencing big-endian FITS binary tables → Icechunk). There is no write to swap, so the reopened-native metadata misreads the big-endian bytes, silently.

The serialization still drops per-field endianness (pure zarr, no extras)

import numpy as np
from zarr.core.dtype import parse_data_type
from zarr.core.metadata.v3 import ArrayV3Metadata

be = np.dtype([("flux", ">f4"), ("ivar", ">f4")])
meta = ArrayV3Metadata(
    shape=(3,), data_type=parse_data_type(be, zarr_format=3),
    chunk_grid={"name": "regular", "configuration": {"chunk_shape": (3,)}},
    chunk_key_encoding={"name": "default"}, fill_value=np.zeros((), be)[()],
    codecs=[{"name": "bytes", "configuration": {"endian": "little"}}],
    attributes={}, dimension_names=None,
)
print(meta.data_type)                                          # Struct(... Float32(endianness='big'))
print(ArrayV3Metadata.from_dict(meta.to_dict()).data_type)     # Struct(... Float32(endianness='little'))  ← dropped

The serialized form has nowhere to record byte order — each field is a bare type name:

"data_type": {"name": "struct", "configuration": {"fields": [
    {"name": "flux", "data_type": "float32"}, {"name": "ivar", "data_type": "float32"}]}}

so from_dict rebuilds every field as native. parse_data_type itself preserves >f4; the loss is purely in to_dict → from_dict.

Why #4142 covers the write path but not virtual/reference reads

check	zarr 3.2.1	post-#4142 (main)
to_dict → from_dict preserves struct endianness	❌	❌ (unchanged)
reopened data_type label	little	little
create_array(dtype=">f4 struct") → reopen values correct	❌	✅

On main the reopened label is still little, but the values are correct because encode swapped the bytes to native on the way to disk — write-swap and native-label are symmetric. A virtual/reference store never encodes through zarr (it points at the original big-endian FITS bytes in place), so that swap never runs, and the read is wrong. The only recourse for a consumer today is to reinterpret each field with .view('>f4') at the numpy boundary.

Why this one is harder (and needs a backward-compatible representation)

In the v3 model, byte order for scalars lives in the BytesCodec (endian), not the data type. A structured array has a single BytesCodec for all fields, so it cannot express per-field byte order — and the struct's serialized field spec ({"name", "data_type"}) has no slot for it either. So per-field endianness is currently unrepresentable in the serialized form.

A fix likely needs the structured-dtype field representation to carry per-field byte order in a backward-compatible way — e.g. an optional endianness (or a richer per-field data_type object) that defaults to native when absent, so existing serialized structs keep reading as they do today. Since the v3 structured dtype is still flagged unstable/unspecified (the UnstableSpecificationWarning points at zarr-extensions), this may also warrant a small spec decision on where per-field byte order lives. Happy to help test whatever shape you land on.

Real-world impact

Reading archival big-endian FITS binary tables (astronomy) in place via VirtualiZarr → Icechunk, with no reshard. The bytes are referenced, never rewritten, so the write-side swap from #4142 doesn't apply and consumers get silently wrong values without the .view('>f4') workaround.

Versions

Reproduced on released zarr==3.2.1 and on main at the #4142 merge commit (bbe27d7). numpy 2.x, Python 3.12.

[d-v-b]

thanks for this report, it might be helpful here to establish that the zarr V3 format uses a different model of data types than zarr V2 used. In Zarr V2, like NumPy, the byte order of a data type was part of its definition. So a data type could be little- or big-endian. In Zarr V3, the data type itself contains no byte order information. In fact, Zarr V3 gives implementations complete freedom to use whatever in-memory data structure they want for decoded scalars of a given data type. That's because the decoded form is confined to a particular implementation. What the spec requires is that implementations agree on the encoded representation of data. This is why the bytes codec takes an endian parameter. It defines the encoded representation of scalars with byte-order semantics, but it doesn't define the decoded representation! That's an implementation concern. So we won't be adding byte order information to the basic struct metadata (that's a stored-representation-concern), but we can change what zarr python does.

In Zarr Python, our current practice is to decode zarr data into numpy arrays that use the platform endianness. We could expose this via configuration as a knob users can adjust, and it would allow decodes of big-endian data to be a bit more efficient because a copy could be avoided. We could also allow users to declare the numpy data type they want to read into when they access the array (probably better than a global config). But before doing that, it would be good to know why the byte order of the numpy arrays is important for your application.