fix(codecs): don't skip outer AA/BB codecs in FusedCodecPipeline partial IO

[d-v-b]

🤖 AI text below 🤖

Fixes F2 from the v3.3.0 evaluation — silent wrong data under FusedCodecPipeline.

The bug

FusedCodecPipeline.supports_partial_decode/encode passed require_no_aa_bb=False; BatchedCodecPipeline passes True. Partial IO hands the whole chunk to the array→bytes codec, so it is only sound when that codec is the entire pipeline. With an outer codec wrapping a sharding codec, the fused pipeline still took the partial path and the outer codec was silently skipped.

Two corrections to the finding as written

1. It is reachable from create_array. The evaluation says this "requires explicit codecs= parameter or metadata from another implementation; create_array never produces this layout internally." That isn't right — filters are applied outside the serializer, so the ordinary documented API produces it:

zarr.create_array(
    ..., filters=[TransposeCodec(order=(1, 0))],
    serializer=ShardingCodec(chunk_shape=(5, 5)), compressors=None,
)
# -> codecs: [TransposeCodec, ShardingCodec]   <- outer AA codec

2. It is a write bug, not just a read bug. Each pipeline is self-consistent — fused drops the transpose symmetrically on encode and decode — so a fused-only roundtrip looks perfectly fine. The corruption only surfaces when the data crosses pipelines:

write	read	result
batched	batched	ok
batched	fused	transposed
fused	batched	transposed
fused	fused	ok

So fused writes bytes that don't match their own metadata. Batched — and any other Zarr implementation — reads that array back transposed. The self-consistency is exactly what makes it silent.

The fix

Pass require_no_aa_bb=True, matching batched. Since the parameter then has a single value at every call site, it is removed rather than left as an attractive nuisance for the same divergence to reappear. The two # Same condition and dispatch as BatchedCodecPipeline comments at the dispatch sites become true statements.

Cost: none for anything create_array produces

create_array nests every AA/BB codec inside the shard, so the outer lists are empty and the predicate is unchanged. Measured on the fused pipeline after this change:

layout	partial decode/encode
shards=	still enabled
shards= + order="F"	still enabled
shards= + filters=	still enabled
shards= + gzip	still enabled
outer transpose + sharding serializer	now declines

zarr-developers#3885's perf story is untouched. Only the layout that was returning wrong data falls back to the general path — fast-and-silently-wrong traded for correct.

Tests

test_outer_array_array_codec_around_sharding in test_pipeline_parity.py, across all four write/read pipeline pairings. The two cross-pipeline cells fail on main and pass here; the two same-pipeline cells pass either way, which is the point.

Full suite: 6731 passed, 0 failed. mypy clean.

Note on origin

Both this and #224 trace to fe5030156 (zarr-developers#3885). The False looks like a refactoring artifact — the helper and FusedCodecPipeline were created in that same commit, and the parameter appears to have been introduced to let the new pipeline's behavior coexist with the old one behind a shared helper, rather than as a deliberate relaxation. Worth a look at whether anything else from that refactor made the same assumption.