feat: URL pipeline core — URLPipeline, adapters, registry, zarr.open_url

changes/4192.feature.md

@@ -0,0 +1,37 @@
+Added core support for URL pipelines (https://github.com/jbms/url-pipeline):
+`|`-chained URLs that address zarr data through nested storage layers, e.g.
+`s3://bucket/data.zip|zip:|zarr3:`. A parsed `zarr.storage.URLPipeline` (built with
+`URLPipeline.from_url`) can be passed to any function that accepts a `StoreLike`, and
+the new `zarr.open_url(url, mode="r" | "r+")` opens the existing node a pipeline string
+addresses without ever creating or overwriting anything. Adapters for the
+sub-URLs after the first are provided by third-party packages (e.g. Icechunk) through
+the `zarr.url_adapters` entry-point group and the single-method
+`zarr.abc.url_pipeline.URLPipelineAdapter` interface, which receives a
+`zarr.storage.PipelineSegment` and `zarr.storage.PipelineContext` and returns a
+`zarr.storage.StorePath`; adapters are loaded lazily and individually, and
+`zarr.registry.register_url_adapter` registers one without an entry point. A builtin
+`zip:` adapter will follow separately.
+
+Behavior notes:
+
+- Plain strings are never interpreted as pipelines: `zarr.open("data|x.zarr")` still
+  addresses a local directory of that name, whatever adapters are installed. Only
+  `zarr.open_url` and `URLPipeline.from_url` read a string as a pipeline.
+- A trailing `zarr2:`/`zarr3:` segment selects the zarr format of the open or create
+  call, and its body is the path of the node to open. A `zarr_format` argument that
+  conflicts with it raises `ValueError`; functions whose `zarr_format` defaults to 3
+  (`zarr.create_array`, `Group.from_store`, `Group.open`, `Array.open`) need an explicit
+  `zarr_format=None` to defer to the pipeline. `Array.open` gains a `zarr_format`
+  parameter mirroring `Group.open`.
+- Inside a pipeline, `memory:` and `file:` roots follow the URL pipeline spec's spelling
+  rules (`memory:` ≡ `memory:/` ≡ `memory://`; `file:` must carry an absolute path with
+  at most a `localhost` authority). Percent-escapes are not decoded, matching
+  `LocalStore`. A pipeline's `memory:` root is zarr's `ManagedMemoryStore`, which does
+  not share data with the fsspec in-memory filesystem a plain `memory://name` string
+  resolves to.
+- Wrapper adapters can open a local *file* root only in mode `"r"` (via
+  `dataclasses.replace(context, mode="r").resolve_preceding()`); writing through a
+  container file is not yet supported.
+- A `zarr.url_adapters` entry point whose scheme is already registered is ignored with a
+  `ZarrUserWarning`, as are duplicate entry-point names; an entry point that fails to
+  import raises `URLPipelineError` naming it, and stays discoverable for a retry.

docs/api/zarr/abc/index.md

@@ -11,3 +11,4 @@ Abstract base classes for extending Zarr-Python.
 - **[zarr.abc.metadata](./metadata.md)** - Creating metadata classes compatible with the Zarr API
 - **[zarr.abc.numcodec](./numcodec.md)** - Protocols and classes for modeling codec interface used by numcodecs
 - **[zarr.abc.store](./store.md)** - ABC for implementing Zarr stores and managing getting and setting bytes in a store
+- **[zarr.abc.url_pipeline](./url_pipeline.md)** - ABC for implementing [URL pipeline](https://github.com/jbms/url-pipeline) adapters

docs/api/zarr/abc/url_pipeline.md

@@ -0,0 +1,5 @@
+---
+title: url_pipeline
+---
+
+::: zarr.abc.url_pipeline

docs/api/zarr/functions/open_url.md

@@ -0,0 +1,5 @@
+---
+title: zarr.open_url
+---
+
+::: zarr.open_url

docs/user-guide/extending.md

@@ -85,6 +85,70 @@ implementation.
 Custom stores can be created by implementing the [`zarr.abc.store.Store`][] interface.
 See [developing custom stores](storage.md#developing-custom-stores) for more information.
 
+## Custom URL pipeline adapters
+
+[URL pipelines](storage.md#user-guide-url-pipelines) are `|`-chained URLs such as
+`s3://bucket/data.zip|zip:|zarr3:`. Each sub-URL after the first names an *adapter*
+that reinterprets everything to its left. A package provides an adapter for a scheme by
+subclassing [`zarr.abc.url_pipeline.URLPipelineAdapter`][] and implementing its single
+classmethod, which resolves the adapter's segment into an open store and a residual path,
+returned as a [`StorePath`][zarr.storage.StorePath]:
+
+```python test="true" session="url-adapter"
+from zarr.abc.url_pipeline import URLPipelineAdapter
+from zarr.storage import PipelineContext, PipelineSegment, StorePath, WrapperStore
+
+
+class MyAdapter(URLPipelineAdapter):
+    @classmethod
+    async def open_pipeline_segment(
+        cls, segment: PipelineSegment, context: PipelineContext
+    ) -> StorePath:
+        # A *wrapper* adapter opens the resource to its left and wraps it, joining the
+        # preceding residual path with this segment's own path.
+        preceding = await context.resolve_preceding()
+        store = WrapperStore(preceding.store)
+        path = "/".join(part.strip("/") for part in (preceding.path, segment.body) if part)
+        return StorePath(store, path)
+```
+
+To resolve the preceding pipeline with a different mode or options (for example
+read-only, or without the adapter's own `storage_options` keys), resolve through a derived
+context: `await dataclasses.replace(context, mode="r").resolve_preceding()`.
+
+Wrapper adapters compose: in `memory://x|mypackage.myscheme:a|mypackage.myscheme:b` the
+inner adapter sees the outer one's residual path `a` and the pipeline resolves to `a/b`.
+
+```python test="true" session="url-adapter"
+import zarr
+from zarr.registry import register_url_adapter
+from zarr.storage import URLPipeline
+
+register_url_adapter("mypackage.myscheme", MyAdapter)
+url = "memory://x|mypackage.myscheme:a|mypackage.myscheme:b"
+created = zarr.create_group(URLPipeline.from_url(url))
+group = zarr.open_url(url)
+assert group.path == "a/b"
+```
+
+Register the class under the `zarr.url_adapters` entry-point group, using the URL
+scheme as the entry-point name. Nonstandard schemes should be vendor-prefixed
+(`vendor.scheme`), per the URL pipeline specification:
+
+```toml
+[project.entry-points."zarr.url_adapters"]
+"mypackage.myscheme" = "mypackage.zarr_adapter:MyAdapter"
+```
+
+Adapters are loaded lazily, only when a pipeline naming their scheme is resolved. An
+adapter may also be used as a *root* scheme (e.g. `mypackage.myscheme://org/repo`), in
+which case `context.preceding` is empty; such a URL reaches the adapter only through
+`zarr.open_url` or a `URLPipeline`, never as a plain string store specification. The returned store must honor
+`context.read_only`, and `open_pipeline_segment` runs on zarr's internal event loop, so
+it must not block or call zarr's synchronous API. See
+[`zarr.abc.url_pipeline`][] and [`PipelineContext`][zarr.storage.PipelineContext] for the
+full contract.
+
 ## Custom array buffers
 
 Zarr-python provides control over where and how arrays are stored in memory through

docs/user-guide/storage.md

@@ -100,9 +100,65 @@ print(group)
    group = zarr.open_group(UPath('s3://noaa-nwm-retro-v2-zarr-pds', anon=True), mode='r')
    ```
 
+- a [`URLPipeline`][zarr.storage.URLPipeline], the parsed form of a
+  [URL pipeline](#user-guide-url-pipelines) such as `s3://bucket/data.zip|zip:`, which
+  is resolved through registered adapters. Strings are never interpreted as pipelines.
+
 - a [`Store`][zarr.abc.store.Store] or [`StorePath`][zarr.storage.StorePath] -
   see explicit store creation below.
 
+## URL Pipelines {#user-guide-url-pipelines}
+
+Zarr supports [URL pipelines](https://github.com/jbms/url-pipeline): `|`-chained URLs
+that address zarr data through nested storage layers, read left to right. The first
+sub-URL locates a resource with a conventional URL; each subsequent sub-URL names an
+*adapter* that reinterprets everything to its left (e.g.
+`s3://bucket/data.zip|zip:|zarr3:`). Adapters are provided by packages through the
+`zarr.url_adapters` entry-point group — see
+[`zarr.abc.url_pipeline`][zarr.abc.url_pipeline] for the adapter interface. A builtin
+`zip:` adapter is under development and will expand this section.
+
+A parsed [`URLPipeline`][zarr.storage.URLPipeline] can be passed to any function that
+accepts a `StoreLike`, and [`zarr.open_url`][] opens the *existing* node a pipeline
+string addresses, read-only unless `mode="r+"` is given:
+
+```python exec="true" session="storage" source="above" result="ansi"
+from zarr.storage import URLPipeline
+
+pipeline = URLPipeline.from_url("memory://pipeline-demo|zarr2:")
+group = zarr.open_group(pipeline, mode="w")
+print(group.metadata.zarr_format)
+# the same thing, as a one-liner
+print(zarr.open_url("memory://pipeline-demo|zarr2:").metadata.zarr_format)
+```
+
+Only these two entry points read a string as a pipeline. Plain string store
+specifications keep their existing meaning everywhere else, so a local path that
+happens to contain `|` still names a directory, whatever adapters are installed.
+Unlike `zarr.open`, `zarr.open_url` never creates or overwrites: to create a node at a
+pipeline-addressed location, pass the `URLPipeline` to `zarr.create_array` or
+`zarr.create_group`.
+
+A trailing `zarr2:` or `zarr3:` segment is a *format segment*. It is not an adapter:
+zarr-python reads it from the pipeline itself and uses it as the `zarr_format` of the
+open or create call, and its body (`zarr3:path/to/node`) is the path of the node to
+open. A `zarr_format` argument that conflicts with the format segment raises
+`ValueError`; functions whose `zarr_format` defaults to 3 (such as `zarr.create_array`
+and `Group.open`) need an explicit `zarr_format=None` to defer to the pipeline.
+
+`storage_options` apply to the *root* sub-URL (e.g. fsspec options for `s3://...`);
+adapters may consume adapter-specific, namespaced keys. No percent-escape is decoded
+anywhere in a pipeline. Registered adapters cannot intercept zarr's native `file:` and
+`memory:` root schemes.
+
+Inside a pipeline, a `memory:` root is zarr's managed in-memory store (`memory:`,
+`memory:/` and `memory://` are equivalent, and `memory:name` selects a named store).
+When fsspec is installed, a plain `memory://name` string store specification is still
+routed to fsspec's in-memory filesystem, which is a different store. A `file:` root
+must carry an absolute path; percent-escapes are not decoded, matching the
+[local store](#local-store). The root always carries a scheme: a local path is spelled
+as a `file:` URL, never as a bare path.
+
 ## Explicit Store Creation
 
 In some cases, it may be helpful to create a store instance directly. Zarr-Python offers

mkdocs.yml

@@ -43,6 +43,7 @@ nav:
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.metadata</code>': api/zarr/abc/metadata.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.numcodec</code>': api/zarr/abc/numcodec.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.store</code>': api/zarr/abc/store.md
+        - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.url_pipeline</code>': api/zarr/abc/url_pipeline.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.api</code>':
         - api/zarr/api/index.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.api.asynchronous</code>': api/zarr/api/asynchronous.md
@@ -80,6 +81,7 @@ nav:
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.open_consolidated</code>': api/zarr/functions/open_consolidated.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.open_group</code>': api/zarr/functions/open_group.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.open_like</code>': api/zarr/functions/open_like.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.open_url</code>': api/zarr/functions/open_url.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.print_debug_info</code>': api/zarr/functions/print_debug_info.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.registry</code>': api/zarr/registry.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.save</code>': api/zarr/functions/save.md

src/zarr/__init__.py

@@ -27,6 +27,7 @@
     open_consolidated,
     open_group,
     open_like,
+    open_url,
     save,
     save_array,
     save_group,
@@ -175,6 +176,7 @@ def set_format(log_format: str) -> None:
     "open_consolidated",
     "open_group",
     "open_like",
+    "open_url",
     "print_debug_info",
     "save",
     "save_array",

src/zarr/abc/url_pipeline.py

@@ -0,0 +1,76 @@
+"""
+Abstract base class for URL pipeline adapters.
+
+A URL pipeline is a `|`-separated chain of sub-URLs, read outer-to-inner,
+as specified by https://github.com/jbms/url-pipeline. The first sub-URL (the
+*root*) locates a resource using a conventional URL, and each subsequent
+sub-URL names an *adapter* that reinterprets everything to its left:
+
+    s3://bucket/data.zip|zip:path/inside|zarr3:
+
+Third-party packages provide adapters by subclassing
+[`URLPipelineAdapter`][zarr.abc.url_pipeline.URLPipelineAdapter] and
+registering the class under the `zarr.url_adapters` entry-point group,
+using the URL scheme as the entry-point name. The values an adapter receives
+and returns, [`PipelineSegment`][zarr.storage.PipelineSegment],
+[`PipelineContext`][zarr.storage.PipelineContext] and
+[`StorePath`][zarr.storage.StorePath], live in `zarr.storage`.
+"""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from zarr.storage import PipelineContext, PipelineSegment, StorePath
+
+__all__ = ["URLPipelineAdapter"]
+
+
+class URLPipelineAdapter(ABC):
+    """
+    Handler for one URL pipeline scheme.
+
+    Subclasses implement a single classmethod,
+    [`open_pipeline_segment`][zarr.abc.url_pipeline.URLPipelineAdapter.open_pipeline_segment],
+    and are registered under the `zarr.url_adapters` entry-point group with
+    the URL scheme as the entry-point name:
+
+        [project.entry-points."zarr.url_adapters"]
+        mypackage.myscheme = "mypackage.zarr_adapter:MyAdapter"
+
+    Nonstandard schemes should be vendor-prefixed (`vendor.scheme`) per the
+    URL pipeline specification.
+
+    An adapter is used in two positions:
+
+    - as an *adapter segment*: `s3://bucket/repo|icechunk://tag.v1` — the
+      context carries the preceding sub-URLs;
+    - as a *root scheme*: `gh://org/repo` — `context.preceding` is empty.
+    """
+
+    @classmethod
+    @abstractmethod
+    async def open_pipeline_segment(
+        cls, segment: PipelineSegment, context: PipelineContext
+    ) -> StorePath:
+        """
+        Resolve `segment` (in the context of the pipeline to its left)
+        into a store and a residual path within that store.
+
+        The returned store must already be open and must honor
+        `context.read_only` (the resolver additionally enforces it by
+        downgrading — or rejecting — a writable store when the caller
+        required read-only). The path of the returned
+        [`StorePath`][zarr.storage.StorePath] is the location the pipeline
+        addresses inside the store; it is empty for the store root.
+
+        This coroutine runs on zarr's internal I/O event loop. It must not
+        block (do I/O through async APIs or a thread executor) and must not
+        call zarr's synchronous API (`zarr.open`, `Group.open`, or anything
+        else that uses `zarr.core.sync.sync`) — doing so raises
+        `SyncError`. To open the preceding pipeline, use
+        [`resolve_preceding`][zarr.storage.PipelineContext.resolve_preceding].
+        """
+        ...