store: support memoryview values additionally to bytes

README.rst

@@ -7,7 +7,8 @@ Overview
 --------
 
 Keys are simple strings like `config/main` or `data/0123456789abcdef` `[str]`
-(config and data are namespaces here). Values are binary objects `[bytes]`.
+(config and data are namespaces here). Values are binary objects `[bytes]` -
+when storing, a `memoryview` is accepted as well, so callers can avoid copying.
 
 The `Store` class is the high-level API, so you can comfortably work with the
 kv store without caring for low-level details.

docs/backends.rst

@@ -4,6 +4,10 @@ Backends
 The backend API is rather simple; one only needs to provide some very
 basic operations.
 
+``store(name, value)`` accepts a ``bytes`` or a ``memoryview`` value, see
+:ref:`store-values`. Backends that can not give a ``memoryview`` to the library
+they use internally create a ``bytes`` object (one copy) - this is noted below.
+
 Existing backends are listed below; more might come in the future.
 
 See also :doc:`store_caching` for optional Store-level caching with a secondary backend.
@@ -115,6 +119,8 @@ Use storage on an SFTP server:
   - Users must know the full absolute path of the space they are permitted to use.
 - Namespaces: directories
 - Values: in key-named files
+- store: a ``memoryview`` value is copied into a ``bytes`` object first, because
+  paramiko does not accept a ``memoryview``.
 - hash: runs the hexdigest computation server-side (if server supports check-file).
   "blake3" is not part of the check-file extension, so it is always computed client-side.
 
@@ -155,6 +161,8 @@ Use storage on an S3-compliant cloud service:
 
 - Namespaces: directories
 - Values: in key-named files
+- store: a ``memoryview`` value is copied into a ``bytes`` object first, because
+  boto3 does not accept a ``memoryview``.
 
 
 REST (http/https)

docs/changes.rst

@@ -11,6 +11,11 @@ New features:
   For backends that hash server-side, it needs to be installed on the server.
   Note: sftp always computes blake3 client-side, because the SFTP "check-file"
   extension does not support blake3.
+- store: accept a memoryview value additionally to bytes, #200.
+  This enables callers to avoid copying, e.g. by giving a slice of a bigger
+  buffer they already have. posixfs, rest and rclone store it without copying,
+  sftp and s3 internally create a bytes object first, because paramiko / boto3
+  do not accept a memoryview.
 
 
 Version 0.5.5 (2026-07-10)

docs/store.rst

@@ -112,10 +112,37 @@ Please note:
    can't add another namespace later, because that would create
    nested namespaces.
 
+.. _store-values:
+
 Values
 ------
 
-Values can be any arbitrary binary data (bytes).
+Values can be any arbitrary binary data.
+
+``store(name, value)`` accepts the value either as ``bytes`` or as a
+``memoryview`` of bytes, so a caller that already has the data inside a bigger
+buffer can avoid copying it:
+
+.. code-block:: python
+
+    buffer = bytearray(...)  # e.g. a big buffer, filled with multiple objects
+    store.store("data/0123456789abcdef", memoryview(buffer)[offset : offset + size])
+
+Notes about ``memoryview`` values:
+
+- The view must be C-contiguous (a plain slice of a buffer is). Otherwise, a
+  ``ValueError`` is raised - the value must be storable as one consecutive
+  sequence of bytes.
+- If the view has an itemsize > 1 (e.g. a view of an ``array.array("I", ...)``),
+  the bytes it consists of are stored, so the stored size is
+  ``len(value) * value.itemsize``.
+- The value is only used while ``store`` runs. Afterwards, the caller may reuse
+  or release the underlying buffer.
+- Not all backends can hand a ``memoryview`` to the library they use: ``posixfs``,
+  ``rest`` and ``rclone`` store it without copying, ``sftp`` and ``s3`` internally
+  create a ``bytes`` object (one copy) first.
+
+Loading always gives ``bytes``.
 
 Automatic Nesting
 -----------------

src/borgstore/backends/_base.py

@@ -14,6 +14,35 @@
 # atime is the last read access UNIX timestamp [s] or 0 if not implemented
 ItemInfo = namedtuple("ItemInfo", "name exists size directory atime", defaults=(0,))
 
+# type of a value given to store: a memoryview is accepted additionally to bytes,
+# so callers can avoid copying (e.g. give a slice of a big buffer they already have).
+StoreValue = bytes | memoryview
+
+
+def validate_value(value: StoreValue) -> StoreValue:
+    """Validate/normalize a value given to store.
+
+    bytes (and other bytes-like objects) are returned unchanged.
+    A memoryview is cast to a 1-dimensional view of bytes, so len(value) always
+    gives the number of bytes (even if the caller had a view with a bigger itemsize).
+    """
+    if isinstance(value, memoryview):
+        try:
+            return value.cast("B")
+        except TypeError:
+            # cast only works for C-contiguous views, but the backends need to
+            # write the value out as one consecutive sequence of bytes.
+            raise ValueError("value must be a C-contiguous memoryview") from None
+    return value
+
+
+def to_bytes(value: StoreValue) -> bytes:
+    """Get a bytes object for code that can not deal with a memoryview.
+
+    Note: this copies the data, except if it already is a bytes object.
+    """
+    return value if isinstance(value, bytes) else bytes(value)
+
 
 def validate_name(name):
     """Validate a backend key/name."""
@@ -108,8 +137,13 @@ def load(self, name: str, *, size=None, offset=0) -> bytes:
         """
 
     @abstractmethod
-    def store(self, name: str, value: bytes) -> None:
-        """store <value> into <name>"""
+    def store(self, name: str, value: StoreValue) -> None:
+        """store <value> into <name>
+
+        <value> is either bytes or a memoryview of bytes (see StoreValue).
+        Backends must not keep a reference to a memoryview value after returning,
+        because the caller may reuse or release the underlying buffer.
+        """
 
     @abstractmethod
     def delete(self, name: str) -> None:

src/borgstore/backends/posixfs.py

@@ -21,7 +21,7 @@
 except ImportError:
     fcntl = None  # not available on Windows
 
-from ._base import BackendBase, ItemInfo, validate_name
+from ._base import BackendBase, ItemInfo, validate_name, validate_value
 from .errors import BackendError, BackendAlreadyExists, BackendDoesNotExist, BackendMustNotBeOpen, BackendMustBeOpen
 from .errors import ObjectNotFound, PermissionDenied, QuotaExceeded
 from ..constants import TMP_SUFFIX, QUOTA_STORE_NAME, QUOTA_PERSIST_DELTA, QUOTA_PERSIST_INTERVAL
@@ -232,6 +232,7 @@ def _write_to_tempfile(self, path, value, suffix=TMP_SUFFIX, do_fsync=False):
     def store(self, name, value):
         if not self.opened:
             raise BackendMustBeOpen()
+        value = validate_value(value)  # writing a memoryview is supported, no copy needed
         path = self._validate_join(name)
         overwrite = path.exists()
         self._check_permission(name, "W" if overwrite else "wW")

src/borgstore/backends/rclone.py

@@ -16,7 +16,7 @@
 except ImportError:
     requests = None
 
-from ._base import BackendBase, ItemInfo, validate_name
+from ._base import BackendBase, ItemInfo, validate_name, validate_value, StoreValue
 from ._utils import make_range_header, ignore_sigint
 from .errors import (
     BackendError,
@@ -288,9 +288,11 @@ def load(self, name: str, *, size=None, offset=0) -> bytes:
             content = content[:size]
         return content
 
-    def store(self, name: str, value: bytes) -> None:
+    def store(self, name: str, value: StoreValue) -> None:
         """Store <value> into <name>."""
         validate_name(name)
+        # requests puts a memoryview into the multipart body as it is, no copy needed.
+        value = validate_value(value)
         files = {"file": (os.path.basename(name), value, "application/octet-stream")}
         params = {"fs": self.fs, "remote": os.path.dirname(name)}
         self._rpc("operations/uploadfile", None, tries=self.TRIES, params=params, files=files)

src/borgstore/backends/rest.py

@@ -29,7 +29,7 @@
 except ImportError:
     pass
 
-from ._base import BackendBase, ItemInfo, validate_name
+from ._base import BackendBase, ItemInfo, validate_name, validate_value, StoreValue
 from ._utils import make_range_header, ignore_sigint
 from .errors import (
     ObjectNotFound,
@@ -557,9 +557,11 @@ def load(self, name: str, *, size=None, offset=0) -> bytes:
         return content
 
     @with_reconnect
-    def store(self, name: str, value: bytes) -> None:
+    def store(self, name: str, value: StoreValue) -> None:
         self._assert_open()
         validate_name(name)
+        # requests sends a memoryview body as it is (with a correct Content-Length), no copy needed.
+        value = validate_value(value)
         algorithm = "sha256"
         headers = {f"X-Content-hash-{algorithm}": hashlib.new(algorithm, value).hexdigest()}
         response = self._request("post", self._url(name), data=value, headers=headers)

src/borgstore/backends/s3.py

@@ -12,7 +12,7 @@
 from typing import Optional
 import urllib.parse
 
-from ._base import BackendBase, ItemInfo, validate_name
+from ._base import BackendBase, ItemInfo, validate_name, validate_value, to_bytes
 from ._utils import make_range_header
 from .errors import BackendError, BackendMustBeOpen, BackendMustNotBeOpen, BackendDoesNotExist, BackendAlreadyExists
 from .errors import ObjectNotFound
@@ -179,6 +179,8 @@ def store(self, name, value):
         if not self.opened:
             raise BackendMustBeOpen()
         validate_name(name)
+        # botocore rejects a memoryview Body (parameter validation), so we need a copy.
+        value = to_bytes(validate_value(value))
         key = self.base_path + name
         self.s3.put_object(Bucket=self.bucket, Key=key, Body=value)
 

src/borgstore/backends/sftp.py

@@ -19,7 +19,7 @@
 except ImportError:
     paramiko = None
 
-from ._base import BackendBase, ItemInfo, validate_name
+from ._base import BackendBase, ItemInfo, validate_name, validate_value, to_bytes
 from .errors import BackendError, BackendMustBeOpen, BackendMustNotBeOpen, BackendDoesNotExist, BackendAlreadyExists
 from .errors import ObjectNotFound
 from ..constants import TMP_SUFFIX
@@ -437,6 +437,9 @@ def _write_to_tmpfile():
         if not self.opened:
             raise BackendMustBeOpen()
         validate_name(name)
+        # paramiko's file.write only deals with str/bytes, a memoryview would blow up
+        # deeply inside paramiko ("Unknown type for encoding"), so we need a copy here.
+        value = to_bytes(validate_value(value))
         tmp_dir = Path(name).parent
         # write to a differently named temp file in same directory first,
         # so the store never sees partially written data.

src/borgstore/store.py

@@ -19,7 +19,7 @@
 from typing import Iterator, NamedTuple, Optional
 
 from .utils.nesting import nest, unnest
-from .backends._base import ItemInfo, BackendBase
+from .backends._base import ItemInfo, BackendBase, StoreValue, validate_value
 from .backends.errors import ObjectNotFound, NoBackendGiven, BackendURLInvalid, ReadRangeError  # noqa
 from .backends.posixfs import get_file_backend
 from .backends.rclone import get_rclone_backend
@@ -433,7 +433,7 @@ def load(self, name: str, *, size=None, offset=0, deleted=False) -> bytes:
             self._stats_update_volume("load", len(result))
             return result
 
-    def _cache_store(self, nested_name: str, value: bytes) -> None:
+    def _cache_store(self, nested_name: str, value: StoreValue) -> None:
         if self.cache_backend is None or self._cache_disabled:
             return
         self._stats["cache_store_calls"] += 1
@@ -444,10 +444,19 @@ def _cache_store(self, nested_name: str, value: bytes) -> None:
             logger.warning(f"borgstore: cache store failed for {nested_name!r}: {err!r}")
             self._stats["cache_errors"] += 1
 
-    def store(self, name: str, value: bytes) -> None:
+    def store(self, name: str, value: StoreValue) -> None:
+        """
+        store <value> into item <name>.
+
+        <value> is either bytes or a memoryview of bytes - a memoryview enables callers
+        to avoid copying (e.g. by giving a slice of a bigger buffer they already have).
+        The value is only used while this method runs, so the caller may reuse or
+        release the underlying buffer afterwards.
+        """
         # note: using .find here will:
         # - overwrite an existing item (level stays same)
         # - write to the last level if no existing item is found.
+        value = validate_value(value)  # normalize, so len(value) is the number of bytes
         with self._stats_updater("store", f"store({name!r})"):
             nested_name = self.find(name)
             self._backend_call(lambda: self.backend.store(nested_name, value), key="store", volume=len(value))

tests/test_backends.py

@@ -2,6 +2,7 @@
 Generic tests for the backend implementations.
 """
 
+import array
 import hashlib
 import os
 import sys
@@ -37,7 +38,7 @@
 from borgstore.backends.posixfs import PosixFS, get_file_backend
 from borgstore.backends.sftp import Sftp, get_sftp_backend
 from borgstore.backends.rclone import get_rclone_backend
-from borgstore.backends.s3 import S3, get_s3_backend
+from borgstore.backends.s3 import S3, boto3, get_s3_backend
 from borgstore.backends.rest import REST, get_rest_backend
 from borgstore.constants import ROOTNS
 
@@ -564,6 +565,79 @@ def test_scalability_size(tested_backends, exp, request):
         assert backend.load(key) == value
 
 
+def test_store_memoryview(tested_backends, request):
+    with get_backend_from_fixture(tested_backends, request) as backend:
+        # a memoryview of a slice of a bigger buffer (that's what callers use to avoid copies)
+        buffer = bytearray(b"0123456789" * 10)
+        value = memoryview(buffer)[10:30]
+        backend.store("key", value)
+        assert backend.load("key") == bytes(value)
+
+        # a memoryview with itemsize > 1: we store the bytes it consists of
+        array_value = array.array("I", range(16))  # itemsize 4
+        backend.store("key_array", memoryview(array_value))
+        assert backend.load("key_array") == array_value.tobytes()
+
+        # empty memoryview
+        backend.store("key_empty", memoryview(b""))
+        assert backend.load("key_empty") == b""
+
+        # a non-contiguous memoryview can not be stored (we would have to copy it first)
+        with pytest.raises(ValueError, match="C-contiguous"):
+            backend.store("key_invalid", memoryview(buffer)[::2])
+
+
+def test_sftp_store_memoryview_gives_bytes_to_paramiko():
+    # paramiko can not deal with a memoryview, so the sftp backend must convert it.
+    written = []
+
+    class FakeFile:
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *args):
+            return False
+
+        def set_pipelined(self, pipelined):
+            pass
+
+        def write(self, data):
+            written.append(data)
+
+    class FakeClient:
+        def open(self, name, mode):
+            return FakeFile()
+
+        def posix_rename(self, curr_name, new_name):
+            pass
+
+    backend = Sftp(hostname="localhost", path="/some/path")
+    backend.opened = True
+    backend.client = FakeClient()
+    buffer = bytearray(b"0123456789" * 10)
+    backend.store("key", memoryview(buffer)[10:30])
+    assert written == [b"0123456789" * 2]
+    assert type(written[0]) is bytes
+
+
+@pytest.mark.skipif(boto3 is None, reason="boto3 is not installed")
+def test_s3_store_memoryview_gives_bytes_to_boto3():
+    # boto3 rejects a memoryview Body (parameter validation), so the s3 backend must convert it.
+    put_objects = []
+
+    class FakeS3:
+        def put_object(self, **kwargs):
+            put_objects.append(kwargs)
+
+    backend = S3(bucket="bucket", path="/path", is_b2=False)
+    backend.opened = True
+    backend.s3 = FakeS3()
+    buffer = bytearray(b"0123456789" * 10)
+    backend.store("key", memoryview(buffer)[10:30])
+    assert put_objects[0]["Body"] == b"0123456789" * 2
+    assert type(put_objects[0]["Body"]) is bytes
+
+
 def test_load_partial(tested_backends, request):
     with get_backend_from_fixture(tested_backends, request) as backend:
         backend.store("key", b"0123456789")