store: support memoryview values additionally to bytes - #203
Merged
Conversation
…kup#200 This enables callers to avoid copying, e.g. by giving a slice of a bigger buffer they already have. A memoryview is normalized (cast to a 1-dimensional view of bytes), so len(value) always is the number of bytes - that matters for quota tracking and the volume statistics. A non-contiguous memoryview is rejected, because the value must be stored as one consecutive sequence of bytes. posixfs, rest and rclone hand the memoryview to the code doing the write / the request without copying. sftp and s3 create a bytes object first, because paramiko (only accepts str/bytes) and boto3 (parameter validation rejects a memoryview Body) can not deal with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ThomasWaldmann
force-pushed
the
memoryview
branch
from
July 27, 2026 23:29
304f95c to
de1807e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #200.
Store.store(name, value)andbackend.store(name, value)now accept amemoryviewof bytes additionally tobytes, so callers can avoid copying:Implementation
backends/_base.pygets theStoreValue = bytes | memoryviewtype alias and two helpers:validate_value(value): casts a memoryview to a 1-dimensional view of bytes, solen(value)always is the number of bytes (relevant for quota tracking and thevolume statistics, e.g. if the caller has a view with itemsize > 1). A
non-contiguous memoryview is rejected with a
ValueError, because the value mustbe stored as one consecutive sequence of bytes.
to_bytes(value): for the backends that can not deal with a memoryview.Per backend (checked against what the used libraries actually do):
write()ends up inutil.asbytes(), which raises "Unknown type for encoding" for a memoryviewParamValidationError: Invalid type for parameter BodyStore.storenormalizes the value once, so the volume statistics and thewrite-through/mirror cache path are correct, too.
Tests
test_store_memoryviewfor all configured backends: a slice of a bigger buffer,a view with itemsize 4, an empty view, and the rejection of a non-contiguous view.
caller's buffer afterwards does not change the stored item.
HTTP round-trip via the REST server (which also validates the
X-Content-hash-sha256header computed from the memoryview).fake client asserting that paramiko / boto3 get a
bytesobject.Docs: "Values" section in
docs/store.rst(incl. which backends copy), notes forthe sftp/s3 backends, README and CHANGES.