Skip to content

store: support memoryview values additionally to bytes - #203

Merged
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:memoryview
Jul 28, 2026
Merged

store: support memoryview values additionally to bytes#203
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:memoryview

Conversation

@ThomasWaldmann

Copy link
Copy Markdown
Member

Fixes #200.

Store.store(name, value) and backend.store(name, value) now accept a
memoryview of bytes additionally to bytes, so callers can avoid copying:

buffer = bytearray(...)  # e.g. a big buffer, filled with multiple objects
store.store("data/0123456789abcdef", memoryview(buffer)[offset : offset + size])

Implementation

backends/_base.py gets the StoreValue = bytes | memoryview type alias and two helpers:

  • validate_value(value): casts a memoryview to a 1-dimensional view of bytes, so
    len(value) always is the number of bytes (relevant for quota tracking and the
    volume statistics, e.g. if the caller has a view with itemsize > 1). A
    non-contiguous memoryview is rejected with a ValueError, because the value must
    be stored as one consecutive sequence of bytes.
  • to_bytes(value): for the backends that can not deal with a memoryview.

Per backend (checked against what the used libraries actually do):

backend memoryview why
posixfs no copy the tempfile write accepts it
rest no copy requests sends it as request body, with a correct Content-Length
rclone no copy requests puts it into the multipart body
sftp copied to bytes paramiko's default file is unbuffered, so write() ends up in util.asbytes(), which raises "Unknown type for encoding" for a memoryview
s3 copied to bytes botocore rejects it: ParamValidationError: Invalid type for parameter Body

Store.store normalizes the value once, so the volume statistics and the
write-through/mirror cache path are correct, too.

Tests

  • test_store_memoryview for all configured backends: a slice of a bigger buffer,
    a view with itemsize 4, an empty view, and the rejection of a non-contiguous view.
  • Store level: also checks the item size, the stats volume and that modifying the
    caller's buffer afterwards does not change the stored item.
  • Also covered: the write-through cache path, posixfs quota accounting and a real
    HTTP round-trip via the REST server (which also validates the
    X-Content-hash-sha256 header computed from the memoryview).
  • sftp and s3 can not be tested here without a server, so there are tests with a
    fake client asserting that paramiko / boto3 get a bytes object.

Docs: "Values" section in docs/store.rst (incl. which backends copy), notes for
the sftp/s3 backends, README and CHANGES.

…kup#200

This enables callers to avoid copying, e.g. by giving a slice of a bigger
buffer they already have.

A memoryview is normalized (cast to a 1-dimensional view of bytes), so
len(value) always is the number of bytes - that matters for quota tracking
and the volume statistics. A non-contiguous memoryview is rejected, because
the value must be stored as one consecutive sequence of bytes.

posixfs, rest and rclone hand the memoryview to the code doing the write /
the request without copying. sftp and s3 create a bytes object first,
because paramiko (only accepts str/bytes) and boto3 (parameter validation
rejects a memoryview Body) can not deal with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ThomasWaldmann
ThomasWaldmann merged commit c8562ac into borgbackup:master Jul 28, 2026
8 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the memoryview branch July 28, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

store methods: support memoryview additionally to bytes

1 participant