Skip to content

fix(storage): make every ObjectStore answer a ranged read the same way - #50

Merged
herbie-bot merged 1 commit into
mainfrom
fix/storage-range-contract
Sep 25, 2026
Merged

herbie-bot merged 1 commit into
mainfrom
fix/storage-range-contract

Conversation

@herbie-bot

Copy link
Copy Markdown
Collaborator

Stacked on #49 — merge that one first, or merge this one and #49 becomes redundant.

ObjectStore promises that a range extending past the end returns the available bytes rather than failing, and that reading a missing key throws ObjectNotFoundException. FileObjectStore kept both promises. The other two did not, so the same call gave different answers depending on which implementation was configured.

The divergences

FileObjectStore S3ObjectStore InMemoryObjectStore
range starting at/past the end empty stream ✅ raw S3Exception (HTTP 416) ❌ empty stream ✅
zero-length read of a missing key ObjectNotFoundException ✅ empty stream ❌ ObjectNotFoundException ✅
negative offset / length IllegalArgumentException ✅ IllegalArgumentException ✅ silently accepted ❌
length = Long.MAX_VALUE reads to the end ✅ overflowed range header ❌ negative slice size ❌

All four now agree. Both stores clamp in long arithmetic, and the S3 store sends an open-ended bytes=offset- where the last byte would overflow.

ObjectStoreContractTest

The fix on its own would not stop this happening again — a shared suite would. ObjectStoreContractTest holds the interface's promises and runs against all three implementations, 22 cases each.

The S3 store runs against a real S3 server (adobe/s3mock), not a stub. What a server answers to an out-of-range read is exactly the thing a stub would have to assume, and that assumption is what was wrong in the first place. The client is built through S3Clients.withoutChunkedEncoding, the same helper the auto-configuration uses, so the test can never pass against a more lenient client than an application gets.

Each implementation also gets what only it promises: the file store's traversal guard (8 rejected key shapes), its scratch-then-move write and upload sweep; the S3 store's multipart boundary at exactly PART_SIZE_BYTES, and a ranged read reaching across a part boundary.

The tests were checked against the bug

Reverting each fix and re-running:

  • old S3ObjectStore → rangeStartingAtTheEnd, rangeStartingPastTheEnd, zeroLengthOfAMissingKey, readToTheEnd fail
  • old InMemoryObjectStore → negativeOffset, negativeLength, readToTheEnd fail

Verification

./mvnw -Pfull-build clean verify -Dmaven.javadoc.failOnWarnings=true — green across all twelve modules; 91 tests in the storage module.

Docs

docs/releases/upgrade-to-1.5.md gains the cross-backend guarantee under "What you get", and its maturity note is corrected — the implementations are covered now. docs/TODO.md drops the two resolved items and records two that surfaced while fixing: the untested abort paths, and S3ObjectStore still letting raw SDK exceptions escape where the package documents ObjectStoreException.

🤖 Generated with Claude Code

ObjectStore promises that a range extending past the end returns the
available bytes rather than failing, and that a read of a missing key
throws ObjectNotFoundException. FileObjectStore kept both promises; the
other two did not, so the same call gave different answers depending on
which implementation was configured.

- S3ObjectStore let the server's 416 out as a raw S3Exception when the
  first byte lay at or past the object's end, and returned an empty
  stream for a zero-length read of a key that does not exist instead of
  reporting the wrong key.
- InMemoryObjectStore accepted a negative offset or length, and computed
  its slice with offset + length, which overflows for a caller asking
  for everything from an offset with Long.MAX_VALUE.
- Both now clamp in long arithmetic; S3ObjectStore sends an open-ended
  range where the last byte would overflow.

Tests: ObjectStoreContractTest holds the interface's promises and runs
against all three implementations, which is the only thing that keeps
three backends honest about one interface. The S3 store runs against a
real S3 server rather than a stub — what a server answers to an
out-of-range read is exactly the thing a stub would have to assume, and
that assumption was the bug. Reverting either fix turns the suite red:
four cases for S3, three for the in-memory store.

Also covers what each implementation promises alone: the file store's
traversal guard, its scratch-then-move write, the upload sweep, and the
S3 store's multipart boundary at PART_SIZE_BYTES.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hendrikebbers
hendrikebbers force-pushed the fix/storage-range-contract branch from 71655fd to 92a5e04 Compare September 25, 2026 07:45
@herbie-bot
herbie-bot merged commit f0ccfdc into main Sep 25, 2026
1 check passed
@herbie-bot
herbie-bot deleted the fix/storage-range-contract branch September 25, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants