Skip to content

Decode Snappy streams with copies further back than 1 MiB - #67

Open
mahach666 wants to merge 1 commit into
feature/issue-15-pinpoint-dataobject-countfrom
feature/issue-16-snappy-window
Open

mahach666 wants to merge 1 commit into
feature/issue-15-pinpoint-dataobject-countfrom
feature/issue-16-snappy-window

Conversation

@mahach666

Copy link
Copy Markdown
Contributor

Closes #16

Based on #66 (the branch feature/issue-15-pinpoint-dataobject-count); merge the earlier pull requests first.

What changed

  • The Snappy decoder no longer keeps a fixed 1 MiB window. The window is now the declared uncompressed length rounded up
    to a power of two, capped at 8 MiB per stream, so memory use stays bounded while streams with copies up to 8 MiB back
    decode. A stream whose copies reach back further than its window is still reported as unsupported instead of being
    decoded wrongly, with a message that names the window and the cap.
  • Small streams now rent small windows instead of a full megabyte.

Files changed

  • src/PclIO/Internal/SnappyReader.cs: adaptive window sized in the constructor; the window size is stored separately
    from the rented buffer (the pool may hand back a larger array); the preamble is parsed before anything is rented so its
    failure only disposes the source.
  • tests/PclIO.Tests/SnappyReaderTests.cs: the unsupported-offset test now uses an offset beyond the new cap; new tests for
    a 2 MiB-back copy and for correctness when the capped window wraps.
  • tests/PclIO.Tests/PinPointPackedTests.cs: new end-to-end test with a points channel that copies 2 MiB back.
  • README.md, CHANGELOG.md: the new bound (8 MiB) and the per-stream window cap.

Tests

  • dotnet build — 0 warnings, 0 errors.
  • dotnet test — 636 passed, 0 failed, 10 skipped (the tests that need private scan files) on .NET 8 and .NET 10.

Limits

  • A stream whose copies reach back more than 8 MiB still fails with NotSupportedException when its points are read,
    without an earlier warning at open time: detecting it would need a full pass over the compressed elements, which the
    open path deliberately avoids. This is now documented.
  • Up to five channel streams of one scan can each hold an 8 MiB window while a scan is read.

@mahach666
mahach666 force-pushed the feature/issue-15-pinpoint-dataobject-count branch from 19c19c6 to d4cd705 Compare November 17, 2026 15:01
@mahach666
mahach666 force-pushed the feature/issue-16-snappy-window branch from 02cc715 to 535d93f Compare November 17, 2026 15:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant