Skip to content

Decode mode 2 (LZ) payloads - #17

Open
mahach666 wants to merge 4 commits into
mainfrom
feat/lz-chunks
Open

mahach666 wants to merge 4 commits into
mainfrom
feat/lz-chunks

Conversation

@mahach666

@mahach666 mahach666 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #7 — continues from the reverted #14 (chunk-chain model was an artifact, not the format).

What changed

The chunk-chain model was wrong. What looked like internal chunk headers in a mode 2 payload is the encoder writing its own sizes as the first literals of a single token stream; the "chunk boundaries" were artifacts of a self-synchronizing walk. ReadChunks, PlaLzChunk and LzChunkWalker are removed.

The real mode 2 format is decoded (LzPayloadDecoder, internal). The payload body after the block header is a chain of sequences producing exactly the declared unpacked size:

  • token byte: high nibble = literal count (15 = extended by 255-chained bytes), literals follow;
  • then a 16-bit big-endian distance and a length byte (0 → 256; otherwise the token's low nibble is the high byte of the match length, up to 4095);
  • distance 0 emits the match length as zero bytes (RLE); a positive distance copies from that many bytes back in the window byte by byte, so overlapping copies replicate a repeating pattern;
  • the body tail after the output is complete is writer slack.

Global window. A mode 2 stream may reference the output of any preceding block of the file, so ReadPayload for a mode 2 block decodes the file from the start into one global window (raw payloads, full zero-mask decodes, mode-2 outputs, in block order) and returns the requested block's output. The cost of the call grows with the size of the file. Preceding blocks that do not decode (false entries of the scan) are skipped and rolled back out of the window.

Validation

  • The format was established on controlled saves (see repo issue Decode mode 2 (LZ) payloads #7 research notes) and cross-checked against the object directories: the decoded global stream's directory records abut with no overlaps, confirming the window matches the real stream.
  • End-to-end run of the shipped code on the six controlled saves: every mode 2 block decodes exactly to its declared size — 2432 of 2432 blocks (m2-a: 401, m2-a2/b/c: 405 each, m2-d/e: 408 each).
  • Synthetic tests (LzPayloadTests): literals, back-references across the raw/mode-2 boundary, overlapping copies, RLE zeros, the length-byte conventions (0 → 256, low nibble as high byte), trailing literals + slack, and the two rejections (back-reference beyond the window, truncated stream).
  • The sample-gated test now asserts every mode 2 block decodes and only false entries are skipped (replaces the old NotSupportedException expectation).

Notes for review

  • PlaDocument.ReadPayload mode-2 branch and the new DecodeGlobalWindow/AppendToWindow/TryAppendToWindow internals in PlaDocument.cs are the core of the change.
  • Known remainder (not in this PR): one ~22 MB RLE-heavy block kind 0x00E4 on a large sample diverges mid-decode under the current literal-extension rule; every other block of that file decodes exactly. Documented in the format spec, tracked for follow-up.
  • docs/public-api.md and CHANGELOG updated; PlaDocument.ReadPayload XML docs now describe all three modes.

ReadChunks walks the chunk chain of an LZ block payload without decompressing it: each chunk reports its offset, header length, body size, output size and compression flag, and the chain is verified against the block's declared size (in real files the chunk output sizes add up to the block's unpacked size; a chain that does not add up is a scanner false positive and throws InvalidDataException). The compressed runs themselves are not decoded yet.
The chunk-chain model was an artifact of the walk: what looked like internal chunk headers is the encoder writing its own sizes as the first literals of one token stream. ReadPayload now decodes the real mode 2 format: a token byte (high nibble = literal count, 15 extended by 255-chained bytes), the literals, a big-endian 16-bit distance and a length byte (0 means 256, otherwise the token's low nibble is the high byte of the length). Distance 0 emits zeros; a positive distance copies from that many bytes back in the global window, byte by byte, so overlapping copies replicate a repeating pattern. The stream produces exactly the block's declared unpacked size; the body tail is writer slack.

A mode 2 token stream may reference the output of any preceding block, so ReadPayload decodes the file from the start into one global window (raw payloads, full zero-mask decodes, mode-2 outputs, in block order) and returns the requested block's output; the cost of the call grows with the size of the file. Blocks that do not decode (false entries of the scan) are skipped and rolled back out of the window; the cross-check against the object directories confirmed the resulting window matches the real stream.

ReadChunks, PlaLzChunk and LzChunkWalker are removed. On the six controlled saves every mode 2 block (2432 total) decodes to its declared size; the sample test now asserts that instead of the old NotSupportedException.
@mahach666 mahach666 changed the title Expose the mode 2 chunk chain Decode mode 2 (LZ) payloads Nov 16, 2026
…buffer blocks

Three refinements established on the big sample corpus (85,429 mode 2 blocks of five files decode with zero failures; the six controlled saves still decode exactly):

- A payload may start with a 9-byte stream header [u8][u32][u32]: present when the u32 at offset 5 equals the block's packed or unpacked size, the token stream starts after it. Without skipping it the parse "self-synchronizes" through the header and occasionally converges on the wrong reading - the old chunk-chain artifact came from exactly this.
- A final match may omit the length byte: the body ends after the distance and the match length is the remainder of the output.
- Some blocks declare an output buffer larger than the stream produces (sizes near whole mebibytes; the stream just ends with the body). The produced bytes fill the head of the buffer, the tail stays zero, and the global window holds the buffer space so directory positions keep addressing it. Detected by retrying the decode with the relaxed ending when the strict decode runs out of body.

A stream that ends with the body is therefore a valid buffer block, not truncation; a back-reference beyond the decoded stream remains the one hard error.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decode mode 2 (LZ) payloads

1 participant