Repository navigation
Conversation
ReadChunks walks the chunk chain of an LZ block payload without decompressing it: each chunk reports its offset, header length, body size, output size and compression flag, and the chain is verified against the block's declared size (in real files the chunk output sizes add up to the block's unpacked size; a chain that does not add up is a scanner false positive and throws InvalidDataException). The compressed runs themselves are not decoded yet.
The chunk-chain model was an artifact of the walk: what looked like internal chunk headers is the encoder writing its own sizes as the first literals of one token stream. ReadPayload now decodes the real mode 2 format: a token byte (high nibble = literal count, 15 extended by 255-chained bytes), the literals, a big-endian 16-bit distance and a length byte (0 means 256, otherwise the token's low nibble is the high byte of the length). Distance 0 emits zeros; a positive distance copies from that many bytes back in the global window, byte by byte, so overlapping copies replicate a repeating pattern. The stream produces exactly the block's declared unpacked size; the body tail is writer slack. A mode 2 token stream may reference the output of any preceding block, so ReadPayload decodes the file from the start into one global window (raw payloads, full zero-mask decodes, mode-2 outputs, in block order) and returns the requested block's output; the cost of the call grows with the size of the file. Blocks that do not decode (false entries of the scan) are skipped and rolled back out of the window; the cross-check against the object directories confirmed the resulting window matches the real stream. ReadChunks, PlaLzChunk and LzChunkWalker are removed. On the six controlled saves every mode 2 block (2432 total) decodes to its declared size; the sample test now asserts that instead of the old NotSupportedException.
…buffer blocks Three refinements established on the big sample corpus (85,429 mode 2 blocks of five files decode with zero failures; the six controlled saves still decode exactly): - A payload may start with a 9-byte stream header [u8][u32][u32]: present when the u32 at offset 5 equals the block's packed or unpacked size, the token stream starts after it. Without skipping it the parse "self-synchronizes" through the header and occasionally converges on the wrong reading - the old chunk-chain artifact came from exactly this. - A final match may omit the length byte: the body ends after the distance and the match length is the remainder of the output. - Some blocks declare an output buffer larger than the stream produces (sizes near whole mebibytes; the stream just ends with the body). The produced bytes fill the head of the buffer, the tail stays zero, and the global window holds the buffer space so directory positions keep addressing it. Detected by retrying the decode with the relaxed ending when the strict decode runs out of body. A stream that ends with the body is therefore a valid buffer block, not truncation; a back-reference beyond the decoded stream remains the one hard error.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7 — continues from the reverted #14 (chunk-chain model was an artifact, not the format).
What changed
The chunk-chain model was wrong. What looked like internal chunk headers in a mode 2 payload is the encoder writing its own sizes as the first literals of a single token stream; the "chunk boundaries" were artifacts of a self-synchronizing walk.
ReadChunks,PlaLzChunkandLzChunkWalkerare removed.The real mode 2 format is decoded (
LzPayloadDecoder, internal). The payload body after the block header is a chain of sequences producing exactly the declared unpacked size:Global window. A mode 2 stream may reference the output of any preceding block of the file, so
ReadPayloadfor a mode 2 block decodes the file from the start into one global window (raw payloads, full zero-mask decodes, mode-2 outputs, in block order) and returns the requested block's output. The cost of the call grows with the size of the file. Preceding blocks that do not decode (false entries of the scan) are skipped and rolled back out of the window.Validation
LzPayloadTests): literals, back-references across the raw/mode-2 boundary, overlapping copies, RLE zeros, the length-byte conventions (0 → 256, low nibble as high byte), trailing literals + slack, and the two rejections (back-reference beyond the window, truncated stream).NotSupportedExceptionexpectation).Notes for review
PlaDocument.ReadPayloadmode-2 branch and the newDecodeGlobalWindow/AppendToWindow/TryAppendToWindowinternals inPlaDocument.csare the core of the change.PlaDocument.ReadPayloadXML docs now describe all three modes.