Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,25 @@ All notable changes are documented here. The format follows [Keep a Changelog](h
and the project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html); versions before 1.0 may change the
API between minor releases.

## [Unreleased]

### Added

- `PlaDocument.ReadPayload` decodes mode 2 (LZ) payloads. A payload may start with a 9-byte stream header holding the
packed and/or unpacked size. The token stream produces the block's declared unpacked size and may reference the
decompressed output of any preceding block of the file, so `ReadPayload` for a mode 2 block decodes the file from the
start into one global window; the cost of the call grows with the size of the file. Preceding blocks that do not
decode (false entries of the scan) are skipped and leave the window unchanged. A final match may omit the length byte
(the match length is the remainder of the output). Some blocks declare an output buffer larger than the stream
produces: the stream simply ends with the body, the produced bytes fill the head of the buffer and the tail stays
zero — the global window holds the buffer space, so directory positions keep addressing it. A back-reference beyond
the decoded stream throws `InvalidDataException`.

### Changed

- `PlaDocument.ReadPayload` no longer throws `NotSupportedException` for mode 2 blocks: all three storage modes
(raw, zero mask, LZ) decode now.

## [0.1.0] - 2026-11-12

### Added
Expand Down
4 changes: 2 additions & 2 deletions docs/public-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ The `PlaIO` package is the supported surface listed here; every public type is i
| `PlaDocument.Length` | Length of the file in bytes. |
| `PlaDocument.Header` | The `FdbHeader`. |
| `PlaDocument.Blocks` | The block index in file order, found by scanning from offset 32. No on-disk block map is known, so completeness is not claimed. The scan can take a byte run for a block header, so the index may contain false entries. In real files a small share of mode 1 blocks does not decode to its declared size, and `ReadPayload` throws `InvalidDataException` for them. |
| `PlaDocument.ReadPayload(PlaBlock block)` | The decoded payload of a block of this document. Decodes mode 0 and mode 1. Throws `NotSupportedException` for mode 2 (LZ), `InvalidDataException` when a mode 1 payload does not decode to the declared size or the file is cut short, and `ArgumentException` when the block is not from this document. |
| `PlaDocument.ReadPayload(PlaBlock block)` | The decoded payload of a block of this document. Decodes all three storage modes: raw (mode 0), zero masks (mode 1) and the LZ token stream (mode 2). A mode 2 payload may start with a 9-byte stream header (packed and/or unpacked size). The token stream produces the block's declared unpacked size — some blocks declare a buffer larger than the stream produces, the tail of the buffer stays zero — and may reference the output of any preceding block, so the payload is produced by decoding every preceding block of the file in order into one global window; the cost of the call grows with the size of the file. Preceding blocks that do not decode (false entries of the scan) are skipped and leave the window unchanged. Throws `InvalidDataException` when a back-reference reaches beyond the decoded stream or the file is cut short, and `ArgumentException` when the block is not from this document. |
| `PlaDocument.FindDirectories()` | Searches every raw (mode 0) block large enough for 64 records of 30 bytes for an object directory, at most one per block. Recomputed on each call; mode 1 payloads are not searched. |
| `PlaDocument.Dispose()` | Releases the stream unless the document was opened with `leaveOpen`. |
| `FdbHeader` | `Size` (64), `LayoutVersion` (u16 at offset 8), `FormatVersion` (u16 at offset 10), `Field12` (u32 at offset 12; meaning not documented), `Raw` (the first min(64, file length) bytes; bytes 32..63 overlap the first block). |
| `PlaBlockMode` | `Raw` (0), `ZeroMask` (1, flag 0x100), `Lz` (2, flag 0x200; recognised, not decoded). |
| `PlaBlockMode` | `Raw` (0), `ZeroMask` (1, flag 0x100), `Lz` (2, flag 0x200). |
| `PlaBlock` | One indexed block: `Offset`, `Tag` (`BL` or `DY`), `Id` and `Kind` (raw), `Version` (0 or 1), `Flags` (raw), `SizeClass` (`Flags & 0xF`), `Mode`, `Guid` (present when flag 0x20 is set), `PackedSize`, `UnpackedSize`, `Extra` (present when flag 0x400 is set; raw), `PayloadOffset`, `HeaderLength`, `End`. |
| `ObjectDirectory` | A run of directory records inside a raw block: `Block`, `RecordOffset` (payload offset of the first record), `Phase` (`RecordOffset % 30`), `Records`, `AbuttingPairs`, `OverlappingPairs`, `DuplicatePositions`. The run is an estimate and can be shorter than the real directory: it stops after more than 8 consecutive records without an abutting neighbour. |
| `ObjectDirectoryRecord` | The 30-byte record `[u16 class][u64 pos][u32 count][GUID16]` as `ClassNumber`, `Position`, `Count`, `ObjectId`. The units of position and count are not documented, and the class number is not mapped to a class name. |
Expand Down
147 changes: 147 additions & 0 deletions src/PlaIO/Internal/LzPayloadDecoder.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
namespace PlaIO.Internal;

/// <summary>Decodes the mode 2 (LZ) token stream of one block payload and appends every produced byte to the window —
/// the global decompressed stream of the whole file. Back-references reach anywhere inside that stream, including the
/// output of preceding blocks. Returns the number of body bytes consumed.</summary>
/// <remarks>
/// <para>The payload may start with a 9-byte stream header <c>[u8][u32 packed][u32 unpacked]</c>: it is present when the
/// u32 at offset 5 equals the block's declared unpacked size, and the token stream starts after it.</para>
/// <para>The stream is a chain of sequences. A sequence starts with a token byte: the high nibble is the number of
/// literal bytes that follow (15 = extended: the following bytes are added while they equal 255). After the literals, if
/// the output is not complete yet, come the match fields: a 16-bit big-endian distance and a length byte. A distance of 0
/// emits the match length as zero bytes (RLE); a positive distance copies the match length from that many bytes back in
/// the global window, byte by byte, so overlapping copies replicate a repeating pattern. The low nibble of the token is
/// the high byte of the match length, so a match spans up to 4095 bytes (a length byte of 0 with a non-zero high nibble
/// still means 256). When the output is full the sequence ends; the remaining bytes of the body are slack the writer
/// reserved and are not consumed.</para>
/// <para>Two endings depart from the strict scheme. A final match may omit the length byte: the body ends after the
/// distance, and the match length is the remainder of the output. And some blocks (repeating kinds, sizes near whole
/// mebibytes) declare an output buffer larger than the stream actually produces: the stream simply ends with the body,
/// and the real output is what was produced so far. Such a block is detected by retrying with this relaxed ending when
/// the strict decode runs out of body; the produced count then fills only part of the output and the caller pads the
/// rest (the directory addresses the buffer space, so the block's contribution to the global window is the declared
/// size). A back-reference that reaches beyond the decoded stream is always an error.</para>
/// </remarks>
internal static class LzPayloadDecoder
{
/// <summary>Decodes one mode 2 payload. Returns the consumed source length; <paramref name="produced"/> receives the
/// number of output bytes the stream actually generated (equal to <c>output.Length</c> except for a buffer block).</summary>
public static int Decode(ReadOnlySpan<byte> body, Span<byte> output, List<byte> window, long blockOffset, out int produced)
{
var result = DecodeAttempt(body, output, window, blockOffset, bufferMode: false);
if (!result.ok)
{
result = DecodeAttempt(body, output, window, blockOffset, bufferMode: true);
if (!result.ok)
throw new InvalidDataException($"Mode-2 block at 0x{blockOffset:x}: {result.error}");
}
produced = result.produced;
return result.consumed;
}

private static (bool ok, int consumed, int produced, string error) DecodeAttempt(
ReadOnlySpan<byte> body, Span<byte> output, List<byte> window, long blockOffset, bool bufferMode)
{
int src = 0, o = 0;
int windowStart = window.Count;
uint declared = (uint)output.Length;
if (body.Length >= 9)
{
uint field = body[5] | ((uint)body[6] << 8) | ((uint)body[7] << 16) | ((uint)body[8] << 24);
if (field == declared || field == (uint)body.Length)
src = 9; // stream header [u8][u32][u32] holding the packed and/or unpacked size, the token stream starts after it
}

void RollBack()
{
window.RemoveRange(windowStart, window.Count - windowStart);
}

try
{
while (o < output.Length)
{
if (src >= body.Length)
{
if (bufferMode) break; // the stream simply ends with the body
throw new InvalidDataException($"the token stream ends after {o} of {output.Length} output bytes.");
}
int token = body[src++];
int litLen = token >> 4;
if (litLen == 15)
{
byte x;
do
{
if (src >= body.Length)
{
if (bufferMode) return End(o, src, output.Length); // the stream simply ends with the body
throw new InvalidDataException("the literal length extension runs past the payload.");
}
x = body[src++];
litLen += x;
}
while (x == 255 && src < body.Length);
}
int litTake = Math.Min(litLen, body.Length - src);
int litEnd = Math.Min(o + litLen, output.Length);
litTake = Math.Min(litTake, litEnd - o);
for (int i = 0; i < litTake; i++, o++)
{
byte v = body[src++];
output[o] = v;
window.Add(v);
}
if (o >= output.Length)
return End(o, src, output.Length); // trailing literals completed the block; the body tail is slack
if (litTake < litLen && src >= body.Length)
{
// the declared literal run reaches past the body
if (bufferMode) return End(o, src, output.Length); // the stream simply ends with the body
throw new InvalidDataException($"the literal run of {litLen} bytes reaches past the payload at {src} ({o} of {output.Length} output bytes).");
}
if (src + 2 > body.Length)
{
if (bufferMode) return End(o, src, output.Length); // the stream simply ends with the body
throw new InvalidDataException($"the distance of the sequence at {src - 1} runs past the payload ({o} of {output.Length} output bytes).");
}
int dist = (body[src] << 8) | body[src + 1];
src += 2;
int matchLen;
if (src >= body.Length)
{
// final match without a length byte: the match length is the remainder of the output
matchLen = output.Length - o;
}
else
{
int lenByte = body[src++];
matchLen = lenByte == 0 ? 256 : (((token & 0xF) << 8) | lenByte);
}
if (o + matchLen > output.Length)
matchLen = output.Length - o; // the final match is clipped at the declared size
if (dist == 0)
{
for (int i = 0; i < matchLen; i++, o++) { output[o] = 0; window.Add(0); }
continue;
}
if (dist > window.Count)
throw new InvalidDataException($"the back-reference distance {dist} exceeds the decoded stream ({window.Count} bytes).");
for (int i = 0; i < matchLen; i++, o++)
{
byte v = window[window.Count - dist];
output[o] = v;
window.Add(v);
}
}
return End(o, src, output.Length);
}
catch (InvalidDataException e)
{
RollBack();
return (false, src, o, e.Message);
}

static (bool, int, int, string) End(int o, int src, int declared) => (true, src, o, null!);
}
}
107 changes: 104 additions & 3 deletions src/PlaIO/PlaDocument.cs
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,11 @@ public static PlaDocument Open(Stream stream, bool leaveOpen = false)
}

/// <summary>The decoded payload of a block of this document.</summary>
/// <exception cref="NotSupportedException">The block uses mode 2 (<see cref="PlaBlockMode.Lz"/>).</exception>
/// <exception cref="InvalidDataException">A mode-1 stream does not decode to the declared size, or the file is cut
/// <remarks>Reading a mode 2 (<see cref="PlaBlockMode.Lz"/>) block decodes every block of the file in order into
/// one global window first, because a mode 2 token stream may reference the output of any preceding block; the cost
/// of the call therefore grows with the size of the file. A preceding block that does not decode to its declared
/// size (a false entry of the scan) is skipped.</remarks>
/// <exception cref="InvalidDataException">A payload does not decode to its declared size, or the file is cut
/// short.</exception>
/// <exception cref="ArgumentException">The block is not one of the <see cref="Blocks"/> of this document.</exception>
public byte[] ReadPayload(PlaBlock block) => ReadPayload(block, out _);
Expand Down Expand Up @@ -120,9 +123,107 @@ internal byte[] ReadPayload(PlaBlock block, out int consumed)
}
return payload;
}
case PlaBlockMode.Lz:
{
// The mode-2 token stream references the decompressed output of all preceding blocks, so the whole
// file is decoded from the start and the requested block's output is returned. The cost of a call
// therefore grows with the size of the file. A preceding block that does not decode is a scanner
// false positive and not part of the real stream, so it is skipped.
var window = new List<byte>();
foreach (var b in _blocks)
{
if (ReferenceEquals(b, block)) break;
TryAppendToWindow(b, window);
}
var blockPayload = new byte[block.PackedSize];
StreamReads.ReadExactlyAt(_stream, block.PayloadOffset, blockPayload, $"the payload of the block at 0x{block.Offset:x}");
// Mode 2 (LZ): the token stream produces the declared unpacked size (some blocks declare a buffer
// larger than the stream produces — the tail of the buffer stays zero) and may reference the window;
// the body tail after the output is complete is writer slack.
var output = new byte[block.UnpackedSize];
int lzConsumed;
try
{
lzConsumed = LzPayloadDecoder.Decode(blockPayload, output, window, block.Offset, out _);
}
catch (InvalidDataException e)
{
throw new InvalidDataException($"Block at 0x{block.Offset:x}: {e.Message}", e);
}
consumed = lzConsumed;
return output;
}
default:
throw new NotSupportedException($"Block at 0x{block.Offset:x} uses payload mode 2 (LZ). Its compression is not documented yet; only modes 0 and 1 can be decoded.");
throw new NotSupportedException($"Block at 0x{block.Offset:x} uses an unknown payload mode {block.Mode}.");
}
}

/// <summary>Appends the decoded payload of one block to the mode-2 global window, or leaves the window unchanged when
/// the block does not decode to its declared size (a false entry of the scan is not part of the real stream, so a
/// partial decode is rolled back). Returns whether the block was appended.</summary>
private bool TryAppendToWindow(PlaBlock b, List<byte> window)
{
int before = window.Count;
try
{
AppendToWindow(b, window);
return true;
}
catch (InvalidDataException)
{
window.RemoveRange(before, window.Count - before);
return false;
}
}

/// <summary>Decodes the whole file once into the global window: the raw payloads, the full zero-mask decodes and the
/// mode-2 outputs, in block order. Blocks that do not decode to their declared size (false entries of the scan) are
/// skipped and reported in <paramref name="skipped"/>; the window stays as if they were not there.</summary>
internal List<byte> DecodeGlobalWindow(out IReadOnlyList<PlaBlock> skipped)
{
ObjectDisposedException.ThrowIf(_disposed, this);
var window = new List<byte>();
var failed = new List<PlaBlock>();
foreach (var b in _blocks)
{
if (!TryAppendToWindow(b, window)) failed.Add(b);
}
skipped = failed.AsReadOnly();
return window;
}

/// <summary>Appends the decoded payload of one block to the mode-2 global window. Raw payloads are copied as they
/// are; a zero-mask payload contributes its full decode; a mode 2 payload is decoded and the decoder appends every
/// produced byte. Throws <see cref="InvalidDataException"/> when the block does not decode to its declared size.</summary>
private void AppendToWindow(PlaBlock b, List<byte> window)
{
var payload = new byte[b.PackedSize];
StreamReads.ReadExactlyAt(_stream, b.PayloadOffset, payload, $"the payload of the block at 0x{b.Offset:x}");
if (b.Mode == PlaBlockMode.Raw)
{
window.AddRange(payload);
return;
}
if (b.Mode == PlaBlockMode.ZeroMask)
{
// A mask byte yields at most 8 bytes, so a larger declared size cannot decode; reject it before allocating.
if ((b.UnpackedSize + 7) / 8 > b.PackedSize)
throw new InvalidDataException($"Mode-1 block at 0x{b.Offset:x} declares {b.UnpackedSize} unpacked bytes from {b.PackedSize} packed bytes; a mask byte yields at most 8.");
long needed = Math.Min(b.PackedSize, b.UnpackedSize + (b.UnpackedSize + 7) / 8);
var source = new byte[needed];
StreamReads.ReadExactlyAt(_stream, b.PayloadOffset, source, $"the payload of the block at 0x{b.Offset:x}");
var decoded = new byte[b.UnpackedSize];
PayloadDecoder.DecodeZeroMask(source, decoded);
window.AddRange(decoded);
return;
}
// Mode 2 (LZ): the token stream produces the declared unpacked size (some blocks declare a buffer
// larger than the stream produces — the tail of the buffer stays zero) and the decoder appends every
// produced byte to the window; the body tail after the output is complete is writer slack.
var output = new byte[b.UnpackedSize];
int produced;
LzPayloadDecoder.Decode(payload, output, window, b.Offset, out produced);
for (long i = produced; i < b.UnpackedSize; i++) window.Add(0); // the directory addresses the buffer space
}

/// <summary>Searches every raw block large enough for 64 records for an object directory (at most one per block).
Expand Down
Loading
Loading