diff --git a/CHANGELOG.md b/CHANGELOG.md index 865a60d..9e853cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,16 @@ API between minor releases. - `PlaDocument.ReadPayload` no longer throws `NotSupportedException` for mode 2 blocks: all three storage modes (raw, zero mask, LZ) decode now. +### Added + +- `PlaDocument.OpenStream` and `PlaStream`: the decoded global stream of the document (every block's payload decoded in + block order) and the extents of it. A directory record's position and count name a byte extent of the stream — the + object's data — so `PlaStream.ReadExtent(ObjectDirectoryRecord)` returns an object's bytes; extents can also be read + by position and count. The stream is a snapshot that holds the whole decoded file in memory and the document may be + disposed afterwards. +- The position and count of `ObjectDirectoryRecord` are now documented: a byte extent of the decoded global stream, + confirmed against the object directories of real files. + ## [0.1.0] - 2026-11-12 ### Added diff --git a/docs/public-api.md b/docs/public-api.md index 0fb864f..8dd1c61 100644 --- a/docs/public-api.md +++ b/docs/public-api.md @@ -23,12 +23,16 @@ The `PlaIO` package is the supported surface listed here; every public type is i | `PlaDocument.Blocks` | The block index in file order, found by scanning from offset 32. No on-disk block map is known, so completeness is not claimed. The scan can take a byte run for a block header, so the index may contain false entries. In real files a small share of mode 1 blocks does not decode to its declared size, and `ReadPayload` throws `InvalidDataException` for them. | | `PlaDocument.ReadPayload(PlaBlock block)` | The decoded payload of a block of this document. Decodes all three storage modes: raw (mode 0), zero masks (mode 1) and the LZ token stream (mode 2). A mode 2 payload may start with a 9-byte stream header (packed and/or unpacked size). The token stream produces the block's declared unpacked size — some blocks declare a buffer larger than the stream produces, the tail of the buffer stays zero — and may reference the output of any preceding block, so the payload is produced by decoding every preceding block of the file in order into one global window; the cost of the call grows with the size of the file. Preceding blocks that do not decode (false entries of the scan) are skipped and leave the window unchanged. Throws `InvalidDataException` when a back-reference reaches beyond the decoded stream or the file is cut short, and `ArgumentException` when the block is not from this document. | | `PlaDocument.FindDirectories()` | Searches every raw (mode 0) block large enough for 64 records of 30 bytes for an object directory, at most one per block. Recomputed on each call; mode 1 payloads are not searched. | +| `PlaDocument.OpenStream()` | The document's decoded global stream (`PlaStream`): every block's payload decoded in block order. Blocks that do not decode (false entries of the scan) are skipped. The stream is a snapshot — it holds the whole decoded file in memory and the document may be disposed afterwards; decoding it once costs like a full mode-2 read. | | `PlaDocument.Dispose()` | Releases the stream unless the document was opened with `leaveOpen`. | | `FdbHeader` | `Size` (64), `LayoutVersion` (u16 at offset 8), `FormatVersion` (u16 at offset 10), `Field12` (u32 at offset 12; meaning not documented), `Raw` (the first min(64, file length) bytes; bytes 32..63 overlap the first block). | | `PlaBlockMode` | `Raw` (0), `ZeroMask` (1, flag 0x100), `Lz` (2, flag 0x200). | | `PlaBlock` | One indexed block: `Offset`, `Tag` (`BL` or `DY`), `Id` and `Kind` (raw), `Version` (0 or 1), `Flags` (raw), `SizeClass` (`Flags & 0xF`), `Mode`, `Guid` (present when flag 0x20 is set), `PackedSize`, `UnpackedSize`, `Extra` (present when flag 0x400 is set; raw), `PayloadOffset`, `HeaderLength`, `End`. | | `ObjectDirectory` | A run of directory records inside a raw block: `Block`, `RecordOffset` (payload offset of the first record), `Phase` (`RecordOffset % 30`), `Records`, `AbuttingPairs`, `OverlappingPairs`, `DuplicatePositions`. The run is an estimate and can be shorter than the real directory: it stops after more than 8 consecutive records without an abutting neighbour. | -| `ObjectDirectoryRecord` | The 30-byte record `[u16 class][u64 pos][u32 count][GUID16]` as `ClassNumber`, `Position`, `Count`, `ObjectId`. The units of position and count are not documented, and the class number is not mapped to a class name. | +| `ObjectDirectoryRecord` | The 30-byte record `[u16 class][u64 pos][u32 count][GUID16]` as `ClassNumber`, `Position`, `Count`, `ObjectId`. The position and count name a byte extent of the decoded global stream — the object's data (see `PlaDocument.OpenStream`). The class number is not mapped to a class name. | +| `PlaStream.Length` | Length of the decoded stream in bytes. | +| `PlaStream.ReadExtent(long position, long count)` | The bytes of one extent of the stream. `ArgumentOutOfRangeException` for negative bounds or an extent past the stream. | +| `PlaStream.ReadExtent(ObjectDirectoryRecord record)` | The bytes of the extent one directory record addresses — the object's data. `InvalidDataException` when the record's extent reaches past the stream. | ## BIMx file (`.bimx`) @@ -45,8 +49,9 @@ The `PlaIO` package is the supported surface listed here; every public type is i ## Resources and threads `PlaDocument` and `BimxDocument` never load the whole file into memory: the block index is built by scanning through the -stream once, at `Open`, and payloads and the cover are read on request. A document is not thread-safe; use one per thread. -After `Dispose`, `ReadPayload`, `FindDirectories` and `ReadCover` throw `ObjectDisposedException`. +stream once, at `Open`, and payloads and the cover are read on request. The exception is `OpenStream`: its decoded global +stream holds the whole decoded file in memory by design. A document is not thread-safe; use one per thread. +After `Dispose`, `ReadPayload`, `FindDirectories`, `OpenStream` and `ReadCover` throw `ObjectDisposedException`. ## Diagnostics diff --git a/src/PlaIO/ObjectDirectory.cs b/src/PlaIO/ObjectDirectory.cs index e00fe85..9ba00b6 100644 --- a/src/PlaIO/ObjectDirectory.cs +++ b/src/PlaIO/ObjectDirectory.cs @@ -1,10 +1,10 @@ namespace PlaIO; -/// One 30-byte directory record: [u16 class][u64 pos][u32 count][GUID16]. The units of -/// and are not documented. +/// One 30-byte directory record: [u16 class][u64 pos][u32 count][GUID16]. The position and count name a +/// byte extent of the decoded global stream (see ), the object's data. /// The class number of the object, raw. It is not mapped to a name. -/// Start of the object's extent, raw. -/// Length of the object's extent, raw. +/// Start of the object's extent in the decoded global stream. +/// Length of the object's extent in bytes. /// The object GUID. public readonly record struct ObjectDirectoryRecord(ushort ClassNumber, ulong Position, uint Count, Guid ObjectId); diff --git a/src/PlaIO/PlaDocument.cs b/src/PlaIO/PlaDocument.cs index 9f4b87b..57b2770 100644 --- a/src/PlaIO/PlaDocument.cs +++ b/src/PlaIO/PlaDocument.cs @@ -158,6 +158,16 @@ internal byte[] ReadPayload(PlaBlock block, out int consumed) } } + /// Decodes the whole file once into the global stream: the raw payloads, the full zero-mask decodes and the + /// mode-2 outputs, in block order. Blocks that do not decode to their declared size (false entries of the scan) are + /// skipped. The stream is a snapshot: it holds the whole decoded file in memory and the document may be disposed + /// afterwards. Object directory records address the stream by position and count. + public PlaStream OpenStream() + { + ObjectDisposedException.ThrowIf(_disposed, this); + return new PlaStream(DecodeGlobalWindow(out _)); + } + /// Appends the decoded payload of one block to the mode-2 global window, or leaves the window unchanged when /// the block does not decode to its declared size (a false entry of the scan is not part of the real stream, so a /// partial decode is rolled back). Returns whether the block was appended. diff --git a/src/PlaIO/PlaStream.cs b/src/PlaIO/PlaStream.cs new file mode 100644 index 0000000..e3df82d --- /dev/null +++ b/src/PlaIO/PlaStream.cs @@ -0,0 +1,41 @@ +namespace PlaIO; + +/// The decoded global stream of a document: every block's payload decoded in block order — raw payloads as +/// they are, zero-mask payloads as their full decode, mode 2 payloads as their token-stream output. The object +/// directories () address this stream: a record's +/// and name a byte extent of the +/// stream, the object's data. The stream is a snapshot: it holds the whole decoded file in memory and never touches the +/// document again, so the document may be disposed. Not thread-safe. +public sealed class PlaStream +{ + private readonly List _bytes; + + internal PlaStream(List bytes) => _bytes = bytes; + + /// Length of the decoded stream in bytes. + public long Length => _bytes.Count; + + /// The bytes of one extent of the stream. + /// The bounds are negative or reach past the stream. + public byte[] ReadExtent(long position, long count) + { + ArgumentOutOfRangeException.ThrowIfNegative(position); + ArgumentOutOfRangeException.ThrowIfNegative(count); + if (position > Length || count > Length - position) + throw new ArgumentOutOfRangeException(nameof(count), $"The extent at {position}+{count} reaches past the decoded stream ({Length} bytes)."); + var extent = new byte[count]; + for (long i = 0; i < count; i++) extent[i] = _bytes[(int)(position + i)]; + return extent; + } + + /// The bytes of the extent one directory record addresses. + /// The record's extent reaches past the decoded stream. + public byte[] ReadExtent(ObjectDirectoryRecord record) + { + long position = (long)record.Position; + long count = record.Count; + if (position > Length || count > Length - position) + throw new InvalidDataException($"The directory record of {record.ObjectId} (class {record.ClassNumber}) addresses {position}+{count}, past the decoded stream ({Length} bytes)."); + return ReadExtent(position, count); + } +} diff --git a/tests/PlaIO.Tests/PlaStreamTests.cs b/tests/PlaIO.Tests/PlaStreamTests.cs new file mode 100644 index 0000000..7821c21 --- /dev/null +++ b/tests/PlaIO.Tests/PlaStreamTests.cs @@ -0,0 +1,73 @@ +using Xunit; + +namespace PlaIO.Tests; + +/// The decoded global stream and the extent reads: the stream is the file decoded in block order, and a +/// directory record's position and count name a byte extent of it — including bytes produced by a mode 2 block. +public class PlaStreamTests +{ + [Fact] + public void ExtentsAddressTheDecodedStreamNotTheRawFile() + { + // Block A: raw "ABCDEFGH". Block B: mode 2 producing "wxyz" (token 0x40 = 4 literals). + var lz = Synthetic.Concat([0x40], "wxyz"u8.ToArray(), [0xFF, 0xFE]); + using var document = PlaDocument.Open(new MemoryStream(Synthetic.Fdb( + Synthetic.Block(PlaBlockMode.Raw, "ABCDEFGH"u8.ToArray()), + Synthetic.Block(PlaBlockMode.Lz, lz, unpacked: 4)))); + + var stream = document.OpenStream(); + Assert.Equal(12, stream.Length); + Assert.Equal("CDE"u8.ToArray(), stream.ReadExtent(2, 3)); // inside the raw block + Assert.Equal("wxyz"u8.ToArray(), stream.ReadExtent(8, 4)); // inside the mode-2 output + } + + [Fact] + public void RecordExtentReadsTheObjectData() + { + var data = "0123456789"u8.ToArray(); + var record = new ObjectDirectoryRecord(7, 4, 5, Guid.Parse("00112233-4455-6677-8899-aabbccddeeff")); + using var document = PlaDocument.Open(new MemoryStream(Synthetic.Fdb( + Synthetic.Block(PlaBlockMode.Raw, data)))); + + var stream = document.OpenStream(); + Assert.Equal("45678"u8.ToArray(), stream.ReadExtent(record)); + } + + [Fact] + public void AZeroLengthExtentAtTheEndIsEmpty() + { + using var document = PlaDocument.Open(new MemoryStream(Synthetic.Fdb( + Synthetic.Block(PlaBlockMode.Raw, "AB"u8.ToArray())))); + + var stream = document.OpenStream(); + Assert.Empty(stream.ReadExtent(2, 0)); + Assert.Empty(stream.ReadExtent(new ObjectDirectoryRecord(0, 2, 0, Guid.NewGuid()))); + Assert.Equal("AB"u8.ToArray(), stream.ReadExtent(0, stream.Length)); + } + + [Fact] + public void ExtentBoundsPastTheStreamAreRejected() + { + using var document = PlaDocument.Open(new MemoryStream(Synthetic.Fdb( + Synthetic.Block(PlaBlockMode.Raw, "AB"u8.ToArray())))); + + var stream = document.OpenStream(); + Assert.Throws(() => stream.ReadExtent(-1, 1)); + Assert.Throws(() => stream.ReadExtent(0, -1)); + Assert.Throws(() => stream.ReadExtent(1, 2)); + Assert.Throws(() => stream.ReadExtent(3, 0)); + } + + [Fact] + public void ARecordExtentPastTheStreamIsRejected() + { + var guid = Guid.NewGuid(); + using var document = PlaDocument.Open(new MemoryStream(Synthetic.Fdb( + Synthetic.Block(PlaBlockMode.Raw, "AB"u8.ToArray())))); + + var stream = document.OpenStream(); + var e = Assert.Throws(() => + stream.ReadExtent(new ObjectDirectoryRecord(3, 1, 10, guid))); + Assert.Contains(guid.ToString(), e.Message); + } +} diff --git a/tests/PlaIO.Tests/SampleTests.cs b/tests/PlaIO.Tests/SampleTests.cs index 61d7473..a3ea88f 100644 --- a/tests/PlaIO.Tests/SampleTests.cs +++ b/tests/PlaIO.Tests/SampleTests.cs @@ -148,6 +148,31 @@ public void EveryLzBlockDecodesAndOnlyFalseEntriesAreSkipped() } } + [SkippableFact] + public void EveryDirectoryRecordFitsTheGlobalStream() + { + foreach (var sample in corpus.Fdb) + { + var stream = sample.Document.OpenStream(); + foreach (var d in sample.Document.FindDirectories()) + { + string where = $"{sample.Name}: directory in block 0x{d.Block.Offset:x}"; + int checkedReads = 0; + foreach (var r in d.Records) + { + Assert.True(r.Position <= (ulong)stream.Length && r.Count <= (ulong)stream.Length - r.Position, + $"{where}: record {r.ObjectId} (class {r.ClassNumber}) addresses {r.Position}+{r.Count} of a {stream.Length}-byte stream"); + // A few full reads exercise the copy, not every record (the streams are large). + if (r.Count > 0 && checkedReads < 3) + { + Assert.Equal((long)r.Count, stream.ReadExtent(r).Length); + checkedReads++; + } + } + } + } + } + [SkippableFact] public void EveryFdbSampleHasADirectoryOfMostlyAbuttingExtents() {