Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`ApplyTransform` is false. A record that does not hold usable values is left unread, and the setup keeps an unknown pose.
- The point count of a Leica scan is taken from the container when it states one, so that reading it does not decode the
point data. Only a container without that number still needs the decoding.
- The Leica reader counts the data it cannot use (the points that are dropped as padding, the compressed streams that are
skipped and the point data that ends at an unknown block) and reports it in the warnings of the document. The new
`PclReadOptions.Strict` makes such a read fail with `InvalidDataException` instead.

## [0.1.0] - 2026-11-15

Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,17 +89,19 @@ These readers drop data without an error or a warning, so a scan can yield fewer
data file in neither layout, or in the sectioned layout with an element type or header value the reader does not support, cannot
be read: the project lists it with a warning and reading its points throws `NotSupportedException`. A compressed stream whose
copies reach back more than 1 MiB is not decoded either: reading its points throws `NotSupportedException`, without an earlier warning.
- Leica: padding entries and points that coincide with a block's origin can be dropped, a compressed stream above its size limit
is skipped, and a block that is not recognised ends the scan. The result can therefore be incomplete.
- Leica: padding entries and points that coincide with a block's origin are dropped, a compressed stream above its size limit is
skipped, and a block that is not recognised ends the scan. The reader counts what it dropped or skipped and reports it in the
warnings of the document, so that an incomplete result is visible. With `PclReadOptions.Strict` the read fails instead.
- EdgeWise: a database node whose point count is 0 is skipped.

### Limits

A file that exceeds one of these limits, or whose declared sizes and counts do not fit the file, is rejected with
`InvalidDataException`, with the exceptions below. A FARO scan grid above its limit gives `NotSupportedException` when the file is
opened. An E57 point prototype with more than 1024 fields gives `NotSupportedException` when the points are read; the scan still
lists. A compressed Leica stream above its size limit is not rejected: its data is skipped, which is one of the Leica data filters
above. An unsupported PinPoint feature (see the data filters) gives `NotSupportedException` when the points are read. Point data is streamed and there is no limit on the file size, except for FARO container files: each compressed data stream is
lists. A compressed Leica stream above its size limit is not rejected: its data is skipped and the skip is reported in the warnings
of the document, which is one of the Leica data filters above. An unsupported PinPoint feature (see the data filters) gives
`NotSupportedException` when the points are read. Point data is streamed and there is no limit on the file size, except for FARO container files: each compressed data stream is
read into memory as a whole, so its size must fit a .NET array (about 2 GB), otherwise reading fails with
`NotSupportedException`. The XML of E57 and ReCap project files is read with DTDs prohibited, so no entity is expanded and nothing
outside the file is fetched.
Expand Down
2 changes: 1 addition & 1 deletion docs/public-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ format-specific members can change when the support for that format changes.
|---|---|
| `PclFile` | Static entry point. `DetectKind(path)` and `DetectKind(stream, extension)` return a `PclFileKind`; `Open(path, options)` detects the format and returns a `PclDocument`. |
| `PclFileKind` | `Unknown`, `E57`, `RecapProject`, `RecapScan`, `LeicaLgsx`, `LeicaLgs`, `FaroFls`, `PinPoint` and `EdgeWise`. |
| `PclReadOptions` | `MaxPoints` (limit of one read call, null for all, 0 for none), `SubsampleStep` (1 keeps every point, N keeps about one in N) and `ApplyTransform` (default true). Negative `MaxPoints` and a step below 1 throw `ArgumentOutOfRangeException`. `PclReadOptions.Default` has no limit. |
| `PclReadOptions` | `MaxPoints` (limit of one read call, null for all, 0 for none), `SubsampleStep` (1 keeps every point, N keeps about one in N), `ApplyTransform` (default true) and `Strict` (default false: a reader that finds incomplete data fails with `InvalidDataException` instead of adding a warning). Negative `MaxPoints` and a step below 1 throw `ArgumentOutOfRangeException`. `PclReadOptions.Default` has no limit. |

`Open` throws `FileNotFoundException` for a missing path, `DirectoryNotFoundException` when a directory the reader needs is
missing or disappears while reading, `NotSupportedException` for an unrecognised format or an unsupported feature, and
Expand Down
158 changes: 144 additions & 14 deletions src/PclIO/Formats/Leica/LgsxDocument.cs
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,18 @@ private LgsxDocument(string path, long fileSize, string guid, string version, bo
_metadata["Format"] = isLegacy ? "Leica LGS" : "Leica LGSx";
}

/// <summary>
/// Adds a warning to the document. The same message is added only once, so that reading the same scan twice
/// does not repeat it. Reads of different scans may add warnings at the same time, so the list is guarded.
/// </summary>
internal void AddWarning(string message)
{
lock (_warnings)
{
if (!_warnings.Contains(message, StringComparer.Ordinal)) _warnings.Add(message);
}
}

/// <summary>Opens a Leica container and reads its table of contents.</summary>
/// <param name="path">Path of the .lgsx or .lgs file.</param>
/// <param name="options">Read options; currently unused when opening.</param>
Expand Down Expand Up @@ -235,7 +247,8 @@ public static LgsxDocument Open(string path, PclReadOptions? options = null)
blockStarts: blockStarts,
transform: setup.Translation is { } translation ? new PclTransform { Translation = translation } : null,
bounds: setup.Bounds,
declaredPointCount: setup.PointCount >= 0 ? setup.PointCount : null));
declaredPointCount: setup.PointCount >= 0 ? setup.PointCount : null,
document: document));
}

document._warnings.Add(
Expand All @@ -251,7 +264,8 @@ public static LgsxDocument Open(string path, PclReadOptions? options = null)
name: Path.GetFileNameWithoutExtension(path),
path: path,
pointCloudEntry: pointCloudEntry,
declaredPointCount: declaredCount));
declaredPointCount: declaredCount,
document: document));
}

if (scanSetups.Count > 0)
Expand Down Expand Up @@ -522,6 +536,7 @@ public sealed class LgsxScan : PclScan

private readonly string _filePath;
private readonly LgsxEntry? _entry;
private readonly LgsxDocument? _document;

// Start offsets of the blocks that belong to this scan; null when the scan holds all blocks of the container.
private readonly long[]? _blockStarts;
Expand All @@ -532,6 +547,21 @@ public sealed class LgsxScan : PclScan
private long _pointCount = -1;
private (bool Colors, bool Intensities, bool Normals)? _channels;

/// <summary>
/// What one read did not return. Each enumeration keeps its own state, so interleaved or parallel reads of the
/// same scan do not mix their counts; the state is reported when the read is done.
/// </summary>
private sealed class ReadState
{
public long DroppedPoints;
public long SkippedStreams;
public long BlocksRead;
public long UnusableBlocks;
public long IncompleteBlocks;
public bool ChainEndedEarly;
public long ChainBlocks;
}

/// <inheritdoc />
public override string Id { get; }

Expand All @@ -558,14 +588,15 @@ public sealed class LgsxScan : PclScan
private (bool Colors, bool Intensities, bool Normals) Channels => _channels ??= DetectChannels();

internal LgsxScan(string id, string name, string path, LgsxEntry? pointCloudEntry, long[]? blockStarts = null,
PclTransform? transform = null, PclBounds? bounds = null, long? declaredPointCount = null)
PclTransform? transform = null, PclBounds? bounds = null, long? declaredPointCount = null, LgsxDocument? document = null)
{
Id = id;
Name = name;
_filePath = path;
_entry = pointCloudEntry;
_blockStarts = blockStarts;
_declaredPointCount = declaredPointCount;
_document = document;
if (transform != null) Transform = transform;
if (bounds != null) Bounds = bounds.Value;
}
Expand All @@ -580,7 +611,7 @@ private sealed record Block(double ScaleX, double ScaleY, double ScaleZ, double
if (_entry == null) return (false, false, false);

using var fs = File.OpenRead(_filePath);
foreach (var block in ReadBlocks(fs))
foreach (var block in ReadBlocks(fs, new ReadState()))
{
int n = block.Frames.Count;
return (n >= ColorFrames, n >= IntensityFrames, n >= NormalFrames);
Expand All @@ -591,13 +622,22 @@ private sealed record Block(double ScaleX, double ScaleY, double ScaleZ, double

/// <summary>Streams the points of this scan.</summary>
/// <remarks>
/// <para>
/// Positions are decoded from the packed position planes of each block. A code of zero at any slot except the
/// first of a block is treated as unused padding and is not returned, so a real point that sits exactly at the
/// block origin is dropped. A block yields as many points as its shortest position plane holds. Returned points
/// keep consecutive <see cref="PclPoint.Index"/> values.
/// </para>
/// <para>
/// Data that the reader cannot use is counted and reported in the warnings of the document: the points that were
/// dropped as padding, the compressed streams that were skipped because they are too large, the point blocks that
/// carry no positions or whose frame list ends before the block does, and the point data that ends at a block the
/// reader does not know. With <see cref="PclReadOptions.Strict"/> the read fails with
/// <see cref="InvalidDataException"/> instead of the warning.
/// </para>
/// </remarks>
/// <param name="options">Read options; null uses defaults.</param>
/// <exception cref="InvalidDataException">The point data is malformed or declares oversized data.</exception>
/// <exception cref="InvalidDataException">The point data is malformed, declares oversized data or is incomplete in strict mode.</exception>
public override IEnumerable<PclPoint> ReadPoints(PclReadOptions? options = null)
{
options ??= PclReadOptions.Default;
Expand All @@ -608,8 +648,11 @@ public override IEnumerable<PclPoint> ReadPoints(PclReadOptions? options = null)
int step = Math.Max(1, options.SubsampleStep);
long yielded = 0;

// The counters describe this read alone; they start at zero every time.
var state = new ReadState();

using var fs = File.OpenRead(_filePath);
foreach (var block in ReadBlocks(fs))
foreach (var block in ReadBlocks(fs, state))
{
var frames = block.Frames;
byte[] p0 = frames[0];
Expand All @@ -629,7 +672,12 @@ public override IEnumerable<PclPoint> ReadPoints(PclReadOptions? options = null)
if (i % step != 0) continue;

uint val = (uint)(p0[i] | (p1[i] << 8) | (p2[i] << 16) | (p3[i] << 24));
if (val == 0 && i > 0 && p0[i] == 0 && p1[i] == 0) continue;
if (val == 0 && i > 0 && p0[i] == 0 && p1[i] == 0)
{
// The slot holds no point: it is padding or a point at the origin of the block.
state.DroppedPoints++;
continue;
}

// Unpack the three 10-bit coordinates from the packed code.
uint xm = 0, ym = 0, zm = 0;
Expand Down Expand Up @@ -694,17 +742,55 @@ public override IEnumerable<PclPoint> ReadPoints(PclReadOptions? options = null)
if (yielded >= targetCount) yield break;
}
}

// The whole point data was read, so what it did not hold can be reported.
ReportIncompleteData(options.Strict, state);
}

/// <summary>
/// Reports the data that the point data did not hold: the points that were dropped as padding, the compressed
/// streams that were skipped, the point blocks that carry no positions or whose frame list is cut short, and the
/// point data that ends at a block the reader does not know. In strict mode the read fails instead.
/// </summary>
private void ReportIncompleteData(bool strict, ReadState state)
{
var problems = new List<string>();

if (state.DroppedPoints > 0)
problems.Add($"{state.DroppedPoints} points were dropped because their slot holds no point");

if (state.SkippedStreams > 0)
problems.Add($"{state.SkippedStreams} compressed streams were skipped because they are larger than the accepted size");

if (state.UnusableBlocks > 0)
problems.Add($"{state.UnusableBlocks} point blocks were skipped because they hold fewer than four planes");

if (state.IncompleteBlocks > 0)
problems.Add($"{state.IncompleteBlocks} point blocks have a frame list that ends before the block does");

if (state.ChainEndedEarly)
problems.Add($"the point data ends after {state.ChainBlocks} blocks at a block that is not a point block");

if (problems.Count == 0) return;

string message = "The point data is incomplete: " + string.Join("; ", problems) + ".";
if (strict) throw FormatError.Corrupted(message);

_document?.AddWarning(message);
}

/// <summary>Walks the chain of point blocks and yields those that carry point planes and belong to this scan.</summary>
private IEnumerable<Block> ReadBlocks(FileStream fs)
private IEnumerable<Block> ReadBlocks(FileStream fs, ReadState state)
{
long fileLength = fs.Length;
long decodedLimit = DecodedAllowanceBytes + (fileLength * DecodedBytesPerFileByte);
long decodedBytes = 0;
byte[] blockHeader = new byte[128];

foreach (var (blockStart, blockLength) in LgsxDocument.ReadBlockChain(fs, _entry!))
var chain = LgsxDocument.ReadBlockChain(fs, _entry!);
long payloadEnd = (long)_entry!.Offset + (long)_entry.Size;

foreach (var (blockStart, blockLength) in chain)
{
if (!BelongsToScan(blockStart)) continue;

Expand Down Expand Up @@ -738,7 +824,7 @@ private IEnumerable<Block> ReadBlocks(FileStream fs)
if (firstFrame >= 8)
{
long firstFrameOffset = blockStart + firstFrame;
var frames = ReadAllFrames(fs, firstFrameOffset, blockEnd);
var frames = ReadAllFrames(fs, firstFrameOffset, blockEnd, state);

// Many small blocks that each expand a lot would otherwise make the file cost unbounded time.
foreach (byte[] plane in frames) decodedBytes += plane.Length;
Expand All @@ -753,8 +839,34 @@ private IEnumerable<Block> ReadBlocks(FileStream fs)
double.IsFinite(minX) && double.IsFinite(minY) && double.IsFinite(minZ)))
throw FormatError.Corrupted("point block has an invalid coordinate scale or origin.");

state.BlocksRead++;
yield return new Block(scaleX, scaleY, scaleZ, minX, minY, minZ, frames);
}
else state.UnusableBlocks++;
}
else state.UnusableBlocks++;
}

// Data that is left in the point data but that is not a point block means that the reader did not read
// everything the container holds. The gap belongs to the payload, so only the unified scan or the scan
// that owns the last block reports it, and the count is the count of the whole chain.
state.ChainBlocks = chain.Count;
if (_blockStarts == null || (chain.Count > 0 && BelongsToScan(chain[^1].Start)))
{
if (chain.Count > 0)
{
long lastEnd = chain[^1].Start + chain[^1].Length;
if (payloadEnd - lastEnd >= 8 + 128) state.ChainEndedEarly = true;
}
else
{
// An empty chain means the very first chunk is not a point block. When a full block could have
// fit there, the point data ends at a block the reader does not know, just as with a later one.
fs.Position = (long)_entry!.Offset;
byte[] payloadHeader = new byte[64];
fs.ReadExactly(payloadHeader);
long firstBlock = (long)_entry.Offset + Bytes.ReadUInt32LE(payloadHeader.AsSpan(0x20, 4));
if (payloadEnd - firstBlock >= 8 + 128) state.ChainEndedEarly = true;
}
}
}
Expand All @@ -763,7 +875,7 @@ private IEnumerable<Block> ReadBlocks(FileStream fs)
private bool BelongsToScan(long blockStart) =>
_blockStarts == null || Array.BinarySearch(_blockStarts, blockStart) >= 0;

private static List<byte[]> ReadAllFrames(FileStream fs, long offset, long maxOffset)
private List<byte[]> ReadAllFrames(FileStream fs, long offset, long maxOffset, ReadState state)
{
var list = new List<byte[]>();
long limit = Math.Min(maxOffset, fs.Length);
Expand All @@ -777,14 +889,32 @@ private static List<byte[]> ReadAllFrames(FileStream fs, long offset, long maxOf
{
fs.ReadExactly(lenBuf);

// A zero or out-of-range length marks the end of the frame list.
// A zero length marks the end of the frame list.
ulong fLen = Bytes.ReadUInt64LE(lenBuf);
if (fLen == 0 || fLen > MaxCompressedFrameBytes || (long)fLen > limit - fs.Position) break;
if (fLen == 0) break;

// A stream that is larger than the accepted size is skipped: the rest of the block is lost with it.
if (fLen > MaxCompressedFrameBytes)
{
state.SkippedStreams++;
break;
}

// A frame list that ends before the block does leaves the rest of the block unread.
if ((long)fLen > limit - fs.Position)
{
state.IncompleteBlocks++;
break;
}

byte[] fData = new byte[(int)fLen];
fs.ReadExactly(fData);

if (!fData.AsSpan().StartsWith(FrameMagic)) break;
if (!fData.AsSpan().StartsWith(FrameMagic))
{
state.IncompleteBlocks++;
break;
}

if (list.Count >= MaxFramesPerBlock)
throw FormatError.Corrupted($"point block has more than {MaxFramesPerBlock} planes.");
Expand Down
7 changes: 7 additions & 0 deletions src/PclIO/PclReadOptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,11 @@ public int SubsampleStep
/// or the 4x4 matrix when the format supplies one. Rotation values are not applied.
/// </summary>
public bool ApplyTransform { get; init; } = true;

/// <summary>
/// Whether a reader that finds incomplete data fails instead of reporting it: a reader that would add a warning
/// about dropped or skipped data throws <see cref="InvalidDataException"/> then. Readers that cannot detect
/// incomplete data ignore this option. The default is false, so that a read never fails because of it.
/// </summary>
public bool Strict { get; init; }
}
Loading
Loading