Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Points that the E57 validity flags mark as invalid are skipped, and flagged colour or intensity channels are left empty.
Colour with a wider Integer range from zero (16 or 32 bits) and intensity as a wider Integer, a Float or a scaled
integer are decoded; other shapes stay unread.
- The E57 reader decodes plain bit-packing only and names any other codec of the scan in the rejection message. The point
prototype limit is the packet header bound of 65535 fields.

## [0.1.0] - 2026-11-15

Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ The library targets .NET 8 and .NET 10. It is managed code with no dependencies

| Format | Extensions | Support level |
| --- | --- | --- |
| ASTM E57 | `.e57` | Points decoded for a defined subset: uncompressed bit-packed streams with single or double precision Cartesian coordinates or scaled-integer coordinates (raw value times scale plus offset), or spherical range, azimuth and elevation converted to Cartesian points, with 8-bit colour and 8-bit intensity, or colour with a wider Integer range from zero and intensity as a wider Integer, a Float or a scaled integer. Pose translation and rotation are applied. Other codecs are not supported: a scan that uses them lists normally and fails when its points are read. Per-point validity flags skip invalid points and clear invalid channels when their streams hold one flag byte per record; other shapes are not interpreted. Images are not read, so a scan with them still lists and reads its points. Page checksums are verified. |
| ASTM E57 | `.e57` | Points decoded for a defined subset: uncompressed bit-packed streams with single or double precision Cartesian coordinates or scaled-integer coordinates (raw value times scale plus offset), or spherical range, azimuth and elevation converted to Cartesian points, with 8-bit colour and 8-bit intensity, or colour with a wider Integer range from zero and intensity as a wider Integer, a Float or a scaled integer. Pose translation and rotation are applied. Only plain bit-packing is decoded: a scan whose codecs vector names another codec lists normally and fails when its points are read, and the message names it. Per-point validity flags skip invalid points and clear invalid channels when their streams hold one flag byte per record; other shapes are not interpreted. Images are not read, so a scan with them still lists and reads its points. Page checksums are verified. |
| ReCap scan | `.rcs` | Experimental. The header is read; point decoding covers the node layouts that are described in the code, which is the one of major version 3. A file of another version is listed with its header, but reading its points fails with a message that names the version. A further layout is described only when the files of that version may be used for the tests, so that its offsets can be verified. The channels of the points follow from the size of the payload of the nodes: colour and intensity when the records are long enough, and normals when the node holds the plane of them. |
| ReCap project | `.rcp` | Lists the scans of the project manifest and opens the `.rcs` files it finds next to the project. The transform of a scan node places the scan in the project and is applied to its points unless `ApplyTransform` is false; when the manifest states no transform, the transform of the scan file is used. |
| Leica scan container | `.lgsx`, `.lgs` | Experimental. Points are decoded from the compressed point blocks: position, and colour, intensity and normals when the block has them. When the container assigns its points to its setups, there is one scan per setup, named after the setup, with the pose and the bounds of the setup when its record holds them; the pose is applied unless `ApplyTransform` is false. When the container does not assign its points, all points of the container form one scan and a warning says so. The point count is taken from the container when it states one; otherwise it is found by decoding the data, which is slow on large files. |
Expand Down Expand Up @@ -103,8 +103,8 @@ These readers drop data without an error or a warning, so a scan can yield fewer

A file that exceeds one of these limits, or whose declared sizes and counts do not fit the file, is rejected with
`InvalidDataException`, with the exceptions below. A FARO scan grid above its limit gives `NotSupportedException` when the file is
opened. An E57 point prototype with more than 1024 fields gives `NotSupportedException` when the points are read; the scan still
lists. A compressed Leica stream above its size limit is not rejected: its data is skipped and the skip is reported in the warnings
opened. An E57 point prototype with more than 65535 fields gives `NotSupportedException` when the points are read; the scan still
lists. A scan whose codecs vector names another codec than plain bit-packing is rejected the same way, and the message names it. A compressed Leica stream above its size limit is not rejected: its data is skipped and the skip is reported in the warnings
of the document, which is one of the Leica data filters above. An unsupported PinPoint feature (see the data filters) gives
`NotSupportedException` when the points are read. Point data is streamed and there is no limit on the file size, except for FARO container files: each compressed data stream is
read into memory as a whole, so its size must fit a .NET array (about 2 GB), otherwise reading fails with
Expand All @@ -113,7 +113,7 @@ outside the file is fetched.

| Format | Limits |
| --- | --- |
| E57 | XML section 32 MiB, nesting depth 64, 1,000,000 elements; declared record counts up to 2^40; at most 1024 fields in a point prototype; at most 4096 consecutive packets without points. |
| E57 | XML section 32 MiB, nesting depth 64, 1,000,000 elements; declared record counts up to 2^40; at most 65535 fields in a point prototype; at most 4096 consecutive packets without points. |
| ReCap project | Manifest 64 MiB (declared and real size), nesting depth 64, 1,000,000 elements, 65,536 scan nodes. |
| ReCap scan | 500,000 nodes in the node table, which must fit in the file together with the node payloads. |
| Leica | 10,000 container entries; per point block at most 64 compressed streams, 10,000,000 compressed bytes per stream, 16 MiB per decompressed stream (checked before allocation) and 128 MiB decompressed in all; over all blocks of one read 1 GiB plus 64 times the file size decompressed. |
Expand Down
18 changes: 12 additions & 6 deletions src/PclIO/Formats/E57/E57Document.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ namespace PclIO.Formats.E57;
/// with a range from zero (8, 16 or 32 bits), Float fields, or scaled-integer intensity; per-point validity flags
/// skip invalid points and clear invalid channels. Images are not read.
/// Not supported (reported with <see cref="NotSupportedException"/> when points are read): integer coordinates,
/// other codecs, and packets whose streams hold different record counts.
/// other codecs (the scan must use plain bit-packing: an empty codecs vector), and packets whose streams hold
/// different record counts.
/// Per-point validity flags of another shape, images and 2D data are not interpreted.
/// </summary>
public sealed class E57Document : PclDocument
Expand Down Expand Up @@ -272,8 +273,8 @@ private static PclTransform ParsePose(XElement scanElem, XNamespace ns)
/// </summary>
internal sealed class E57PointLayout
{
// Reader limit on the number of fields in a prototype; packet headers would allow 65535 streams.
private const int MaxFields = 1024;
// Reader limit on the number of fields in a prototype: a data packet header holds at most 65535 streams.
private const int MaxFields = 65535;

public int StreamCount { get; }

Expand Down Expand Up @@ -354,9 +355,9 @@ internal sealed class E57PointLayout
public static E57PointLayout? Create(XElement points, XNamespace ns, out string? unsupportedReason)
{
unsupportedReason = null;
if (points.Element(ns + "codecs")?.HasElements == true)
if (points.Element(ns + "codecs") is { HasElements: true } codecs)
{
unsupportedReason = "the scan uses a compressed vector codec other than plain bit-packing";
unsupportedReason = $"the scan uses compressed vector codecs other than plain bit-packing ({DescribeCodecs(codecs)})";
return null;
}

Expand All @@ -370,7 +371,7 @@ internal sealed class E57PointLayout
var fields = prototype.Elements().ToList();
if (fields.Count == 0 || fields.Count > MaxFields)
{
unsupportedReason = "the point prototype has no fields or too many fields";
unsupportedReason = "the point prototype has no fields or more than 65535 fields";
return null;
}

Expand Down Expand Up @@ -659,6 +660,11 @@ private static bool TryIntensityField(XElement field, out int size, out bool isF
// unknown field, so files the reader cannot interpret keep today's behaviour instead of failing.
private static bool TryValidityField(XElement field) =>
IntegerSize(field, out _, out _, out _) == 1;

// Names the codecs of a scan for the rejection message without assuming their shape: the distinct element
// names inside the codecs vector, at most eight of them.
private static string DescribeCodecs(XElement codecs) =>
string.Join(", ", codecs.DescendantNodes().OfType<XElement>().Select(e => e.Name.LocalName).Distinct().Take(8));
}

/// <summary>An individual scan inside an ASTM E57 file.</summary>
Expand Down
5 changes: 4 additions & 1 deletion tests/PclIO.Tests/E57Spec.cs
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ public sealed class E57Spec

public string PrototypeXml { get; init; } = DefaultPrototype;

/// <summary>Elements placed inside the codecs vector; empty for plain bit-packing.</summary>
public string CodecsXml { get; init; } = "";

public ulong PageSize { get; init; } = 1024;

/// <summary>Text inserted between the XML declaration and the root element, for example a document type declaration.</summary>
Expand Down Expand Up @@ -202,7 +205,7 @@ private string BuildXml(ulong sectionOffset)
{PoseXml}
<points type="CompressedVector" fileOffset="{sectionOffset.ToString(CultureInfo.InvariantCulture)}" recordCount="{count.ToString(CultureInfo.InvariantCulture)}">
{PrototypeXml}
<codecs type="Vector" allowHeterogeneousChildren="1"/>
<codecs type="Vector" allowHeterogeneousChildren="1">{CodecsXml}</codecs>
</points>
</vectorChild>
</data3D>
Expand Down
50 changes: 50 additions & 0 deletions tests/PclIO.Tests/FormatTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -968,6 +968,56 @@ public void E57_With_A_Partial_Colour_Record_Is_Rejected()
Assert.Throws<InvalidDataException>(() => doc.Scans[0].ReadPoints().ToList());
}

[Fact]
public void E57_With_A_Named_Codec_Lists_But_Is_Not_Decoded()
{
var spec = new E57Spec { CodecsXml = "<codec><mysteryCodec/></codec>" };
using var dir = new TempDirectory();
using var doc = E57Document.Open(dir.Write("codec.e57", spec.Build()));

Assert.Single(doc.Scans);
var error = Assert.Throws<NotSupportedException>(() => doc.Scans[0].ReadPoints().ToList());
Assert.Contains("mysteryCodec", error.Message, StringComparison.Ordinal);
}

[Fact]
public void E57_With_More_Prototype_Fields_Than_Packets_Can_Hold_Is_Not_Decoded()
{
var many = string.Concat(Enumerable.Repeat("<padding type=\"Integer\" minimum=\"0\" maximum=\"0\"/>", 70000));
string prototype = $"<prototype type=\"Structure\"><cartesianX type=\"Float\" precision=\"single\"/><cartesianY type=\"Float\" precision=\"single\"/><cartesianZ type=\"Float\" precision=\"single\"/>{many}</prototype>";
using var dir = new TempDirectory();
using var doc = E57Document.Open(dir.Write("wide.e57", new E57Spec { PrototypeXml = prototype }.Build()));

Assert.Single(doc.Scans);
var error = Assert.Throws<NotSupportedException>(() => doc.Scans[0].ReadPoints().ToList());
Assert.Contains("65535", error.Message, StringComparison.Ordinal);
}

[Fact]
public void E57_With_A_Prototype_Of_More_Than_A_Thousand_Fields_Is_Decoded()
{
// 1500 unknown fields ride along with the coordinates; the packets hold one stream per field.
var many = string.Concat(Enumerable.Range(0, 1500).Select(i => $"<padding{i} type=\"Integer\" minimum=\"0\" maximum=\"0\"/>"));
string prototype = $"<prototype type=\"Structure\"><cartesianX type=\"Float\" precision=\"single\"/><cartesianY type=\"Float\" precision=\"single\"/><cartesianZ type=\"Float\" precision=\"single\"/>{many}</prototype>";
var streams = new List<byte[]>
{
BitConverter.GetBytes(1.0f), BitConverter.GetBytes(2.0f), BitConverter.GetBytes(3.0f),
};
streams.AddRange(Enumerable.Repeat(new byte[1], 1500));
var spec = new E57Spec
{
PrototypeXml = prototype,
Points = [],
RecordCount = 1,
RawPackets = E57Spec.DataPacket(streams),
};
using var dir = new TempDirectory();
using var doc = E57Document.Open(dir.Write("wide.e57", spec.Build()));

var point = Assert.Single(doc.Scans[0].ReadPoints(NoTransform));
AssertNear(new PclVector3(1, 2, 3), point.Position);
}

// ---- Leica -----------------------------------------------------------------------------------------------

[Fact]
Expand Down
6 changes: 3 additions & 3 deletions tests/PclIO.Tests/InputHardeningTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1071,12 +1071,12 @@ public void An_E57_Prototype_With_More_Fields_Than_The_Limit_Cannot_Be_Decoded()
static string Prototype(int fields) =>
"<prototype type=\"Structure\">" + string.Concat(Enumerable.Range(0, fields).Select(i => $"<f{i} type=\"Integer\" minimum=\"0\" maximum=\"1\"/>")) + "</prototype>";

using var atLimit = E57Document.Open(dir.Write("limit.e57", new E57Spec { PrototypeXml = Prototype(1024) }.Build()));
using var beyond = E57Document.Open(dir.Write("beyond.e57", new E57Spec { PrototypeXml = Prototype(1025) }.Build()));
using var atLimit = E57Document.Open(dir.Write("limit.e57", new E57Spec { PrototypeXml = Prototype(65535) }.Build()));
using var beyond = E57Document.Open(dir.Write("beyond.e57", new E57Spec { PrototypeXml = Prototype(65536) }.Build()));

// Both open; reading says why the points cannot be decoded. The first lacks the coordinate fields, the second has too many fields.
Assert.Contains("lacks cartesianX", Assert.Throws<NotSupportedException>(() => atLimit.Scans[0].ReadPoints().ToList()).Message);
Assert.Contains("too many fields", Assert.Throws<NotSupportedException>(() => beyond.Scans[0].ReadPoints().ToList()).Message);
Assert.Contains("65535", Assert.Throws<NotSupportedException>(() => beyond.Scans[0].ReadPoints().ToList()).Message);
}

// ---- PinPoint limits ------------------------------------------------------------------------------------------
Expand Down
Loading