diff --git a/CHANGELOG.md b/CHANGELOG.md index 298c20d..ee4d870 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ All notable changes to the `NwcIO` library, the CLI and the `NwcIO.Scene` adapte ### Added - Cancellation and progress overloads for `NwcDocument.Open`, `NwcAssembly.LoadFromDirectory` (also with several patterns and recursion) and `NwcAssembly.OpenNwf` and `NwcAssembly.OpenManifest`, plus `OpenAsync`, `LoadFromDirectoryAsync`, `OpenNwfAsync` and `SaveAsync` (`.nwc`) wrappers on `Task.Run`. `NwcProgress` reports `read-container`, `read-geometry`, `open` and per-model `load-models` counts (`OpenNwf` counts every reference, so `Completed` reaches `Total`). The token is checked between phases and models; a running container read is not interrupted, and `SaveAsync` checks it only before it starts. - `meshes-skipped` reader diagnostic: `NwcDocument.Diagnostics` reports geometry records that the lenient reader path skipped (vertex count out of range, truncated or malformed record). `nwcio --report` prints `NwcDocument.Diagnostics` and `NwcAssembly.Diagnostics` for a single file, a folder and an `.nwf`. +- Reader diagnostics also cover property-scanner failures (`property-scan-failed`, Warning: the first malformed string a property scan met, with its stream index and byte offset; not raised on the strict DbCache path) and are written to document and assembly JSON as a `diagnostics` array. `NwcReaderOptions.Strict` throws `InvalidDataException` when an error-level diagnostic is recorded. - `HierarchyParser` keeps linked Revit placements instead of throwing: a repeated element under another parent is marked `NwcNode.IsLinkPlacement` and points at the first placement. `NwcSceneReader` writes each placement as its own `SceneModel`, so its geometry and `SourceId` are preserved without duplicate ids inside one model. - `NwcIO.Scene` converts an `NwcAssembly` into one scene with one `SceneModel` per component document, unique keys from file names, and scene-wide element, geometry and material ids. The `.nwf` document itself is skipped when components exist. - `NwcIO.Scene` writes Revit level elevations in metres from the level's `LcRevitData_Parameter` length property; a missing, non-finite or unknown-unit value stays null. diff --git a/docs/public-api.md b/docs/public-api.md index 2c18136..21e9015 100644 --- a/docs/public-api.md +++ b/docs/public-api.md @@ -14,9 +14,9 @@ The supported surface is the `NwcIO` and `NwcIO.Model.*` namespaces listed here | `NwcProgress` | Progress payload for the cancellation-aware overloads: a phase name, completed units, and total units (0 when unknown). | | `NwcDocumentBuilder` | Fluent builder API for creating Navisworks documents and geometry programmatically from code. | | `NwcWriter` | Direct `#LcUStream-V1.1` binary container serializer and continuous zlib chunk writer. Throws `ArgumentException` before writing for input it cannot store faithfully: an unrecognised header, scaled transforms, line meshes, triangle indices outside the vertex buffer, out-of-range vertex counts or coordinates, records or nodes without a name/class, and viewpoint names the reader would rename. Mesh ids, names and triangle indices are stored; normals and materials are not. `Materials`, `XRefs`, `DbCache`, `AppearanceProfiler`, `ClashTests` and `GridSystem` are not written; the output is read back by NwcIO but has not been verified in Autodesk Navisworks. | -| `NwcReaderOptions` | Per-open options: which streams to read (property records, GUIDs, transforms, meshes, viewpoints, selection sets, TimeLiner), whether to read materials (`ReadMaterials`: decoded from the file on the strict path, named in property records elsewhere, with a neutral inferred colour unless a `revit_Color` record gives one), whether to build the scene tree (`BuildSceneTree`), catalog retention. | +| `NwcReaderOptions` | Per-open options: which streams to read (property records, GUIDs, transforms, meshes, viewpoints, selection sets, TimeLiner), whether to read materials (`ReadMaterials`: decoded from the file on the strict path, named in property records elsewhere, with a neutral inferred colour unless a `revit_Color` record gives one), whether to build the scene tree (`BuildSceneTree`), catalog retention, and `Strict` (throw `InvalidDataException` when an error-level diagnostic is recorded). | | `NwcAssembly` | Federated multi-model assembly: `Create`, `LoadFromDirectory` (several patterns, recursion), `OpenManifest` (`.nwf.xml`), `OpenNwf` (with `UnresolvedXRefs`/`SkippedXRefs` reporting), combined meshes/materials/bounds, per-assembly exports and clash runs. | -| `NwcDiagnostic` / `NwcDiagnosticSeverity` | Reader warning for skipped or substituted data (`Severity`, stable `Code`, `Message`, optional `StreamIndex` and `ByteOffset`), listed in `NwcDocument.Diagnostics` and `NwcAssembly.Diagnostics`. Codes: `meshes-skipped` (geometry records dropped because their vertex count is out of range or they are truncated or malformed; not raised on the strict DbCache path), `ContainerTrailingGap` (bytes after the last container stream, including a truncated zlib stream), `NwfUnresolvedXref` and `NwfSkippedXref` (from `NwcAssembly.OpenNwf`). `nwcio --report` prints them. | +| `NwcDiagnostic` / `NwcDiagnosticSeverity` | Reader warning for skipped or substituted data (`Severity`, stable `Code`, `Message`, optional `StreamIndex` and `ByteOffset`), listed in `NwcDocument.Diagnostics` and `NwcAssembly.Diagnostics`. Codes: `meshes-skipped` (geometry records dropped because their vertex count is out of range or they are truncated or malformed; not raised on the strict DbCache path), `property-scan-failed` (the first malformed string met by the lenient property scan, with its stream index and byte offset; not raised on the strict DbCache path), `ContainerTrailingGap` (bytes after the last container stream, including a truncated zlib stream), `NwfUnresolvedXref` and `NwfSkippedXref` (from `NwcAssembly.OpenNwf`). They are included in JSON export (`diagnostics`) and printed by `nwcio --report`. | | `NwcNode` | Scene graph tree node representing layers, categories, composite elements, and geometry instances with optional transformations and properties. For Navisworks model files read through their DbCache directory (the strict path), `Meshes` lists every mesh placed on a leaf and `Mesh` the first, ids are depth-first preorder indices, `Transform` is null; `IsElement` marks Revit element nodes, `ElementGuid` is their export GUID from `LcOpGuidStore`, `IfcGlobalId` is the GlobalId the Revit IFC export gives the element: its valid `IFC_GUID` parameter, else its export GUID in the IFC base-64 form (the rule of `RvtIO.Scene`), and `Element` points every node inside an element (and the second node of a sibling pair that repeats one element) to its element node; properties include the tabs stored in `LcOaPartitionProps`; other files leave `Meshes` empty and `Mesh` null. | | `NwcPropertyRecord` | Property record (`Class`, `Name`, `Value`, `ValueType`, `Tag`, `FirstInt`; typed `IntValue`, `DoubleValue`, `ReferenceId`, `ReferenceTag`, `ReferenceClass`). On node properties of the strict path, `Class` is the property tab name (category, for example `Element`), `Name` the property name and `FirstInt` the key object id. Records scanned from property streams (`NwcDocument.PropertyRecords`, and node properties of other files) keep their internal class names. `TabInternalName` and `InternalName` are the internal names of the tab and the key on strict-path node properties, null on scanned records. | | `NwcPropertyTab` | One property tab of a strict-path node (`NwcNode.PropertyTabs`, in stored slot order): display `Name`, `InternalName`, `Properties`. A tab object shared by several nodes is one instance. | diff --git a/src/NwcIO/Internal/Json/JsonExporter.cs b/src/NwcIO/Internal/Json/JsonExporter.cs index 8e91590..854c385 100644 --- a/src/NwcIO/Internal/Json/JsonExporter.cs +++ b/src/NwcIO/Internal/Json/JsonExporter.cs @@ -291,6 +291,28 @@ public static void Write(NwcDocument document, Stream output, NwcJsonExportOptio writer.WriteEndArray(); } + if (document.Diagnostics.Count > 0) + { + writer.WriteStartArray("diagnostics"); + foreach (var diagnostic in document.Diagnostics) + { + writer.WriteStartObject(); + writer.WriteString("severity", diagnostic.Severity.ToString()); + writer.WriteString("code", diagnostic.Code); + writer.WriteString("message", diagnostic.Message); + if (diagnostic.StreamIndex is int streamIndex) + { + writer.WriteNumber("streamIndex", streamIndex); + } + if (diagnostic.ByteOffset is long byteOffset) + { + writer.WriteNumber("byteOffset", byteOffset); + } + writer.WriteEndObject(); + } + writer.WriteEndArray(); + } + if (options.IncludeStreamCatalog) { WriteStreamCatalog(writer, document); diff --git a/src/NwcIO/Internal/Parsing/PropertyRecordScanner.cs b/src/NwcIO/Internal/Parsing/PropertyRecordScanner.cs index 8b6a82e..a25c6c0 100644 --- a/src/NwcIO/Internal/Parsing/PropertyRecordScanner.cs +++ b/src/NwcIO/Internal/Parsing/PropertyRecordScanner.cs @@ -36,12 +36,16 @@ internal sealed class PropertyRecordScanner public string? FailureReason { get; private set; } + /// Byte offset of the first malformed probe; null when no probe failed. + public int? FailureOffset { get; private set; } + public List Scan() { var records = new List(); _pos = 0; StoppedAt = 0; FailureReason = null; + FailureOffset = null; while (_pos + 8 <= _data.Length) { @@ -207,12 +211,24 @@ private bool TryReadLengthPrefixedString(ref int pos, out string? value) value = null; if (pos + 4 > _data.Length) { + FailureReason ??= $"A length-prefixed string at byte {pos} runs past the end of the stream."; + FailureOffset ??= pos; return false; } int len = ReadInt32At(pos); - if (len is < 0 or > 65536 || pos + 4 + len > _data.Length) + if (len == -1) + { + // The format uses -1 for a null string; it is valid in both strict and lenient readers. + value = null; + pos += 4; + return true; + } + + if (len < 0 || len > 65536 || pos + 4 + len > _data.Length) { + FailureReason ??= $"A length-prefixed string at byte {pos} has invalid length {len}."; + FailureOffset ??= pos; return false; } diff --git a/src/NwcIO/Model/NwcAssembly.cs b/src/NwcIO/Model/NwcAssembly.cs index 0dcd887..1a353f5 100644 --- a/src/NwcIO/Model/NwcAssembly.cs +++ b/src/NwcIO/Model/NwcAssembly.cs @@ -930,6 +930,28 @@ public void SaveJson(Stream stream, NwcJsonExportOptions? options = null) writer.WriteString("size", BoundingBox.Size.ToString()); writer.WriteEndObject(); + if (Diagnostics.Count > 0) + { + writer.WriteStartArray("diagnostics"); + foreach (var diagnostic in Diagnostics) + { + writer.WriteStartObject(); + writer.WriteString("severity", diagnostic.Severity.ToString()); + writer.WriteString("code", diagnostic.Code); + writer.WriteString("message", diagnostic.Message); + if (diagnostic.StreamIndex is int streamIndex) + { + writer.WriteNumber("streamIndex", streamIndex); + } + if (diagnostic.ByteOffset is long byteOffset) + { + writer.WriteNumber("byteOffset", byteOffset); + } + writer.WriteEndObject(); + } + writer.WriteEndArray(); + } + writer.WriteStartArray("models"); foreach (var m in Models) { diff --git a/src/NwcIO/NwcDocument.cs b/src/NwcIO/NwcDocument.cs index 61ac269..b1b8afa 100644 --- a/src/NwcIO/NwcDocument.cs +++ b/src/NwcIO/NwcDocument.cs @@ -184,6 +184,7 @@ public static NwcDocument Open( string rootName = Path.GetFileNameWithoutExtension(fileInfo.SourcePath ?? fallbackName ?? "Model"); + var diagnostics = new List(); var records = new List(); cancellationToken.ThrowIfCancellationRequested(); if (options.ReadPropertyRecords) @@ -192,6 +193,16 @@ public static NwcDocument Open( { var scanner = new PropertyRecordScanner(s.Data); records.AddRange(scanner.Scan()); + if (segments is null && scanner.FailureReason is not null) + { + // The lenient scanner searches forward; a rejected probe is not a lost record. + diagnostics.Add(new NwcDiagnostic( + NwcDiagnosticSeverity.Warning, + "property-scan-failed", + scanner.FailureReason, + s.Index, + scanner.FailureOffset)); + } } } @@ -311,9 +322,6 @@ public static NwcDocument Open( } cancellationToken.ThrowIfCancellationRequested(); - progress?.Report(new NwcProgress("open", 1, 1)); - var document = new NwcDocument(fileInfo, records, guids, transforms, meshes, materials, rootNode, xRefs, dbCache, savedViewpoints, selectionSets, timeLiner, container); - var diagnostics = new List(); if (skippedMeshRecords > 0) { diagnostics.Add(new NwcDiagnostic( @@ -332,6 +340,17 @@ public static NwcDocument Open( ByteOffset: container.FileLength - container.TrailingGap)); } + if (options.Strict) + { + var errors = diagnostics.Where(d => d.Severity == NwcDiagnosticSeverity.Error).ToList(); + if (errors.Count > 0) + { + throw new InvalidDataException(string.Join(" ", errors.Select(d => d.Message))); + } + } + + progress?.Report(new NwcProgress("open", 1, 1)); + var document = new NwcDocument(fileInfo, records, guids, transforms, meshes, materials, rootNode, xRefs, dbCache, savedViewpoints, selectionSets, timeLiner, container); document.Diagnostics = diagnostics; return document; diff --git a/src/NwcIO/NwcReaderOptions.cs b/src/NwcIO/NwcReaderOptions.cs index 9b3fd26..ff8757e 100644 --- a/src/NwcIO/NwcReaderOptions.cs +++ b/src/NwcIO/NwcReaderOptions.cs @@ -63,4 +63,10 @@ public sealed class NwcReaderOptions /// Model files read through their DbCache directory use the strict geometry parser and its own bounds. /// public int MaxMeshVertices { get; set; } = Internal.Parsing.MeshParser.MaxPlausibleVertices; + + /// + /// Throw when the reader records an error-level diagnostic + /// (default: false, diagnostics are only recorded). + /// + public bool Strict { get; set; } } diff --git a/tests/NwcIO.Tests/DiagnosticsExportTests.cs b/tests/NwcIO.Tests/DiagnosticsExportTests.cs new file mode 100644 index 0000000..d0bc93b --- /dev/null +++ b/tests/NwcIO.Tests/DiagnosticsExportTests.cs @@ -0,0 +1,105 @@ +using System.Buffers.Binary; +using System.Text; +using NwcIO.Internal.Container; +using NwcIO.Model; +using Xunit; + +namespace NwcIO.Tests; + +public sealed class DiagnosticsExportTests +{ + [Fact] + public void Open_MalformedPropertyRecord_ReportsPropertyScanFailedWarning() + { + using var stream = new MemoryStream(MalformedSceneTreeFile()); + + var document = NwcDocument.Open(stream); + + var diagnostic = Assert.Single(document.Diagnostics, d => d.Code == "property-scan-failed"); + Assert.Equal(NwcDiagnosticSeverity.Warning, diagnostic.Severity); + Assert.Contains("invalid length", diagnostic.Message); + Assert.Equal(0, diagnostic.StreamIndex); + Assert.Equal(16L, diagnostic.ByteOffset); + } + + [Fact] + public void Open_StrictWithMalformedPropertyRecord_DoesNotThrow() + { + using var stream = new MemoryStream(MalformedSceneTreeFile()); + + var document = NwcDocument.Open(stream, new NwcReaderOptions { Strict = true }); + + Assert.DoesNotContain(document.Diagnostics, d => d.Severity == NwcDiagnosticSeverity.Error); + } + + [Fact] + public void JsonExport_IncludesDocumentAndAssemblyDiagnostics() + { + using var stream = new MemoryStream(MalformedSceneTreeFile()); + var document = NwcDocument.Open(stream); + + string json = document.ToJson(); + Assert.Contains("\"diagnostics\"", json); + Assert.Contains("property-scan-failed", json); + + var assembly = NwcAssembly.Create("Fed", document); + using var assemblyStream = new MemoryStream(); + assembly.SaveJson(assemblyStream); + string assemblyJson = Encoding.UTF8.GetString(assemblyStream.ToArray()); + Assert.Contains("\"diagnostics\"", assemblyJson); + Assert.Contains("property-scan-failed", assemblyJson); + } + + [Fact] + public void Open_NullPropertyString_IsValidAndStrictDoesNotThrow() + { + var data = new byte[20]; + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(0), 100); + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(4), 86); + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(8), 0); + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(12), 53); + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(16), -1); + byte[] file = NwcContainerWriter.Write([data]); + + var document = NwcDocument.Open(new MemoryStream(file), new NwcReaderOptions { Strict = true }); + + Assert.DoesNotContain(document.Diagnostics, d => d.Code == "property-scan-failed"); + } + + [Fact] + public void Scan_NullValueString_KeepsRecordAndAdvancesPastPrefix() + { + // tag 52 record: name "a", class "b", valueType 4 (string) with a null (-1) value, then tag 53 node "n". + var ms = new MemoryStream(); + using (var w = new BinaryWriter(ms, Encoding.UTF8, leaveOpen: true)) + { + w.Write(1); w.Write(52); + w.Write(1); w.Write(new byte[] { (byte)'a', 0, 0, 0 }); + w.Write(1); w.Write(new byte[] { (byte)'b', 0, 0, 0 }); + w.Write(4); w.Write(-1); + w.Write(2); w.Write(53); + w.Write(1); w.Write(new byte[] { (byte)'n', 0, 0, 0 }); + } + + var scanner = new NwcIO.Internal.Parsing.PropertyRecordScanner(ms.ToArray()); + var records = scanner.Scan(); + + Assert.Equal(2, records.Count); + Assert.Null(records[0].Value); + Assert.Equal("n", records[1].Name); + Assert.Null(scanner.FailureReason); + } + + private static byte[] MalformedSceneTreeFile() + { + // The first two ints classify the stream as SceneTree; tag 53 at byte 12 has an invalid -1 length. + var data = new byte[24]; + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(0), 100); + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(4), 86); + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(8), 0); + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(12), 53); + BinaryPrimitives.WriteInt32LittleEndian(data.AsSpan(16), -2); + return NwcContainerWriter.Write([data]); + } +} +