diff --git a/CHANGELOG.md b/CHANGELOG.md index ddbd4f3..619d4a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ All notable changes to the `NwcIO` library, the CLI and the `NwcIO.Scene` adapte ## [Unreleased] ### Added +- `meshes-skipped` reader diagnostic: `NwcDocument.Diagnostics` reports geometry records that the lenient reader path skipped (vertex count out of range, truncated or malformed record). `nwcio --report` prints `NwcDocument.Diagnostics` and `NwcAssembly.Diagnostics` for a single file, a folder and an `.nwf`. - `HierarchyParser` keeps linked Revit placements instead of throwing: a repeated element under another parent is marked `NwcNode.IsLinkPlacement` and points at the first placement. `NwcSceneReader` writes each placement as its own `SceneModel`, so its geometry and `SourceId` are preserved without duplicate ids inside one model. - `NwcIO.Scene` converts an `NwcAssembly` into one scene with one `SceneModel` per component document, unique keys from file names, and scene-wide element, geometry and material ids. The `.nwf` document itself is skipped when components exist. - `NwcIO.Scene` writes Revit level elevations in metres from the level's `LcRevitData_Parameter` length property; a missing, non-finite or unknown-unit value stays null. diff --git a/docs/public-api.md b/docs/public-api.md index fec7821..7420e99 100644 --- a/docs/public-api.md +++ b/docs/public-api.md @@ -15,7 +15,7 @@ The supported surface is the `NwcIO` and `NwcIO.Model.*` namespaces listed here | `NwcWriter` | Direct `#LcUStream-V1.1` binary container serializer and continuous zlib chunk writer. Throws `ArgumentException` before writing for input it cannot store faithfully: an unrecognised header, scaled transforms, line meshes, triangle indices outside the vertex buffer, out-of-range vertex counts or coordinates, records or nodes without a name/class, and viewpoint names the reader would rename. Mesh ids, names and triangle indices are stored; normals and materials are not. `Materials`, `XRefs`, `DbCache`, `AppearanceProfiler`, `ClashTests` and `GridSystem` are not written; the output is read back by NwcIO but has not been verified in Autodesk Navisworks. | | `NwcReaderOptions` | Per-open options: which streams to read (property records, GUIDs, transforms, meshes, viewpoints, selection sets, TimeLiner), whether to read materials (`ReadMaterials`: decoded from the file on the strict path, named in property records elsewhere, with a neutral inferred colour unless a `revit_Color` record gives one), whether to build the scene tree (`BuildSceneTree`), catalog retention. | | `NwcAssembly` | Federated multi-model assembly: `Create`, `LoadFromDirectory` (several patterns, recursion), `OpenManifest` (`.nwf.xml`), `OpenNwf` (with `UnresolvedXRefs`/`SkippedXRefs` reporting), combined meshes/materials/bounds, per-assembly exports and clash runs. | -| `NwcDiagnostic` / `NwcDiagnosticSeverity` | Reader warning for skipped or substituted data (`Severity`, stable `Code`, `Message`, optional `StreamIndex` and `ByteOffset`), listed in `NwcDocument.Diagnostics` and `NwcAssembly.Diagnostics`. Codes: `ContainerTrailingGap` (bytes after the last container stream, including a truncated zlib stream), `NwfUnresolvedXref` and `NwfSkippedXref` (from `NwcAssembly.OpenNwf`). | +| `NwcDiagnostic` / `NwcDiagnosticSeverity` | Reader warning for skipped or substituted data (`Severity`, stable `Code`, `Message`, optional `StreamIndex` and `ByteOffset`), listed in `NwcDocument.Diagnostics` and `NwcAssembly.Diagnostics`. Codes: `meshes-skipped` (geometry records dropped because their vertex count is out of range or they are truncated or malformed; not raised on the strict DbCache path), `ContainerTrailingGap` (bytes after the last container stream, including a truncated zlib stream), `NwfUnresolvedXref` and `NwfSkippedXref` (from `NwcAssembly.OpenNwf`). `nwcio --report` prints them. | | `NwcNode` | Scene graph tree node representing layers, categories, composite elements, and geometry instances with optional transformations and properties. For Navisworks model files read through their DbCache directory (the strict path), `Meshes` lists every mesh placed on a leaf and `Mesh` the first, ids are depth-first preorder indices, `Transform` is null; `IsElement` marks Revit element nodes, `ElementGuid` is their export GUID from `LcOpGuidStore`, `IfcGlobalId` is the GlobalId the Revit IFC export gives the element: its valid `IFC_GUID` parameter, else its export GUID in the IFC base-64 form (the rule of `RvtIO.Scene`), and `Element` points every node inside an element (and the second node of a sibling pair that repeats one element) to its element node; properties include the tabs stored in `LcOaPartitionProps`; other files leave `Meshes` empty and `Mesh` null. | | `NwcPropertyRecord` | Property record (`Class`, `Name`, `Value`, `ValueType`, `Tag`, `FirstInt`; typed `IntValue`, `DoubleValue`, `ReferenceId`, `ReferenceTag`, `ReferenceClass`). On node properties of the strict path, `Class` is the property tab name (category, for example `Element`), `Name` the property name and `FirstInt` the key object id. Records scanned from property streams (`NwcDocument.PropertyRecords`, and node properties of other files) keep their internal class names. `TabInternalName` and `InternalName` are the internal names of the tab and the key on strict-path node properties, null on scanned records. | | `NwcPropertyTab` | One property tab of a strict-path node (`NwcNode.PropertyTabs`, in stored slot order): display `Name`, `InternalName`, `Properties`. A tab object shared by several nodes is one instance. | diff --git a/src/NwcIO.Cli/Program.cs b/src/NwcIO.Cli/Program.cs index 7cb0880..9ece2f7 100644 --- a/src/NwcIO.Cli/Program.cs +++ b/src/NwcIO.Cli/Program.cs @@ -1170,6 +1170,7 @@ int ProcessLoadedAssembly(NwcAssembly preloaded, string displayPath, string? out int totalTriangles = doc.Meshes.Sum(m => m.TriangleCount); @err.WriteLine("=== NwcIO Execution Report ==="); + PrintDiagnostics(doc.Diagnostics, @err); @err.WriteLine($"Input: {inputPath}"); @err.WriteLine($"Source document: {doc.FileInfo.SourcePath ?? ""}"); @err.WriteLine($"Loader plugin: {doc.FileInfo.LoaderPlugin ?? ""}"); @@ -1493,6 +1494,7 @@ private static int ProcessAssemblyDirectory( if (printReport) { @err.WriteLine("=== NwcIO Assembly Report ==="); + PrintDiagnostics(assembly.Diagnostics, @err); @err.WriteLine($"Assembly: {assembly.Name}"); @err.WriteLine($"Models: {assembly.Models.Count}"); @err.WriteLine($"Total Meshes: {assembly.AllMeshes.Count}"); @@ -1800,6 +1802,7 @@ private static int ProcessNwfAssembly( @err.WriteLine($"Assembly: {assembly.Name}"); @err.WriteLine($"Models: {assembly.Models.Count}"); } + PrintDiagnostics(assembly.Diagnostics, @err); @err.WriteLine($"Total Meshes: {assembly.AllMeshes.Count}"); @err.WriteLine($"Total Materials: {assembly.AllMaterials.Count}"); @err.WriteLine($"Saved viewpoints: {assembly.SavedViewpoints.Count}"); @@ -1924,6 +1927,22 @@ private static void ApplyClashGrouping(Model.Clash.NwcClashTest test, string? mo } } + private static void PrintDiagnostics(IReadOnlyList diagnostics, TextWriter err) + { + // Unresolved and skipped references already have their own "Warning:" lines in the NWF report. + var shown = diagnostics.Where(d => d.Code is not ("NwfUnresolvedXref" or "NwfSkippedXref")).ToList(); + if (shown.Count == 0) + { + return; + } + + err.WriteLine($"Diagnostics: {shown.Count}"); + foreach (var diagnostic in shown) + { + err.WriteLine($" [{diagnostic.Severity}] {diagnostic.Code}: {diagnostic.Message}"); + } + } + private static void EnsureParentDirectory(string path) { var dir = Path.GetDirectoryName(path); diff --git a/src/NwcIO/Internal/Parsing/MeshParser.cs b/src/NwcIO/Internal/Parsing/MeshParser.cs index b493ecf..0fa0f00 100644 --- a/src/NwcIO/Internal/Parsing/MeshParser.cs +++ b/src/NwcIO/Internal/Parsing/MeshParser.cs @@ -24,6 +24,18 @@ public static IReadOnlyList Parse( IReadOnlyList? transforms = null, int maxVertices = MaxPlausibleVertices, IReadOnlyList? transformIndices = null) + => Parse(container, transforms, maxVertices, transformIndices, out _); + + /// + /// Parses all 3D meshes and reports how many geometry records were recognised but skipped + /// (vertex count out of range, record truncated or malformed). + /// + public static IReadOnlyList Parse( + NwcContainer container, + IReadOnlyList? transforms, + int maxVertices, + IReadOnlyList? transformIndices, + out int skippedRecords) { ArgumentNullException.ThrowIfNull(container); if (maxVertices < MinMeshVertices) @@ -32,20 +44,24 @@ public static IReadOnlyList Parse( } var meshes = new List(); + int tocSkipped = 0; + int scanSkipped = 0; // 1. Primary path: parse via Table of Contents (TOC) stream and chunked geometry block. var tocStream = container.Streams.FirstOrDefault(s => s.Kind == NwcStreamKind.Toc); if (tocStream is not null) { - ParseViaToc(container, tocStream, meshes, transforms, maxVertices, transformIndices); + ParseViaToc(container, tocStream, meshes, transforms, maxVertices, transformIndices, ref tocSkipped); } // 2. Secondary fallback: if TOC yielded no meshes, scan individual streams for vertex buffers. if (meshes.Count == 0) { - ParseViaHeuristicScan(container, meshes, transforms, maxVertices, transformIndices); + ParseViaHeuristicScan(container, meshes, transforms, maxVertices, transformIndices, ref scanSkipped); } + // The fallback replaces the TOC attempt; take the larger count so neither path hides a skipped record. + skippedRecords = Math.Max(tocSkipped, scanSkipped); return meshes; } @@ -81,7 +97,8 @@ private static void ParseViaToc( List meshes, IReadOnlyList? transforms, int maxVertices, - IReadOnlyList? transformIndices) + IReadOnlyList? transformIndices, + ref int skippedRecords) { var tocData = tocStream.Data; if (tocData.Length < 16) @@ -192,6 +209,14 @@ private static void ParseViaToc( meshes.Add(mesh); } + else + { + skippedRecords++; + } + } + else + { + skippedRecords++; } } } @@ -205,7 +230,8 @@ private static void ParseViaHeuristicScan( List meshes, IReadOnlyList? transforms, int maxVertices, - IReadOnlyList? transformIndices) + IReadOnlyList? transformIndices, + ref int skippedRecords) { foreach (var stream in container.Streams) { @@ -228,12 +254,14 @@ private static void ParseViaHeuristicScan( int minVertices = lines ? 2 : MinMeshVertices; if (vCount < minVertices || vCount > maxVertices) { + skippedRecords++; continue; } // vCount can reach int.MaxValue through MaxMeshVertices: compare without multiplying. if (vCount > (data.Length - (i + 16)) / 12) { + skippedRecords++; continue; } @@ -245,6 +273,10 @@ private static void ParseViaHeuristicScan( meshes.Add(mesh); i += 12 + vCount * 12; } + else + { + skippedRecords++; + } } } } diff --git a/src/NwcIO/NwcDocument.cs b/src/NwcIO/NwcDocument.cs index c95627d..ba4244b 100644 --- a/src/NwcIO/NwcDocument.cs +++ b/src/NwcIO/NwcDocument.cs @@ -183,6 +183,7 @@ public static NwcDocument Open(Stream stream, NwcReaderOptions? options = null, var transforms = new List(); var meshes = new List(); + int skippedMeshRecords = 0; var materials = new List(); NwcNode? rootNode = null; if (segments is not null) @@ -216,7 +217,7 @@ public static NwcDocument Open(Stream stream, NwcReaderOptions? options = null, transformIndices = TransformLinkParser.TryParse(linksStream); } - meshes.AddRange(MeshParser.Parse(container, transforms, options.MaxMeshVertices, transformIndices)); + meshes.AddRange(MeshParser.Parse(container, transforms, options.MaxMeshVertices, transformIndices, out skippedMeshRecords)); } if (options.BuildSceneTree) @@ -279,18 +280,27 @@ public static NwcDocument Open(Stream stream, NwcReaderOptions? options = null, } var document = new NwcDocument(fileInfo, records, guids, transforms, meshes, materials, rootNode, xRefs, dbCache, savedViewpoints, selectionSets, timeLiner, container); + var diagnostics = new List(); + if (skippedMeshRecords > 0) + { + diagnostics.Add(new NwcDiagnostic( + NwcDiagnosticSeverity.Warning, + "meshes-skipped", + $"{skippedMeshRecords} geometry records were skipped: their vertex count is outside " + + $"{MeshParser.MinMeshVertices}..{options.MaxMeshVertices} or the record is malformed.")); + } + if (container.TrailingGap > 0) { - document.Diagnostics = - [ - new NwcDiagnostic( - NwcDiagnosticSeverity.Warning, - "ContainerTrailingGap", - $"Unrecognized {container.TrailingGap} byte tail after the last stream.", - ByteOffset: container.FileLength - container.TrailingGap), - ]; + diagnostics.Add(new NwcDiagnostic( + NwcDiagnosticSeverity.Warning, + "ContainerTrailingGap", + $"Unrecognized {container.TrailingGap} byte tail after the last stream.", + ByteOffset: container.FileLength - container.TrailingGap)); } + document.Diagnostics = diagnostics; + return document; } diff --git a/tests/NwcIO.Tests/CliArgumentValidationTests.cs b/tests/NwcIO.Tests/CliArgumentValidationTests.cs index 91c2488..85a35e4 100644 --- a/tests/NwcIO.Tests/CliArgumentValidationTests.cs +++ b/tests/NwcIO.Tests/CliArgumentValidationTests.cs @@ -740,4 +740,97 @@ public void ManifestInput_IsAnAssembly() Directory.Delete(dir, true); } } + + private static void AppendTail(string path) + { + using var stream = new FileStream(path, FileMode.Append, FileAccess.Write); + stream.Write(new byte[8]); + } + + [Fact] + public void Report_PrintsDocumentDiagnostics_OnlyWhenPresent() + { + string model = TempModel(); + string clean = Path.Combine(Path.GetTempPath(), "cli-diag-" + Guid.NewGuid().ToString("N") + ".json"); + string output = Path.Combine(Path.GetTempPath(), "cli-diag-" + Guid.NewGuid().ToString("N") + ".json"); + try + { + var (cleanExit, _, cleanErr) = Run([model, clean, "--report"]); + Assert.Equal(0, cleanExit); + Assert.DoesNotContain("Diagnostics:", cleanErr); + + AppendTail(model); + var (exit, _, err) = Run([model, output, "--report"]); + + Assert.Equal(0, exit); + Assert.Contains("Diagnostics: 1", err); + Assert.Contains("[Warning] ContainerTrailingGap", err); + } + finally + { + foreach (var f in new[] { model, clean, output }) + { + if (File.Exists(f)) File.Delete(f); + } + } + } + + [Fact] + public void Report_PrintsAssemblyDiagnostics_ForFolderAndManifest() + { + string dir = Path.Combine(Path.GetTempPath(), "cli-diag-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + try + { + string a = Path.Combine(dir, "a.nwc"); + WriteModel(a); + AppendTail(a); + WriteModel(Path.Combine(dir, "b.nwc")); + + var (folderExit, _, folderErr) = Run([dir, "--report"]); + Assert.True(folderExit == 0, folderErr); + Assert.Contains("Diagnostics: 1", folderErr); + Assert.Contains("ContainerTrailingGap", folderErr); + + string manifest = Path.Combine(dir, "fed.nwf.xml"); + var m = new NwcIO.Model.Clash.NwcNwfManifest("Fed"); + m.ModelPaths.Add("a.nwc"); + m.ModelPaths.Add("b.nwc"); + m.Save(manifest); + + var (exit, _, err) = Run([manifest, "--report"]); + Assert.True(exit == 0, err); + Assert.Contains("Diagnostics: 1", err); + Assert.Contains("ContainerTrailingGap", err); + } + finally + { + Directory.Delete(dir, true); + } + } + + [Fact] + public void Report_UnresolvedReference_IsReportedOnce() + { + string dir = Path.Combine(Path.GetTempPath(), "cli-diag-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(dir); + try + { + WriteModel(Path.Combine(dir, "a.nwc")); + string manifest = Path.Combine(dir, "fed.nwf.xml"); + var m = new NwcIO.Model.Clash.NwcNwfManifest("Fed"); + m.ModelPaths.Add("a.nwc"); + m.ModelPaths.Add("missing.nwc"); + m.Save(manifest); + + var (exit, _, err) = Run([manifest, "--report"]); + + Assert.True(exit == 0, err); + Assert.Equal(1, err.Split("missing.nwc").Length - 1); + } + finally + { + Directory.Delete(dir, true); + } + } } diff --git a/tests/NwcIO.Tests/MeshParserTests.cs b/tests/NwcIO.Tests/MeshParserTests.cs index f1d5f59..0a8e5eb 100644 --- a/tests/NwcIO.Tests/MeshParserTests.cs +++ b/tests/NwcIO.Tests/MeshParserTests.cs @@ -58,6 +58,101 @@ public void Open_MalformedTocRecord_SkipsMesh(int p1, int p2, int recordLength) Assert.Empty(MeshParser.Parse(container)); } + [Fact] + public void Parse_TocSkippedRecordNextToValidMesh_CountsSkippedRecord() + { + // The valid first record keeps the heuristic fallback from running, so only the TOC path counts. + var triangle = Int32Bytes(0, 94, 0, 60, 3, 9, + 0, 0, 0, + BitConverter.SingleToInt32Bits(1f), 0, 0, + 0, BitConverter.SingleToInt32Bits(1f), 0); + var skipped = Int32Bytes(0, 94, 0, 60, 100, 0, 0, 0); + var chunk = triangle.Concat(skipped).ToArray(); + var toc = Int32Bytes(2, 1, chunk.Length, triangle.Length, skipped.Length); + var container = NwcContainerReader.Read(NwcContainerWriter.Write([chunk, toc])); + container.Streams[1].Kind = NwcStreamKind.Toc; + + var meshes = MeshParser.Parse(container, null, MeshParser.MaxPlausibleVertices, null, out int skippedRecords); + + Assert.Single(meshes); + Assert.Equal(1, skippedRecords); + } + + [Fact] + public void Parse_ValidRecords_CountNoSkippedRecords() + { + var triangle = Int32Bytes(0, 94, 0, 60, 3, 9, + 0, 0, 0, + BitConverter.SingleToInt32Bits(1f), 0, 0, + 0, BitConverter.SingleToInt32Bits(1f), 0); + var toc = Int32Bytes(1, 1, triangle.Length, triangle.Length); + var container = NwcContainerReader.Read(NwcContainerWriter.Write([triangle, toc])); + container.Streams[1].Kind = NwcStreamKind.Toc; + + MeshParser.Parse(container, null, MeshParser.MaxPlausibleVertices, null, out int skippedRecords); + + Assert.Equal(0, skippedRecords); + } + + [Fact] + public void Parse_LoneTruncatedRecord_CountsSkippedRecord() + { + // A vertex count in range but more bytes than present is the typical truncation. No mesh is found, + // so the heuristic fallback replaces the TOC attempt and must count the record as well. + var chunk = Int32Bytes(0, 94, 0, 60, 100, 0, 0, 0); + var toc = Int32Bytes(1, 1, chunk.Length, 1224); + var container = NwcContainerReader.Read(NwcContainerWriter.Write([chunk, toc])); + container.Streams[1].Kind = NwcStreamKind.Toc; + + var meshes = MeshParser.Parse(container, null, MeshParser.MaxPlausibleVertices, null, out int skippedRecords); + + Assert.Empty(meshes); + Assert.Equal(1, skippedRecords); + } + + [Fact] + public void Open_TruncatedGeometryRecord_ReportsMeshesSkipped() + { + var chunk = Int32Bytes(0, 94, 0, 60, 100, 0, 0, 0); + var toc = Int32Bytes(1, 1, chunk.Length, 1224); + using var file = new MemoryStream(NwcContainerWriter.Write([chunk, toc])); + + var diagnostic = Assert.Single(NwcDocument.Open(file).Diagnostics); + + Assert.Equal("meshes-skipped", diagnostic.Code); + Assert.Equal(NwcDiagnosticSeverity.Warning, diagnostic.Severity); + Assert.StartsWith("1 geometry records were skipped", diagnostic.Message); + } + + [Fact] + public void Open_OutOfRangeVertexCount_ReportsCapFromOptions() + { + var chunk = Int32Bytes(0, 94, 0, 60, 5, 0, 0, 0); + var toc = Int32Bytes(1, 1, chunk.Length, chunk.Length); + using var file = new MemoryStream(NwcContainerWriter.Write([chunk, toc])); + + var diagnostic = Assert.Single(NwcDocument.Open(file, new NwcReaderOptions { MaxMeshVertices = 4 }).Diagnostics); + + Assert.Equal("meshes-skipped", diagnostic.Code); + Assert.Contains("3..4", diagnostic.Message); + } + + [Fact] + public void Open_ModelWithoutSkippedRecords_HasNoDiagnostics() + { + var triangle = Int32Bytes(0, 94, 0, 60, 3, 9, + 0, 0, 0, + BitConverter.SingleToInt32Bits(1f), 0, 0, + 0, BitConverter.SingleToInt32Bits(1f), 0); + var toc = Int32Bytes(1, 1, triangle.Length, triangle.Length); + using var file = new MemoryStream(NwcContainerWriter.Write([triangle, toc])); + + var doc = NwcDocument.Open(file); + + Assert.Single(doc.Meshes); + Assert.Empty(doc.Diagnostics); + } + [Theory] [InlineData(536870912)] [InlineData(int.MaxValue)]