ENH: Out-of-core architecture rewrite and filter optimizations - #1568
Open
joeykleingers wants to merge 7 commits into
Open
ENH: Out-of-core architecture rewrite and filter optimizations#1568joeykleingers wants to merge 7 commits into
joeykleingers wants to merge 7 commits into
Conversation
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
March 24, 2026 18:13
b4ef97f to
99b49ed
Compare
joeykleingers
requested review from
JDuffeyBQ,
imikejackson,
jmarquisbq,
mmarineBlueQuartz and
nyoungbq
March 24, 2026 18:17
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
2 times, most recently
from
March 24, 2026 18:51
b4ef97f to
bb09048
Compare
This was referenced Mar 24, 2026
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
4 times, most recently
from
April 2, 2026 00:55
102c436 to
b4c1358
Compare
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
6 times, most recently
from
April 8, 2026 17:41
2bd614a to
110c054
Compare
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
4 times, most recently
from
April 16, 2026 13:03
35aecd0 to
3a88bbf
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
April 27, 2026 18:18
bdfed87 to
6fbfc8d
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
June 12, 2026 15:32
ec3a5e7 to
3c8903e
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
2 times, most recently
from
June 25, 2026 17:25
ed64934 to
a56f168
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
3 times, most recently
from
July 13, 2026 18:29
c6bb7fe to
ee6b370
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
2 times, most recently
from
July 15, 2026 17:22
5a8c281 to
b61d3fd
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
5 times, most recently
from
July 30, 2026 14:22
9e11033 to
03ac304
Compare
joeykleingers
marked this pull request as ready for review
July 30, 2026 14:27
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
7 times, most recently
from
August 10, 2026 20:33
89e66d2 to
f106b35
Compare
Add storage-neutral backing-store selection and bulk I/O contracts so core
filters can choose between direct in-memory and bounded-memory algorithms at
runtime. Split affected SimplnxCore and OrientationAnalysis implementations
into direct, scanline, BFS, CCL, and worklist paths without depending on a
concrete out-of-core backend. Expand algorithm-path tests, storage-policy
coverage, documentation, and build wiring.
663 files changed, +84,941 / -20,875 lines.
================================================================================
1. Define storage policy and bulk data-store interfaces
================================================================================
Files: src/simplnx/Core/{Application,Preferences}.{cpp,hpp},
src/simplnx/DataStructure/{AbstractDataStore,DataStore,EmptyDataStore,
IDataStore}.{hpp}, src/simplnx/DataStructure/IO/{Generic,HDF5}/...,
src/simplnx/Filter/Actions/...
Introduce Adaptive, ForceInCore, and ForceOutOfCore storage modes with migration
from the legacy preference keys. Let each DataStructure resolve registered
storage formats through a storage-neutral IDataStoreFormatResolver, and expose
format registration, base-directory, and manager lookup behavior through the
application and I/O collection APIs.
Add contiguous copy and N-dimensional extent operations to the common data-store
contract, with in-memory, empty, HDF5, string, and NeighborList implementations.
Route array and geometry creation actions through the selected format so storage
intent applies consistently to newly allocated data.
================================================================================
2. Add runtime dispatch and bounded-memory infrastructure
================================================================================
Files: docs/AlgorithmDispatch.md, src/simplnx/Common/Extent.hpp,
src/simplnx/Utilities/{AlgorithmDispatch,BoundedRecordPageCache,
ExternalEquivalence,InMemoryTemporaryRecordStore,MemoryBudgetManager,
SliceBufferedTransfer,UnionFind}.*,
src/simplnx/DataStructure/IO/Generic/{IExternalSort,
ITemporaryRecordStore}.hpp
Extend algorithm dispatch to mixed IArray targets, including NeighborLists, and
select implementations from storage residency plus explicit test overrides.
Record the algorithm/store combination that actually executes so tests can
witness dispatch rather than infer it from configuration.
Add reusable extent, temporary-record, external-equivalence, page-cache,
memory-budget, slice-transfer, and union-find primitives for algorithms that
must bound resident memory and preserve cancellation and error propagation.
Update shared data-array, segmentation, geometry, clustering, meshing, and file
utilities to use the common bulk-access contracts.
================================================================================
3. Provide storage-aware SimplnxCore algorithm paths
================================================================================
Files: src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/...,
src/Plugins/SimplnxCore/src/SimplnxCore/Filters/...,
src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp
Route array statistics and histograms, clustering, thresholding, tuple transfer,
feature operations, partitioning, silhouette generation, and surface meshing
through direct or scanline implementations selected from the participating
stores. This includes dedicated paths for DBSCAN, MultiThresholdObjects,
QuickSurfaceMesh, RemoveFlaggedFeatures, SurfaceNets, and related filters.
Add BFS and connected-component alternatives for FillBadData and IdentifySample,
and move affected readers, writers, segmentation utilities, and geometry
algorithms onto buffered or bulk data access so the scanline paths avoid
full-volume materialization.
================================================================================
4. Provide storage-aware OrientationAnalysis algorithm paths
================================================================================
Files: src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/
Algorithms/..., src/Plugins/OrientationAnalysis/src/
OrientationAnalysis/Filters/...,
src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/...
Add direct and scanline implementations for GBCD pole figures, IPF colors,
kernel average misorientations, quaternion conjugation, and other
orientation-array operations. Add scanline and worklist processing for bad-data
neighbor correction, and adapt feature segmentation, section alignment,
crystallographic calculations, EBSD readers, and export algorithms to the
storage-neutral access layer.
================================================================================
5. Expand path-specific tests, documentation, and build integration
================================================================================
Files: test/..., test/UnitTestCommon/..., src/Plugins/SimplnxCore/test/...,
src/Plugins/OrientationAnalysis/test/...,
src/Plugins/{SimplnxCore,OrientationAnalysis}/docs/...,
CMakeLists.txt, cmake/..., vcpkg.json
Add AlgorithmTestScope and dispatch-state support that run explicitly selected
in-core and out-of-core algorithm implementations on in-memory stores and verify
the implementation that executed. Extend filter coverage for both paths and add
focused tests for extents, storage-mode migration, format resolution, temporary
records, memory budgets, memory safety, and DREAM3D/HDF5 loading behavior.
Register the new sources, test modes, exported configuration, and dependencies,
and document the storage-sensitive behavior of the migrated filters and dispatch
APIs.
================================================================================
Verification
================================================================================
- clang-format verified the three conflict-resolved C++ files using the
repository-root .clang-format.
- git diff --check passed for the rebased tree.
- CreateFeatureArrayFromElementArray passed 12/12 in-core tests and 12/12
OOC-build tests, including all upstream AF-1 through AF-9 V&V cases.
- The private 200x200x200 disk-backed OOC test passed 3,495 assertions.
- The in-core all-target build passed. The OOC simplnx_test target compiled
successfully against the live SimplnxOoc backend.
- Dream3dLoadingApi and ComputeCAxisLocations passed 15/15 in-core tests
and 19/19 OOC-build tests, including the private disk-backed cases.
Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
August 11, 2026 14:38
f106b35 to
d54fab8
Compare
* Allocate bounded OIM and OINA scratch pages on the heap to keep them resident without exhausting the default Windows thread stack. * Apply the clang-format 15 layout required by CI. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Serialize HDF5 C API access within file, group, dataset, object, attribute, and DREAM3D reader leaf operations through the shared non-recursive ApiLock. Keep nested self-locking calls outside leaf critical sections and close transient HDF5 identifiers under the same lock. 9 files changed, +966 / -479 lines. ================================================================================ 1. HDF5 leaf-level serialization ================================================================================ Files: H5Support.hpp, DatasetIO.cpp, FileIO.cpp, GroupIO.cpp, ObjectIO.cpp, ObjectIO.hpp Wrap file, group, dataset, dataspace, property-list, object, and attribute HDF5 calls with the process-wide API mutex. Resolve lazy-opening and other self-locking operations before entering leaf critical sections to preserve the public-locks/private-unlocked contract and avoid recursive deadlocks. Keep related HDF5 resource lifetimes and error-handler changes within the serialized scopes while translating errors after releasing the lock. ================================================================================ 2. DREAM3D attribute import ================================================================================ Files: Dream3dIO.cpp, ObjectIO.cpp, ObjectIO.hpp Inspect legacy attribute types under the shared HDF5 lock, then release it before dispatching to self-locking string and vector readers. Serialize attribute enumeration, type and dataspace queries, reads, writes, deletes, and cleanup while preserving typed DataArray conversion behavior. ================================================================================ 3. Concurrent HDF5 regression coverage ================================================================================ Files: test/CMakeLists.txt, test/H5IOTest.cpp Add a multithreaded regression test that repeatedly opens one fixture file, queries groups and datasets, and validates full-array and hyperslab reads. Exercise concurrent open, query, read, close, and destructor paths to detect HDF5 races, data corruption, and non-recursive-lock deadlocks. Verification ============ clang-format 15.0.7 --dry-run --Werror passed for all changed C++ files. git diff --cached --check passed. Builds and tests were not rerun as part of this history-only squash.
* Remove progress and status emissions introduced by storage-neutral paths * Preserve pre-existing diagnostics and output behavior Signed-off-by: Joey Kleingers <joey.kleingers@gmail.com>
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
August 13, 2026 18:45
ec2e32c to
09d552a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch does two things at once:
Reworks the core out-of-core (OOC) data-storage architecture in
simplnxso that disk-backed storage is supplied by a runtime-registered IO manager through clean interfaces, with the core library holding no reference to any concrete OOC format or symbol. Core gains a bulk-I/O data-store API (copyIntoBuffer/copyFromBuffer), an injectable store-format resolver, IO-manager lifecycle hooks, a tri-state storage-mode preference, a memory-budget manager, and memory-safety guards. The OOC implementation itself (chunked HDF5 stores, the chunk cache, deflate decompression) lives in a separate privateSimplnxOocsource set and is not part of this diff — it plugs into core at runtime through these interfaces.Adds chunk-aware (OOC-optimized) algorithm variants for ~50 filters across the SimplnxCore and OrientationAnalysis plugins, plus shared core utilities that benefit many more. Random-access algorithms that thrash the chunk cache on disk-backed arrays now either stream data with bulk I/O or dispatch to a sequential "scanline / CCL" variant, eliminating per-element virtual dispatch and HDF5 chunk churn. In-core behavior and performance are preserved via a runtime storage-type check, so optimized filters keep two code paths: one for in-memory data, one for disk-backed data.
Scope of this diff
develop…HEAD: 378 files, +42,853 / −13,112.src/simplnx(core library)Plugins/SimplnxCorePlugins/OrientationAnalysisHow to review
The diff is large but highly patterned, and the two layers are independent:
src/simplnx/DataStructure/IO/Generic/*,Core/Preferences.*,Utilities/{MemoryBudgetManager,AlgorithmDispatch,SegmentFeatures,UnionFind,SliceBufferedTransfer}.*, andFilter/IFilter.cpp(Part 1).Part 1 — Core out-of-core architecture (
src/simplnx)1.1 Bulk-I/O data-store API
copyIntoBuffer/copyFromBufferonAbstractDataStore<T>(a flat range form and a tuple-addressed form) are the single mechanism for bulk reads/writes, implemented byDataStore<T>andEmptyDataStore<T>; the OOC implementation supplies its own override.IDataStore::StoreType { InMemory, OutOfCore, Empty }, queried viagetStoreType(). The chunk-traversal API (loadChunk,getNumberOfChunks,getChunkLowerBounds,getChunkUpperBounds,getChunkShape) and the chunk-shape-based OOC detection are removed — bulk I/O plusgetStoreType()replace them entirely.EmptyStringStore,AbstractStringStore/StringStoreplaceholder support, andStringArray::isPlaceholder().1.2 Runtime store-format resolution and IO-manager hooks
OOC capability is supplied entirely at runtime; core never names the concrete format.
IDataStoreFormatResolver: a const, thread-safe policy interface deciding which registered format a soon-to-be-created array uses (""= in-memory).InMemoryFormatResolveris the default policy.ArrayCreationUtilities::ResolveStorageFormatis the single decision point for every array-creation call site, with a fixed precedence: the unstructured/poly-geometry gate (ParentGeometrySupportsOoc) forces in-core, then an explicit per-filter format override, then theDataStructure's resolver.DataStructurecarries a per-instance resolver plus a lazily-seeded process-wide default (neither serialized).IDataIOManagerlifecycle hooks (default no-ops):finalizesImport,onImportFinalize,onRecoveryWrite,onFinalizeStores,setBaseDirectory,shutdownManager.DataIOCollectionfans these out to every registered manager, so an OOC manager participates in import finalization, recovery writes, store read-only transition, and shutdown without core knowing the specifics.CoreDataIOManager: the always-present default manager that registers the in-memory data-store/list-store factories; an OOC manager registers on top at runtime.CreateDataStore/CreateListStoresingle entry points;CreateNeighborListAction/CreateNeighborsthread an explicitdataFormatoverride through to the store.1.3 Storage-mode preference (+ legacy migration)
DataStorageMode { Adaptive, ForceInCore, ForceOutOfCore }(persisted asdata_storage_mode) is the single source of truth for storage placement, replacing thelarge_data_format/force_ooc_datapreferences. The enum is deliberately OOC-vocabulary-free — core states user intent; the registered manager maps it to a concrete format.dataStorageMode()migrates older preference files from the retained legacy keys;useOocData()is a convenience view (true unlessForceInCore).Preferencesseeds the OOC base directory and default format on startup via the IO-collection hooks, and gains aremoveValuehelper.1.4 Memory budget + memory safety
MemoryBudgetManager: tracks the budget governing in-core vs OOC placement.defaultBudgetBytes()(50% of RAM, ≥1 GiB, clamped to the cap),maxBudgetBytes()(max(min(total−6 GiB, 0.95·total), 1 GiB)), andsetBudgetBytes()which clamps to the cap and reports whether it clamped. Total-RAM detection is centralized inMemory::GetTotalMemory(). The default is clamped to the cap so it never exceeds it on <12 GiB machines (e.g. CI runners).nxrunner --memory-budget <GB>: CLI flag wired through to the budget manager so the override takes effect in headless runs, with parsing hardened against NaN/inf/trailing garbage.-264total-RAM hard block):-271— non-blocking preflight warning when an in-core array would exceed currently-available RAM. OOC arrays are excluded (EmptyDataStore::memoryUsage()reports 0 for OOC placeholders; the format is resolved in preflight via the sharedResolveStorageFormathelper).-272— astd::bad_allocsafety net at the singleIFilter::execute → executeImplboundary, turning an out-of-memory condition into a clean pipeline error instead of a crash.1.5 HDF5 streaming write + compression
DatasetIOprovides the streaming OOC write path —createEmptyDataset+writeSpanHyperslabfor arrays too large to be resident — alongside the single-shotwriteSpan; reads usereadChunk/readChunkIntoSpan.createEmptyDatasetbuilds the chunked-deflate creation property list (BuildChunkedDeflateDcpl), so a large OOC array taking the two-step streaming write is compressed identically to the single-shot path. Contiguous storage is still used for compression level 0 and for arrays below the small-array threshold.1.6
.dream3dloader APIDream3dIOexposes aLoadDataStructurefamily:LoadDataStructure,LoadDataStructureMetadata(metadata-only for preflight),LoadDataStructureArrays(array subset), plus resolver-aware overloads that stamp a per-DataStructureresolver before import finalization (so a read-only visualization load can direct arrays to disk for fast first-show). Import is eager or deferred based onanyManagerFinalizesImport(); writes run under a recovery-write guard supplied by the registered manager.ImportH5ObjectPathsActionuses this API: metadata-only on preflight, full load on execute, with a selective shortest-path-first merge of only the requested paths.1.7 Core algorithm infrastructure
AlgorithmDispatch.hpp—DispatchAlgorithm<InCoreAlgo, OocAlgo>(arrays, args…), a free function template selecting between an in-core and an OOC algorithm class at runtime. Priority:ForceInCoreAlgorithm()> any array OOC >ForceOocAlgorithm()> in-core default. RAII test guardsForceOocAlgorithmGuard(bool)andForceInCoreAlgorithmGuardlet a single build exercise both paths.SegmentFeaturesOOC path —executeCCL(): Z-slice connected-component labeling with a 2-slice rolling buffer andUnionFindequivalence tracking (Face and FaceEdgeVertex connectivity, optional periodic BCs), replacing random-access BFS/DFS flood-fill on disk-backed data.UnionFind— vector-based disjoint set with union-by-rank and path-halving.SliceBufferedTransfer— type-dispatched Z-slice buffered tuple copy for morphological / neighbor-replacement transfer phases.Extent— region/range math helper (with unit tests).AlignSectionsOOC path — bulk slice read/write transfer for the align-sections family.1.8 Core utility bulk-I/O conversions
These live in core utilities and benefit every caller; each is guarded by a runtime storage-type check that preserves the original in-core code path:
DataArrayUtilities—ImportFromBinaryFile,AppendData,CopyData, and the mirrorswap_rangesops route through chunked bulk I/O when OOC. (PowersReadRawBinaryandAppendImageGeometry's mirror.)DataGroupUtilities::RemoveInactiveObjects— chunked featureIds renumbering viacopyIntoBuffer/copyFromBuffer.ClusteringUtilities::RandomizeFeatureIds— chunked bulk I/O (both overloads; benefits segmentation filters, SharedFeatureFace, MergeTwins).GeometryHelpers—FindElementsContainingVert/FindElementNeighborsuse 65K-element chunked passes with a current-chunk cache-hit check before falling back to per-element reads.ImageRotationUtilities— Z-slab source cache for nearest-neighbor and a ±2-slice trilinear margin, sliding-window slab updates (memmove + delta reads), and intra-slice parallelism. This is howApplyTransformationToGeometryandRotateSampleRefFrameget their OOC speedups (no plugin algorithm file changes for ApplyTransformation).TriangleUtilities— bulk-load triangles/labels for winding repair.H5DataStore— streaming row-batchFillOocDataStorereplacing full-dataset allocation on import.RectGridGeom/ImageGeomfindElementSizes— route through the resolver-awareCreateDataStore(voxel-sizes array can go OOC); the RectGrid inner loop refactored to per-axis precompute + Z-slicecopyFromBuffer.Part 2 — Filter optimizations
Optimization patterns
Every filter optimization is one of these four shapes:
DispatchAlgorithm<…Direct, …Scanline>: an in-coreDirectclass (unchanged or parallel) and an OOCScanlineclass that streams Z-slices / chunks. The originalFoo.cppbecomes a thin dispatcher.DispatchAlgorithm<…BFS, …CCL>(or an in-file branch): random-access flood-fill in-core, sequential Z-slice connected-component labeling for OOC.std::vectors, with a runtime storage-type check so in-core stays optimal.SliceBufferedTransfer, or an OOC-correctness guard (e.g. disabling threading for OOC stores) / progress-and-cancel additions.Algorithm structure: in-core + out-of-core variants
For dispatch-split filters the original algorithm file is renamed to the in-core variant and the OOC variant is added beside it; the original filename remains as a thin dispatcher:
FillBadDataBFSFillBadDataCCLIdentifySampleBFSIdentifySampleCCL…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Worklist…Scanline…Direct…ScanlineNew shared header
IdentifySampleCommon.hpp(SimplnxCore) provides theVectorUnionFindand per-slice functor shared by the BFS/CCL variants;TupleTransfer.hppgainsquickSurfaceTransferBatch/surfaceNetsTransferBatchbulk APIs used by the mesh Scanline variants.SimplnxCore inventory
DispatchAlgorithm<FillBadDataBFS, FillBadDataCCL>; CCL streams Z-slices with coreUnionFindDispatchAlgorithm<IdentifySampleBFS, IdentifySampleCCL>; scanline labeling viaVectorUnionFindSegmentFeatures::executeCCL()(Z-slice CCL)copyIntoBuffer(256K-tuple)canMergefindClusters; bounded per-cluster peak memorynodeIdsfor rolling 2-plane node buffers;quickSurfaceTransferBatchsurfaceNetsTransferBatchstd::vectoraccumulators (no DataStore); 64K-tuple chunked featureIdsassignBadVoxelsvoting; sparse changed-voxel trackingtriMaskbitset + sparsetriPrefixSumpopcount (~6.4× less memory); 65K-element streamed passesZSliceWorkerparallel Z-slice rasterize → mutex-guarded bulkcopyFromBufferSliceBufferedTransferper-Z commit + Z-slice neighbor readsSliceBufferedTransferper-Z commitSliceBufferedTransferper-Z best-neighbor commitfindShiftsOoc()with per-Z-slice mask readscopyIntoBufferfor coordinate/data outputOrientationAnalysis inventory
DispatchAlgorithm<…Direct, …Scanline>; Direct keeps parallel in-core, Scanline 65K-tuple chunks + cached crystal structures; color key forwarded to the OOC pathDispatchAlgorithm<…Worklist, …Scanline>; bool-mask reads routed through bulk I/OSegmentFeatures::executeCCL()slice-by-slice, replacing DFS flood-fillfindShiftsOoc(), 2-slice buffered quats/phases/mask + cached crystal structuresSliceBufferedTransfer; cached crystal structures[plane−kZ, plane+kZ]slab reads; cached crystal structurescopyFromBuffertriIncluded); feature-level caching; raw-pointer parallel selectorfmt::memory_buffercopyFromBufferfor all cell arrays; chunked Euler interleave; in-place phase validationcopyFromBufferin CopyData template, phase copy, Euler interleavecopyFromBufferfrom raw HDF5 reader bufferscopyFromBufferfor intensity/image outputsCancel + progress
In-core and OOC variants gained
m_ShouldCancelchecks at the top of major loops andThrottledMessenger-based progress reporting with per-phase messages and percent-complete.Part 3 — Performance results
All benchmarks use arm64 Release builds. OOC measurements force the disk-backed path through
DataStorageMode::ForceOutOfCoreorForceOocAlgorithmGuard.CRITICAL filter optimizations (existing unit-test data, filter.execute() only)
Small existing datasets do not expose every asymptotic win: mesh bucketing/pruning and bounded-memory streaming can add fixed overhead at this scale. All 17 filters passed their affected suites in both configurations; the table reports measured values rather than hiding those small-test regressions.
Large synthetic filter benchmarks (filter.execute() only)
HIGH and MEDIUM benchmarks use 8,000,000 cells (200³) except the 2D-only tests (2048²), the generated Channel5 reader dataset (8,000 × 1,000), and CSV's 8,000,000 rows. Figures are medians of five execute-only runs for the MEDIUM batches and repeated runs for HIGH where practical;
>values are conservative timeout bounds. Every listed filter passed its normal affected suite and hidden large benchmark in both configurations.Additional 200³ execute-only OOC benchmarks
Full-test wall-clock OOC benchmarks
Geometry / Mesh / Phase Filters
ImageRotationUtilitiesslab cachePart 4 — Test infrastructure
UnitTestCommon.hpp):CompareDataArrays,CompareDataArraysByComponent,CompareArrays,CompareFloatArraysWithNansstream both arrays in 40K-element chunks viacopyIntoBufferinstead of per-elementoperator[](per-element access on OOC arrays is pathologically slow).ExpectedStoreType()(derives the expectedStoreTypefrom the activeDataStorageMode+ whether an OOC manager is registered) andRequireExpectedStoreType().PreferencesSentineltakes aDataStorageModeand restores the original preferences on destruction.TestFileSentinelreference-counts archive extraction via per-process holder files, so parallel test runs don't delete shared decompressed data prematurely.LoadDataStructuretest helper callsDREAM3D::LoadDataStructure(path)directly.SegmentFeaturesTestUtils.hpp(~638 lines): shared builders/verifiers for the SegmentFeatures family.ForceOocAlgorithmGuard+GENERATE(from_range(k_ForceOocTestValues))runs each case in both in-core and forced-OOC modes — adopted by 23 plugin test files (16 SimplnxCore, 7 OrientationAnalysis).test/:DataStoreFormatResolverTest,DataStorageModeMigrationTest,Dream3dLoadingApiTest,EmptyStringStoreTest,ExtentTest,MemoryBudgetManagerTest,MemorySafetyTest,IParallelAlgorithmTest.RotateSampleRefFrame/RotateEulerRefFrametest paths use slab/chunked bulk I/O.Part 5 — Build system
cmake/SimplnxConfig.hpp.inis a generated PUBLIC, ODR-safe config header.SIMPLNX_TEST_ALGORITHM_PATHcache option (default0):0=Both,1=OOC-only,2=InCore-only; plumbed into every plugin test as a compile definition (cmake/Plugin.cmake). An in-core build can validate both algorithm paths (forcing the Scanline path against in-core data still runs correctly and fast).SIMPLNX_UNIT_TEST_TARGETSglobal property —create_simplnx_plugin_unit_testrecords each test target so consumer (add_subdirectory) builds can attach extra sources/settings.zlibas a direct vcpkg dependency — the OOC layer compiled into consumers uses it directly for parallel deflate-chunk decompression off the global HDF5 mutex (andUnitTestCommonuses it for the in-house tar.gz extractor)./bigobjfor the MSVC build ofDream3dIO.cpp(exceeds the COMDAT limit / C1128 in Debug).Extent,EmptyStringStore,IDataStoreFormatResolver,InMemoryFormatResolver,MemoryBudgetManager,AlgorithmDispatch,SliceBufferedTransfer,UnionFind,IdentifySampleCommon).Part 6 — Documentation
src/Plugins/*/docs/(27 SimplnxCore, 21 OrientationAnalysis; ~+1,177 lines), each adding an## Algorithmsection with### Performanceand paired In-Core / Out-of-Core subsections that explain the dual implementation, memory footprint, and chunk/slab streaming strategy. No docs deleted.Part 7 — Test data archives
Three
download_test_data()entries added:fill_bad_data_exemplars.tar.gz(SimplnxCore)identify_sample_exemplars.tar.gz(SimplnxCore)segment_features_exemplars.tar.gz(referenced by both SimplnxCore and OrientationAnalysis)No existing archive entries were removed.
Related PR
Test Plan
SIMPLNX_TEST_ALGORITHM_PATH=2) passesSIMPLNX_TEST_ALGORITHM_PATH=1) passesSIMPLNX_TEST_ALGORITHM_PATH=0) pass