Skip to content

Commit f81c675

Browse files
committed
GPU Metal: do not let Metal track hazards on the memory buffers
All memory of the reconstruction lives in three MTLBuffers, so with Metal's default hazard tracking every encoder that writes device memory conflicts with every other one, and Metal ordered them even across the streams' command queues: in a Metal System Trace the per-sector kernels the chain spreads over 8 streams did not overlap at all (kernel time summed to the GPU busy time). The backend orders the work itself, as CUDA streams do: a fence between the encoders of a stream, an event between its command buffers, and the chain's events between streams. So create the buffers untracked; the kernels of different streams now overlap by about 150 ms per event. On an M1 Max, interleaved over 3 rounds of 10 runs (load average 4 to 7), the 30-collision Pb-Pb time frame takes 975 ms per event instead of 1046, and a 10-collision Pb-Pb time frame dumped from the full system test 754 ms instead of 850. The clusters are byte-identical to the CPU's on the 30-collision time frame and on sim5pp.
1 parent 49dec7f commit f81c675

1 file changed

Lines changed: 3 additions & 3 deletions

File tree

‎GPU/GPUTracking/Base/metal/GPUReconstructionMetal.mm‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -116,14 +116,14 @@ uint32_t metalGPUCoreCount()
116116
}
117117

118118
// We only support internal GPUs, so the ownership of the memory is always shared
119-
mInternals->mem_gpu = [mInternals->device newBufferWithLength:mDeviceMemorySize options:MTLResourceStorageModeShared];
119+
mInternals->mem_gpu = [mInternals->device newBufferWithLength:mDeviceMemorySize options:MTLResourceStorageModeShared | MTLResourceHazardTrackingModeUntracked];
120120
if (mInternals->mem_gpu == nil) {
121121
GPUErrorReturn("Metal Memory Allocation Error");
122122
}
123123

124124
// FIXME: until I understand how to enable the gGPUConstantMemBufferSize constexpr
125125
int32_t tmpGPUConstantMemBufferSize = 100000; // gGPUConstantMemBufferSize
126-
mInternals->mem_constant = [mInternals->device newBufferWithLength:tmpGPUConstantMemBufferSize options:MTLResourceStorageModeShared];
126+
mInternals->mem_constant = [mInternals->device newBufferWithLength:tmpGPUConstantMemBufferSize options:MTLResourceStorageModeShared | MTLResourceHazardTrackingModeUntracked];
127127
if (mInternals->mem_constant == nil) {
128128
GPUErrorReturn("Metal Constant Memory Allocation Error");
129129
}
@@ -143,7 +143,7 @@ uint32_t metalGPUCoreCount()
143143
mInternals->streamHasWork[i] = false;
144144
}
145145

146-
mInternals->mem_host = [mInternals->device newBufferWithLength:mHostMemorySize options:MTLResourceStorageModeShared];
146+
mInternals->mem_host = [mInternals->device newBufferWithLength:mHostMemorySize options:MTLResourceStorageModeShared | MTLResourceHazardTrackingModeUntracked];
147147
if (mInternals->mem_host == nil) {
148148
GPUErrorReturn("Error allocating pinned host memory");
149149
}

0 commit comments

Comments
 (0)