馃悰 Describe the bug
On Adreno, a single Conv2d lowered to the Vulkan delegate produces a different result on almost every execution, for a specific band of output channel counts. Identical .pte, identical input, same already-encoded command buffer replayed in one process.
Probe: Conv2d(48, c_out, 3, padding=1, bias=False) on a (1, 48, 64, 64) input, 200 in-process replays per run, order rotated across runs, cool device.
| c_out |
excess distinct outputs |
replays |
runs with any deviation |
| 144 |
0 |
1600 |
0 / 8 |
| 160 |
0 |
1600 |
0 / 8 |
| 168 |
1142 |
1800 |
9 / 9 |
| 176 |
427 |
800 |
4 / 4 |
| 184 |
770 |
1000 |
5 / 5 |
| 192 |
772 |
1200 |
6 / 6 |
| 200 |
0 |
1400 |
0 / 7 |
| 224 |
0 |
600 |
0 / 3 |
Perfect separation. Every run of every c_out in {168, 176, 184, 192} deviates; no run of {144, 160, 200, 224} deviates across 5200 replays.
The band lies inside the im2col/GEMM convolution path: should_use_conv2d_im2col() routes any conv with c_out >= kIm2colMinCOut (128) there. Forcing that function to return false, so the direct conv2d shader is used, makes all of these deterministic. Note 160 and 200 are also in that path and are clean, so it is a size band inside the path, not the threshold itself.
This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug
Three independent checks point away from ExecuTorch:
- A different GPU vendor is clean. The same binaries and the same
.pte files on a Galaxy S10+ (SM-G975F, Exynos 9820, Mali-G76, Android 12): all eight c_out values, 200 replays each, zero deviations. This is the strongest signal, because should_use_conv2d_im2col() returns graph.device_is_mali() || c_out >= kIm2colMinCOut, so on Mali every conv takes the im2col path, including 144 and 160 which do not on Adreno. If the im2col shader itself were wrong, Mali should have been the worse device.
Confirmed at the application level as well, not just with executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.
Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.
- Output is numerically correct on Mali, not merely stable. Exporting a Vulkan
.pte and a CPU .pte from the same model instance and comparing both against the eager reference gives 9.9e-07 (Vulkan) and 3.6e-07 (CPU), i.e. fp32 rounding.
- Khronos synchronization validation reports zero hazards on the failing workload (layer injected into a debuggable app via
gpu_debug_layers, sync validation confirmed active by the layer's own settings message). No SYNC-HAZARD, no VUID errors.
So the return false workaround most likely helps by avoiding whatever the Adreno driver miscompiles, rather than by avoiding an incorrect shader.
Reproducing
Counting distinct md5 of executor_runner stdout across repeated process launches is a poor measure: it is noisy, and on this device the failure rate of the related issue #21939 depends strongly on GPU power state. Patch executor_runner to dump outputs on every iteration of --num_executions (stock writes only the last), then count distinct outputs across in-process replays. This one is stable and reproduces at 215 to 262 MHz on a cool device, so unlike #21939 it is not sensitive to clock.
Possibly the same underlying defect as #21939, which is also Adreno-only but low-rate and power-state dependent.
Versions
ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.
馃悰 Describe the bug
On Adreno, a single
Conv2dlowered to the Vulkan delegate produces a different result on almost every execution, for a specific band of output channel counts. Identical.pte, identical input, same already-encoded command buffer replayed in one process.Probe:
Conv2d(48, c_out, 3, padding=1, bias=False)on a(1, 48, 64, 64)input, 200 in-process replays per run, order rotated across runs, cool device.Perfect separation. Every run of every
c_outin {168, 176, 184, 192} deviates; no run of {144, 160, 200, 224} deviates across 5200 replays.The band lies inside the im2col/GEMM convolution path:
should_use_conv2d_im2col()routes any conv withc_out >= kIm2colMinCOut(128) there. Forcing that function toreturn false, so the directconv2dshader is used, makes all of these deterministic. Note 160 and 200 are also in that path and are clean, so it is a size band inside the path, not the threshold itself.This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug
Three independent checks point away from ExecuTorch:
.ptefiles on a Galaxy S10+ (SM-G975F, Exynos 9820, Mali-G76, Android 12): all eightc_outvalues, 200 replays each, zero deviations. This is the strongest signal, becauseshould_use_conv2d_im2col()returnsgraph.device_is_mali() || c_out >= kIm2colMinCOut, so on Mali every conv takes the im2col path, including 144 and 160 which do not on Adreno. If the im2col shader itself were wrong, Mali should have been the worse device.Confirmed at the application level as well, not just with
executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.
.pteand a CPU.ptefrom the same model instance and comparing both against the eager reference gives 9.9e-07 (Vulkan) and 3.6e-07 (CPU), i.e. fp32 rounding.gpu_debug_layers, sync validation confirmed active by the layer's own settings message). NoSYNC-HAZARD, no VUID errors.So the
return falseworkaround most likely helps by avoiding whatever the Adreno driver miscompiles, rather than by avoiding an incorrect shader.Reproducing
Counting distinct md5 of
executor_runnerstdout across repeated process launches is a poor measure: it is noisy, and on this device the failure rate of the related issue #21939 depends strongly on GPU power state. Patchexecutor_runnerto dump outputs on every iteration of--num_executions(stock writes only the last), then count distinct outputs across in-process replays. This one is stable and reproduces at 215 to 262 MHz on a cool device, so unlike #21939 it is not sensitive to clock.Possibly the same underlying defect as #21939, which is also Adreno-only but low-rate and power-state dependent.
Versions
ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.