馃悰 Describe the bug
On Adreno, CNNs lowered to the Vulkan delegate produce different results run to run, at a low but persistent rate, even with the im2col/GEMM path from #21938 disabled. EfficientNetV2-S gives 24 distinct results in 25 replays of the same already-encoded command buffer inside a single process, which rules out load-time state, prepack and weight packing.
Deviation magnitude: EfficientNetV2-S logits span [-1.44, 2.75]; typical deviations are 0.001 to 0.009, with rare excursions to about 4.0. argmax was stable across 30 runs for one input, so class flips need the rare large excursions.
This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug
- A different GPU vendor is clean. Same binaries, same
.pte, Galaxy S10+ (Exynos 9820, Mali-G76): 20 runs, 4020 in-process replays, zero deviations. The probe below that deviates on Adreno gives 0 across 1200 replays there. Output is also numerically correct on Mali, not merely stable: matched Vulkan and CPU exports from one model instance both land within fp32 rounding of the eager reference (9.9e-07 and 3.6e-07).
Confirmed at the application level as well, not just with executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.
Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.
- Khronos synchronization validation reports zero hazards on the failing workload. Layer injected into a debuggable app via
gpu_debug_layers, sync validation confirmed active by the layer's own settings message. No SYNC-HAZARD, no VUID errors of any kind.
- Unconditional full memory barriers change nothing. Patching
CommandBuffer::insert_barrier to always emit a global ALL_COMMANDS -> ALL_COMMANDS barrier with VK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BIT before every dispatch, on top of the computed per-resource barriers, does not help.
The dispatch sequence is also deterministic: an env-gated trace in DispatchNode::encode gives byte-identical traces across runs.
Measurement note, important for anyone reproducing
The failure rate is strongly dependent on GPU power state and swings by more than an order of magnitude for the same .pte and same binary within minutes. Same build, EfficientNetV2-S: 79 distinct / 101 at ~1000 MHz, 13 / 101 at ~826 MHz, 1 / 101 pinned at 461 MHz.
adb shell cmd power set-fixed-performance-mode-enabled true pins the GPU at 461 MHz without root on this device (devfreq/{min,max}_freq are unavailable). Over 4 alternating pairs of 100 in-process replays: 3 excess distinct / 404 pinned versus 30 / 404 unpinned.
Consequences for methodology:
- Any A/B comparison that is not interleaved within a single round is unreliable.
- Patch
executor_runner to dump outputs on every iteration of --num_executions; stock writes only the last. This is far more sensitive than relaunching the process.
- Probe models must be activation calibrated. Untrained stacked convs collapse the signal to ~1e-19, where "distinct output" counts last-bit noise on values carrying no information.
Smallest structure that shows it
12 rounds, order rotated each round, 200 in-process replays per run, calibrated probes, input (1, 48, 64, 64), im2col disabled. Excess distinct outputs per 2400 replays:
| graph |
run A |
run B |
| conv3x3 -> SiLU -> conv1x1 |
43 |
18 |
| conv3x3 -> ReLU -> conv1x1 |
3 |
11 |
| conv3x3 -> Sigmoid -> conv1x1 |
- |
2 |
| conv3x3 -> conv1x1 |
1 |
- |
| conv3x3 -> SiLU -> conv3x3 |
0 |
0 |
conv -> SiLU -> conv3x3 is 0 across 4800 replays while the same graph with a 1x1 consumer is the worst case. With no activation at all, kernel size alone does nothing (0 to 6 per 2400 for all four combinations of 1x1/3x3 producer and consumer). Absolute rates are low, so treat this as a lead rather than a conclusion. It is only visible with DVFS active; pinned at 461 MHz all four collapse to the noise floor.
Forcing 1x1 convs away from Conv2dMethod::Pointwise into the generic sliding window shader does not fix it, so it is not shader identity. The same generic shader is clean at 3x3 and dirty at 1x1.
Ruled out, each tested
Memory aliasing (add_tensor(..., -1) instead of mem_obj_id(); made it worse), storage type, command buffer splitting (execute_threshold_node_count set to SIZE_MAX), descriptor pool recycling (descriptor_pool_safety_factor 1.25 to 64.0), caller threading, host cache coherency, node count.
Possibly the same underlying defect as #21938, which is also Adreno-only but is 100% reproducible and independent of clock.
Versions
ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.
馃悰 Describe the bug
On Adreno, CNNs lowered to the Vulkan delegate produce different results run to run, at a low but persistent rate, even with the im2col/GEMM path from #21938 disabled. EfficientNetV2-S gives 24 distinct results in 25 replays of the same already-encoded command buffer inside a single process, which rules out load-time state, prepack and weight packing.
Deviation magnitude: EfficientNetV2-S logits span [-1.44, 2.75]; typical deviations are 0.001 to 0.009, with rare excursions to about 4.0. argmax was stable across 30 runs for one input, so class flips need the rare large excursions.
This is very likely an Adreno driver or hardware problem, not an ExecuTorch bug
.pte, Galaxy S10+ (Exynos 9820, Mali-G76): 20 runs, 4020 in-process replays, zero deviations. The probe below that deviates on Adreno gives 0 across 1200 replays there. Output is also numerically correct on Mali, not merely stable: matched Vulkan and CPU exports from one model instance both land within fp32 rounding of the eager reference (9.9e-07 and 3.6e-07).Confirmed at the application level as well, not just with
executor_runner: a demo app built from the stock Vulkan backend runs both object detection (YOLO26) and image classification (EfficientNetV2-S) on the S10 with correct and stable results.Separately, and worth knowing for anyone weighing the Vulkan delegate on Mali: on this S10 the Vulkan delegate is close to 2x slower than the XNNPACK equivalent for the same models. That is the opposite of the Adreno device, where Vulkan is clearly faster. The S10 is a 2019 part so this is not necessarily representative of current Mali hardware, but it does mean the delegate is fast on the GPU where it is incorrect and slow on the GPU where it is correct.
gpu_debug_layers, sync validation confirmed active by the layer's own settings message. NoSYNC-HAZARD, no VUID errors of any kind.CommandBuffer::insert_barrierto always emit a globalALL_COMMANDS -> ALL_COMMANDSbarrier withVK_ACCESS_MEMORY_READ_BIT | VK_ACCESS_MEMORY_WRITE_BITbefore every dispatch, on top of the computed per-resource barriers, does not help.The dispatch sequence is also deterministic: an env-gated trace in
DispatchNode::encodegives byte-identical traces across runs.Measurement note, important for anyone reproducing
The failure rate is strongly dependent on GPU power state and swings by more than an order of magnitude for the same
.pteand same binary within minutes. Same build, EfficientNetV2-S: 79 distinct / 101 at ~1000 MHz, 13 / 101 at ~826 MHz, 1 / 101 pinned at 461 MHz.adb shell cmd power set-fixed-performance-mode-enabled truepins the GPU at 461 MHz without root on this device (devfreq/{min,max}_freqare unavailable). Over 4 alternating pairs of 100 in-process replays: 3 excess distinct / 404 pinned versus 30 / 404 unpinned.Consequences for methodology:
executor_runnerto dump outputs on every iteration of--num_executions; stock writes only the last. This is far more sensitive than relaunching the process.Smallest structure that shows it
12 rounds, order rotated each round, 200 in-process replays per run, calibrated probes, input
(1, 48, 64, 64), im2col disabled. Excess distinct outputs per 2400 replays:conv -> SiLU -> conv3x3is 0 across 4800 replays while the same graph with a 1x1 consumer is the worst case. With no activation at all, kernel size alone does nothing (0 to 6 per 2400 for all four combinations of 1x1/3x3 producer and consumer). Absolute rates are low, so treat this as a lead rather than a conclusion. It is only visible with DVFS active; pinned at 461 MHz all four collapse to the noise floor.Forcing 1x1 convs away from
Conv2dMethod::Pointwiseinto the generic sliding window shader does not fix it, so it is not shader identity. The same generic shader is clean at 3x3 and dirty at 1x1.Ruled out, each tested
Memory aliasing (
add_tensor(..., -1)instead ofmem_obj_id(); made it worse), storage type, command buffer splitting (execute_threshold_node_countset toSIZE_MAX), descriptor pool recycling (descriptor_pool_safety_factor1.25 to 64.0), caller threading, host cache coherency, node count.Possibly the same underlying defect as #21938, which is also Adreno-only but is 100% reproducible and independent of clock.
Versions
ExecuTorch 1.4.1. Failing: Galaxy S26 Ultra (SM-S948B), Adreno 840v2. Clean: Galaxy S10+ (SM-G975F), Mali-G76.