[maleadt]

Since AcceleratedKernels 0.5.7 (released 2026-10-17), every ALCF job fails, and main's next one will too. On Aurora with Julia 1.13, cumprod and cumprod! of a ComplexF32 vector return wrong results (GPUArrays test/testsuite/accumulate.jl:13 and :15). This PR skips ComplexF32 in the accumulate tests when ONEAPI_LTS=1 and documents the bug as a known LTS issue in docs/src/lts.md. The skip uses the existing per-test supported_eltypes filter, so other test files still exercise ComplexF32 on LTS. Locally, the accumulate tests run 328 tests with ONEAPI_LTS=1 and 362 without. The difference is exactly the ComplexF32 testset.

This only gets CI green. Aurora users still get wrong ComplexF32 cumprod results until IGC is fixed.

@michel2323, a report for you below: what we know, what to run on Aurora, and what to ask Intel.

Failure

Seen in both Julia 1.13 runs since AK 0.5.7: 39018094747 (#709) and 39060435368 (#704, GitLab pipeline 70454, PBS job 8939625).

Component	Version
Device	Data Center GPU Max 1550, ZE_FLAT_DEVICE_HIERARCHY=FLAT, one tile per worker
Driver	system LTS driver, reports v1.3.33578 (NEO 25.18.33578)
IGC	the system IGC configured by the ALCF julia modulefile (2.11.x, LLVM 15; please confirm the exact build)
SPIR-V generator	Khronos SPIRV-LLVM-Translator 23.1.1 (ONEAPI_LTS=1)
Julia	1.13.2 fails, 1.10.12 passes
Packages	oneAPI.jl 2.9.2, AcceleratedKernels 0.5.7, GPUArrays 12.1.1, KernelInterface 0.4.1

What passes and what fails

Configuration	Result
Aurora, Julia 1.13, cumprod ComplexF32	wrong (2 of 2 runs)
Aurora, Julia 1.13, cumsum ComplexF32 (up to 100 000 elements)	passes
Aurora, Julia 1.13, cumprod Complex{Int32}, ComplexF16, ComplexF64, Float32	passes
Aurora, Julia 1.10, same packages	passes
Aurora, Julia 1.13, AK ≤ 0.5.6	passed (every run before 2026-10-17)
Iris Xe, rolling NEO 26.35 / IGC 2.41.5, LTS or rolling code path	0/200 wrong
Iris Xe, NEO 25.18.33578.6 + IGC 2.11.7 (public LTS release), Julia 1.13, ONEAPI_LTS=1	0/100 wrong
PVC with the rolling driver or a newer IGC	untested

What we know about the kernel

• AK 0.5.7 added sub-group block scans (Faster scans: sub-group block scans, and reduce-then-scan in ScanPrefixes AcceleratedKernels.jl#163). The failing kernel is _slice_scan_tiles!, compiled with SubgroupSize 32. ComplexF32 has no native sub-group shuffle, so each value is shuffled as two 32-bit words with OpGroupNonUniformShuffle.
• Julia 1.13's LLVM 20 emits different SPIR-V than 1.10's LLVM 15. The scan loop is unrolled further (184 vs 64 OpFMul), and the shuffle lane index max(lane - 1, 0) becomes the OpenCL builtin u_sub_sat.
• Neither difference explains the failure by itself. The passing Julia 1.13 kernels for Float32 and Complex{Int32} also contain u_sub_sat and the same two-word shuffles. What only the failing kernel has is complex float multiplication combined with those shuffles.
• Compiled offline with the 25.18 ocloc -device pvc, every kernel is SIMD32 with 128 GRFs and similar flag spills. Nothing stands out, so finding the bug needs PVC hardware.

What to run on Aurora first

Run this on one tile, in an environment with oneAPI.jl and AcceleratedKernels 0.5.7 (also in the gist):

repro.jl

# ComplexF32 cumprod on the Aurora LTS stack. Usage:
#   ONEAPI_LTS=1 julia +1.13 --project=<env with oneAPI, AcceleratedKernels 0.5.7> repro.jl [nosubgroups]
# `nosubgroups` makes AcceleratedKernels scan in local memory instead of with sub-group shuffles.
using oneAPI
import AcceleratedKernels as AK
if "nosubgroups" in ARGS
    AK.scan_tuning(::oneAPIBackend, ::Type) = AK.ScanTuning(subgroups=false)
end
println("Julia ", VERSION, ", AK ", pkgversion(AK), ", LTS=", oneAPI.oneL0.LTS[],
        ", subgroups=", !("nosubgroups" in ARGS))
println("device: ", oneAPI.properties(device()).name)
for T in (ComplexF32, Complex{Int32}, Float32)
    bad = 0
    for i in 1:100
        x = T <: Complex{<:Integer} ? T.(rand(-3:3, 10), rand(-3:3, 10)) : rand(T, 10)
        y = Array(cumprod(oneArray(x)))
        if !(y ≈ cumprod(x))
            bad += 1
            bad == 1 && println("  first mismatch for $T:\n    input:    ", x,
                                "\n    expected: ", cumprod(x), "\n    got:      ", y)
        end
    end
    println(rpad(string(T), 16), bad, "/100 wrong")
end

ONEAPI_LTS=1 julia +1.13 --project=ENV repro.jl              # expected: ComplexF32 wrong
ONEAPI_LTS=1 julia +1.13 --project=ENV repro.jl nosubgroups  # AK scans in local memory instead
ONEAPI_LTS=1 julia +1.10 --project=ENV repro.jl              # control: expected to pass

1. Confirm the failure on a single tile, outside the test suite.
2. nosubgroups tells us whether the sub-group shuffle path is involved. If it passes, AK could switch that path off on LTS as a stopgap.
3. Rolling driver on PVC. Drop the system-library preferences so oneAPI.jl uses its bundled NEO 26.35 / IGC 2.41.5, and keep ONEAPI_LTS=1 so the SPIR-V is identical. This only works if Aurora's kernel driver accepts the newer user-mode driver. If it passes, the bug is fixed upstream and the fix needs a backport. If it fails, it's a current IGC bug on PVC.
4. If (3) isn't possible, an IGC knob that changes the result also helps Intel narrow it down. Candidates: IGC_DisableLoopUnroll=1, IGC_DisableCodeSinking=1, IGC_VISAOptions="-noSchedule".

What to ask Intel

New, wrong results: IGC miscompiles a SIMD32 sub-group scan on Xe-HPC (PVC). A SIMD32 kernel that scans ComplexF32 values with complex multiplication and OpGroupNonUniformShuffle gives wrong results on a Data Center GPU Max 1550 with the 25.18 LTS driver (IGC 2.11.x). The same SPIR-V gives correct results on Xe-LP with the same driver. SPIR-V for the same kernel from an older LLVM gives correct results on PVC. Ask Intel to reproduce it on PVC and to check whether current IGC is affected. If it's fixed upstream, ask them to backport the fix to the LTS branch Aurora runs. If not, ask them to fix it and backport. Include the results of steps 1–3. The reproducers are in this gist: repro.jl, plus the SPIR-V of the failing kernel, of its passing Julia 1.10 version, and of two passing Julia 1.13 kernels, as the exact binaries the driver received (clone the gist to get them) and as disassembly, with the command to compile them for PVC. The kernel's arguments are Julia structs, so repro.jl is the runnable reproducer and the SPIR-V is for compiler-side inspection.

Other LTS items that need a fix or backport from Intel (oneAPI.jl works around all of them):

Issue	Upstream status	Ask
NEO SIGSEGV handler race: using oneAPI plus a multithreaded GC crashes (#607). The shared handlerIndex in PageFaultManagerLinux goes out of bounds.	Fix proposed in intel/compute-runtime#1025; shipped in NEO_jll 26.35.39758+1	Backport to the 25.18 LTS driver
IGC segfaults on a private array whose address escapes to a non-inlined function (LowerGEPForPrivMem SOA promotion). We skip KernelAbstractions' "Private" tests on LTS.	Masked since IGC 2.22.0 (3e2f5492a); still latent on master	Backport, or set IGC_DisableSOAPromotion=1 on Aurora
IGC rejects SPIR-V from LLVM's SPIR-V back-end (undefined reference to _Z17__spirv_IAddCarryll, #708). This is why LTS must use the Khronos translator.	Fixed by KhronosGroup/SPIRV-LLVM-Translator#2877, backported to llvm_release_160; IGC needs to build against a translator that includes it	Rebuild the LTS IGC against such a translator
Strided (non-coalesced) global reads in reduction kernels give wrong results.	Not seen on the rolling stack	Find and backport the fix (we materialize strided inputs instead)
ZE_extension_memory_free_policies is advertised, but BLOCKING_FREE is not honored: a free with work in flight faults and bans the context.	Not seen on the rolling stack	Backport (we drain streams before every free)
Declaring SPV_KHR_bfloat16 crashes the runtime; the translator path can't codegen bfloat.	Works on the rolling stack	Low priority; BFloat16 is disabled on LTS

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (8460d3f) to head (09841a8).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #710      +/-   ##
==========================================
- Coverage   83.70%   83.07%   -0.64%     
==========================================
  Files          52       51       -1     
  Lines        4376     4253     -123     
==========================================
- Hits         3663     3533     -130     
- Misses        713      720       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.